The far future argument for confronting catastrophic threats to humanity: Practical significance and alternatives

Introduction Over several decades, scholars from a variety of fields have advanced an argument for confronting catastrophic threats to humanity, rooted in the far future benefits of doing so. 1 In this context, the far future can loosely be defined as anything beyond the next several millennia, but will often emphasize timescales of millions or billions of years, or even longer. 2 Likewise, the catastrophic threats in question-also known as global catastrophic risks (GCRs) and existential risks, among other things-are those that would affect the trajectory of human civilization over these timescales. The simplest case is catastrophes resulting in human extinction, which is a permanent result and thus affects the trajectory of human civilization into the far future. More subtle but comparably relevant cases include catastrophes resulting in the permanent collapse of human civilization, preventing humanity from ever achieving certain very great things, and catastrophes resulting in delays Sufficiently large catastrophes can affect human civilization into the far future: thousands, millions, or billions of years from now, or even longer. The far future argument says that people should confront catastrophic threats to humanity in order to improve the far future trajectory of human civilization. However, many people are not motivated to help the far future. They are concerned only with the near future, or only with themselves and their communities. This paper assesses the extent to which practical actions to confront catastrophic threats require support for the far future argument and proposes two alternative means of motivating actions. First, many catastrophes could occur in the near future; actions to confront them have near-future benefits. Second, many actions have cobenefits unrelated to catastrophes, and can be mainstreamed into established activities. Most actions, covering most of the total threat, can be motivated with one or both of these alternatives. However, some catastrophe-confronting actions can only be justified with reference to the far future. Attention to the far future can also sometimes inspire additional action. Confronting catastrophic threats best succeeds when it considers the specific practical actions to confront the threats and the various motivations people may have to take these actions. ß 2015 Elsevier Ltd. All rights reserved. in the subsequent rise of civilization toward these achievements. The scholarship argues that people should care about human civilization into the far future, and thus, to achieve far future benefits, should seek to confront these catastrophic threats. Call this the far future argument for confronting catastrophic threats to humanity. In this paper, I will not dispute the basic validity of the far future argument. Indeed, I agree with it, and have advanced it repeatedly in my own work (Baum, 2009 (Baum, , 2010 Maher and Baum, 2013) . Instead, I assess the extent to which the far future argument is necessary or helpful for actually confronting the threats. In other words, what is the practical significance of the far future argument? I also propose and assess two alternative approaches to confronting the threats. One alternative emphasizes near future benefits of avoiding near future catastrophes. The other alternative emphasizes other (unrelated) benefits of actions that also help confront the threats, creating opportunities even for people who have zero care about the threats. It would be important if the threats can be confronted without the far future argument, because many people do not buy the argument. That people do not is suggested by a range of research. An extensive time discounting literature assess how much people value future costs and benefits. Most discounting studies use time scales of days to decades and focus on future benefits to oneself (Frederick, Loewenstein, & O'Donoghue, 2002) ; these studies are of limited relevance to valuations of the far future of human civilization. One more relevant time discounting study finds that people discount lives saved 20 years later at a 25% annual rate and lives saved 100 years later at an 8% annual rate (Johannesson & Johansson, 1996) ; extrapolating this suggests negligible concern for lives saved in the far future. Similarly, Tonn, Conrad, & Hemrick (2006, p. 821) find that people believe humanity should plan mainly for the upcoming 20 years or so and should plan less for time periods over 1000 years. In a study on social discounting, Jones and Rachlin (2006) find that people are willing to forgo more money to help close friends and family than distant acquaintances; they presumably would forgo even less for members of far future generations. Finally, there are considerations rooted in how societies today are structured. Several researchers have argued that current electoral structures favor the short-term (Ekeli, 2005; Tonn, 1996; Wolfe, 2008) . Similarly, Karlsson (2005) suggests that the rise of decentralized capitalist/democratic political economies and the fall of authoritarian (notably communist) political economies has diminished major long-term planning. While none of these studies directly assess the extent to which people buy the far future argument, the studies all suggest that many people do not buy the argument to any significant degree. To the extent that efforts to confront catastrophic threats can be made synergistic with what people already care about, a lot more can be done. This would seem to be an obvious point, but it has gone largely overlooked in prior research on catastrophic threats. One exception is Posner (2004) , who argues that some actions to reduce the risk of human extinction can be justified even if only the current generation and its immediate successor are valued. Another is Baum (2015) , who proposes to confront the threat of catastrophic nuclear winter in terms that could appeal to nuclear weapon states; Baum calls this ''ethics with strategy''. But most of the prior research, including the studies cited above, emphasize the far future argument. This paper expands Posner's argument to further argue that some actions can be taken even for those who only care about their immediate communities or even just themselves. This paper also makes progress toward assessing the total practical significance of the far future by presenting a relatively comprehensive survey of GCRs and GCR-reducing actions. Such surveys are also scarce; one example is Leggett (2006) , who surveys the space of GCRs to identify priorities for action. The present paper also has commonalities with Tonn and Stiefel (2014) , who evaluate different levels of sacrifice that society should make in response to GCRs of different magnitude. The present paper also considers levels of sacrifice, but instead argues that, from a practical standpoint, it is better to start with those actions that require less sacrifice or are in other ways more desirable. Indeed, actions requiring large sacrifice may only be justifiable with reference to far future benefits. The paper is organized as follows. Section 2 briefly reviews the space of GCRs. All actions to reduce the risk must help on one or more of these so as to result in a net risk reduction. The space of GCRs likewise provides an organizing framework for subsequent sections, as summarized in Table 1 . Section 3 discusses the timing of GCRs. For catastrophes that could happen earlier, actions to avoid them will include the earlier benefits of catastrophe avoidance. Almost all GCR reduction actions have near-future GCR reduction benefits. Section 4 discusses co-benefits and mainstreaming of GCR reduction actions. Co-benefits are benefits unrelated to GCR reduction. Mainstreaming is integrating GCR reduction into established Table 1 Summary of global catastrophic risk categories (Section 2), their timing (Section 3), co-benefits and mainstreaming opportunities (Section 4), and high-cost GCR reduction actions that may only be justifiable with reference to far future benefits (Section 5). The co-benefits and mainstreaming opportunities and high-cost actions are illustrative examples, not complete listings. activities. Co-benefits and mainstreaming are both ways to facilitate GCR reduction for those who are not specifically motivated by the far future. Section 5 discusses GCR reduction actions that can only be justified in reference to the farfuture benefits of GCR reduction. While these actions will typically not be the best place to start, they can play an important role in overall GCR reduction efforts. Section 6 discusses the ways in which attention to the far future can inspire additional GCR reduction action. This includes both analytical inspiration and emotional inspiration. Section 7 concludes.
 GCR category
 The global catastrophic risks Which actions can help reduce the risk of global catastrophe depend on what the global catastrophic risks are in the first place. This section briefly overviews the risks. The risks have been described in more detail elsewhere (Asimov, 1979; Barrett, 2007; Bostrom & C ´irkovic ´, 2008; Guterl, 2012; Jha, 2011; Leslie, 1996; Rees, 2003; Tonn & MacGregor, 2009) . While this section lists GCRs in distinct categories, the risks are often interconnected both within and across categories. For example, the emerging technology GCR of geoengineering is developed in response to the environmental change GCR of climate change, and the geoengineering risk is in turn affected by other GCRs such as large-scale violence or pandemics (Baum, Maher, & Haqq-Misra, 2013) . Similarly, GCR reduction actions can often affect risks in multiple categories. So while the GCR reduction actions discusses in Sections 3-5 are organized in terms of the categories presented here, it should be understood that specific actions often spill across categories. All this suggests a systems approach to studying GCR.
 Environmental change By environmental change, I mean to refer to human-driven global environmental changes; natural disasters are discussed below. Climate change is perhaps the most commonly cited environmental change GCR; worst-case climate change scenarios attract considerable attention (e.g. Sherwood & Huber, 2010) . Other environmental change GCRs could include biodiversity loss, biogeochemical flows (interference with the nitrogen and phosphorus cycles), stratospheric ozone depletion, ocean acidification, global fresh water use, land use change, chemical pollution, and atmospheric aerosol loading (Rockstro ¨m et al., 2009a; Rockstro ¨m et al., 2009b) . While any of these phenomena could dramatically alter the global environment, it is less clear whether the impacts would be catastrophic for humanity (Baum & Handoh, 2014; Raudsepp-Hearne et al., 2010) . For this paper, it is important that many pro-environmental actions simultaneously help across a broad set of global environmental changes, lessening the need to distinguish which changes could be catastrophic for humanity.
 Emerging technologies Several emerging technologies could cause global catastrophe, including artificial intelligence (Bostrom, 2014; Eden, Moor, Soraker, & Steinhart, 2013) , biotechnology (Vogel, 2013) , geoengineering (Baum et al., 2013; Caldeira, Bala, & Cao, 2013) , and nanotechnology for atomically precise manufacturing (Drexler, 2013) . These risks are relatively uncertain given the unprecedented nature of emerging technology, but they may constitute a significant portion of total risk.
 Large-scale violence A sufficiently large global war could be catastrophic regardless of the technologies used-for comparison, hundreds of thousands died from attacks with machetes and other unmechanized weapons in the Rwandan genocide. Weapons of mass destruction make the job much easier. Nuclear weapons can be catastrophic both through direct explosions and the indirect effects of nuclear winter (Mills, Toon, Lee-Taylor, & Robock, 2014) . Biological weapons can also readily cause global catastrophe, in particular if they are contagious; indeed, nonstate actors or even single individuals may be able to cause global catastrophes with engineered contagions (Nouri & Chyba, 2008; Rees, 2003) . The pressures of conflict can also lead actors to take larger risks, as occurred during World War II when the Americans proceeded with the first nuclear weapon test despite concerns that it could ignite the atmosphere, killing everyone (Konopinski, Marvin, & Teller, 1946) . Finally, major global violence could also result from an oppressive global totalitarian government (Caplan, 2008) .
 Pandemics Pandemics can be of natural or artificial origin, or both. Humans catch disease from the environment, in particular from other species. The development and transmission of zoonotic diseases can be enhanced by human activities including wild habitat destruction and factory farming. While it is clear that global pandemics can occur, their exact severity is a matter of ongoing analysis and debate (Germann, Kadau, Longini, & Macken, 2006; Koblentz, 2009) .
 Natural disasters Global catastrophes can result from several natural disasters including asteroid and comet impacts (Bucknam & Gold, 2008; Sleep & Zahnle, 1998) , supervolcano eruptions (Driscoll, Bozzo, Gray, Robock, & Stenchikov, 2012; Rampino, Self, & Stothers, 1988) , solar storms (NRC, 2008), and gamma ray bursts (Atri, Melott, & Karam, 2013) . While these natural disasters generally have lower probabilities, they nonetheless can be worth some effort to confront. Another natural disaster is the gradual warming of the Sun, which will (with very high probability) make Earth uninhabitable for humanity in a few billion years (O'Malley-James, Cockell, Greaves, & Raven, 2014). Other long-term astronomical risks, such as the Milky Way collision with Andromeda (increasing the rate of dangerous supernovae) and the death of all stars (removing a major energy source) play out on similar or longer time scales (Adams, 2008) .
 Physics experiments Certain types of physics experiments have raised concerns that the experiments could go wrong, obliterating Earth and its vicinity. This notably includes high-energy particle physics experiments such as at the CERN Large Hadron Collider. Physicists evaluating this risk have argued that the risk is vanishingly small; however, they may be underestimating the risk by neglecting the possibility that their analysis is mistaken (Ord, Hillerbrand, & Sandberg, 2010) .
 Extraterrestrial encounter It is not presently known if there is any extraterrestrial life, let alone intelligent extraterrestrial civilizations. However, if extraterrestrial civilizations exist, then the result could be catastrophic for humanity (Baum, Haqq-Misra, & Domagal-Goldman, 2011; Michaud, 2007) . Non-civilization extraterrestrial life could also harm humanity with catastrophic contaminations (Conley & Rummel, 2008) .
 Unknowns There may be entire categories of GCR not yet identified.
 The timing of the global catastrophic risks If a global catastrophe could occur during the near future, then there will be near-future benefits to reducing the risk. The sooner the catastrophe could occur, the larger the near-future benefits would be. In general, it will be easiest to motivate action to confront the most imminent catastrophes-hence Posner's (2004) argument that much GCR-reducing action can be justified even if one only cares about the present generation and the next one to come. It is thus worth examining the timing of the catastrophes.
 Specific global catastrophic risks
 Environmental change Major environmental changes are already visible, with larger changes expected on time scales of decades to tens of millennia. Climate change is among the more long-term of these, with some impacts already visible, and the worst climatic effects contained within the next 25,000 years or so. 3 Another possible long-term environmental change is an oceanic anoxic event, which is caused by phosphorus runoff and would in turn cause major die-off of marine species. An oceanic anoxic event could occur on time scales of millennia (Handoh & Lenton, 2003) ; more localized effects of phosphorus runoff are already visible.
 Emerging technologies Dangerous biotechnology already exists, and is steadily increasing in capability. Early design work for geoengineering is already underway, with deployments suggested to occur in upcoming decades (Keith, Parsons, & Morgan, 2010) . Experts give a significant probability to GCR-level artificial intelligence occurring within this century or next (Baum, Goertzel, & Goertzel, 2011; Mu ¨ller & Bostrom, 2014) . Nanotechnology for atomically precise manufacturing may have similar time horizons.
 Large-scale violence Large-scale violence can happen at any time. The ongoing Ukraine crisis is a firm reminder that significant tensions linger between major nuclear weapons states. Nuclear war could even occur inadvertently, due to false alarm events that can occur at any time (Barrett, Baum, & Hostetler, 2013) . Risks from biological weapons could increase in upcoming decades as biotechnology advances. However, overall risk from large-scale violence may be gradually declining, following a general trend toward less violence (Pinker, 2011) and an increasing sophistication of global peacekeeping capability (Goldstein, 2011) .
 Pandemics Pandemics can also break out at any time. Recent outbreaks of SARS, H5N1 and H1N1 flus, MERS, and currently Ebola have so far not reached a high degree of global lethality, but they are clear reminders that the threat of pandemics persists. Advances in biotechnology can lead to increasing risk through both intentional use and mishaps, as can increasing global connectivity. On the other hand, advances in public health can reduce the risk.
 Natural disasters Many natural disasters can also occur at any time. Risk from impact events, supervolcano eruptions, solar storms, and gamma ray bursts is roughly constant over long periods of time, into the far future. NASA's near-Earth objects survey has significantly reduced estimates of the risk of large impacts occurring over the next century or so (Harris, 2008) . Several astronomical risks, including the Sun's gradual warming, the Milky Way collision with Andromeda, and the death of all stars, are GCRs that exists exclusively in the far future.
 Physics experiments The risk from physics experiments depends critically on which physics experiments are conducted. The risk could increase as the capability to conduct experiments increases.
 Extraterrestrial encounter Humanity could encounter extraterrestrials at any time, including through ongoing searches for extraterrestrial intelligence (SETI). Some risk (especially contamination risk) comes mainly from human or robotic travel in space. Some risk comes from messaging to extraterrestrial intelligence (METI; Haqq-Misra, Busch, Som, & Baum, 2013) . The timing of METI risk depends on the distance from Earth to the location being messaged. METI to sufficiently distant locations is another GCR that exists exclusively in the far future.
 Unknowns Unknown GCRs could occur in both the near and far future. Indeed, more future GCRs are less likely to be already identified.
 Discussion Relatively few identified GCRs exist exclusively in the far future: certain astronomical risks and METI to distant locations. For all other GCRs-and this constitutes almost all of the total identifiable risk-the catastrophes could occur in the near future. The identified risks from emerging technologies and physics experiments could only occur in the near future. The identified risks from environmental change, large-scale violence, pandemics, some natural disasters, some extraterrestrial encounter risks, and unknowns could occur in the near or far future. The preponderance of near future risks suggests that a lot of actions to reduce these risks can be done without reference to their far future benefits. On the other hand, these near future benefits may not always be enough, especially when the catastrophes would occur several decades or centuries later, as people often care little about even these earlier times. It is thus worth pursuing other means of motivating GCR reduction.
 Co-benefits and mainstreaming GCR-reducing actions Insight on how to motivate GCR reduction can be found from outside the core GCR literature, in some related literatures. The climate change mitigation community has developed the concept of co-benefits, defined as benefits besides the target goal (Hosking, Mudu, & Dora, 2011; Miyatsuka & Zusman, n.d.) . For climate change mitigation, the target goal is greenhouse gas emissions reductions. Research assesses how communities can reduce their emissions while improving their economic development, public health, and wellbeing. The co-benefits concept readily applies GCR. Some actions to reduce GCR will also be profitable, fun, healthy, satisfying, safe, or otherwise desirable, often to those who perform the actions. Similarly, the natural disaster management community has a robust practice of mainstreaming disaster management into established goals and procedures, especially those regarding development (Benson, 2009; Twigg & Steiner, 2002) . Disaster management actions will often only be taken when they can be integrated into established goals and procedures; otherwise, the actions will be too impractical or undesirable. For example, urban design steps to reduce a town's vulnerability to hurricane storm surge can be mainstreamed into the town's broader urban planning processes (Frazier, Wood, & Yarnal, 2010) . GCR reduction actions can likewise also be mainstreamed into whatever people are already doing or trying to do. The GCR reduction community would be wise to adopt the approaches of co-benefits and mainstreaming. Doing so requires an understanding of the co-benefits that can come from various GCR-reducing actions and the relevant established goals and procedures. For many of these actions-in particular those with sufficiently large co-benefits and wellestablished goals and procedures-reducing GCR can be a nice ancillary benefit of actions that might as well be taken anyway. For these actions, no concern for the far future is needed; often, no concern beyond one's immediate community is needed. These actions require the least sacrifice (indeed, it is a sacrifice not to take these actions) and likewise will often be the easiest actions to promote. This begs the question of which GCR-reducing actions have significant co-benefits and mainstreaming opportunities.
 . Environmental change Environmental change is largely driven by a wide variety of basic activities, including food consumption, transport, real estate development, and natural resource usage. More environmentally friendly actions can often be justified for nonenvironmental reasons. A recent study by McKinsey (Enkvist, Naucle ´r, & Rosander, 2007) found that many greenhouse gas emission reductions would result in net monetary benefits for those who reduce these emissions, especially in the realm of energy efficiency in buildings and transport. Happiness research has found that people rate their daily commute as being among their least happy activities (Layard, 2003) . Public health research links high-meat diets with obesity and other health problems (Pan et al., 2012) . Buildings, transport, and food are meanwhile three of the most environmentally important sectors (Metz et al., 2007; Steinfeld et al., 2006; USEIA, 2011) . If significant changes in these sectors can be achieved for nonenvironmental reasons, the environmental benefit could be quite large.
 Emerging technologies It is often beneficial to develop regulations for multiple technologies at the same time, due to similarities between the technologies and the regulations (Kuzma & Priest, 2010; Wilson, 2013a) . Concerns about other technologies can thus motivate general technology regulation, which provides a framework for mainstreaming the regulation of emerging technologies GCRs. In addition, some actions specific to certain emerging technologies can have co-benefits. For example, one proposed solution to artificial intelligence risk is to design the AI to be ''friendly'' to humanity. In addition to not causing a catastrophe, such an AI could help with other societal problems (Muehlhauser & Bostrom, 2014) . If such an AI can be achieved with sufficient confidence, then this could be an attractive action even for those who are not concerned about AI risk.
 Large-scale violence Achieving peace avoids violence at all scales and also brings a variety of co-benefits. One co-benefit is economic growththe so-called ''peace dividend'' (Knight, Loayza, & Villanueva, 1996; Ward & Davis, 1992) . Another co-benefit is psychological. Recent research finds that conflict is often driven by humiliation, and likewise that giving people a sense of dignity can help (Lindner, 2006; Stern, 2003) . Finally, other research suggests that reducing domestic violence against women could lead to less interstate war (Hudson, Ballif-Spanvill, Caprioli, & Emmett, 2012) . Emphasizing these co-benefits could justify much action to reduce large-scale violence. Another worthy point of focus is violent nonstate actors, which continue to receive extensive attention in the wake of the 9-11 attacks. While nonstate actors may not be able to cause violence large enough to result in global catastrophe, 4 actions to confront them may have co-benefits and mainstreaming opportunities for large-scale violence. For example, the annual Nuclear Security Summits initiated by US President Barack Obama aim to prevent nonstate actors from acquiring nuclear weapons, but they also strengthen norms against nuclear weapon use more generally.
 Pandemics As noted above, there is some debate about how severe pandemics could be, including whether they would impact the far future. To a large extent, this debate is irrelevant. Regardless of how severe pandemics would be, there already exists a significant global public health infrastructure that responds to pandemics of all sizes. Improving this infrastructure can further improve the response. The case for improving this infrastructure is strengthened by the possibility of catastrophic pandemics, but the case is not dependent on this possibility (McKibbin & Sidorenko, 2006) .
 Natural disasters The GCR literature has proposed certain measures to increase society's resilience to a wide range of global catastrophes, including natural disasters. These measures include food stockpiles , underground refuges (Jebari, 2014) , and space colonies and refuges (Abrams et al., 2007; Shapiro, 2009) . These measures also tend to increase society's resilience to smaller catastrophes. Indeed, many actions taken to prepare for smaller catastrophes also benefit GCR reduction. In addition, while space colonies and refuges have been criticized for their high cost relative to other means of reducing global catastrophic risk (Baum, 2009; Sandberg, Matheny, & C ´irkovic ´, 2008) , some space missions are already underway or in planning for a variety of other reasons, including science, political prestige, and economic opportunity (e.g. in asteroid mining). Space colonies or refuges could be mainstreamed into these missions (Baum, Denkenberger, & Haqq-Misra, 2014) .
 Physics experiments Physics experiments are a curious case, because the relevant experiments are quite expensive (hundreds of millions to billions of dollars) and the social benefits somewhat limited. As Parson (2007, p. 155) puts it, ''this research is remote from practical application and serves largely to indulge national pride and the intellectual passion of a tiny elite group''. Arguably, the co-benefits of reducing physics experiment risk include the money saved by not doing the experiments, and by tasking the money to a worthier cause, analogous to the peace dividend, though this is likely to be a controversial view among those who value the physics experiments.
 Extraterrestrial encounter Protection against extraterrestrial contamination has the co-benefit of protecting extraterrestrial environments from contamination by humans, which is of significant scientific value (Conley & Rummel, 2008) . The costs of SETI and METI are small relative to the big physics experiments, so while there are dollar savings to realize from skipping them, these are less of an issue. Perhaps the extraterrestrial-risk-reducing action with the most co-benefits would be research and public education into what the risks could be. Discussions of ETI are very popular, as seen in the extensive popular media and entertainment attention to ETI.
 Unknowns Actions likely to reduce unknown GCRs will typically be generic actions that also help reduce other GCRs or even smaller risks, such as building refuges (Jebari, 2014) and stockpiling resources .
 Discussion Many GCR-reducing actions, covering the full breadth of GCRs, have sizable co-benefits, and can also be mainstreamed into existing activities. Many of these actions will often be desirable even without reference to GCR, let alone to the far future benefits of GCR reduction. These ''easy'' actions will typically be the lowest hanging fruit, the easiest GCR reductions to promote. They offer a sensible starting point for those seeking to reduce GCR.
 Actions with significant cost Ideally, all GCR-reducing actions would have low costs and large co-benefits, such that it would be easy to persuade people to take the actions, and such that the totality of GCR could be reduced with minimal burden to those taking the actions and to society at large. As discussed above, many such actions exist. However, this is not the case for all GCR-reducing actions. Some of these other actions require considerable sacrifice, especially the most aggressive GCR-reduction efforts. Tonn and Stiefel's (2014) levels of societal actions are instructive here. The levels range from doing nothing to an extreme war footing in which society is organized specifically to reduce GCR. Actions requiring more sacrifice, especially those at or near the level of extreme war footing, might only be justifiable with reference to the far-future benefits. While these actions will typically not be the lowest hanging fruit, they could be important components of an overall portfolio of GCR-reducing actions.
 Specific global catastrophic risks
 Environmental change The most aggressive pro-environmental actions include public policies like a high carbon tax, personal behaviors requiring great inconvenience and sacrifice, and restructuring the entire global industrial economy away from fossil fuels and other pollutants. To achieve a larger reduction in environmental change GCR, some of these more aggressive actions may be needed. That these more aggressive actions may only be justifiable with reference to far-future benefits is a core point from debates about discounting in environmental policy (Nordhaus, 2008).
 Emerging technologies One way to reduce emerging technology GCR is to simply abstain from developing the technologies, i.e., to relinquish them (Joy, 2000) . However, if these technologies do not cause catastrophe, they sometimes come with great benefits: geoengineering can avoid the worst effects of climate change; AI can solve a variety of social problems; biotechnology can help cure disease. Thus relinquishing the technologies can require a large sacrifice (Baum, 2014) . This sacrifice may sometimes only be justifiable given the far-future benefits of GCR reduction.
 Large-scale violence While nuclear weapons can cause great harm, they are also often attributed with helping maintain peace, through the doctrine of nuclear deterrence: countries hesitate to attack each other for fear of being destroyed in nuclear retaliation. There are questions about the efficacy of nuclear deterrence (Wilson, 2013b) and there are proposals to achieve deterrence with out large nuclear arsenals (Baum, 2015) . However, a common view posits that nuclear deterrence is necessary until international relations are peaceful enough for a world without nuclear weapons (Obama, 2009) . Following this logic, immediate nuclear disarmament might reduce GCR, but it might also increase the preponderance of smaller conflicts and other geopolitical instabilities. Depending on the details, immediate nuclear disarmament might only be justifiable with reference to the far future.
 Pandemics One aggressive action to reduce pandemics risk would be aggressive quarantine, such as blockading the major islands of Indonesia, Japan, the United Kingdom, and other countries. Travel restrictions could keep populations in these places safe. During a sufficiently severe outbreak, populations in these places could even request to be blockaded. A safer but costlier and less desirable policy would blockade them at first alert of a possible outbreak, or even keep them blockaded on a permanent basis. Doing so might lower GCR, but might only be justifiable with reference to the far future.
 Natural disasters One proposed aggressive action for natural disasters could be to drill the ground around potential supervolcanoes to extract the heat, although the technological feasibility of this proposal has not yet been established. 5 This could be a very costly project, but, if it works, it could also reduce supervolcanoes GCR. The project would come with a co-benefit of geothermal energy, but this is likely not nearly enough to justify the expense. Another possibility is advanced surfaceindependent refuges, which could protect against a variety of GCRs, including many of the natural disasters, but again could come at great expense Beckstead, 2014) .
 Physics experiments and extraterrestrial encounter I am not aware of any actions to reduce near-future GCR from physics experiments and extraterrestrial encounter that have significant cost and can only be justified with reference to far-future benefits. To the contrary, many actions to reduce these risks save money (Section 4.1). Protection against contamination does have a cost, and shutting METI programs down could cost the public a source of popular entertainment. On the other hand, the shut down itself could also create an entertaining controversy. Regardless, the costs involved are not large.
 Unknowns One action that might only be justifiable with reference to far future benefits is a far-future version of the ''extraterrestrial time capsule'' proposed in . These capsules contain artifacts of benefit to catastrophe survivors for a range of known and unknown catastrophe scenarios. The capsules are launched into space in trajectories designed to return to Earth at some future date. suggest a return date 100 years into the future, but it may be possible (and expensive) to have return dates in the far future.
 Discussion For those who wish to keep humanity highly safe from catastrophe, there are actions that can only be justified with reference to the far-future benefits of GCR reduction. While these actions are typically not the best place to start, they can offer additional GCR reductions beyond what the easier actions offer. Given the enormous far-future benefits of GCR reduction, arguably these actions merit consideration. However, hopefully GCR can be essentially eliminated without resorting to these actions. If these actions are necessary, it will likewise be necessary to appeal to the importance of the far future.
 Far future as inspiration The paper thus far has focused on how to avoid appeals to the far future argument, in recognition of the fact that many people are not motivated by what will benefit the far future. But some GCR reduction actions can only be justified with reference to far future benefits. Additionally, some people are motivated to benefit the far future. Other people could be too. Tapping the inspirational power of the far future can enable more GCR reduction. There are at least two ways that the far future can inspire action: analytical and emotional. Both are consistent with the far future argument, but the argument is typically inspired by analytical considerations. The analytical inspiration is found in works analyzing how to maximize the good or achieve related objectives. Most of the scholarly works invoking the far future argument are of this sort. 6 Such ideas have the potential to resonate not just with other scholars, but with people in other professions as well, and also the lay public. Thus there can be some value to disseminating analysis about the importance of the far future and its relation to GCR. Analytical inspiration can also come from analyzing specific actions in terms of their far-future importance. Such analysis can help promote these actions, even if the actions could be justified without reference to the far future. However, the analysis should be careful to connect with actual decision makers, and not just evaluate hypothetically optimal actions that no one ever takes. For example, there has been now multiple decades of research analyzing what the optimal carbon tax should be (for an early work, see Nordhaus, 1992 ), yet throughout this period, for most of the world, the actual carbon tax has been zero. Analytical inspiration has its limits. Research effort may be more productively spent on what policies and other actions people are actually willing to implement. The other far future inspiration is emotional. The destruction of human civilization can itself be a wrenching emotional idea. In The Fate of the Earth, Jonathan Schell writes ''The thought of cutting off life's flow, of amputating this future, is so shocking, so alien to nature, and so contradictory to life's impulse that we can scarcely entertain it before turning away in revulsion and disbelief'' (Schell, 1982 (Schell, /2000 ). In addition, there is a certain beauty to the idea of helping shape the entire arch of the narrative of humanity, or even the universe itself. People often find a sense of purpose and meaning in contributing to something bigger than themselves-and it does not get any bigger than this. Carl Sagan's (1994) Pale Blue Dot and James Martin's (2007) The Meaning of the 21st Century both capture this well, painting vivid pictures of the special place of humanity in the universe and the special opportunities people today have to make a difference of potentially cosmic significance. This perspective says that humanity faces great challenges. It says that if these challenges are successfully met, then humanity can go on to some amazing achievements. It is a worthy perspective for integrating the far future into our lives, not just for our day-to-day actions but also for how we understand ourselves as human beings alive today. This may be worth something in its own right, but it can also have a practical value in motivating additional actions to confront catastrophic threats to humanity.
 Conclusion The far future argument is sound. The goal of helping the far future is a very worthy one, and helping the far future often means helping reduce the risk of those global catastrophes that could diminish the far-future success of human civilization. However, in practical terms, reducing this risk will not always require attention to its far-future significance. This is important because many people are not motivated to help the far future, but they could nonetheless be motivated to take actions that reduce GCR and in turn help the far future. They may do this because the actions reduce the risk of near-future GCRs, or because the actions have co-benefits unrelated to GCRs and can be mainstreamed into established activities. This paper surveys GCRs and GCR-reducing actions in terms of how much these actions require support for the far future argument for confronting catastrophic threats to humanity. The analysis suggests that a large portion of total GCR, probably a large majority, can be reduced without reference to the far future and with reference to what people already care about, be it the near future or even more parochial concerns. These actions will often be the best to promote, achieving the largest GCR reduction relative to effort spent. On the other hand, some significant GCR reducing actions (especially those requiring large sacrifice) can only be justified with reference to their far-future benefits. For these actions in particular, it is important to emphasize how the far future can inspire action. Several priorities for future research are apparent. Quantitative GCR analysis could help identify which actions best reduce GCR and also what portion of GCR can be reduced without reference to the far future. Analysis covering the breadth of GCRs would be especially helpful. Social scientific research could study how to effectively engage stakeholders so as to leverage co-benefits and mainstream GCR reductions into existing programs. Social scientific research could also examine how to effectively tap the inspirational power of the far future, especially for emotional inspiration, which has received limited prior attention. Progress in these research areas could go a long way toward identifying how to, in practice, achieve large GCR reductions. The overall message of this paper is that helping the far future requires attention to which specific actions can help the far future and likewise to what can motivate these actions. The actions are not necessarily motivated by their far-future impact. This is fine. The far future does not care why people acted to help it-the far future only cares that it was helped. And people taking these actions will rarely mind that their actions also help the far future. Most people will probably view this as at least a nice ancillary benefit. Additionally, people will appreciate that those promoting the far future have taken the courtesy to consider what they care about and fit the far future into that. It can be disrespectful and counterproductive to expect people to drop everything they are doing just because some research concluded that the far future is more important. This means that those who seek to promote actions to benefit the far future must engage on an interpersonal level with the people who will take these actions, to understand what these people care about and how far-future-benefiting actions can fit in. This is an important task to pursue, given the enormity of what human civilization can accomplish from now into the far future. Futures
 The 25,000 year figure is derived from Archer and Ganopolski (2005) , Fig.3C, which shows a rapid temperature spike that declines most of the way back to current temperatures within 25,000 years and then remains at similar temperatures for another 500,000 years. However, this refers specifically to climatic effects; the human effects could persist longer, especially if the climate change causes a civilization-ending global catastrophe. S.D.Baum / Futures 72 (2015) 86-96
 Nuclear terrorism would likely be too small to cause a far-future-impacting global catastrophe, unless it catalyzed a large-scale interstate nuclear war (Ayson, 2010) . Biological terrorism could more readily cause a global catastrophe, as discussed above.S.D.Baum / Futures 72 (2015) 86-96
 S.D.Baum / Futures 72 (2015) 86-96
 An idea to this effect is briefly discussed inLeggett (2006, p. 794). 6 See citations in Footnote 1. S.D. Baum / Futures 72 (2015) 86-96


How long until human-level AI? Results from an expert assessment

Introduction The field of Artificial Intelligence was founded in the mid 1950s with the aim of constructing "thinking machines"computer systems with human-like general intelligence, and capability equaling and eventually exceeding that of human beings. After a few initial successes, early AI researchers predicted that human-level AI would soon be achieved. In 1965, Herbert Simon predicted that "machines will be capable, within twenty years, of doing any work that a man can do" [1, p.108] . Two years later, Marvin Minsky predicted that "within a generation… few compartments of intellect will remain outside the machine's realm" [1, p.109 ]; see also [2] . But the task proved recalcitrant and the failure of their optimistic timing projections was discouraging. Subsequent timing predictions became far more muted and AI research shifted focus towards producing AI systems demonstrating intelligence on specific tasks in relatively narrow domains. Much of this "narrow AI" research has been dramatically successful, with applications in many areas of science and industry. But one lesson learned through this work is that the magnitude of the difference between narrow, task-specific AI capability and more general, human-like intelligence (referred to throughout this paper as artificial general intelligence, or AGI) including the ability to generalize across divergent contexts and to maintain a sense of self. For most recent and contemporary AI researchers, building AI with human-like general intelligence is simply not a concrete research goal. Over the past few years, however, there has been a resurgence of research interest in the original goals of AI [3] [4] [5] [6] [7] . This resurgence is based on the assessment that advances in computer hardware, computer science, cognitive psychology, neuroscience, and domainspecific AI put contemporary researchers in a far better position to approach these goals than were the founders of AI. Recent years have seen a growing number of special sessions, workshops, and conferences devoted specifically to topics such as human-level AI and AGI. This includes the annual Biologically Inspired Cognitive Architectures AAAI Symposium, a series of workshops on integrative cognitive architectures; the most recent is the AAAI-10: Special Track on Integrated Intelligence (II) [8] . Another major AGI event is the international AGI conference series (AGI-08, AGI-09, AGI-10, etc.) [9] . Some recent convocations of leading AI researchers, such as the AI@50 conference [10], the 50th Anniversary Summit of Artificial Intelligence [11] , and the annual Singularity Summits at Stanford University, San Francisco, and New York City [12, 13] have also focused largely on the AGI problem. A report from AI@50 notes that these days "much of the original optimism is back, driven by rapid progress in artificial intelligence technologies" [14] . Meanwhile, MIT has recently launched a 5-year initiative aimed at exploring novel approaches to creating advanced thinking machines [15] . And noted futurist and AI inventor Ray Kurzweil has widely publicized his view that human-level AGI will be achieved within the next few decades [16, 17] a view recently supported by Intel Chief Technology Officer Justin Rattner [18] . We summarize these recent trends as a resurgence in "AI optimism", whereby "optimism" in this context we mean specifically "timing optimism". Timing optimism is of course not the same as optimism about the beneficial effects of human-level AGI. One can be optimistic that AGI will be achieved soon, but pessimistic that the consequences of AGI will be harmful. Likewise, one can be pessimistic that AGI will not be achieved soon, but optimistic that the consequences of AGI will be beneficial, whenever AGI is achieved. Of course, many in the field are less optimistic about the timing of human-level AGI. Not all experts believe that the time is ripe for a return to the original goals. Craig Silverstein of Google (a company that carries out a huge amount of narrow-AI research under the supervision of AI guru Peter Norvig) recently told a reporter that such computers are "hundreds of years away" [19] . Mark Andreesen, the founder of Netscape, said that "we are no closer to a computer that thinks like a human than we were fifty years ago" [19] . Some AI researchers, such as Selmer Bringsjord (chair of RPI's Department of Cognitive Science) even doubt that computers will ever display humanlike intelligence [20] . And the standard university AI curriculum continues to focus almost entirely on narrow AI, with only passing reference to the field's original grand goals. While AI experts with less optimistic views on timing rarely take the time to elaborate the reasons for their views, several relevant perspectives are well articulated in the literature. Hubert Dreyfus's classic critique of AI [21, 22] and John Searle's perspective of "biological naturalism" [23] both argue that achieving human-like intelligence would require human-like physical embodiment and social context. Philosopher William Hurlbut [24] argues that (very roughly speaking) human mind emerges from human brain according to patterns and processes too complex for humans to fully understand anytime in the foreseeable future. Roger Penrose [25] and Stuart Hameroff [26] believe that human intelligence relies on quantum or quantum gravity phenomena that don't exist in ordinary digital computers. Finally, some experts may believe that there is no fundamental barrier to creating human-level AGI on digital computers, but that it's simply a very hard problem which would require one or more radical breakthroughs so far beyond current understanding that they will probably take the AI community many centuries to solve. Anecdotally, based on conversations with survey participants and others at AGI-09, our impression is that many AI researchers who are timing pessimists don't have any highly specific objection to AGI optimism in mind, but rather feel that there is a large amount of uncertainty involved so that there are probably many large "hidden rocks" along the path to human-level AGI, which might or might not prove insurmountable. If the path to AGI is laden with unforeseen obstacles, then there may be no way to predict when or if the requisite scientific breakthrough might occur. On the other hand, AGI timing optimists generally believe that human-level intelligence is a straightforward although complex "mechanical" process that can be understood by extending today's scientific ideas. If this is correct, then it is reasonable to consider detailed projections regarding how much time will pass until human-level AGI is built. One issue on which there is minimal disagreement is that achievement of AGI would have a radically transformative effect on multiple aspects of science, industry, commerce, government, and arts. In the most extreme case, an intelligence explosion would ensue in which initial AGIs would design and build ever-smarter AGIs, leaving humans in the dust. In this eventuality, superintelligent AGIs might use their vast intelligence to either solve a great many of humanity's problems or to destroy (perhaps inadvertently) their human creators [27] . But whether the results are good or bad, the achievement of human-level AGI would be among the most important events in human history. It is thus of considerable importance and interest to make the best estimates possible of when and how AGI milestones may occur. Unfortunately, the database for making such estimates is quite limited. We can extrapolate hardware trends such as Moore's Law into the future, but there is no guarantee that these trends will persist, or that the requisite software will accompany them. Nor is it certain that hardware equivalent to the wetware in the human brain will be necessary or sufficient for human-level AI. There may be software paradigms that will permit human-level AGI on sub-human-level hardware, or perhaps more advanced hardware will be required for some reason. Given the limitations of hardware and software data, expert opinion can be an important information source for our understanding of AGI. Expert opinion refers simply to the views and estimates of people who have exceptional knowledge about some topic, in this case AGI. Expert opinion, if carefully elicited, can yield meaningful insights on what is and isn't known about likely developments in a range of phenomena, including AGI. Expert assessment is a particularly insightful methodology whenas is the case with AGIexperts lack consensus on important parameters. We are aware of only two previous studies explicitly exploring expert opinion on the future of AGI. In 2006, a seven-question poll was taken of participants at the AI@50 conference [28] . Four of the seven questions are particularly relevant. Asked "when will computers be able to simulate every aspect of human intelligence?", 41% said "More than 50 years" and 41% said "Never". Thus approximately 80% of the participants at the conference were AI "timing pessimists" and 20% were "timing optimists". Seventynine percent said that an accurate model of thinking is impossible without further discoveries about brain functioning. Sixty percent of the participants strongly agreed that "AI should take a multidisciplinary approach, incorporating stats, machine learning, linguistics, computer science, cognitive psychology, philosophy, and biology". Finally, 71% said that "statistical/ probabilistic methods are most accurate in representing how the brain works". The other survey, taken in 2007 by futurist entrepreneur Bruce Klein [29] , was an online survey that garnered 888 responses, asking one question: "When will AI surpass human-level intelligence?" Most of those who chose to respond to this survey were AI "optimists" who believed that human-level artificial intelligence would be achieved during the next half century. The distribution of responses is reproduced in Fig. 1 . While the results show that a substantial number of optimists exist, Klein's study has several important limitations. First, the sample population is broad, containing many non-experts. Second, the study only asks for point estimates for when superhuman AI will occur, neglecting any uncertainty in study participants' beliefs. Finally, the study only asked one question, despite the plethora of other important aspects of AGI. The AI@50 and Klein studies are interesting because they show that significant numbers of experts and interested persons believe that AGI with intelligence at least equaling the human level will exist within upcoming decades. The current study probes more deeply into the thinking of people with substantial AGI expertise. Many (though not all) of these experts can be classified as timing optimists. While their views are not representative of the larger population of AI experts, they are nonetheless worth considering. In many cases, these experts are working seriously on AGI because they believe there is a strong likelihood of achieving success with the fundamental theoretical and hardware tools at hand or likely to be developed in the next few decades. This study is based on an expert assessment survey distributed to attendees of the Artificial General Intelligence 2009 (AGI-09) conference [30] . This was a gathering of AI researchers and other individuals interested specifically in pursuing theoretical or practical work on AGI. The survey covered the timing of AGI milestones, the effectiveness of various technical approaches to achieving AI, the potential for AGI to help or harm humanity, and the possibility of an AI being conscious.
 Methods Expert assessments have been used for several decades to provide timely information and aid decision making. Expert assessments have been used in the highest levels of study, including the United Nations-sponsored Intergovernmental Panel on Climate Change [31] . A range of assessment methods exist [32] [33] [34] . There is no single optimal methodology for assessing a group of experts. Each assessment should be customized to meet the questions of the research, the abilities of the experts, and any other relevant factors. Our study draws heavily on recent expert elicitation methodology that emphasizes highlighting the diversity of expert opinion instead of reaching consensus [34, p.146-154] . While this expert elicitation has been used primarily for estimation of static parameter values, it has also been used for several technological forecasting study [35] [36] [37] [38] . Our study used a two-part survey. The first part is a standardized survey instrument distributed at the AGI-09 conference. This survey is adapted from previous expert elicitation surveys, in particular [39] . It features primarily quantitative questions. Many expert elicitation studies only use a survey along these lines. However, we felt that our quantitative survey results would be insightfully complemented by qualitative follow-up questions. So, the second part of our survey consists of follow-up questions customized for each survey participant and distributed to participants via email. Unlike follow-up questions used in Delphi studies, ours were focused on clarifying and expanding original survey responses instead of working towards consensus. These follow-up questions give us a richer understanding of the meanings and reasons behind the quantitative responses given in the initial survey.
 Initial survey The initial survey was in a single standardized form printed across 18 pages for each of the survey participants. Participants reported spending an average of 28 min on the survey. All questions were quantitative, closed-ended questions, asking for numbers or binary true/false or yes/no answers. However, participants were encouraged to write any clarifying or other relevant remarks anywhere on the survey instrument; most participants took advantage of this option. The full survey instrument that we used for our investigation is available online [40] . Here we briefly summarize the most important questions that were posed. Extensive empirical research shows that experts and non-experts alike tend to be overconfident in their probability estimates [34] . In rough terms, this means that for any given parameter, individuals tend to underestimate the probability that the actual value of the parameter is quite different from their best guess estimate. For example, someone might estimate that there is a 90% chance of the parameter values being within some range when that the actual values are within the range only 50% of the time. Our survey took several steps to reduce this overconfidence bias. Before any technical questions were asked, the survey explained overconfidence so as to sensitize participants to be cautious about it when answering questions. Additionally, some questions were worded so as to reduce overconfidence. The first set of questions elicited experts' beliefs about when AI would reach each of four milestones: passing the Turing test, 1 performing Nobel quality work, passing third grade, and becoming superhuman. These specific milestones were selected to span a variety of levels of advanced general intelligence. For each milestone, two question versions were askedwith and without additional fundingmaking for eight total milestone questions. These two versions explore the possibility that when the milestones are reached depends on the resources available to researchers. The amount of additional funding listed, $100 billion per year, is obviously more than could be used for AGI research; the intent with this figure is to ensure that money would not be a scarce resource in this hypothetical AGI development scenario. For each of the eight milestone questions, we asked the respondents to give us estimates representing 10%, 25%, 75%, and 90% confidence intervals, as well as their best estimate dates. The 10% point represents the date in which the expert estimates that there is a 10% chance that the milestone will have been achieved, and so on. We asked experts to estimate the 10% and 90% intervals first in order to reduce overconfidence. Collectively, the five points correspond at least roughly with points on participants' probability density functions for the uncertain milestone achievement dates. The five point approach represents a trade-off between gaining deeper insight into experts' beliefs and demanding more detail from the experts. Subsequent questions requested only single point estimates instead of multiple points on a probability density function. This choice was made to keep the overall survey instrument relatively short. The subsequent questions covered four topics. Three questions asked what embodiment the first AGIs would have: physical or virtual robot bodies or a minimal text-or voice-only embodiment. Eight questions asked what AI software paradigm the first AGIs would be based on: formal neural networks, probability theory, uncertain logic, evolutionary learning, a large hand-coded knowledge-base, mathematical theory, nonlinear dynamical systems, or an integrative design combining multiple paradigms. Three questions asked the likelihood of a strongly negative-to-humanity outcome if the first AGIs were created by: an open-source project, the US military, or a private for-profit software company. Two true/false questions asked if quantum computing or hypercomputing would be required for AGI. Two yes/ no questions asked if AGIs emulating the human brain conceptually or near-exactly would be conscious in the sense that humans are. Finally, 14 questions asked experts to evaluate their own expertise in several subjects: cognitive science, neural networks, probability theory, uncertain logic, expert systems, theories of ethics, evolutionary learning, quantum theory, quantum gravity theory, robotics, virtual worlds, software engineering, computer hardware design, and cognitive neuroscience.
 Follow-up questions After analyzing results of the initial survey, we wrote and distributed follow-up questions customized for each participant. Whereas the initial survey consisted entirely of quantitative, closed-ended questions, the follow-up questions were all qualitative, open-ended questions, asking for prose explanations and clarifications of their initial survey responses. Participants received between two and six questions. While the initial survey responses raised a large number of issues we were interested in following up on, we chose only the most important questions so as to minimize the burden on survey participants' time.
 Study participants Participants were recruited at the AGI-09 conference. Due to the highly specialized nature of the AGI-09 conference, no conference attendees were excluded. Participants thus have a range of levels of expertise, from graduate students to senior researchers. This inclusive approach risked eliciting low-quality responses from non-experts who happen to be at the conference. On the other hand, more exclusive approaches risk missing important ideas or information and also risk biasing results in certain directions. All of the participants in our study displayed a depth of thinking demonstrative of significant AGI expertise. Twenty-one people participated in the study. Nineteen people completed the initial survey at the AGI-09 conference; two completed it later, submitting it via postal mail. Also, 18 people (including both of those who submitted via postal mail) completed the follow-up questions. Such levels of participation are common for expert elicitationone recent study [38] had just 7 experts; another study [41] had just 12 experts. While a larger number of participants may bring additional information, our participant group is very much adequate to yield meaningful insights. It should be emphasized here that the purpose of our study is ultimately to gain insight about AGI itself, not about AGI experts. Thus our sample of AGI experts must provide rich insights about AGI, but it does not need to be in some way representative of the broader population of AGI experts. After all, there is no guarantee that an accurate assessment of the opinions of the broader on AGI population correlates with the realities of AGI. It is entirely possible that the deepest insights about AGI come from outliers from the broader population of experts. Indeed, the very existence of disagreement among experts is grounds for reflection on the state of our knowledge. As our results make clear, our sample succeeds quite well at providing rich insight about AGI. While we believe our sample to be at least somewhat representative of the broader AGI expert population, this is of secondary importance. Study participants have a broad range of backgrounds and experience, all with significant prior thinking about AGI. Eleven are in academia, including six Ph.D. students, four faculty members, and one visiting scholar, all in AI or allied fields. Three lead research at independent AI research organizations and three do the same at information technology organizations. Two are researchers at major corporations. One holds a high-level administrative position at a relevant non-profit organization. One is a patent attorney. All but four participants reported being actively engaged in conducting AI research. We are not listing the names of participating experts for several reasons. First, prior expert assessments vary in their approach to anonymity. Many studies do not list experts' names at all (e.g. [42, 43] ). Many other studies list experts' names but do not link specific names to specific responses (e.g. [35] [36] [37] [38] [39] 41] ). Second, due to the sensitive nature of AGI, several of our experts requested that their names not be listed. The others participated on the condition that their names not be linked to their responses. Listing experts' names and discussing results in this way could enable some readers to link specific names to specific responses.
 Results Although the majority of the experts who participated in our study were clearly AGI timing optimists, there was a significant minority of timing pessimists. While we don't know how the views of study participants compare to those of the rest of the conference attendees, it seems unsurprising to find an optimist majority at an AGI conference. Most AGI pessimists may see little reason to bother attending an AGI conference, although this is not universalsome known AGI pessimists do regularly attend AGI conferences. Finally, all experts in our study, including those who could be classified as timing pessimists, gave at least a 10% chance of some AI milestones being achieved within a few decades. Additionally, the amount of agreement between the experts varied considerably from one question to another. On several questions, experts disagreed stronglymore so than is typically found in expert assessmentsindicating a lack of professional consensus and the presence of multiple competing mental models of the future of AGI. Meanwhile, on other questions, much more agreement existed, though rarely full consensus.
 Milestone timing without additional funding Much of the initial survey and follow-up questions focused on the timing of four AI milestones: passing the Turing test, performing Nobel-quality research, passing the third grade, and gaining dramatically superhuman capabilities. The range of the best guess estimates given for these milestones without massive additional funding is summarized in Fig. 2 . This figure clearly shows the dichotomy between timing optimists and timing pessimists. It also shows that a substantial number of AGI experts expect that key milestones will be reached within upcoming decades. The overlap in the best guess estimates for the milestones was surprising. Using human cognitive development as a model, one might think that being able to do Nobel level science would take much longer than being able to conduct a social conversation, as in the Turing test. It is true that many complex problems can be solved effectively by artificial expert systems. But performing Nobel level science requires more than just complex problem solving: it also requires creativity in the formulation of new research questions and ideas, something that expert systems cannot achieve. Thus we might expect that performing Nobel level science would be more difficult than passing the Turing test. However, some experts thought it would be quite different for an AGI. One expert observed that "making an AGI capable of doing powerful and creative thinking is probably easier than making one that imitates the many, complex behaviors of a human mindmany of which would actually be hindrances when it comes to creating Nobel-quality science". This expert observed that "humans tend to have minds that bore easily, wonder [sic] away from a given mental task, and that care about things such as sexual attraction, all [of] which would probably impede scientific ability, rather that promote it". To successfully emulate a human, a computer might have to disguise many of its abilities, masquerading as being less intelligent, in certain ways, than it actually was. This is one reason that the Turing test milestone might be reached later than the Nobel-quality science milestone. We computed median estimates for each milestone (Table 1 ). The medians would be only slightly lower if these less optimistic respondents were excluded from the analysis. Medians were used, instead of means, because these are less susceptible to distortion via statistical outliers. 2
 Milestone estimates with massive new funding Another interesting result concerns the role of fundingthe predicted impact on the AGI field of a hypothetical $100 billion per year funding infusion. The range of the best guess estimates given for these milestones with massive additional funding is summarized in Fig. 3 . Most experts estimated that a massive funding increase of this nature would cause the AI milestones to be reached sooner. However, for many of these experts, the difference in milestone timing with and without massive funding was quite smalljust a few years. Furthermore, several experts estimated that massive funding would actually cause the AI milestones to be reached later. One reason given for this is that with so much funding, "many scholars would focus on making money and administration" instead of on research. Another reason given is that "massive funding increases corruption in a field and its oppression of dissenting views in the long term". Of those who thought that funding would make little difference, a common reason was that AGI progress requires theoretical breakthroughs from just a few dedicated, capable researchers, something that does not depend on massive funding. Another common reason was that the funding would not be wisely targeted. Several noted that funding could be distributed better if there was a better understanding of what paradigms could produce AGI, but such an understanding is either not presently available or not likely to be understood by those who would distribute funds. Because of the lack of agreement on a single paradigm, several experts recommended that modest amounts of funding should be distributed to a variety of groups following different approaches, instead of large amounts of funding being given to a "Manhattan Project" type crash program following one approach. Only one expert recommended concentrating funds in a fewer number of groups. Similarly, several observed that well-funded efforts guided by a single paradigm had failed in the past, including the Japanese Fifth Generation Computer Systems project. While most experts did not address the question of how to distribute funds across groups, our initial impression is that there would be some consensus for distributing funds to a larger variety of groups. This result could have major implications for how available funds are to be allocated. Several experts gave technical reasons for why funding would not have a significant impact. On this, one said, "AGI requires more theoretical study than real investment." Another said, "I believe the development of AGI's to be more of a tool and evolutionary problem than simply a funding problem. AGI's will be built upon tools that have been developed from previous tools. This evolution in tools will take time. Even with a crash project and massive funding, these tools will still need time to develop and mature." Given that these experts are precisely those who would benefit most from increased funding, their skeptical views of the impact of hypothetical massive funding are very likely sincere.
 Milestone timing: estimates of certainty Since we do not know for sure when milestones will be reached, it is helpful to characterize the uncertainty about this. Some insights can be gained from considering the distributions of best guess estimates of experts (as in Figs. 2 and 3 ) or of broader populations (as in Fig. 1 ). However, each individual has uncertainty about his/her own estimates. These uncertainties are not found from presentations of best guess estimates, but they can be found from the estimates across the range of confidence intervals elicited in our survey. Participants showed a broad range of certainty levels in their timing estimates. Several participants showed little uncertainty, indicating that they believed it was very likely that milestones would be achieved within a narrow range of dates. The participants giving the narrowest ranges were all strong timing optimists. Meanwhile, many of the participants showed much uncertainty. Some of these individuals gave substantial probabilities for both early and late milestone achievement dates. For these individuals, the timing optimist/pessimist dichotomy does not readily apply. Other individuals showing much uncertainty could be classified as timing pessimists in that they considered it most likely that milestones would be achieved much later. The fact that these individuals are not certain about their pessimism could explain why they thought that attending a conference on AGI would be worthwhile. One expert estimated that there was at least a 25% chance that the Turing test would never be achieved, but he also thought that there was a 10% chance that it would be achieved by the year 2020. Another estimated that there was a 25% chance it would not be achieved until the year 3000, but that there was a 10% chance it would happen by 2050. The stronger optimists at the conference were much more certain of their opinions. One estimated that there was a 90% chance that the Turing test would be achieved by 2016 and a 10% chance that it would be achieved as soon as 2010. The range of variation in the estimates for the four milestones without massive additional funding is shown in Fig. 4 . Dates after 2100 are removed to improve resolution in the rest of the plots. There was substantial agreement among the more optimistic participants in the study. Most thought that there was a substantial likelihood that the various milestones would be passed sometime between 2020 and 2040. Sixteen experts gave a date before 2050 as their best estimate for when the Turing test would be passed. Thirteen experts gave a date between 2020 and 2060 as their best estimate for when superhuman AI would be achieved. (The others all estimated later dates, ranging from 2100 to never.) The opinions of the optimists were similar to or perhaps somewhat more optimistic than Kurzweil's well-known projections. As noted above, optimism in this sense is about the timing of AI milestones. It does not imply a belief that achieving AGI would be a good thing. To the contrary, one can be optimistic that AGI will happen soon yet believe that AGI would have negative outcomes. Indeed, several experts reported this pair of beliefs. Results about the likelihood of negative outcomes are discussed further below.
 Milestone order There was little agreement among experts on the order in which the four milestones (Turing test; third grade; Nobel; and superhuman) would be achieved. The only area of consensus was that the superhuman milestone would be achieved either last or at the same time as other milestones. Meanwhile, there was significant divergence regarding the order of the other three milestones. One expert argued that the Nobel milestone would be easier precisely because it is more sophisticated: to pass the Turing test, an AI must "skillfully hide such mental superiorities". Another argued that a Turing test-passing AI needs the same types of intelligence as a Nobel AI "but additionally needs to fake a lot of human idiosyncrasies (irrationality, imperfection, emotions)". Finally, one expert noted that the third grade AI might come first because passing a third grade exam might be achieved "by advances in natural language processing, without actually creating an AI as intelligent as a third-grade child". This diversity of views on milestone order suggests a rich, multidimensional understanding of intelligence. It may be that a range of milestone orderings are possible, depending on how AI development proceeds. The milestone order results highlight the fact that many experts do not consider it likely that the first human-level AGI systems will closely mimic human intelligence. Analogy to human intelligence would suggest that achieving an AGI capable of Nobel level science would take much longer than achieving an AGI capable of conducting a social conversation. However, as discussed above, an AGI would not necessarily mimic human intelligence. This could enable it to achieve the intelligence milestones in other orders.
 Technical approaches Our initial survey asked about eight technical approaches used in constructing AI systems: formal neural networks, probability theory, uncertain logic, evolutionary learning, a large hand-coded knowledge-base, mathematical theory, nonlinear dynamical systems, and integrative designs combining multiple paradigms. For each, it requested a point estimate of the odds that the approach would be critical in the creation of human-level AGI. They survey also asked about the odds that physical robotics, virtual agent control, or minimal text-or voice-based embodiments would play a critical role. The most striking finding in this section was a strong majority opinion in favor of integrative designs. Of the 18 respondents who answered this question, 13 gave a probability of 0.5 or greater; the mean response across all 18 was 0.63. 3 None of the specific technical approaches mentioned received strong support from any more than a small plurality of respondents. Probability theory received the strongest support, with a mean probability of 0.35 and with 7 out of 20 respondents answering the question giving a probability of 0.5 or greater. The question about robotics vs. other forms of embodiment received a range of responses (Fig. 5 ). There were responses of 0.9, 0.94, 0.89, and 0.60 for physical robot embodiment, but the mean response was only 0.28. The handful of participants who felt robotics was crucial were all relatively optimistic, as seen by their mean date of 2034 for the Turing test without massive additional funding. The virtual agents option received a similar but considerably less strong response, with top responses of 0.80, 0.65, and 0.60 and a mean response of 0.34. While some of the virtual worlds enthusiasts were optimistic about timing, this was much less of a strong pattern than in the robotics case. The preliminary impression one obtains is that a few researchers are highly bullish on robotics as the correct path to AGI in the relatively near term, whereas the rest feel robotics is probably not necessary for AGI. Meanwhile, a larger number of researchers are moderately optimistic about the role of virtual worlds for AGI, but with milder opinions about the importance of the approach.
 Impacts of AGI In science fiction, intelligent computers frequently become dangerous competitors to humanity, sometimes even seeking to exterminate humanity as an inferior life form. Several researchers have recently expressed similar concerns [27, 44] . Motivated by these concerns, we asked experts to estimate the probability of a negative-to-humanity outcome occurring if an AGI passes the Turing test. Our question was broken into three parts, for each of three possible development scenarios: if the first AGI that can pass the Turing test is created by an open source project, by the United States military, or by a private company focused on commercial profit. This set of questions marked another instance in which the experts lacked consensus (see Fig. 6 ). Five experts estimated a less than 20% chance of a negative outcome, regardless of the development scenario. Four experts estimated a greater than 60% chance of a negative outcome, regardless of the development scenario. Only four experts gave the same estimate for all three development scenarios. Finally, several experts were more concerned about the risk from AGI itself, whereas others were more concerned that AGI could be misused by humans who controlled it. Interesting insights can be found in the experts' orderings of the riskiness of the development scenarios. Of the 11 experts who gave different estimates for each of the three scenarios, 10 gave the private company scenario the middle value. Of these 10, 6 gave the US military scenario the highest value and 4 gave it the lowest value. Thus the open source scenario and the US military scenario tend to perceived as relatively safe or relatively dangerous, but experts are divided on which is which. Experts who estimated that the US military scenario is relatively safe noted that the US military faces strong moral constraints, has experience handling security issues, and is very reluctant to develop technologies that may backfire (such as biological weapons), whereas open source development lacks these features and cannot easily prevent the "deliberate insertion of malicious code". In contrast, 3 In this section the survey did not require that the probabilities all add up to one, because the technical approaches are not necessarily mutually exclusive. When the numbers are normalized such that each participant's set of probabilities add up to one, then only 2 experts give integrated design a probability of 0.5 or greater, but 12 still rank it the most likely, including 1 who gave the highest probability to "none of the above". After normalization, the average for integrated design is 0.32; all other approaches ranged 6% (mathematical theory) and 12% (probability theory). Several experts noted potential impacts of AGI other than the catastrophic. One predicted that "in thirty years, it is likely that virtually all the intellectual work that is done by trained human beings such as doctors, lawyers, scientists, or programmers, can be done by computers for pennies an hour. It is also likely that with AGI the cost of capable robots will drop, drastically decreasing the value of physical labor. Thus, AGI is likely to eliminate almost all of today's decently paying jobs." This would be disruptive, but not necessarily bad. Another expert thought that, "societies could accept and promote the idea that AGI is mankind's greatest invention, providing great wealth, great health, and early access to a long and pleasant retirement for everyone." Indeed, the experts' comments suggested that the potential for this sort of positive outcome is a core motivator for much AGI research.
 Conclusion Despite decades of setbacks, the development of AI with the general intelligence of humans remains an important possibility both for the AI research community and for society at large. Given the paucity of hardware and software data enabling detailed extrapolations regarding the future of such AI, expert opinion is an important source of information. In this article, we have presented the most careful assessment of expert opinion on AGI to date. Given the long history of AI researchers making erroneous predictions, our results must be interpreted cautiously, but they nonetheless add much to our understanding of the future of AGI. In the broadest of terms, our results on AGI timing concur with those of the two previous studies mentioned in the Introduction [28, 29] . All three studies suggest that significant numbers of interested, informed individuals believe it is likely that AGI at the human level or beyond will occur around the middle of this century, and plausibly even sooner. Due to the greater depth of the questions, our survey also revealed some interesting additional information, such as the disagreement among experts over the likely order of AGI milestones and the relative safety of different AGI development scenarios. The experts' suggestions regarding funding (in particular, skepticism about the importance of funding and a preference for distributing funds among a broad range of research groups) are also potentially valuable. Our results reinforce the idea that there is a significant likelihood of AGI being achieved within upcoming decades. This idea has profound societal implications given the strong potential for AGI to act as a transformative agent, either to massively help humanity or to destroy it. The experts in our study, like others who have commented on the issue, were divided on whether the impacts of AGI would be positive or negative for humanity. Our results add depth to our understanding of AGI impacts by exploring multiple development scenarios. However, the jury remains out regarding how to achieve positive outcomes and avoid negative outcomes. This is a crucial area for future research. Currently, AGI experts hold a rich diversity of views, a situation which challenges our ability to make confident predictions about the future of AGI, but is not surprising given the state of development of the field. It would be interesting to carry out similar studies in future and see how expert views evolve and converge or diverge as AI technology advances. Participants are displayed in the same order as in Fig. 4 , such that Participant 1 in Fig. 6 is the same person as Participant 1 in Fig. 4 . Fig. 1 . 1 Fig.1. Results from the survey taken by Klein [29] in 2007 asking the question "When will AI surpass human-level intelligence?".
 Fig. 2 . 2 Fig. 2. Milestone best estimate guesses without massive additional funding. Estimates are for when AI would achieve four milestones: the Turing test (horizontal lines), third grade (white), Nobel-quality work (black), and superhuman capability (grey).
 Fig. 3 . 3 Fig. 3. Milestone best estimate guesses with massive additional funding. Estimates are for when AI would achieve four milestones: the Turing test (horizontal lines), third grade (white), Nobel-quality work (black), and superhuman capability (grey).
 Fig. 4 . 4 Fig. 4. Estimates for when AI would achieve four milestones without massive additional funding. Estimates are given at five confidence intervals. Asterisks represent 10% and 90% confidence bounds. Horizontal lines represent 25% and 75% confidence bounds. Diamonds represent best guess estimates. In each plot, participants are ordered by their Turing test best guess estimates.
 Fig. 5 . 5 Fig.5. Number of respondents for different embodiment approaches. Respondents are grouped into probability intervals for each of three different embodiment approaches: physical robot (black), virtual agent (white), and minimal embodiment (grey). Intervals are labeled on the horizontal axis by their lower bound. For example, the interval labeled "0.1" represents the interval 0.1 through 0.199.
 Fig. 6 . 6 Fig. 6. Probability of a negative-to-humanity outcome for different development scenarios. The three development scenarios are if the first AGI that can pass the Turing test is created by an open source project (x's), the United States military (squares), or a private company focused on commercial profit (triangles).Participants are displayed in the same order as in Fig.4, such that Participant 1 in Fig.6is the same person as Participant 1 in Fig.4.
 Table 1 1 Median estimates for each AGI milestone at each level of confidence. Milestone 10% 25% 50% 75% 90% Turing test 2020 2030 2040 2050 2075 Nobel science 2020 2030 2045 2080 2100 Third grade 2020 2025 2030 2045 2075 Superhuman 2025 2035 2045 2080 2100
 Participants expressed concern that the Turing test milestone is ambiguous, due to the numerous variations of the test. In response to this, several participants specified the Turing test variant their responses are based on. At the time of survey distribution, a verbal suggestion was given to consider the "one hour" rather than the "five minute" Turing test as some potential participants felt the latter could too easily be "gamed" by narrow-AI chatbots without significant general intelligence.
 In this case, the sample size is so small that it's hard to rigorously speak about "outliers," but it's clear qualitatively that the very long estimates given by some experts, including those who thought that AGI might never be achieved, would have a disproportionate effect upon the means.

Seth D. Baum
Ben Goertzel
Ted G. Goertzel