Preliminary survey of prescient actions

*Published 3 April 2020*

 In a 10-20 hour exploration, we did not find clear examples of ‘prescient actions’—specific efforts to address severe and complex problems decades ahead of time and in the absence of broader scientific concern, experience with analogous problems, or feedback on the success of the effort—though we found six cases that may turn out to be examples on further investigation.

Details
-------

 We briefly investigated 20 leads on historical cases of actions taken to eliminate or mitigate a problem a decade or more in advance, evaluating them for their ‘prescience’. None were clearly as prescient as the [actions of Leó Szilárd](https://intelligence.org/files/SzilardNuclearWeapons.pdf), which were previously the best examples of such actions that we found. The primary ways in which these actions failed to exhibit prescience were the amount of feedback that was available while developing a solution and the number of years in advance of the threat that the action was taken. Although we are uncertain about most of the cases, we believe that six of them are promising for future investigation.

Background
----------

 Current efforts to prepare for the impacts of artificial intelligence have several features that could make them unlikely to succeed. They typically require us to make complex predictions about novel threats over a timescale of decades, and many of these efforts will receive little feedback on whether they are on the right track, receive little input from the larger scientific community, and produce results that are not useful outside the problem of mitigating AI risk.

It may be useful to search for past cases of preparations that have similar features. It is important to know if humanity has failed to solve problems in advance because the attempts to do so have failed or because solutions were not attempted. If we find failed attempts, we want to know why they failed.  For example, if it turns out that most previous actions were not successful because of failure to accurately predict the future, we may want to focus more of our efforts on forecasting. To this end, we use the following set of criteria for evaluating past efforts for their ‘prescience’, or the extent to which they represent early actions to mitigate a risk in absence of feedback:[1](https://aiimpacts.org/survey-of-prescient-actions/#easy-footnote-bottom-1-2362 "Originally proposed by Alexander Berger in 2015.")
* **Years in Advance:** How many years in advance of the expected emergence of the threat was the action taken?
* **Novelty:** Was the threat novel, or can we re-use (perhaps with modification) the solution to past threats?
* **Scientific Concern:** Was the effort to address the threat endorsed by the larger scientific community?
* **Complex Prediction:** Did the solution require a complex prediction, or is the solution clear and closely related to the problem?
* **Specificity:** Was the solution specific to the threat or is it something that is broadly useful and may be done anyway?
* **Feedback:** Was feedback available while developing a solution, so that we can make mistakes and learn from them, or will we need to get it right on the first try?
* **Severity:** Was it a severe threat of global importance?

 In addition to these criteria, we took note of whether the outcome of the efforts is known, as cases with a known outcome may be more informative and more fruitful for further investigation.

Methodology
-----------

 Potential cases of interest were found by searching the Internet, asking our friends and colleagues, and offering a bounty on promising leads. We compiled a list of topics to research that were sufficiently narrow to allow for evaluation over a short period of time. This list included individual people that took actions (like Clair Patterson), specific actions that were taken (e.g. the installation of the Moscow-Washington Hotline), and the threats themselves (such as the destruction of infrastructure by a geomagnetic storm).

 One researcher spent approximately 30 minutes reviewing each case, and rated them on a scale of 0 to 10 on the criteria described in the previous section.[2](https://aiimpacts.org/survey-of-prescient-actions/#easy-footnote-bottom-2-2362 "All of the ratings were assigned by Rick Korzekwa") A score of 1 indicates the criterion described the case very poorly, while a score of 10 indicates the case demonstrated the criterion extremely well. These ratings were highly subjective, though we made efforts to evaluate the cases in a way that is consistent and which would avoid too many false negatives.[3](https://aiimpacts.org/survey-of-prescient-actions/#easy-footnote-bottom-3-2362 "For example, efforts to reduce the risks of geomagnetic storms and antibiotic resistance both involve some actions that are high in specificity and others that are low in specificity. We evaluated both cases on the most specific-to-the-problem actions that we are aware of.") A composite score was calculated from these by taking a weighted average with the following weights:[4](https://aiimpacts.org/survey-of-prescient-actions/#easy-footnote-bottom-4-2362 " Because we were highly uncertain about our scores given only a half hour of research per case, we assigned scores for our best guess, or ‘median guess’ score, as well as 10th and 90th percentile estimates for each criterion for each case. These should be interpreted as the range of scores which we expect we would arrive at given several hours of investigation, with 80% credence, and equal likelihood of having over- or underestimated the score. We calculated 10th and 90th percentile estimates of the average by modeling the high and low estimates as uncorrelated deviations from the mean, so that they could be added in the usual way for propagating uncorrelated errors.")

|  |  |
| --- | --- |
| **Criterion** | **Weight** |
| Number of years in advance[5](https://aiimpacts.org/survey-of-prescient-actions/#easy-footnote-bottom-5-2362 " This score was calculated directly from the estimated number of years by a root logistic function with values 2.75, 7.1, and 9.6 for 0, 10, and 20 years, respectively ") | 20 |
| Overall severity of threat | 2 |
| Novelty of threat/solution | 3 |
| Overall level of concern from the scientific community at large | 2 |
| Complexity of prediction required to produce a solution | 5 |
| Specificity of solution | 2 |
| Level of feedback available while developing a solution | 10 |

 In addition to these ratings, we rated each one for how promising it was for further research, and annotated the ratings in the spreadsheet as seemed appropriate. We also assigned ratings to two cases that were previously the subject of in-depth investigations, for comparison. These were the [Asilomar Conference](https://intelligence.org/files/TheAsilomarConference.pdf) and [the actions of Leó Szilárd](https://intelligence.org/files/SzilardNuclearWeapons.pdf).

Results
-------

 The following table shows our ratings. The two reference cases are in italics. Our full spreadsheet of ratings and notes can be found [here](https://docs.google.com/spreadsheets/d/12mMQFjgWPjE6agOxD8jDceNTKBzIeplkjrpyTdcMa48/edit?usp=sharing).

|  |  |  |
| --- | --- | --- |
| **Case** | **Score** | **Suitability for Further Research** |
| *Leo Szilard* | *7.24* |  |
| Antibiotic resistance | 7.11 | 7 |
| Open Quantum Safe | 6.80 | 5 |
| Nordic Gene Bank | 6.74 | 4 |
| Geomagnetic Storm Prep | 6.74 | 5 |
| Fukushima Daichii | 6.74 | 5 |
| Swiss Redoubt | 6.60 | 2 |
| Nonproliferation Treaty | 6.14 | 6 |
| Cavendish Banana and TR4 | 6.12 | 5 |
| WIPP | 6.02 | 4 |
| Population Bomb | 5.99 | 3 |
| Y2k | 5.76 | 4 |
| *Asilomar Conference* | *5.70* |  |
| Cold War Civil Defense | 5.29 | 3 |
| Religious Apocalypse | 4.88 | 2 |
| Hurricane Katrina | 4.18 | 4 |
| Iran Nuclear Deal | 4.18 | 4 |
| Moscow-Washington Hotline | 3.90 | 3 |
| England 1800s Policy Reform | 3.89 | 2 |
| Clair Patterson | 3.74 | 2 |
| Missile gap | 3.22 | 2 |
| PQCrypto Conference 2006 |  | 4 |

 For one case, the PQCrypto 2006 conference, we were unable to find sufficient information after 45 minutes of investigation to provide an evaluation.

In general the cases we investigated did not score highly on these criteria. The average score was 5.6 out of 10, with the US-Russia missile gap receiving the minimum score of 3.0 and antibiotic resistance receiving the maximum score of 7.11. None of the cases received a higher score than our reference case, the actions of Leó Szilárd (score = 7.24), which we consider to be sufficiently ‘prescient’ to be worth examining. Just over half (11) of our cases received higher ratings than the Asilomar Conference (rating = 5.6), which was previously judged to be less prescient.

The ratings are highly uncertain, as is natural for thirty minute reviews of complex topics.  On average, our 90th percentile estimates were 80% larger than their corresponding 10th percentile estimate. All but four cases had minimum ratings lower than the best guess for Asilomar, and more than half had maximum ratings higher than the best guess for Leó Szilárd.

The axes on which the cases were least prescient were feedback and years in advance.[6](https://aiimpacts.org/survey-of-prescient-actions/#easy-footnote-bottom-6-2362 "On average, the cases lost 1.35 points from their composite score on each of these criteria. This is partly due to the large weight assigned to these criteria. If we used an unweighted average to compute the scores, cases would lose .77 points for feedback and .39 for years in advance, with years in advance being the axis with the highest average score.") The cases were most analogous on severity, novelty, and specificity of solution, losing on average .20, .30, and .20 points from their composite scores, respectively.

Two cases, antibiotic resistance and the Treaty on the Non-Proliferation of Nuclear Weapons, seemed particularly promising for additional research, and received scores of 7 and 6 accordingly. Five other cases received scores of at least five and seemed less promising, but likely worth some additional research.

Discussion
----------

 Although the very short research time allotted to each case limits our ability to confidently draw conclusions, we ruled out some cases which were clearly not prescient, identified some promising cases, and roughly characterized some ways in which efforts to reduce AI risk may be different from past efforts to reduce risks.

### Irrelevant Cases

 There were four cases that we found to be poor examples of prescient actions: The **US-Russia Missile Gap** of the late 1950’s, the actions of **Clair Patterson** to combat the use of leaded gasoline, **19th century policy reforms in England** that were made in response to the industrial revolution, and the **Moscow-US Nuclear Hotline**. All of these cases involved actions that were taken in response to, rather than in anticipation of, the emergence of a problem (or perceived problem), and for which the solutions were relatively straightforward, with the primary barriers being political.[7](https://aiimpacts.org/survey-of-prescient-actions/#easy-footnote-bottom-7-2362 "Clair Patterson made some impressive inferences about the present state of the world, and seemed to believe that the problems he was observing would continue to get worse without intervention. In this respect, his actions were prescient. But in general, he was working to prevent a present problem from becoming worse, rather than working to avoid a future problem.")
### Questionable Cases

 Two cases involved actions based on highly dubious predictions: Preparations for a **religious apocalypse**[8](https://aiimpacts.org/survey-of-prescient-actions/#easy-footnote-bottom-8-2362 " Preparations for religious apocalypse is a broad category. We attempted to find examples in this category that fell within our target reference class, but we were generally unable to find examples that involved specific actions taken more than a few years in advance. We are not highly confident that there do not exist examples that meet these criteria.") and the book ***The Population Bomb*** and the accompanying actions of author Paul Erhlich. Although the actors in these cases were acting on predictions that have since been shown to be inaccurate, the cases do have some similarity to AI risk. They were addressing predictions of severe consequences from novel threats, they were acting without help from the scientific community, and they did not expect to receive a great deal of feedback along the way. However, the actions were only taken 5-10 years in advance of the threat, and we expect the apparent disconnect between the forecasts and reality to make it more difficult to learn from the actions.

Some cases involved threats that had already emerged, in the sense that they could happen immediately, but had sufficiently low per-year risk for a reasonable person to expect the outcome to be at least a decade in the future. These include  **Hurricane Katrina**, **US civil defense during the cold war**, **Fukushima Daichii**, the comparison case **Asilomar Conference**, and the **Nordic Gene Bank**.[9](https://aiimpacts.org/survey-of-prescient-actions/#easy-footnote-bottom-9-2362 "The Nordic Gene Bank addresses a low per-year risk, so that it seems reasonable to consider it to be addressing a future risk. However, the first withdrawal from the seed vault happened relatively quickly, suggesting that either the risk is near term or that the solution is not highly specific to long term risks.") [10](https://aiimpacts.org/survey-of-prescient-actions/#easy-footnote-bottom-10-2362 " Although geomagnetic storm preparation has a similar quality, it seems that the per-year risk of a catastrophic outcome is low enough, and the preparations for such severe outcomes is specific enough that it qualifies as a promising case, as described in the next section.")
Other cases involved solutions that were easy or not dependent on complex forecasting. The **Swiss National Redoubt** relied on long-range forecasting, but was more of a large investment in defense than a complex search for a solution. The **year 2000 problem** was easy to address, even without taking action until relatively shortly before the event took place. The **Iran Nuclear Deal** (and perhaps also the **Nuclear Non-Proliferation Treaty**) required difficult political negotiations, but did not appear to rely on complex predictions.

### Promising Cases

 We identified six cases that seem promising for further investigation:

**Alexander Fleming** warned, in his 1945 Nobel Lecture, that widespread access to antibiotics without supervision may lead to antibiotic resistance.[11](https://aiimpacts.org/survey-of-prescient-actions/#easy-footnote-bottom-11-2362 "“The time may come when penicillin can be bought by anyone in the shops. Then there is the danger that the ignorant man may easily underdose himself and by exposing his microbes to non-lethal quantities of the drugmake them resistant.” “Wayback Machine,” March 31, 2018. https://web.archive.org/web/20180331001640/https://www.nobelprize.org/nobel_prizes/medicine/laureates/1945/fleming-lecture.pdf.") We are uncertain of the impact of Fleming’s warning, whether he took additional action to mitigate the risk, or how widespread within the scientific community such concerns were, but our impression is that it was not a widely known issue, that his was an early warning, and that his judgement was generally taken seriously by the time of his speech. His warning preceded the first documented cases of penicillin-resistant bacteria by more than 20 years, and the threat of antimicrobial resistance seems to be broadly analogous with AI risk on most of our criteria, though it does seem that feedback was available throughout efforts to reduce the threat.

**The Treaty on the Non-Proliferation of Nuclear Weapons** required many actions from many actors, but it seems to have required a complex prediction about technological development and geopolitics to address a severe threat, was specific to a particular threat, and had limited opportunities for feedback. We are uncertain if any of the specific actions will prove to be prescient on further investigation, but it seems promising.

**Open Quantum Safe** is an open-source project to develop cryptographic techniques that are resistant to the use of quantum computers. The threat of quantum computing to cryptography has several relevant features, including complex forecasting over a decades-time scale of a novel threat. We found limited information on the circumstances surrounding the founding of the project or the related case, the **2006 PQCrypto Conference**, but the problem generally seems prescient.

**Geomagnetic Storm Preparation** addresses the threat caused by severe damage and disruption by solar weather to electronics and power infrastructure, which could be a severe global catastrophe.[12](https://aiimpacts.org/survey-of-prescient-actions/#easy-footnote-bottom-12-2362 " See, for example https://allfed.info/industrial-civilisation/") The expected time between such events is decades or centuries, and mitigating the risk involves actions that may be specific to the particular problem and requires complex predictions about the physics involved and how our infrastructure and institutions would be able to respond. However, we are uncertain about which actions were taken and when, and whether there is evidence that they are working. Additionally, there is substantial investment from the scientific community and we are uncertain how much feedback is available while developing solutions.

**Panama Disease** is a fungal infection that has been spreading globally for decades and threatens the viability of the cavendish banana as a commercial crop. Cavendish bananas account for the vast majority of banana exports, and are integral to the food security of countries such as Costa Rica and Guatemala.[13](https://aiimpacts.org/survey-of-prescient-actions/#easy-footnote-bottom-13-2362 "“export revenue from bananas covered 40 percent of Costa Rica’s food import bill and 27 percent of Guatemala’s in 2014” “EST: Banana Facts.” Accessed February 6, 2020. http://www.fao.org/economic/est/est-commodities/bananas/bananafacts/en/#.XjyilyOIYuV. ") Early action included measures to slow the spread of the fungus, a search for cultivars to replace the Cavendish, calls for greater diversity in banana varietals, and searches for fungicides that are able to kill the fungus. Although these actions have many opportunities for feedback, some of them involve complex predictions and searches for specific technical solutions, and, from the perspective of farmers on continents that have not yet encountered the infection, the arrival of the fungus represents a discrete event at some undetermined time in the future. We are uncertain if these are good examples of prescient actions, but they may be worth additional investigation.

### Presence of Feedback

 The axis on which our cases most differed from efforts to reduce AI risk was the level of feedback available while developing a solution. The average score on feedback was 3.8, and none of the cases received a score higher than 7. Even cases that initially seemed that they would have very little feedback proved to have enough to aid those that were making preparations. Examples include Hurricane Katrina, which benefited from lessons learned from preceding hurricanes, and the National Redoubt of Switzerland, which benefited from the observation of conflicts between other actors, providing information about which military equipment and tactics were viable against likely adversaries. Assuming that these results are representative, here are two ways to interpret these results:

**Feedback is abundant:** Feedback is abundant in a wide variety of situations, so that we should also expect to have opportunities for feedback while preparing for advanced artificial intelligence. In support of this view are the cases mentioned above that were initially expected to lack feedback, even on the part of those making preparations, but which nonetheless benefited from feedback.

**AI risk is unusual:** The common perception that there is very little feedback available to efforts to reduce the risks of advanced AI is correct, and AI risk is unique (or very rare) in this regard. Support for this view comes from [arguments](https://intelligence.org/2018/10/03/rocket-alignment/) for the one-shot nature of solving the AI control problem.[14](https://aiimpacts.org/survey-of-prescient-actions/#easy-footnote-bottom-14-2362 "For instance, Eliezer Yudkowsky obliquely argues this in The Rocket Alignment Problem.  “The Rocket Alignment Problem – Machine Intelligence Research Institute.” Accessed March 26, 2020. https://intelligence.org/2018/10/03/rocket-alignment/.")
*Primary author: Rick Korzekwa*

Notes
-----

AI Impacts is looking into other examples in history where people took actions in order to address a complex, novel, severe future problem, and in hindsight we recognize those actions as prescient. Ideally we could learn lessons for AI alignment from such cases. The survey is so far very preliminary, so I'll summarize it later when it has been further developed, but I thought I'd send it along if you wanted to follow along (I found the six cases they've identified quite interesting).

One particularly interesting finding is that so far, in all of the cases they've looked at, there was a lot of feedback available to develop a solution. The post notes that this could be interpreted in two ways. First, since feedback is abundant in real-world problems, we should expect feedback for AI alignment as well ([the optimistic take](https://mailchi.mp/b3dc916ac7e2/an-80-why-ai-risk-might-be-solved-without-additional-intervention-from-longtermists)). Second, since AI alignment has no opportunity for feedback, it is unlike other problems (<@the pessimistic take@>(@The Rocket Alignment Problem@)). I would add a third option: that any real-world problem without feedback is extremely hard to solve, and so we wouldn't generate any hypotheses of actions that were prescient for such problems (in which case AI risk is not special amongst problems, it is just an instance of a very difficult problem).


Takeaways from safety by default interviews

*By Asya Bergal, 3 April 2020*

Last year, several researchers at AI Impacts (primarily Robert Long and I) interviewed prominent researchers inside and outside of the AI safety field who are relatively optimistic about advanced AI being developed safely. These interviews were originally intended to focus narrowly on reasons for optimism, but we ended up covering a variety of topics, including AGI timelines, the likelihood of current techniques leading to AGI, and what the right things to do in AI safety are right now.

We talked to Ernest Davis, Paul Christiano, Rohin Shah, Adam Gleave, and Robin Hanson.

Here are some more general things I personally found noteworthy while conducting these interviews. For interview-specific summaries, check out our [Interviews Page](https://aiimpacts.org/interviews-on-plausibility-of-ai-safety-by-default/).

**Relative optimism in AI often comes from the belief that AGI will be developed gradually, and problems will be fixed as they are found rather than neglected.**

All of the researchers we talked to seemed to believe in non-discontinuous takeoff.[1](https://aiimpacts.org/takeaways-from-safety-by-default-interviews/#easy-footnote-bottom-1-2350 "Paul Christiano: https://sideways-view.com/2018/02/24/takeoff-speeds/

Rohin Shah: “I don’t know, in a world where fast takeoff is true, lots of things are weird about the world, and I don’t really understand the world. So I’m like, “Shit, it’s quite likely something goes wrong.” I think the slow takeoff is definitely a crux. Also, we keep calling it slow takeoff and I want to emphasize that it’s not necessarily slow in calendar time. It’s more like gradual. … Yeah. And there’s no discontinuity between… you’re not like, “Here’s a 2X human AI,” and a couple of seconds later it’s now… Not a couple of seconds later, but like, “Yeah, we’ve got 2X AI,” for a few months and then suddenly someone deploys a 10,000X human AI. If that happened, I would also be pretty worried. It’s more like there’s a 2X human AI, then there’s like a 3X human AI and then a 4X human AI. Maybe this happens from the same AI getting better and learning more over time. Maybe it happens from it designing a new AI system that learns faster, but starts out lower and so then overtakes it sort of continuously, stuff like that.”
— From Conversation with Rohin Shah

Adam Gleave: “I don’t see much reason for AI progress to be discontinuous in particular. So there’s a lot of empirical records you could bring to bear on this, and it also seems like a lot of commercially valuable interesting research applications are going to require solving some of these problems. You’ve already seen this with value learning, that people are beginning to realize that there’s a limitation to what we can just write a reward function down for, and there’s been a lot more focus on imitation learning recently. Obviously people are solving much narrower versions of what the safety community cares about, but as AI progresses, they’re going to work on broader and broader versions of these problems.”
— From Conversastion with Adam Gleave

Robin Hanson: “That argument was a very particular one, that this would appear under a certain trajectory, under a certain scenario. That was a scenario where it would happen really fast, would happen in a very concentrated place in time, and basically once it starts, it happens so fast, you can’t really do much about it after that point. So the only chance you have is before that point. … But I was doubting that scenario. I was saying that that wasn’t a zero probability scenario, but I was thinking it was overestimated by him and other people in that space. I still think many people overestimate the probability of that scenario. Over time, it seems like more people have distanced themselves from that scenario, yet I haven’t heard as many substitute rationales for why we should do any of this stuff early.”
— From Conversation with Robin Hanson") Rohin gave ‘problems will likely be fixed as they come up’ as his primary reason for optimism,[2](https://aiimpacts.org/takeaways-from-safety-by-default-interviews/#easy-footnote-bottom-2-2350 "“The first one I had listed is that continual or gradual or slow takeoff, whatever you want to call it, allows you to correct the AI system online. And also it means that AI systems are likely to fail in not extinction-level ways before they fail in extinction-level ways, and presumably we will learn from that and not just hack around it and fix it and redeploy it.”
— From Conversation with Rohin Shah") Adam[3](https://aiimpacts.org/takeaways-from-safety-by-default-interviews/#easy-footnote-bottom-3-2350 "

“And then, I also have optimism that yes, the AI research community is going to try to solve these problems. It’s not like people are just completely disinterested in whether their systems cause harm, it’s just that right now, it seems to a lot of people very premature to work on this. There’s a sense of ‘how much good can we do now, where nearer to the time there’s going to just be naturally 100s of times more people working on the problem?’. I think there is still value you can do now, in laying the foundations of the field, but that maybe gives me a bit of a different perspective in terms of thinking, ‘What can we do that’s going to be useful to people in the future, who are going to be aware of this problem?’ versus ‘How can I solve all the problems now, and build a separate AI safety community?’.”
— From Conversation with Adam Gleave") and Paul[4](https://aiimpacts.org/takeaways-from-safety-by-default-interviews/#easy-footnote-bottom-4-2350 "“Before we get to resources or people, I think one of the basic questions is, there’s this perspective which is fairly common in ML, which is like, ‘We’re kind of just going to do a bunch of stuff, and it’ll probably work out’. That’s probably the basic thing to be getting at. How right is that?

This is the bad view of safety conditioned on– I feel like prosaic AI is in some sense the worst– seems like about as bad as things would have gotten in terms of alignment. Where, I don’t know, you try a bunch of shit, just a ton of stuff, a ton of trial and error seems pretty bad. Anyway, this is a random aside maybe more related to the previous point. But yeah, this is just with alignment. There’s this view in ML that’s relatively common that’s like, we’ll try a bunch of stuff to get the AI to do what we want, it’ll probably work out. Some problems will come up. We’ll probably solve them. I think that’s probably the most important thing in the optimism vs pessimism side.”
— From Conversation with Paul Christiano") both mentioned it as a reason.

Relatedly, both Rohin[5](https://aiimpacts.org/takeaways-from-safety-by-default-interviews/#easy-footnote-bottom-5-2350 "“I think I could imagine getting more information from either historical case studies of how people have dealt with new technologies, or analyses of how AI researchers currently think about things or deal with stuff, could change my mind about whether I think the AI community would by default handle problems that arise, which feels like an important crux between me and others.”
— From Conversation with Rohin Shah") and Paul[6](https://aiimpacts.org/takeaways-from-safety-by-default-interviews/#easy-footnote-bottom-6-2350 "“One can learn… I don’t know very much about any of the relevant institutions, I may know a little bit. So you can imagine easily learning a bunch about them by observing how well they solve analogous problems or learning about their structure, or just learning better about the views of people. That’s the second category.”
— From Conversation with Paul Christiano") said one thing that could update their views was gaining information about how institutions relevant to AI will handle AI safety problems– potentially by seeing them solve relevant problems, or by looking at historical examples.

I think this is a pretty big crux around the optimism view; my impression is that MIRI researchers generally think that 1) the development of human-level AI will likely be fast and potentially discontinuous and 2) people will be incentivized to hack around and redeploy AI when they encounter problems. See [Likelihood of discontinuous progress around the development of AGI](https://aiimpacts.org/likelihood-of-discontinuous-progress-around-the-development-of-agi/#Intelligence_explosion) for more on 1). I think 2) could be a fruitful avenue for research; in particular, it might be interesting to look at recent examples of people in technology, particularly ML, correcting software issues, perhaps when they’re against their short-term profit incentives. Adam said he thought the AI research community wasn’t paying enough attention to building safe, reliable, systems.[7](https://aiimpacts.org/takeaways-from-safety-by-default-interviews/#easy-footnote-bottom-7-2350 "“And then, if I look at the state of the art in AI, there’s a number of somewhat worrying trends. We seem to be quite good at getting very powerful superhuman systems in narrow domains when we can specify the objective that we want quite precisely. So AlphaStar, AlphaGo, OpenAI Five, these systems are very much lacking in robustness, so you have some quite surprising failure modes. Mostly we see adversarial examples in image classifiers, but some of these RL systems also have somewhat surprising failure modes. This seems to me like an area the AI research community isn’t paying much attention to, and I feel like it’s almost gotten obsessed with producing flashy results rather than necessarily doing good rigorous science and engineering. That seems like quite a worrying trend if you extrapolate it out, because some other engineering disciplines are much more focused on building reliable systems, so I more trust them to get that right by default.

Even in something like aeronautical engineering where safety standards are very high, there are still accidents in initial systems. But because we don’t even have that focus, it doesn’t seem like the AI research community is going to put that much focus on building safe, reliable systems until they’re facing really strong external or commercial pressures to do so. Autonomous vehicles do have a reasonably good safety track record, but that’s somewhere where it’s very obvious what the risks are. So that’s kinda the sociological argument, I guess, for why I don’t think that the AI research community is going to solve all of the safety problems as far ahead of time as I would like.”
— From Conversation with Adam Gleave")

**Many of the arguments I heard around relative optimism weren’t based on inside-view technical arguments.**

This isn’t that surprising in hindsight, but it seems interesting to me that though we interviewed largely technical researchers, a lot of their reasoning wasn’t based particularly on inside-view technical knowledge of the safety problems. See [the interviews](https://aiimpacts.org/interviews-on-plausibility-of-ai-safety-by-default/) for more evidence of this, but here’s a small sample of the not-particularly-technical claims made by interviewees:

* *AI researchers are likely to stop and correct broken systems rather than hack around and redeploy them.*[8](https://aiimpacts.org/takeaways-from-safety-by-default-interviews/#easy-footnote-bottom-8-2350 "See footnotes 2 – 4 above.")
* *AI has and will progress via* *a cumulation of lots of small things rather than via a sudden important insight.*

My instinct when thinking about AGI is to defer largely to safety researchers, but these reasons felt noteworthy to me in that they seemed like questions that were perhaps better answered by economists or sociologists (or for the latter case, neuroscientists) than safety researchers. I really appreciated Robin’s efforts to operationalize and analyze the second claim above.

(Of course, many of the claims were also more specific to machine learning and AI safety.)

**There are lots of calls for individuals with views around AI risk to engage with each other and understand the reasoning behind  fundamental disagreements.**

This is especially true around views that MIRI have, which many optimistic researchers reported not having a good understanding of.

This isn’t particularly surprising, but there was a strong universal and unprompted theme that there wasn’t enough engagement around AI safety arguments. Adam and Rohin both said they had a much worse understanding than they would like of others viewpoints.[9](https://aiimpacts.org/takeaways-from-safety-by-default-interviews/#easy-footnote-bottom-9-2350 "“I guess, I don’t know if this is really useful, but I do wish I had a better sense of what other people in the safety community and outside of it actually thought and why they were working on it, so I really appreciate you guys doing these interviews because it’s useful to me as well. I am generally a bit concerned about lots of people coming to lots of different conclusions regarding how pessimistic we should be, regarding timelines, regarding the right research agenda.

I think disagreement can be healthy because it’s good to explore different areas. The ideal thing would be for us to all converge to some common probability distribution and we decide we’re going to work on different areas. But it’s very hard psychologically to do this, to say, ‘okay, I’m going to be the person working on this area that I think isn’t very promising because at the margin it’s good’– people don’t work like that. It’s better if people think, ‘oh, I am working on the best thing, under my beliefs’. So having some diversity of beliefs is good. But it bothers me that I don’t know why people have come to different conclusions to me. If I understood why they disagree, I’d be happier at least.”
— From Conversation with Adam Gleave

“Slow takeoff versus fast takeoff…. I feel like MIRI still apparently believes in fast takeoff. I don’t have a clear picture of these reasons, I expect those reasons would move me towards fast takeoff. … Yeah, there’s a lot of just like.. MIRI could say their reasons for believing things and that would probably cause me to update. Actually, I have enough disagreements with MIRI that they may not update me, but it could in theory update me.”
— From Conversation with Rohin Shah") Robin[10](https://aiimpacts.org/takeaways-from-safety-by-default-interviews/#easy-footnote-bottom-10-2350 "“My experience is that I’ve just written on this periodically over the years, but I get very little engagement. Seems to me there’s just a lack of a conversation here. Early on, Eliezer Yudkowsky and I were debating, and then as soon as he and other people just got funding and recognition from other people to pursue, then they just stopped engaging critics and went off on pursuing their stuff.

Which makes some sense, but these criticisms have just been sitting and waiting. Of course, what happens periodically is they are most eager to engage the highest status people who criticize them. So periodically over the years, some high-status person will make a quip, not very thought out, at some conference panel or whatever, and they’ll be all over responding to that, and sending this guy messages and recruiting people to talk to him saying, “Hey, you don’t understand. There’s all these complications.”

Which is different from engaging the people who are the longest, most thoughtful critics. There’s not so much of that going on. You are perhaps serving as an intermediary here. But ideally, what you do would lead to an actual conversation. And maybe you should apply for funding to have an actual event where people come together and talk to each other. Your thing could be a preliminary to get them to explain how they’ve been misunderstood, or why your summary missed something; that’s fine. If it could just be the thing that started that actual conversation it could be well worth the trouble.”
— From Conversation with Robin Hanson") and Paul[11](https://aiimpacts.org/takeaways-from-safety-by-default-interviews/#easy-footnote-bottom-11-2350 "“And I don’t know, I mean this has been a project that like, it’s a hard project. I think the current state of affairs is like, the MIRI folk have strong intuitions about things being hard. Essentially no one in… very few people in ML agree with those, or even understand where they’re coming from. And even people in the EA community who have tried a bunch to understand where they’re coming from mostly don’t. Mostly people either end up understanding one side or the other and don’t really feel like they’re able to connect everything. So it’s an intimidating project in that sense. I think the MIRI people are the main proponents of the everything is doomed, the people to talk to on that side. And then in some sense there’s a lot of people on the other side who you can talk to, and the question is just, who can articulate the view most clearly? Or who has most engaged with the MIRI view such that they can speak to it?”
— From Conversation with Paul Christiano") both pointed to some existing but meaningful unfinished debate in the space.

*3 April 2020*

Notes
-----

This post lists three key takeaways from AI Impacts' conversations with "optimistic" researchers (summarized mainly in [AN #80](https://mailchi.mp/b3dc916ac7e2/an-80-why-ai-risk-might-be-solved-without-additional-intervention-from-longtermists) with one in [AN #63](https://mailchi.mp/533c646a4b21/an-63how-architecture-search-meta-learning-and-environment-design-could-lead-to-general-intelligence)). I'll just name the takeaways here, see the post for more details:

1. Relative optimism in AI often comes from the belief that AGI will be developed gradually, and problems will be fixed as they are found rather than neglected.
2. Many of the arguments I heard around relative optimism weren’t based on inside-view technical arguments.
3. There are lots of calls for individuals with views around AI risk to engage with each other and understand the reasoning behind fundamental disagreements.

As one of the people interviewed, these seem like the right high-level takeaways to me.


Interviews on plausibility of AI safety by default

This is a list of interviews on the plausibility of AI safety by default.

Background
----------

AI Impacts conducted interviews with several thinkers on AI safety in 2019 as part of a project exploring arguments for expecting advanced AI to be safe by default. The interviews also covered other AI safety topics, such as timelines to advanced AI, the likelihood of current techniques leading to AGI, and currently promising AI safety interventions.

List
----

* [Conversation with Ernie Davis](https://aiimpacts.org/conversation-with-ernie-davis/)
* [Conversation with Rohin Shah](https://aiimpacts.org/conversation-with-rohin-shah/)
* [Conversation with Paul Christiano](https://aiimpacts.org/conversation-with-paul-christiano/)
* [Conversation with Adam Gleave](https://aiimpacts.org/conversation-with-adam-gleave/)
* [Conversation with Robin Hanson](https://aiimpacts.org/conversation-with-robin-hanson/)


Atari early

*By Katja Grace, 1 April 2020*

Deepmind [announced](https://deepmind.com/blog/article/Agent57-Outperforming-the-human-Atari-benchmark) that their Agent57 beats the ‘human baseline’ at all 57 Atari games usually used as a benchmark. I think this is probably enough to resolve one of the predictions we had respondents make in our [2016 survey](https://aiimpacts.org/2016-expert-survey-on-progress-in-ai/).

Our question was when it would be feasible to ‘outperform professional game testers on all Atari games using no game specific knowledge’.[1](https://aiimpacts.org/atari-early/#easy-footnote-bottom-1-2351 "

Full question wording:

How many years until you think the following AI tasks will be feasible with:

a small chance (10%)?
an even chance (50%)?
a high chance (90%)?

Let a task be ‘feasible’ if one of the best resourced labs could implement it in less than a year if they chose to. Ignore the question of whether they would choose to.

[…]

Outperform professional game testers on all Atari games using no game-specific knowledge. This includes games like Frostbite, which require planning to achieve sub-goals and have posed problems for deep Q-networks1,2.

1 Mnih et al. (2015). Human-level control through deep reinforcement learning
2 Lake et al. (2015). Building Machines That Learn and Think Like People

small chance (10%)
 even chance (50%)
 high chance (90%)") ‘Feasible’ was defined as meaning that one of the best resourced labs could do it in a year if they wanted to.

As I see it, there are four non-obvious things to resolve in determining whether this task has become feasible:

* Did or could they outperform ‘professional game testers’?
* Did or could they do it ‘with no game specific knowledge’?
* Did or could they do it for ‘all Atari games’?
* Is anything wrong with the result?

**I. Did or could they outperform ‘professional game testers’?**

It looks like yes, for at least for 49 of the games: the ‘human baseline’ appears to have come from ‘professional human games testers’ described in [this paper](https://www.nature.com/articles/nature14236).[2](https://aiimpacts.org/atari-early/#easy-footnote-bottom-2-2351 "“In addition to the learned agents, we also report scores for
 a professional human games tester playing under controlled conditions…”

“The professional human tester used the same emulator engine as the agents, and played under controlled conditions. The human tester was not allowed to pause, save or reload games. As in the original Atari 2600 environment, the emulator was run at 60 Hz and the audio output was disabled: as such, the sensory input was equated between human player and agents. The human performance is the average reward achieved from around 20 episodes of each game lasting a maximum of 5min each, following around 2 h of practice playing each game.”") (What exactly the comparison was for the other games is less clear, but it sounds like what they mean by ‘human baseline’ is ‘professional game tester’, so I guess the other games meet a similar standard.)

I’m not sure how good professional games testers are. It sounds like they were not top-level players, given that the paper doesn’t say that they were, that they were given two hours to practice the games, and that randomly searching for high scores online for a few of these games (e.g. [here](http://highscore.com/)) yields higher ones (though this could be complicated by e.g. their only being allowed a short time to play).

**II. Did or could they do it with ‘no game specific knowledge’?**

My impression is that their system does not involve ‘game specific knowledge’ under likely meanings of this somewhat ambiguous term. However I don’t know a lot about the technical details here or how such things are usually understood, and would be interested to hear what others think.

**III. Did or could they do it for ‘all Atari games’?**

Agent57 only plays 57 [Atari 2600](https://en.wikipedia.org/wiki/Atari_2600) games, whereas [there are hundreds](https://en.wikipedia.org/wiki/List_of_Atari_2600_games) of Atari 2600 games (and [other](https://en.wikipedia.org/wiki/Atari_XEGS) [Atari](https://en.wikipedia.org/wiki/Atari_7800) [consoles](https://en.wikipedia.org/wiki/Atari_Jaguar) with presumably even more games).

Supposing that Atari57 is a longstanding benchmark including only these 57 Atari games, it seems likely that the survey participants interpreted the question as about only those games. Or at least about all Atari 2600 games, rather than every game associated with the company Atari.

Interpreting it as written though, does Agent57’s success suggest that playing all Atari games is now feasible? My guess is yes, at least for Atari 2600 games.

Fifty-five of the fifty-seven games were proposed in [this paper](https://arxiv.org/pdf/1207.4708.pdf)[3](https://aiimpacts.org/atari-early/#easy-footnote-bottom-3-2351 "Section 3.1.2, https://arxiv.org/pdf/1207.4708.pdf"), which describes how they chose fifty of them:

>
> Our testing set was constructed by choosing semi-randomly from the 381 games listed on Wikipedia [http://en.wikipedia.org/wiki/List\_of\_Atari\_2600\_games (July 12, 2012)] at the time of writing. Of these games, 123 games have their own Wikipedia page, have a single player mode, are not adult-themed or prototypes, and can be emulated in ALE. From this list, 50 games were chosen at random to form the test set.
>
>
>

The other five games in that paper were a ‘training set’, and I’m not sure where the other two came from, but as long as fifty of them were chosen fairly randomly, the provenance of the last seven doesn’t seem important.

My understanding is that none of the listed constraints should make the subset of games chosen particularly easy rather than random. So being able to play these games well suggests being able to play any Atari 2600 game well, without too much additional effort.

This might not be true if having chosen those games (about eight years ago), systems developed in the meantime are good for this particular set of games, but a different set of methods would have been needed had a different subset of games been chosen, to the extent that more than an additional year would be needed to close the gap now. My impression is that this isn’t very likely.

In sum, my guess is that respondents usually interpreted the ambiguous ‘all Atari games’ at least as narrowly as Atari 2600 games, and that a well resourced lab could now develop AI that played all Atari 2600 games within a year (e.g. plausibly DeepMind could already do that).

**IV. Is there anything else wrong with it?**

Not that I know of, but let’s wait a few weeks and see if anything comes up.

~

Given all this, I think it is more likely than not that this Atari task is feasible now. Which would be interesting, because the [median 2016 survey response](https://aiimpacts.org/2016-expert-survey-on-progress-in-ai/#Answers-9) put a 10% chance on it being feasible in five years, i.e. by 2021.[4](https://aiimpacts.org/atari-early/#easy-footnote-bottom-4-2351 "Though note that only 19 participants answered the question about when there was a 10% chance.

We surveyed 352 machine learning researchers publishing at top conferences, asking each a random subset of many questions. Some of these questions were about when they expected thirty-two concrete AI tasks would become ‘feasible’. We asked each of those questions in two slightly different ways. The relevant Atari questions had 19 and 20 responses for the two wordings, only one of which gave an answer for 2021.") They more robustly put a median 50% chance on ten years out (2026).[5](https://aiimpacts.org/atari-early/#easy-footnote-bottom-5-2351 "Half the time we asked about chances in N years, and half the time we asked about years until P probability, and people fairly consistently had earlier distributions when asked the second way. Both methods yielded a 50% chance in ten years here, though later the distributions diverge, with a 90% chance in 15 years yet a 60% chance in 20 years. Note that small numbers of different respondents answered each question, so inconsistency is not a huge red flag, though the consistent inconsistency across many questions is highly suspicious.")
It’s exciting to resolve expert predictions about early tasks so we know more about how to treat their later predictions about human-level science research and the obsolescence of all human labor for instance. But we should probably wait for a few more before reading much into it.

At a glance, some other tasks which we are already learning something about, or might soon:

* The ‘reading Aloud’ task[6](https://aiimpacts.org/atari-early/#easy-footnote-bottom-6-2351 "‘Take a written passage and output a recording that can’t be distinguished from a voice actor, by an expert listener.’") [seems to be coming along](https://www.descript.com/lyrebird-ai?source=lyrebird) to my very non-expert ear, but I know almost nothing about it.
* It seems like we are [close on Starcraft](https://aiimpacts.org/the-unexpected-difficulty-of-comparing-alphastar-to-humans/) though as far as I know the prediction hasn’t been exactly resolved as stated.

*1 April 2020*

*Thanks to Rick Korzekwa, Jacob Hilton and Daniel Filan for answering many questions.*

Notes
-----

With DeepMind's Agent57 (summarized below), it seems that it is feasible to outperform professional game testers on all Atari games using no game-specific knowledge. Interestingly, in a 2016 survey, the median response put a small chance (10%) on this being feasible by 2021, and a medium chance (50%) of being feasible by 2026.


Three kinds of competitiveness

*By Daniel Kokotajlo, 30 March 2020*

In this post, I distinguish between three different kinds of competitiveness — Performance, Cost, and Date — and explain why I think these distinctions are worth the brainspace they occupy. For example, they help me introduce and discuss a problem for AI safety proposals having to do with aligned AIs being outcompeted by unaligned AIs.

Distinguishing three kinds of competitiveness and competition
-------------------------------------------------------------

A system is *performance-competitive* insofar as its ability to perform relevant tasks compares with competing systems. If it is better than any competing system at the relevant tasks, it is very performance-competitive. If it is almost as good as the best competing system, it is less performance-competitive.

(For AI in particular, “speed” “quality” and “collective” intelligence as [Bostrom defines them](https://www.lesswrong.com/posts/semvkn56ZFcXBNc2d/superintelligence-5-forms-of-superintelligence) all contribute to performance-competitiveness.)

A system is *cost-competitive* to the extent that it costs less to build and/or operate than its competitors. If it is more expensive, it is less cost-competitive, and if it is much more expensive, it is not at all cost-competitive.

A system is *date-competitive* to the extent that it can be created sooner (or not much later than) its competitors. If it can only be created after a prohibitive delay, it is not at all date-competitive.

A *performance competition* is a competition that performance-competitiveness helps you win. The more important performance-competitiveness is to winning, the more intense the performance competition is.

Likewise for cost and date competitions. Most competitions are all three types, to varying degrees. Some competitions are none of the types; e.g. a “competition” where the winner is chosen randomly.

I briefly searched the AI alignment forum for uses of the word “competitive.” It seems that when people talk about competitiveness of AI systems, they [usually](https://www.alignmentforum.org/posts/H5gXpFtg93qDMZ6Xn/aligning-a-toy-model-of-optimization#oGdcKrWwPfwGzXNjT) mean performance-competitiveness, but [sometimes](https://www.alignmentforum.org/posts/5Kv2qNfRyXXihNrx2/ai-safety-debate-and-its-applications) mean cost-competitiveness, and [sometimes](https://www.alignmentforum.org/posts/ZHXutm7KpoWEj9G2s/an-unaligned-benchmark) both at once. Meanwhile, I suspect that [this important post](https://ai-alignment.com/prosaic-ai-control-b959644d79c2) can be summarized as “We should do prosaic AI alignment in case only prosaic AI is date-competitive.”

Putting these distinctions to work
----------------------------------

First, I’ll sketch some different future scenarios. Then I’ll sketch how different AI safety schemes might be more or less viable depending on which scenario occurs. For me at least, having these distinctions handy makes this stuff easier to think and talk about.

*Disclaimer: The three scenarios I sketch aren’t supposed to represent the scenarios I think most likely; similarly, my comments on the three safety proposals are mere hot takes. I’m just trying to illustrate how these distinctions can be used.*

**Scenario: FOOM:** There is a level of performance which leads to a localized FOOM, i.e. very rapid gains in performance combined with very rapid drops in cost, all within a single AI system (or family of systems in a single AI lab). Moreover, these gains & drops are enough to give decisive strategic advantage to the faction that benefits from them. Thus, in this scenario, *control over the future is mostly a date competition.* If there are two competing AI projects, and one project is building a system which is twice as capable and half the price but takes 100 days longer to build, *that project will lose*.

**Scenario: Gradual Economic Takeover:** The world economy gradually accelerates over several decades, and becomes increasingly dominated by billions of AGI agents. However, no one entity (AI or human, individual or group) has most of the power. In this scenario, *control over the future is mostly a cost and performance competition.* The values which shape the future will be the values of the bulk of the economy, and that in turn will be the values of the most popular and successful AGI designs, which in turn will be the designs that have the best combination of performance- and cost-competitiveness. Date-competitiveness is mostly irrelevant.

**Scenario:** **Final Conflict:** It’s just like the Gradual Economic Takeover scenario, except that several powerful factions are maneuvering and scheming against each other, in a Final Conflict to decide the fate of the world. This Final Conflict takes almost a decade, and mostly involves “cold” warfare, propaganda, coalition-building, alliance-breaking, and that sort of thing. Importantly, the victor in this conflict will be determined not so much by economic might as by clever strategy; a less well resourced faction that is nevertheless more far-sighted and strategic will gradually undermine and overtake a larger/richer but more dysfunctional faction. In this context, having themost *capable* AI advisors is of the utmost importance; having your AIs be cheap is much less important.In this scenario, *control of the future is mostly a performance competition.* (Meanwhile, in this same scenario, popularity in the wider economy is a moderately intense competition of all three kinds.)

**Proposal:** [**Value Learning**](https://www.alignmentforum.org/posts/5eX8ko7GCxwR5N9mN/what-is-ambitious-value-learning)**:** By this I mean schemes that take state-of-the-art AIs and train them to have human values. I currently think of these schemes as not very date-competitive, but pretty cost-competitive and very performance-competitive. I say value learning isn’t date-competitive because my impression is that it is probably harder to get right, and thus slower to get working, than other alignment proposals. Value learning would be better for the gradual economic takeover scenario because the world will change slowly, so we can afford to spend the time necessary to get it right, and once we do it’ll be a nice add-on to the existing state-of-the-art systems that won’t sacrifice much cost or performance.

**Proposal:** [**Iterated Distillation and Amplification:**](https://ai-alignment.com/iterated-distillation-and-amplification-157debfd1616) By this I mean… well, it’s hard to summarize. It involves training AIs to imitate humans, and then scaling them up until they are arbitrarily powerful while still human-aligned. I currently think of this scheme as decently date-competitive but not as cost-competitive or performance-competitive. But lack of performance-competitiveness isn’t a problem in the FOOM scenario because IDA is above the threshold needed to go FOOM; similarly, lack of cost-competitiveness is only a minor problem because if they don’t have enough money already, the first project to build FOOM-capable AI will probably be able to attract a ton of investment (e.g. via being nationalized) without even using their AI for anything, and then reinvest that investment into paying the extra cost of aligning it via IDA.

**Proposal:** [**Impact regularization:**](https://medium.com/@deepmindsafetyresearch/designing-agent-incentives-to-avoid-side-effects-e1ac80ea6107)By this I mean attempts to modify state-of-the-art AI designs so that they deliberately avoid having a big impact on the world. I think of this scheme as being cost-competitive and fairly date-competitive. I think of it as being performance-uncompetitive in some competitions, but performance-competitive in others. In particular, I suspect it would be very performance-uncompetitive in the Final Conflict scenario (because AI advisors of world leaders need to be impactful to do anything), yet nevertheless performance-competitive in the Gradual Economic Takeover scenario.

Putting these distinctions to work again
----------------------------------------

I came up with these distinctions because they helped me puzzle through the following problem:

>
> Lots of people worry that in a vastly multipolar, hypercompetitive AI economy (such as described in Hanson’s *Age of Em* or Bostrom’s “Disneyland without children” scenario) eventually pretty much everything of merely intrinsic value will be stripped away from the economy; the world will be dominated by hyper-efficient self-replicators various kinds, performing their roles in the economy very well and seeking out new roles to populate but not spending any time on art, philosophy, leisure, etc. Some value might remain, but the overall situation will be Malthusian.
> Well, why not apply this reasoning more broadly? Shouldn’t we be pessimistic about *any* AI alignment proposal that involves using aligned AI to compete with unaligned AIs? After all, at least one of the unaligned AIs will be willing to cut various ethical corners that the aligned AIs won’t, and this will give it an advantage.
>
>
>

This problem is more serious the more the competition is cost-intensive and performance-intensive. Sacrificing things humans value is likely to lead to cost- and performance-competitiveness gains, so the more intense the competition is in those ways, the worse our outlook is.

However, it’s plausible that the gains from such sacrifices are small. If so, we need only worry in scenarios of extremely intense cost and performance competition.

Moreover, the extent to which the competition is date-intensive seems relevant. Optimizing away things humans value, and gradually outcompeting systems which didn’t do that, takes time. And plausibly, scenarios which are not at all date competitions are also very intense performance and cost competitions. (Given enough time, lots of different designs will appear, and minor differences in performance and cost will have time to overcome differences in luck.) On the other hand, aligning AI systems might take time too, so if the competition is *too* date-intensive things look grim also. Perhaps we should hope for a scenario in between, where control of the future is a moderate date competition.

Concluding thoughts
-------------------

These distinctions seem to have been useful for me. However, I could be overestimating their usefulness. Time will tell; we shall see if others make use of them.

If you think they would be better if the definitions were rebranded or modified, now would be a good time to say so! I currently expect that a year from now my opinions on which phrasings and definitions are most useful will have evolved. If so, I’ll come back and update this post.

*30 March 2020*

*Thanks to Katja Grace and Ben Pace for comments on a draft.*


AGI in a vulnerable world

*By Asya Bergal, 25 March 2020*

I’ve been thinking about a class of AI-takeoff scenarios where a very large number of people can build dangerous, unsafe AGI before anyone can build safe AGI. This seems particularly likely if:

* It is considerably more difficult to build safe AGI than it is to build unsafe AGI.
* AI progress is software-constrained rather than compute-constrained.
* Compute available to individuals grows quickly and unsafe AGI turns out to be more of a straightforward extension of existing techniques than safe AGI is.
* Organizations are bad at keeping software secret for a long time, i.e. it’s hard to get a considerable lead in developing anything.
	+ This may be because information security is bad, or because actors are willing to go to extreme measures (e.g. extortion) to get information out of researchers.

Another related scenario is one where safe AGI is built first, but isn’t defensively advantaged enough to protect against harms by unsafe AGI created soon afterward.

The intuition behind this class of scenarios comes from an extrapolation of what machine learning progress looks like now. It seems like large organizations make the majority of progress on the frontier, but smaller teams are close behind and [able to reproduce impressive results with dramatically fewer resources](https://www.fast.ai/2018/04/30/dawnbench-fastai/). I don’t think the large organizations making AI progress are (currently) well-equipped to keep software secret if motivated and well-resourced actors put effort into acquiring it. There are strong openness norms in the ML community as a whole, which means knowledge spreads quickly. I worry that there are strong incentives for progress to continue to be very open, since decreased openness can hamper an organization’s ability to recruit talent. If compute available to individuals increases a lot, and building unsafe AGI is much easier than building safe AGI, we could suddenly find ourselves in a [vulnerable world](https://nickbostrom.com/papers/vulnerable.pdf).

I’m not sure if this is a meaningfully distinct or underemphasized class of scenarios within the AI risk space. My intuition is that there is more attention on incentives failures within a small number of actors, e.g. via arms races. I’m curious for feedback about whether many-people-can-build-AGI is a class of scenarios we should take seriously and if so, what things society could do to make them less likely, e.g. invest in high-effort info-security and secrecy work. AGI development seems much more likely to go existentially badly if more than a small number of well-resourced actors are able to create AGI.

*25 March 2020*