How popular is ChatGPT? Part 2: slower growth than Pokémon GO

*Rick Korzekwa, March 3, 2023*

A major theme in reporting on ChatGPT is the rapid growth of its user base. A [commonly](https://www.theguardian.com/technology/2023/feb/02/chatgpt-100-million-users-open-ai-fastest-growing-app) [stated](https://www.cbsnews.com/news/chatgpt-chatbot-tiktok-ai-artificial-intelligence/) [claim](https://www.reuters.com/technology/chatgpt-sets-record-fastest-growing-user-base-analyst-note-2023-02-01/) is that it broke records, with over 1 million users in less than a week and 100 million users in less than two months. It seems not to have broken the record, though I do think ChatGPT’s growth is an outlier.

Checking the claims
-------------------

### ChatGPT growth

From what I can tell, the only source for the claim that ChatGPT had 1 million users in less than a week comes from [this tweet](https://twitter.com/sama/status/1599668808285028353?lang=enhttps://twitter.com/sama/status/1599668808285028353?lang=en) by Sam Altman, the CEO of OpenAI:

[![](https://aiimpacts.org/wp-content/uploads/2023/03/Screenshot-2023-03-02-at-3.34.46-PM-1024x365.png)](https://aiimpacts.org/wp-content/uploads/2023/03/Screenshot-2023-03-02-at-3.34.46-PM.png)
I don’t see any reason to strongly doubt this is accurate, but keep in mind it is an imprecise statement from a single person with an incentive to promote a product, so it could be wrong or misleading.

The claim that it reached 100 million users within two months has [been reported](https://www.reuters.com/technology/chatgpt-sets-record-fastest-growing-user-base-analyst-note-2023-02-01/) by many news outlets, which all seem to bottom out in data from [Similarweb](https://www.similarweb.com/website/chat.openai.com/). I was not able to find a detailed report, but it looks like they have more data behind a paywall. I think it’s reasonable to accept this claim for now, but, again, it might be different in some way from what the media is reporting[1](https://aiimpacts.org/how-popular-is-chatgpt-part-2-slower-growth-than-pokemon-go/#easy-footnote-bottom-1-3444 "I also found some claims that the 100M number was inferred from some other figure, like total site visits, and that it might be an overestimate. I haven’t actually seen any sources doing this, so I’m sticking with the original number for now.").

### Setting records and growth of other apps

#### Claims of record setting

I saw people sharing graphs that showed the number of users over time for various apps and services. Here is a [rather hyperbolic example](https://twitter.com/kylelf_/status/1626273321020198913):

[![](https://aiimpacts.org/wp-content/uploads/2023/03/Screenshot-2023-03-02-at-4.08.37-PM-899x1024.png)](https://aiimpacts.org/wp-content/uploads/2023/03/Screenshot-2023-03-02-at-4.08.37-PM.png)
That’s an impressive curve and it reflects a notable event. But it’s missing some important data and context.

The claim that this set a record seems to originate from a comment by an analyst at investment bank UBS, who [said](https://www.barrons.com/articles/chatgpt-investors-ai-microsoft-stock-51675284609) “We cannot remember an app scaling at this pace”, which strikes me as a reasonable, hedged thing to say. The stronger claim that it set an outright record seems to be misreporting.

#### Data on other apps

I found data on monthly users for all of these apps except Spotify[2](https://aiimpacts.org/how-popular-is-chatgpt-part-2-slower-growth-than-pokemon-go/#easy-footnote-bottom-2-3444 "I skipped Spotify because at first glance it seemed not to be unusually fast, it didn’t seem very easy to find, and I thought the other apps were sufficient to put things in context."). I also searched lists of very popular apps for good leads on something with faster user growth. You can see the full set of data, with sources, [here](https://docs.google.com/spreadsheets/d/1twVx0J6WEc7ViSp8qKeokb9T_cWuqSWO-Z3vPqZyk_s/edit?usp=sharing).[3](https://aiimpacts.org/how-popular-is-chatgpt-part-2-slower-growth-than-pokemon-go/#easy-footnote-bottom-3-3444 "Be warned that, at the time of this writing, the Google sheet is a bit of a mess and the sources are not cited in the most user-friendly way. If you’d like to use the data and you’re having trouble, please don’t hesitate to ask for a cleaner version of it.") I give more details on the data and my methods in the appendix.

From what I can tell, that graph is reasonably accurate, but it’s missing Pokémon GO, which was substantially faster. It’s also missing the Android release of Instagram, which is arguably a new app release, and surpassed 1M within the first day. Here’s a table summarizing the numbers I was able to find, listed in chronological order:

|  |  |  |  |  |
| --- | --- | --- | --- | --- |
| **Service** | **Date launched** | **Days to 1M** | **Days to 10M** | **Days to 100M** |
| Netflix subscribers (all) | 1997-08-29 | 3669 | 4185 | 7337 |
| Facebook | 2004-02-04 | 331 | 950 | 1608 |
| Twitter | 2006-07-15 | 670 | 955 | 1903 |
| Netflix subscribers (streaming) | 2007-01-15 | 1889 | 2351 | 3910 |
| Instagram (all) | 2010-10-01 | 61 | 362 | 854 |
| Instagram (Android) | 2012-04-03 | 1 |  |  |
| Pokemon Go (downloads) | 2016-07-05 |  | 7 | 27 |
| ChatGPT | 2022-11-30 | 4 |  | 61 |

Number of days to reach 1 million, 10 million, and 100 million users, for several apps. Some of the figures are exponentially interpolated, due to a lack of datapoints at the desired values.
It’s a little hard to compare early numbers for ChatGPT and Pokémon GO, since I couldn’t find the days to 1M for Pokémon GO or the days to 10M for ChatGPT, but it seems unlikely that ChatGPT was faster for either.

Analysis
--------

### Scaling by population of Internet users

The total number of people with access to the Internet has been growing rapidly over the last few decades. Additionally, the growth of social networking sites makes it easier for people to share apps with each other. Both of these should make it easier for an app to spread. With that in mind, here’s a graph showing the fraction of all Internet users who are using each app over time (note the logarithmic vertical axis):

[![](https://aiimpacts.org/wp-content/uploads/2023/03/chatgpt-new-colors-1024x791.png)](https://aiimpacts.org/wp-content/uploads/2023/03/chatgpt-new-colors.png)Number of monthly users over time for several applications. The vertical axis is on a log scale.
In general, it looks like these curves have initial slopes that are increasing with time, suggesting that how quickly an app can spread is influenced by more than just an increase in the number of people with access to the Internet. But Pokémon GO and ChatGPT just look like vertical lines of different heights, so here’s another graph, showing the (logarithmic) time since launch for each app:

[![](https://aiimpacts.org/wp-content/uploads/2023/03/apps-since-launch-1-1024x791.png)](https://aiimpacts.org/wp-content/uploads/2023/03/apps-since-launch-1.png)Fraction of total global population with access to the Internet who are using the service vs days since the service launched. The number of users is set somewhat arbitrarily to 1 at t=1 minute
This shows pretty clearly that, while ChatGPT is an outlier, it was nonetheless substantially slower than Pokémon GO[4](https://aiimpacts.org/how-popular-is-chatgpt-part-2-slower-growth-than-pokemon-go/#easy-footnote-bottom-4-3444 "This is still the case if we do not divide by the number of Internet users, which increased by less than a factor of two between 2016 and 2022.").

### Additional comparisons

One more comparison we can make is to other products and services that have a very fast uptake with users and how their reach increases over time:

1. **YouTube views within 24 hours** for newly posted videos gives us a reference point for how quickly a link to something on the Internet can spread and get engagement. The lower barrier to watching a video, compared to making an account for ChatGPT, might give videos an advantage. Additionally, there is presumably more than one view per person. I do not know how big this effect is, but it may be large.
2. **Pay-per-view sales** for live events, in this case for combat sports, are a reference point for something that people are willing to pay for to use at home in a short timeframe. The payment is a higher barrier than making an account, but marketing and sales can happen ahead of time.
3. **Video game sales within 24 hours**, in some cases digital downloads, are similar to pay-per-view, but seem more directly comparable to a service on a website. I would guess that video games benefit from a longer period of marketing and pre-sales than PPV, but I’m not sure.

Here is a graph of records for these things over time, with data taken from Wikipedia[5](https://aiimpacts.org/how-popular-is-chatgpt-part-2-slower-growth-than-pokemon-go/#easy-footnote-bottom-5-3444 "

The relevant Wikipedia pages are:
https://en.wikipedia.org/wiki/List_of_most-viewed_online_videos_in_the_first_24_hours

https://en.wikipedia.org/wiki/Pay-per-view#List_of_pay-per-view_bouts

https://en.wikipedia.org/wiki/List_of_fastest-selling_products

"), which is included in the [data spreadsheet](https://docs.google.com/spreadsheets/d/1twVx0J6WEc7ViSp8qKeokb9T_cWuqSWO-Z3vPqZyk_s/edit?usp=sharing). Each dot is a separate video, PPV event, or game, and I’m only including those that set 24 hour records:

[![](https://aiimpacts.org/wp-content/uploads/2023/03/Screenshot-2023-03-03-at-3.20.37-PM-1024x632.png)](https://aiimpacts.org/wp-content/uploads/2023/03/Screenshot-2023-03-03-at-3.20.37-PM.png)Records for most sales, views, and users within the first 24 hours for video games, PPV bouts, YouTube videos, and apps, plus a few points for users during first week for apps (shown as blue diamonds). Each data point represents one event, game, video, or app. Only those setting records in their particular category are included.
*

It would appear that very popular apps are not as popular as very popular video games or videos. I don’t see a strong conclusion to be drawn from this, but I do think it is helpful context.

### Additional considerations

I suspect the marketing advantage for Pokémon GO and other videogames is substantial. I do not remember seeing ads for Pokémon GO before its release, but I did a brief search for news articles about it before it was released and found lots of hype going back months. I did not find any news articles mentioning ChatGPT before launch. This does not change the overall conclusion, that the claim about ChatGPT setting an outright record is false, but it should change how we think about it.

That ChatGPT was able to beat out most other services without any marketing seems like a big deal. I think it’s hard to sell people on what’s cool about it without lots of user engagement, but the next generation of AI products might not need that, now that people are aware of how far the technology has come. Given this (and the hype around Bing Chat and Bard), I would weakly predict that marketing will play a larger role in future releases.

Appendix – methods and caveats
------------------------------

Most of the numbers I found were for monthly users or, in some cases, monthly active users. I wasn’t always sure what the difference was between these two things. In some cases, all I was able to find was monthly app downloads or annual downloads, both of which I would naively expect to be strictly larger than monthly users. But the annual user numbers reflected longer-term growth anyway, so they shouldn’t affect the conclusions.

Some of the numbers for days to particular user milestones were interpolated, assuming exponential growth. By and large, I do not think this affects the overall story too much, but if you need to know precise numbers, you should check my interpolations or find more direct measurements. None of the numbers is extrapolated.

When searching for data, I tried to use either official sources like SEC filings and company announcements, or measurements from third-party services that seem reputable and have paying customers. But sometimes those were hard to find and I had to use less reliable sources like news reports with dubious citations or studies with incomplete data.

I did not approach this with the intent to produce very reliable data in a very careful way. Overall, this took about 1-2 researcher-days of effort. Given this, it seems likely I made some mistakes, but hopefully not any that undermine the conclusions.

*Thanks to Jeffrey Heninger and Harlan Stewart for their help with research on this. Thanks to the two of them and Daniel Filan for helpful comments.*


Scoring forecasts from the 2016 “Expert Survey on Progress in AI”

*Patrick Levermore, 1 March 2023*

Summary
-------

This document looks at the predictions made by AI experts in [The 2016 Expert Survey on Progress in AI](https://aiimpacts.org/2016-expert-survey-on-progress-in-ai/), analyses the predictions on ‘Narrow tasks’, and gives a Brier score to the median of the experts’ predictions.

My analysis suggests that the experts did a fairly good job of forecasting (Brier score = 0.21), and would have been less accurate if they had predicted each development in AI to generally come, by a factor of 1.5, later (Brier score = 0.26) or sooner (Brier score = 0.29) than they actually predicted.

I judge that the experts expected 9 milestones to have happened by now – and that 10 milestones have now happened.

But there are important caveats to this, such as:

* I have only analysed whether milestones have been publicly met. AI labs may have achieved more milestones in private this year without disclosing them. This means my analysis of how many milestones have been met is probably conservative.
* I have taken the point probabilities given, rather than estimating probability distributions for each milestone, meaning I often round down, which skews the expert forecasts towards being more conservative and unfairly penalises their forecasts for low precision.
* It’s not apparent that forecasting accuracy on these nearer-term questions is very predictive of forecasting accuracy on the longer-term questions.
* My judgements regarding which forecasting questions have resolved positively vs negatively were somewhat subjective (justifications for each question [in the separate appendix](https://docs.google.com/document/d/1CzFaNBnUhN0KIG0GU8RjozdV0S4CsdQlJ8f474HdHXk/edit#bookmark=id.raa8hmybk9b9)).

Introduction
------------

In 2016, AI Impacts published [The Expert Survey on Progress in AI](https://aiimpacts.org/2016-expert-survey-on-progress-in-ai/): a survey of machine learning researchers, asking for their predictions about when various AI developments will occur. The results have been used to inform general and expert opinions on AI timelines.

The survey largely focused on timelines for general/human-level artificial intelligence (median forecast of 2056). However, included in this survey were a collection of questions about shorter-term milestones in AI. Some of these forecasts are now resolvable. Measuring how accurate these shorter-term forecasts have been is probably somewhat informative of how accurate the longer-term forecasts are. More broadly, the accuracy of these shorter-term forecasts seems somewhat informative of how accurate ML researchers’ views are in general. So, how have the experts done so far?

Findings
--------

I analysed the 32 ‘Narrow tasks’ to which the following question was asked:

>
> *How many years until you think the following AI tasks will be feasible with:*
>
>
> * *a small chance (10%)?*
> * *an even chance (50%)?*
> * *a high chance (90%)?*
>
>
> *Let a task be ‘feasible’ if one of the best resourced labs could implement it in less than a year if they chose to. Ignore the question of whether they would choose to.*1
>
>
>

I interpret ‘feasible’ as whether, in ‘less than a year’ before now, any AI models had passed these milestones, and this was disclosed publicly. Since it is now (February 2023) 6.5 years since this survey, I am therefore looking at any forecasts for events happening within 5.5 years of the survey.

Across these milestones, I judge that 10 have now happened and 22 have not happened. My 90% confidence interval is that 7-15 of them have now happened. A full description of milestones, and justification of my judgments, are in [the appendix (separate doc).](https://docs.google.com/document/d/1CzFaNBnUhN0KIG0GU8RjozdV0S4CsdQlJ8f474HdHXk/edit#bookmark=id.raa8hmybk9b9)

The experts forecast that:

* 4 milestones had a <10% chance of happening by now,
* 20 had a 10-49% chance,
* 7 had a 50-89% chance,
* 1 had a >90% chance.

So they expected 6-17 of these milestones to have happened by now. By eyeballing the forecasts for each milestone, my estimate is that they expected ~9 to have happened.2 I did not estimate the implied probability distributions for each milestone, which would make this more accurate.

Using the 10, 50, and 90% point probabilities, we get the following calibration curve:

[![](https://aiimpacts.org/wp-content/uploads/2023/03/graph-10-1024x683.png)](https://aiimpacts.org/wp-content/uploads/2023/03/graph-10.png)
*But*, firstly, the data here is small (there are 7 data points at the 50% mark and 1 at the 90% mark). Secondly, my methodology for this graph, and in the below Brier calculations, is based on rounding down to the nearest given forecast. For example, if a 10% chance was given at 3 years, and a 50% chance at 10 years, the forecast was taken to be 10%, rather than estimating a full probability distribution and finding the 5.5 years point. This skews the expert forecasts towards being more conservative and unfairly penalises a lack of precision.

Brier scores
------------

Overall, across every forecast made, the experts come out with **a Brier score of 0.21**.3 The score breakdown and explanation of the method is [here](https://docs.google.com/spreadsheets/d/175mCZwcZcrFQENcjUL3bd5ksC8ioCo9EZAYdYl8zZjk/edit?usp=sharing).4

For reference, a lower Brier score is better. 0 would mean absolute confidence in everything that eventually happened, 0.25 would mean a series of 50% hedged guesses on anything happening, and randomly guessing from 0% to 100% for every question would yield a Brier score of 0.33.5

Also interesting is the Brier score relative to others who forecast the same events. We don’t have that when looking at the median of our experts – but we could simulate a few other versions:

Bearish6 – if the experts all thought each milestone would take 1.5 times longer than they actually thought, they would’ve gotten a Brier score of 0.27.

Slightly Bearish – if the experts all thought each milestone would take 1.2 times longer than they actually thought, they would’ve gotten a Brier score of 0.25.

Actual forecasts – a Brier score of 0.21.

Slightly Bullish – if the experts all thought each milestone would take 1.2 times less than they actually thought, they would’ve gotten a Brier score of 0.24.

Bullish – if the experts all thought each milestone would take 1.5 times less than they actually thought, they would’ve gotten a Brier score of 0.29.

[![](https://aiimpacts.org/wp-content/uploads/2023/03/Screenshot-2023-03-03-130904-1024x623.png)](https://aiimpacts.org/wp-content/uploads/2023/03/Screenshot-2023-03-03-130904.png)
So, the experts were in general pretty accurate and would have been less so if they had been more or less bullish on the speed of AI development (with the same relative expectations between each milestone).

Taken together, I think this should slightly update us towards the expert forecasts being useful in as yet unresolved cases, and away from the usefulness of estimates which fall outside of 1.5 times further or closer than the expert forecasts.

Randomised – if the experts’ forecast for each specific milestone were randomly assigned to any forecasted date for a different milestone in the collection, they would’ve gotten a Bier score of 0.31 (in the random assignment I received from a random number generator).

I think this should update us slightly towards the surveyed experts generally being accurate on which areas of AI would progress fastest. My assessment is that, compared to the experts’ predictions, AI has progressed more quickly in text generation and coding and more slowly in game playing and robotics. It is not clear now whether this trend will continue, or whether other areas in AI will unexpectedly progress more quickly in the next 5 year period.

Summary of milestones and forecasts
-----------------------------------

*In the below table, the numbers in the cells are the median expert response to “Years after the (2016) survey for which there is a 10, 50 and 90% probability of the milestone being feasible”. The final column is my judgement of whether the milestone was in fact feasible after 5.5 years. Orange shading shows forecasts falling within the 5.5 years between the survey and today.*[*A full description of milestones, and justification of my judgments, are in the appendix.*](https://docs.google.com/document/d/1CzFaNBnUhN0KIG0GU8RjozdV0S4CsdQlJ8f474HdHXk/edit#bookmark=id.raa8hmybk9b9)

|  |  |  |  |  |
| --- | --- | --- | --- | --- |
| **Milestone / Confidence of AI reaching the milestone within X years** | 10 percent | 50 percent | 90 percent | True by Feb 2023? (5.5 + 1 years) |
| Translate a new-to-humanity language | 10 | 20 | 50 | FALSE |
| Translate a new-to-it language | 5 | 10 | 15 | FALSE |
| Translate as well as bilingual humans | 3 | 7 | 15 | FALSE |
| Phone bank as well as humans | 3 | 6 | 10 | FALSE |
| Correctly group unseen objects | 2 | 4.5 | 6.5 | TRUE |
| One-shot image labeling  | 4.5 | 8 | 20 | FALSE |
| Generate video from a photograph | 5 | 10 | 20 | TRUE |
| Transcribe as well as humans | 5 | 10 | 20 | TRUE |
| Read aloud better than humans | 5 | 10 | 15 | FALSE |
| Prove and generate top theorems | 10 | 50 | 90 | FALSE |
| Win Putnam competition | 15 | 35 | 55 | FALSE |
| Win Go with less gametime | 3.5 | 8.5 | 19.5 | FALSE |
| Win Starcraft | 2 | 5 | 10 | FALSE |
| Win any random computer game | 5 | 10 | 15 | FALSE |
| Win angry birds | 2 | 4 | 6 | FALSE |
| Beat professionals at all Atari games | 5 | 10 | 15 | FALSE |
| Win Atari with 20 minutes training | 2 | 5 | 10 | FALSE |
| Fold laundry as well as humans | 2 | 5.5 | 10 | FALSE |
| Beat a human in a 5km race | 5 | 10 | 20 | FALSE |
| Assemble any LEGO | 5 | 10 | 15 | FALSE |
| Efficiently sort very large lists | 3 | 5 | 10 | TRUE |
| Write good Python code | 3 | 10 | 20 | TRUE |
| Answers factoids better than experts | 3 | 5 | 10 | TRUE |
| Answer open-ended questions well | 5 | 10 | 15 | TRUE |
| Answer unanswered questions well | 4 | 10 | 17.5 | TRUE |
| High marks for a high school essay | 2 | 7 | 15 | FALSE |
| Create a top forty song | 5 | 10 | 20 | FALSE |
| Produce a Taylor Swift song | 5 | 10 | 20 | FALSE |
| Write a NYT bestseller | 10 | 30 | 50 | FALSE |
| Concisely explain its game play | 5 | 10 | 15 | TRUE |
| Win World Series of Poker | 1 | 3 | 5.5 | TRUE |
| Output laws of physics of virtual world | 5 | 10 | 20 | FALSE |

Caveats:
--------

**My judgements of which forecasts have turned out true or false are a little subjective.** This was made harder by the survey question asking which tasks were ‘feasible’, where feasible meant*‘if one of the best resourced labs could implement it in less than a year if they chose to. Ignore the question of whether they would choose to.’*I have interpreted this as, one year after the forecasted date, have AI labs achieved these milestones, and disclosed this publicly?

Given (a) ‘has happened’ implies ‘feasible’, but ‘feasible’ does not imply ‘has happened’ and (b) labs may have achieved some of these milestones but not disclosed it, **I am probably being conservative in the overall number of tasks which have been completed by labs**. I have **not** attempted to offset this conservatism by using my judgement of what labs can probably achieve in private. If you disagree or have insider knowledge of capabilities, you may be interested in editing my working [here](https://docs.google.com/spreadsheets/d/175mCZwcZcrFQENcjUL3bd5ksC8ioCo9EZAYdYl8zZjk/copy). Please reach out if you want an explanation of the method, or to privately share updates – patrick at rethinkpriorities dot org.

**It’s not obvious that forecasting accuracy on these nearer-term questions is very predictive of forecasting accuracy on the longer-term questions**. [Dillon (2021)](https://forum.effectivealtruism.org/posts/hqkyaHLQhzuREcXSX/data-on-forecasting-accuracy-across-different-time-horizons#Background) notes *“There is some evidence that forecasting skill generalises across topics (see Superforecasting, Tetlock, 2015 and for a brief overview see*[*here*](https://www.cardrates.com/news/good-judgment-helps-organizations-quantify-risks/)*) and this might inform a prior that good forecasters in the short term will also be good over the long term, but there may be specific adjustments which are worth emphasising when forecasting in different temporal domains.”* I have not found any evidence either way on whether good forecasters in the short term will also be good over the long term, but this does seem possible to analyse from the data that [Dillon](https://forum.effectivealtruism.org/posts/hqkyaHLQhzuREcXSX/data-on-forecasting-accuracy-across-different-time-horizons#PredictionBook_Analysis) and [niplav](https://www.lesswrong.com/posts/MquvZCGWyYinsN49c/range-and-forecasting-accuracy#comments) collect.8

**Finally, there are caveats in the original survey worth noting here, too**. For example, how the question is framed makes a difference to forecasts, even when the meaning is the same. To illustrate this, the authors note

>
> “**People consistently give later forecasts if you ask them for the probability in N years instead of the year that the probability is M**. We saw this in the straightforward HLMI (high-level machine intelligence) question and most of the tasks and occupations, and also in most of these things when we tested them on mturk people earlier. For HLMI for instance, if you ask when there will be a 50% chance of HLMI you get a median answer of 40 years, yet if you ask what the probability of HLMI is in 40 years, you get a median answer of 30%.”
>
>
>

This is commonly true of the ‘Narrow tasks’ forecasts (although I disagree with the authors that it is *consistently* so).9 For example, when asked when there is a 50% chance AI can write a top forty hit, respondents gave a median of 10 years. Yet when asked about the probability of this milestone being reached in 10 years, respondents gave a median of 27.5%.

What does this all mean for us?
-------------------------------

Maybe not a huge amount at this point. It is probably a little too early to get a good picture of the experts’ accuracy, and there are a few important caveats. But this should update you slightly towards the experts’ timelines if you were sceptical of their forecasts. Within another five years, we will have ~twice the data and a good sense of how the experts performed across their 50% estimates.

It is also limiting to have only one comprehensive survey of AI experts which includes both long-term and shorter-term timelines. What would be excellent for assessing accuracy is detailed forecasts from various different groups, including political pundits, technical experts, and professional forecasters, with which we can compare accuracy between groups. It would be easier to analyse the forecasting accuracy of the questions focused on what developments have *happened*, rather than what developments are *feasible*. We could try closer to home, maybe the average EA would be better at forecasting developments than the average AI expert – it seems worth testing now to give us some more data in ten years!

[![](https://aiimpacts.org/wp-content/uploads/2023/03/rethinklogo-1024x385.png)](https://aiimpacts.org/wp-content/uploads/2023/03/rethinklogo.png)
*This is a blog post, not a research report, meaning it was produced quickly and is not to our typical standards of substantiveness and careful checking for accuracy. I’m grateful to Alex Lintz, Amanda El-Dakhakhni, Ben Cottier, Charlie Harrison, Oliver Guest, Michael Aird, Rick Korzekwa, and Zach Stein-Perlman for comments on an earlier draft.*

*If you are interested in RP’s work, please visit our*[*research database*](https://www.rethinkpriorities.org/research)*and subscribe to our*[*newsletter*](https://www.rethinkpriorities.org/newsletter)*.*

*Cross-posted to*[*EA Forum,*](https://forum.effectivealtruism.org/posts/tCkBsT6cAw6LEKAbm/scoring-forecasts-from-the-2016-expert-survey-on-progress-in)*[Lesswrong](https://www.lesswrong.com/posts/tQwjkFT8s2uf2arFN/scoring-forecasts-from-the-2016-expert-survey-on-progress-in), and*[*this google doc*](https://docs.google.com/document/d/1CzFaNBnUhN0KIG0GU8RjozdV0S4CsdQlJ8f474HdHXk/edit)*.*

Footnotes
---------

1. I only analysed this ‘fixed probabilities’ question and not the alternative ‘fixed years’ question, which asked:
“How likely do you think it is that the following AI tasks will be feasible within the next:
– 10 years?
– 20 years?
– 50 years?”
We are not yet at any of these dates, so the analysis would be much more unclear.
2. 9 =  4\*5% + 14\*15% + 6\*30% + 5\*55% + 2\*80% + 1\*90%
3. A*precise* number as a Brier score does not imply an *accurate* assessment of forecasting ability – ideally, we could work with a larger dataset (i.e. more surveys, with more questions) to get more accuracy.
4. My methodology for the Brier score calculations is based on rounding down to the nearest given forecast, or rounding up to the 10% mark. For example, if a 10% chance was given at 3 years, and a 50% chance at 10 years, the forecast was taken to be 10%, rather than estimating a full probability distribution and finding the 5.5 years point. This skews the expert forecasts towards being more conservative and unfairly penalises them. If the experts gave a 10% chance of X happening in 3 years, I didn’t check whether it had happened in 3 years, but instead checked if it had happened by now. I estimate these two factors (the first skewing the forecasts to be more begives a roughly balance 5-10% increase to the Brier score, given most milestones included a probability at the 5 year mark. A better analysis would estimate the probability distributions implied by each 10, 50, 90% point probability, then assess the probability implied at 5.5 years.
5. For more detail, see [Brier score – Wikipedia](https://en.wikipedia.org/wiki/Brier_score).
6. By ‘bearish’ and ‘bullish’ I mean expecting AI milestones to be met later or sooner, respectively.
7. The score breakdown and method for these calculations is also [here](https://docs.google.com/spreadsheets/d/175mCZwcZcrFQENcjUL3bd5ksC8ioCo9EZAYdYl8zZjk/edit?usp=sharing).
8. This seems valuable, and I’m not sure why it hasn’t been analysed yet.
Somewhat relevant sources:<https://forum.effectivealtruism.org/posts/hqkyaHLQhzuREcXSX/data-on-forecasting-accuracy-across-different-time-horizons><https://www.lesswrong.com/posts/MquvZCGWyYinsN49c/range-and-forecasting-accuracy><https://www.openphilanthropy.org/research/how-feasible-is-long-range-forecasting/><https://forum.effectivealtruism.org/topics/long-range-forecasting>
9. I sampled ten forecasts where probabilities were given on a 10 year timescale, and five of them (Subtitles, Transcribe, Top forty, Random game, Explain) gave later forecasts when asked with a ‘probability in N years’ framing rather than a ‘year that the probability is M’ framing, three of them (Video scene, Read aloud, Atari) gave the same forecasts, and two of them (Rosetta, Taylor) gave an earlier forecast. This is why I disagree it leads to *consistently*later forecasts.


How popular is ChatGPT? Part 1: more popular than Taylor Swift

*Harlan Stewart, 23 February 2023*

Introduction
------------

Public attention toward AI seems much higher after the release of ChatGPT at the end of November. But how much higher is it? To better understand this, I looked at search data from Google Trends about ChatGPT, OpenAI, AI, and AI Alignment. Unfortunately, Google Trends only shares relative search volumes instead of the number of searches made for a term or topic. I compared these relative search volumes to other non-AI topics, such as Taylor Swift, to make them more useful. This is similar to adding a familiar “for scale” object in a product photo.

[![](https://aiimpacts.org/wp-content/uploads/2023/02/Magnetic-core_memory_18x24_bits-300x270.jpg)](https://aiimpacts.org/wp-content/uploads/2023/02/Magnetic-core_memory_18x24_bits-scaled.jpg)Magnetic-core memory with a quarter for scale, by [Jud McCranie](https://commons.wikimedia.org/wiki/File:Magnetic-core_memory,_18x24_bits.jpg)
How to read these graphs
------------------------

1. In the first graph, the data is about searches for the terms in quotation marks, which are exact search terms. In the others, the data is about search “topics,” which are collections of various search terms related to a topic, as defined by Google Trends.
2. The vertical axes of these graphs are relative search volume, defined as the percentage of the peak search volume in that graph.

Data
----

### ChatGPT is mainstream

[![](https://aiimpacts.org/wp-content/uploads/2023/02/US-Google-searches-for-_ChatGPT_-or-_Chat-GPT_-_Wordle_-_Taylor-Swift_-and-_Drake_.png)](https://aiimpacts.org/wp-content/uploads/2023/02/US-Google-searches-for-_ChatGPT_-or-_Chat-GPT_-_Wordle_-_Taylor-Swift_-and-_Drake_.png)
For the time that ChatGPT has been publicly available since November 30 2022, [US searches](https://trends.google.com/trends/explore?date=2022-11-30%202023-02-23&geo=US&q=ChatGPT%20%2B%20Chat%20GPT,Wordle,Taylor%20Swift,Drake) for it outnumbered US searches for Taylor Swift or Drake. However, there were only around a third as many searches for ChatGPT as searches for Wordle, and Wordle itself had only [around a third](https://trends.google.com/trends/explore?date=2021-10-01%202023-02-23&geo=US&q=Wordle) of the search volume that it did in Spring 2022.

### Americans suddenly know about OpenAI

[![](https://aiimpacts.org/wp-content/uploads/2023/02/OpenAI-vs-Blockbuster-vs-Samsung.png)](https://aiimpacts.org/wp-content/uploads/2023/02/OpenAI-vs-Blockbuster-vs-Samsung.png)
For the time that OpenAI has existed, since December 10 2015, Americans usually searched for it less than for Blockbuster Video, a retailer that closed in 2014. In the months since ChatGPT was announced, American searches for OpenAI have increased by around 15x to a volume similar to that for Samsung.

### Interest in AI evolved from dinosaurs to birds

[![](https://aiimpacts.org/wp-content/uploads/2023/02/AI-vs-dinosaurs-vs-birds.png)](https://aiimpacts.org/wp-content/uploads/2023/02/AI-vs-dinosaurs-vs-birds.png)
For most of the last decade, there has been a similar number of [global searches](https://trends.google.com/trends/explore?date=2013-01-01%202023-02-21&q=%2Fm%2F0mkz,%2Fm%2F029tx,%2Fm%2F015p6) about AI as about dinosaurs. In the time since DALL-E 2’s beta was announced less than a year ago, global searches about AI have roughly tripled, rising to a volume of global searches similar to that about birds.

### Alignment interest is at an all-time high but still pretty low

[![](https://aiimpacts.org/wp-content/uploads/2023/02/Alignment-vs-scent-tech-vs-moon-colonies.png)](https://aiimpacts.org/wp-content/uploads/2023/02/Alignment-vs-scent-tech-vs-moon-colonies.png)
Over the last 10 years, [global searches](https://trends.google.com/trends/explore?date=2013-01-01%202023-02-21&q=%2Fg%2F11bzrg63f7,%2Fm%2F0gvvmxl,%2Fm%2F03jy3f) about AI alignment have risen from “digital scent technology” level to “colonization of the moon” level and possibly beyond. Searches about AI alignment seem to have roughly quadrupled in the last two years. Eyeballing this graph, it’s unclear to me whether the announcements of DALL-E 2 or ChatGPT had any significant effect on search volume.

Discussion
----------

ChatGPT is receiving mainstream attention. Although I have not done any statistical analysis of these trends, it appears to me that the popularity of ChatGPT is also driving interest in both OpenAI as a company and AI in general. Interest in alignment is also on the rise but still about as obscure an interest as colonization of the moon.

It’s unclear whether interest in AI will continue to grow, plateau, or drop back to previous levels. This will likely depend on what near-term future progress in AI will look like. If you expect that AI-related news as interesting as ChatGPT will be rare, you might expect interest to decline as the hype fizzles out. If you expect that the pace of interesting AI advancements will continue at its current fast rate, you might expect interest in AI to continue to grow, perhaps becoming even more popular than birds.


The public supports regulating AI for safety

*Zach Stein-Perlman, 16 February 2023*

A high-quality American public survey on AI, [Artificial Intelligence Use Prompts Concerns](https://www.monmouth.edu/polling-institute/documents/monmouthpoll_us_021523.pdf/), was released yesterday by Monmouth. Some notable results:

* 9% say AI[1](https://aiimpacts.org/the-public-supports-regulating-ai-for-safety/#easy-footnote-bottom-1-3420 "The survey defined “AI” as “the ability of computers and machines to carry out decision-making and thought processes similar to humans, sometimes referred to as computers being able to think for themselves.” The survey elicited generally more negative opinions on AI than other surveys, possibly because its definition of “AI” is relatively scary or possibly because public opinion has recently become more pessimistic.") would do more good than harm vs 41% more harm than good (similar to responses to a similar survey in 2015)
* 55% say AI could eventually pose an existential threat (up from 44% in 2015)
* 55% favor “having a federal agency regulate the use of artificial intelligence similar to how the FDA regulates the approval of drugs and medical devices”
* 60% say they have “heard about A.I. products – such as ChatGPT – that can have conversations with you and write entire essays based on just a few prompts from humans”

Worries about safety and support of regulation echoes [other surveys](https://wiki.aiimpacts.org/doku.php?id=responses_to_ai:public_opinion_on_ai:surveys_of_public_opinion_on_ai:surveys_of_us_public_opinion_on_ai):

* 71% of Americans agree that there should be national regulations on AI ([Morning Consult 2017](https://morningconsult.com/wp-content/uploads/2017/04/170401_crosstabs_Brands_v3_AG.pdf))
* The public is concerned about some AI policy issues, especially privacy, surveillance, and cyberattacks ([GovAI 2019](https://governanceai.github.io/US-Public-Opinion-Report-Jan-2019/executive-summary.html))
* The public is concerned about various negative consequences of AI, including loss of privacy, misuse, and loss of jobs ([Stevens / Morning Consult 2021](https://assets.stevens.edu/mviowpldu823/46xBb3AAf6Agnd1Avvfill/f1ae8499925ba19621bbbb32bbd29cdd/AI-Summery-Deck.pdf))

Surveys match the anecdotal evidence from talking to Uber drivers: Americans are worried about AI safety and would support regulation on AI. Perhaps there is an opportunity to improve the public’s beliefs, attitudes, and memes and frames for making sense of AI; perhaps better public opinion would enable better policy responses to AI or actions from AI labs or researchers.

Public desire for safety and regulation is far from sufficient for a good government response to AI. But it does mean that the main challenge for improving government response is helping relevant actors believe what’s true, developing good [affordances](https://www.lesswrong.com/posts/AGrBjEb6vFaJBwjTj/framing-ai-strategy-1#Affordances) for them, and helping them take good actions— not making people care enough about AI to act at all.


Whole Bird Emulation requires Quantum Mechanics

*Jeffrey Heninger, 14 February 2023*

*Epistemic status: Written for engagement. More sober analysis coming soon.*

>
> Bird navigation is surprisingly cruxy for the future of AI.
>
>
>  – Zach Stein-Perlman

>
> This seems pretty wrong.
>
>
>  – Richard Korzekwa

Birds are astonishingly good at navigating, even over thousands of miles. The longest migration routes, of the arctic term, are only limited by the size of the globe. Homing pigeons can return home after being released 1800 km (1100 mi) away. White-crowned sparrows have been able to migrate to their wintering grounds after being displaced 3700 km (2300 mi) shortly before they began migration.

How they do this is not entirely understood. There seem to be multiple cues they respond to, which combine to give them an accurate ‘map’ and ‘compass’. Which cues are most important might be different for different species. Some of these cues include watching the stars & sun, low frequency sounds, long-range smells, and detecting the earth’s magnetic field. This last one is the most interesting. Birds can detect magnetic fields, and there is increasing consensus that the detection mechanism involves quantum mechanics (See Appendix for details).

The result is a precise detector of the magnetic field. It is located in the retina and transferred up the optical nerve to the brain, so birds can ‘see’ magnetic fields. Leaving aside questions like “[What is it like to be a [Bird]?](https://warwick.ac.uk/fac/cross_fac/iatl/study/ugmodules/humananimalstudies/lectures/32/nagel_bat.pdf)”, this result has implications for the difficulty of Whole Bird Emulation (WBE).

WBE is important for understanding the future development of artificial intelligence. If we can put an upper bound on the difficulty of WBE, we have an upper bound on the difficulty of making AI that can do everything a bird can do. And birds can do lots of cool things: they know how to fly, they sing pretty songs, and they even [drop nuts in front of cars](https://www.youtube.com/watch?v=BGPGknpq3e0) !

In order to put bounds on WBE, we need to determine how much resolution is needed in order to emulate everything a bird can do. Is it good enough to model a bird at the cellular level? Or at the protein level? Or do you need an even finer resolution?

In order to model the navigational ability of a bird, you need a quantum mechanical description of the spin state of a pair of electrons. This is extremely high resolution.

A few caveats:

* Not all parts of a bird require quantum mechanics to describe their macroscopic behavior. You can likely get away with coarse-graining most of the bird at a much higher level.
* This is a simple quantum system, so it’s not hard to figure out the wave function over the singlet and triplet states.
* What you need to know to determine the behavior of the bird is the concentration of the two final products as a function of the external magnetic field. Once this (quantum mechanical) calculation is done, you likely don’t need to model the subsequent evolution of the bird using quantum mechanics.

On the other hand:

* Birds are extremely complicated things, so it is always somewhat surprising when we understand anything in detail about them.
* If quantum mechanics is necessary to understand the macroscopic behavior of some part of a bird, we should think that it is more likely that quantum mechanics is necessary to understand the macroscopic behavior of other parts of a bird too.
* If there are other parts of a bird which depend on quantum mechanics in a more complicated way, or if the macroscopic response cannot be well modeled using classical probabilities, we almost certainly would not have discovered it. Getting good empirical evidence for even simple models of biological systems is hard. Getting good empirical evidence for complex models of biological systems is much harder.

WBE requires a quantum mechanical calculation in order to describe at least one macroscopic behavior of birds. This dramatically increases the resolution needed for at least parts of WBE and the overall expected difficulty of WBE. If your understanding of artificial intelligence would have predicted that Whole Bird Emulation would be much simpler than this, you should update accordingly.

Unless, of course, Birds Aren’t Real.

Further Reading
---------------

* Lambert et al. *Quantum Biology.* Nature Physics **9**. (2013) <https://quantum.ch.ntu.edu.tw/ycclab/wp-content/uploads/2015/01/Nat-Phys-2013-Lambert.pdf>.
* Holland. *True navigation in birds: from quantum physics to global migration.* Journal of Zoology **293**. (2014) <https://zslpublications.onlinelibrary.wiley.com/doi/pdfdirect/10.1111/jzo.12107>.
* Ritz. *Quantum effects in biology: Bird navigation.* Procedia Chemistry **3**. (2011) <https://www.sciencedirect.com/science/article/pii/S1876619611000738>.

Appendix
--------

Here is a brief description of how a bird’s magnetic sense seems to work:

A bird’s retina contains some pigments called cryptochromes. When blue or green light (<570 nm) is absorbed by the pigment, an electron is transferred from one molecule to another. This electron had previously been paired with a different electron, so after the transfer, there is now an excited radical pair. Initially, the spins of the two electrons are anti-parallel (they initially are in the singlet state). An external magnetic magnetic field can cause one of the electrons to flip so they become parallel (they transition to a triplet state). Transitions can also occur due to interactions with the nuclear spins, so it is better to think of the external magnetic field as changing the rate at which transitions happen instead of introducing entirely new behavior. The excited singlet state decays back to the original state of the cryptochrome, while the excited triplet state decays into a different product. Neurons in the retina can detect the change in the relative concentration of these two products, providing a measurement of the magnetic field.

This model has made several successful predictions. (1) Cryptochromes were originally known from elsewhere in biology. This theory predicted that they, or another pigment which produces radical pairs, would be found in birds’ eyes. (2) Low amplitude oscillating magnetic fields with a frequency of between 1-100 MHz should also affect the transition between the singlet and triplet states. Exposing birds to these fields disrupts their ability to navigate.


Framing AI strategy

*Zach Stein-Perlman, 6 February 2023*

*Strategy* is the activity or project of doing research to inform interventions to achieve a particular goal.[1](https://aiimpacts.org/framing-ai-strategy/#easy-footnote-bottom-1-3406 "So in my usage, strategy includes forecasting and the research component of governance.") *AI strategy* is strategy from the perspective that AI is important, focused on interventions to make AI go better. An analytic *frame* is a conceptual orientation that makes salient some aspects of an issue, including cues for what needs to be understood, how to approach the issue, what your goals and responsibilities are, what roles to see yourself as having, what to pay attention to, and what to ignore.

This post discusses ten strategy frames, focusing on AI strategy. Some frames are comprehensive approaches to strategy; some are components of strategy or prompts for thinking about an aspect of strategy. This post focuses on meta-level exploration of frames, but the second and last sections have some object-level thoughts within a frame.

Sections are overlapping but independent; focus on sections that aren’t already in your toolbox of approaches to strategy.

*Epistemic status: exploratory, brainstormy.*

**Make a plan**
---------------

*See Jade Leung’s* [*Priorities in AGI governance research*](https://www.listennotes.com/podcasts/ea-radio/priorities-in-agi-governance-WM_DUyzNPqR/) *(2022) and* [*How can we see the impact of AI strategy research?*](https://web.archive.org/web/20220702141246/https://www.effectivealtruism.org/articles/jade-leung-how-can-we-see-the-impact-of-ai-strategy-research) *(2019).*

One output of strategy is a plan describing relevant (kinds of) actors’ behavior. More generally, we can aim for a playbook– something like a function from (sets of observations about) world-states to plans. A plan is good insofar as it improves important decisions in the counterfactual where you try to implement it, in expectation.

To make a plan or playbook, identify (kinds of) actors that might be affectable, then figure out

1. what they could do,
2. what it would be good for them to do,
3. what their incentives are (if relevant), and then
4. how to cause them to act better.

It is also possible to focus on decisions rather than actors: determine what decisions you want to affect (presumably because they’re important and affecting them seems tractable) and how you can affect them.

For AI, relevant actors include AI labs, states (particularly America), non-researching non-governmental organizations (particularly standard-setters), compute providers, and the AI risk and EA communities.[2](https://aiimpacts.org/framing-ai-strategy/#easy-footnote-bottom-2-3406 "

Eli Lifland comments:

I think it’s often helpful to be even more granular than this, and identify a particular person whose actions you want to inform. This helps make your goal more concrete, and even if it’s helpful for lots of people making it helpful for a single person is a good start and often a good proxy and provides good feedback loops (e.g. you can literally ask them if it was informative, and iterate based on their feedback, etc.).

I definitely agree in the special case where the person acts optimally given their information. In practice, I fear that most people will (1) act predictably suboptimally, such that you should try to improve their actions beyond just informing them, and (2) predictably incorrectly identify what it would be best for them to be better informed about, such that you should try to inform them about other topics.")
Insofar as an agent (not necessarily an actor that can take directly important actions) has distinctive abilities and is likely to try to execute good ideas you have, it can be helpful to focus on *what the agent can do* or *how to leverage the agent’s distinctive abilities* rather than backchain from *what would be good*.[3](https://aiimpacts.org/framing-ai-strategy/#easy-footnote-bottom-3-3406 "Suppose you have a magical superpower. You should not do strategy as usual, then try to use your superpower to achieve the resulting goals. Instead, you should start by reasoning about your superpower, considering how you can leverage it most effectively. Similarly, insofar as you or your organization or community has distinctive abilities, it can be helpful to focus on those abilities.")
**Affordances**
---------------

As in the previous section, a natural way to improve the future is to identify relevant actors, determine what it would be good for them to do, and cause them to do those things. “Affordances” in strategy are “possible partial future actions that could be communicated to relevant actors, such that they would take similar actions.”[4](https://aiimpacts.org/framing-ai-strategy/#easy-footnote-bottom-4-3406 "Alex Gray, personal communication, 9 Dec. 2022.") The motivation for searching for and improving affordances is that there probably exist actions that would be great and relevant actors would be happy to take, but that they wouldn’t devise or recognize by default. Finding great affordances is aided by a deep understanding of how an actor thinks and its incentives, as well as a deep external understanding of the actor, to focus on its blind spots and identify feasible actions.[5](https://aiimpacts.org/framing-ai-strategy/#easy-footnote-bottom-5-3406 "

 Katja Grace comments:

My guess is that in many cases things one can do in a situation are by default not noticed by people, and there is a small set they do notice, because other people do them. I’m thinking outside of AI strategy, but figure it probably generalizes e.g. nobody had an affordance for making virgin/chad memes, then after someone did it, lots of people developed an affordance. People mostly don’t have an affordance for asking someone else to shut up, but if they see someone do it very smoothly, they might adopt whatever strategy that was. On technology safety, the Asilomar Conference I think gives people the affordance for doing ‘something like the Asilomar Conference’ (in various ways like). i.e. often the particular details of the doer don’t matter that much—the idea of doing a generally useful thing is appealing to a lot of different doers, and just everyone’s visual field is made mostly of blindspots, re actions.

") Separately, the actor’s participation would sometimes be vital.

Affordances are relevant not just to cohesive actors but also to non-structured groups. For example, for AI strategy, discovering affordances for ML researchers (as individuals or for collective action) could be valuable. Perhaps there also exist great possible affordances that don’t depend much on the actor– generally helpful actions that people just aren’t aware of.

For AI, two relevant kinds of actors are states (particularly America) and AI labs. One way to discover affordances is to brainstorm the kinds of actions particular actors can take, then find creative new plans within that list. Going less meta, I made lists of the kinds of actions states and labs can take that may be strategically significant, since such lists seem worthwhile and I haven’t seen anything like them.

Kinds of things states can do that may be strategically relevant (or consequences or characteristics of possible actions):

* Regulate (and enforce regulation in their jurisdiction and investigate possible violations)
* Expropriate property and nationalize companies (in their territory)
* Perform or fund research (notably including through Manhattan/Apollo-style projects)
* Acquire capabilities (notably including military and cyber capabilities)
* Support particular people, companies, or states
* Disrupt or attack particular people, companies, or states (outside their territory)
* Affect what other actors believe on the object level
	+ Share information
	+ Make information salient in a way that predictably affects beliefs
	+ Express attitudes that others will follow
* Negotiate with other actors, or affect other actors’ incentives or meta-level beliefs
* Make agreements with other actors (notably including contracts and treaties)
* Establish standards, norms, or principles
* Make unilateral declarations (as an international legal commitment) [less important]

Kinds of things AI labs[6](https://aiimpacts.org/framing-ai-strategy/#easy-footnote-bottom-6-3406 "It may also be useful to consider possible actors related to but distinct from a lab, such as a lab with a substantial lead, a major lab, or all major labs collectively.") can do—or choose not to do—that may be strategically relevant (or consequences or characteristics of possible actions):

* Deploy an AI system
* Pursue capabilities
	+ Pursue risky (and more or less alignable systems) systems
	+ Pursue systems that enable risky (and more or less alignable) systems
	+ Pursue weak AI that’s mostly orthogonal to progress in risky stuff for a specific (strategically significant) task or goal
		- This could enable or abate catastrophic risks besides unaligned AI
* Do alignment (and related) research (or: decrease the [alignment tax](https://forum.effectivealtruism.org/posts/63stBTw3WAW6k45dY/paul-christiano-current-work-in-ai-alignment) by doing technical research)
	+ Including interpretability and work on solving or avoiding alignment-adjacent problems like [decision theory and strategic interaction](https://www.lesswrong.com/posts/brXr7PJ2W4Na2EW2q/the-commitment-races-problem) and maybe [delegation involving multiple humans or multiple AI systems](http://acritch.com/arches/)
* Advance global capabilities
	+ Publish capabilities research
	+ Cause investment or spending in big AI projects to increase
* Advance alignment (or: decrease the alignment tax) in ways other than doing technical research
	+ Support and coordinate with external alignment researchers
* Attempt to align a particular system (or: try to pay the alignment tax)
* Interact with other labs[7](https://aiimpacts.org/framing-ai-strategy/#easy-footnote-bottom-7-3406 "Holden Karnofsky says:

“Deals with other companies. Magma [a fictional AI lab] might be able to reduce some of the pressure to “race” by making explicit deals with other companies doing similar work on developing AI systems, up to and including mergers and acquisitions (but also including more limited collaboration and information sharing agreements).

– Benefits of such deals might include (a) enabling freer information sharing and collaboration; (b) being able to prioritize alignment with less worry that other companies are incautiously racing ahead; (c) creating incentives (e.g., other labs’ holding equity in Magma) to cooperate rather than compete; and thus (d) helping Magma get more done (more alignment work, more robustly staying ahead of other key actors in terms of the state of its AI systems).
– These sorts of deals could become easier to make once Magma can establish itself as being likely to lead the way on developing transformative AI (compared to today, when my impression is that different companies have radically different estimates of which companies are likely to end up being most relevant in the long run).”")
	+ Coordinate with other labs (notably including coordinating to avoid risky systems)
		- Make themselves transparent to each other
		- Make themselves transparent to an external auditor
		- Merge
		- Effectively commit to share upsides
		- Effectively commit to [stop and assist](https://openai.com/charter/)
	+ Affect what other labs believe on the object level (about AI capabilities or risk in general, or regarding particular memes)
		- Practice [selective information sharing](https://www.lesswrong.com/posts/vZzg8NS7wBtqcwhoJ/nearcast-based-deployment-problem-analysis#SelectiveInformationSharing)
		- Demonstrate AI risk (or provide evidence about it)
	+ Negotiate with other labs, or affect other labs’ incentives or meta-level beliefs
* Affect public opinion, media, and politics
	+ Publish research
	+ Make demos or public statements
	+ Release or deploy AI systems
* Improve their culture or [operational adequacy](https://www.lesswrong.com/s/v55BhXbpJuaExkpcD/p/keiYkaeoLHoKK4LYA)
	+ Improve operational security
	+ Affect attitudes of effective leadership
	+ Affect attitudes of researchers
	+ Make a plan for alignment (e.g., [OpenAI’s](https://openai.com/blog/our-approach-to-alignment-research/)); share it; update and improve it; and coordinate with capabilities researchers, alignment researchers, or other labs if relevant
	+ Make a plan for what to do with powerful AI (e.g., [CEV](https://arbital.com/p/cev/) or some specification of [long reflection](https://forum.effectivealtruism.org/topics/long-reflection)), share it, update and improve it, and coordinate with other actors if relevant
	+ Improve their ability to make themselves (selectively) transparent
* Try to better understand the future, the strategic landscape, risks, and possible actions
* Acquire resources
	+ E.g., money, hardware, talent, influence over states, status/prestige/trust
	+ Capture scarce resources
		- E.g., language data from language model users
* Affect other actors’ resources
	+ Affect the flow of talent between labs or between projects
* Plan, execute, or participate in [pivotal acts](https://arbital.com/p/pivotal/) or [processes](https://www.lesswrong.com/posts/etNJcXCsKC6izQQZj/pivotal-outcomes-and-pivotal-processes)

(These lists also exist on the [AI Impacts wiki](https://wiki.aiimpacts.org/), where they may be improved in the future: [Affordances for states](https://wiki.aiimpacts.org/doku.php?id=responses_to_ai:affordances:state_affordances) and [Affordances for AI labs](https://wiki.aiimpacts.org/doku.php?id=responses_to_ai:affordances:lab_affordances). These lists are written from an alignment-focused and misuse-aware perspective, but prosaic risks may be important too.)

Maybe making or reading lists like these can help you notice good tactics. But innovative affordances are necessarily not things that are already part of an actor’s behavior.

Maybe making lists of *relevant things similar actors have done in the past* would illustrate possible actions, build intuition, or aid communication.

This frame seems like a potentially useful complement to the standard approach [*backchain*](https://www.lesswrong.com/posts/DwoPGM8ytBCXrZpM7/backchaining-in-strategy) *from goals to actions of relevant actors*. And it seems good to understand *actions that should be items on lists like these*—both like understanding these list-items well and expanding or reframing these lists—so you can notice opportunities.

**Intermediate goals**
----------------------

*No great sources are public, but illustrating this frame see “Catalysts for success” and “Scenario variables” in Marius Hobhannon et al.’s* [*What success looks like*](https://forum.effectivealtruism.org/posts/AuRBKFnjABa6c6GzC/what-success-looks-like#Catalysts_for_success) *(2022). On goals for AI labs, see Holden Karnofsky’s* [*Nearcast-based “deployment problem” analysis*](https://www.lesswrong.com/posts/vZzg8NS7wBtqcwhoJ/nearcast-based-deployment-problem-analysis) *(2022).*

An intermediate/instrumental goal is a goal that is valuable because it promotes one or more final/terminal goals. (“Goal” sounds discrete and binary, like “there exists a treaty to prevent risky AI development,” but often should be continuous, like “gain resources and influence.”) Intermediate goals are useful because we often need more specific and actionable goals than “make the future go better” or “make AI go better.”

*Knowing what specifically would be good for people to do* is a bottleneck on *people doing useful things*. If the AI strategy community had better strategic clarity, in terms of knowledge about the future and particularly intermediate goals, it could better utilize people’s labor, influence, and resources. Perhaps an overlapping strategy framing is *finding or unlocking effective opportunities to spend money*. See Luke Muehlhauser’s [A personal take on longtermist AI governance](https://forum.effectivealtruism.org/posts/M2SBwctwC6vBqAmZW/a-personal-take-on-longtermist-ai-governance) (2021).[8](https://aiimpacts.org/framing-ai-strategy/#easy-footnote-bottom-8-3406 "Muehlhauser says:

We lack the strategic clarity and forecasting ability to know which “intermediate goals” are high-ROI or even net-positive to pursue (from a longtermist perspective). If we had more clarity on intermediate goals, we could fund more people who are effectively pursuing those goals, whether they are sympathetic to longtermism or not.

")
It is also sometimes useful to consider *goals about particular actors*.

**Threat modeling**
-------------------

*Illustrating threat modeling for the technical component of AI misalignment, see the DeepMind safety team’s* [*Threat Model Literature Review*](https://www.lesswrong.com/posts/wnnkD6P2k2TfHnNmt/threat-model-literature-review) *and* [*Clarifying AI X-risk*](https://www.lesswrong.com/posts/GctJD5oCDRxCspEaZ/clarifying-ai-x-risk) *(2022), Sam Clarke and Sammy Martin’s* [*Distinguishing AI takeover scenarios*](https://www.lesswrong.com/posts/qYzqDtoQaZ3eDDyxa/distinguishing-ai-takeover-scenarios) *(2021), and GovAI’s* [*Survey on AI existential risk scenarios*](https://forum.effectivealtruism.org/posts/2tumunFmjBuXdfF2F/survey-on-ai-existential-risk-scenarios-1) *(2021).*

The goal of [threat modeling](https://www.lesswrong.com/tag/threat-models) is deeply understanding one or more risks for the purpose of informing interventions. A great causal model of a threat (or *class of possible failures*) can let you identify points of intervention and determine what countering the threat would require.

A related project involves assessing all threats (in a certain class) rather than a particular one, to help account for and prioritize between different threats.

Technical AI safety research informs AI strategy through threat modeling. A causal model of (part of) AI risk can generate a model of AI risk abstracted for strategy, with relevant features made salient and irrelevant details black-boxed. This abstracted model gives us information including necessary and sufficient conditions or intermediate goals for averting the relevant threats. These in turn can inform affordances, tactics, policies, plans, influence-seeking, and more.

**Theories of victory**
-----------------------

*I am not aware of great sources, but illustrating this frame see Marius Hobhannon et al.’s* [*What success looks like*](https://forum.effectivealtruism.org/posts/AuRBKFnjABa6c6GzC/what-success-looks-like) *(2022).*

Considering theories of victory is another natural frame for strategy: consider scenarios where the future goes well, then find interventions to nudge our world toward those worlds. (Insofar as it’s not clear what *the future going well* means, this approach also involves clarifying that.) To find interventions to make our world like a victorious scenario, I sometimes try to find necessary and sufficient conditions for the victory-making aspect of that scenario, then consider how to cause those conditions to hold.[9](https://aiimpacts.org/framing-ai-strategy/#easy-footnote-bottom-9-3406 "One useful output of related analysis is finding different necessary and sufficient conditions for victory than the most straightforward or common ones. For example, it is commonly assumed that there is a natural race for AI capabilities, but if we go back to first principles, we can find that on some views there’s no natural race because ‘winning the race’ is bad even for the winner. This observation can lead to new necessary and sufficient conditions for victory– perhaps in this case we guess that if AI labs appreciate AI risk they shouldn’t race or if they don’t they will, so victory conditions include propositions related to AI labs appreciate AI risk. And then the new framing of victory may be quite informative for interventions.")
Great threat-model analysis can be an excellent input to theory-of-victory analysis, to clarify the threats and what their solutions must look like. And it could be useful to consider scenarios in which the future goes well and scenarios where it doesn’t, then examine the differences between those worlds.

**Tactics and policy development**
----------------------------------

*Collecting progress on possible government policies, see GovAI’s* [*AI Policy Levers*](https://www.governance.ai/research-paper/ai-policy-levers-a-review-of-the-u-s-governments-tools-to-shape-ai-research-development-and-deployment) *(2021) and GCRI’s* [*Policy ideas database*](https://www.gcrpolicy.com/ideas)*.*

Given a model of the world and high-level goals, we must figure out how to achieve those goals in the messy real world. For a goal, what would cause success, which of those possibilities are tractable, and how could they become more likely to occur? For a goal, what are necessary and sufficient conditions for achievement and how could those occur in the real world?

**Memes & frames**
------------------

*I am not aware of great sources on memes & frames in strategy, but see Jade Leung’s* [*How can we see the impact of AI strategy research?*](https://web.archive.org/web/20220702141246/https://www.effectivealtruism.org/articles/jade-leung-how-can-we-see-the-impact-of-ai-strategy-research) *(2019). See also the academic literature on framing, e.g. Robert Entman’s* [*Framing*](https://is.muni.cz/el/1423/podzim2018/POL256/um/Entman_1993_FramingTowardclarificationOfAFracturedParadigm.pdf) *(1993).*

(“Frames” in this context refers to the lenses through which people interpret the world, not the analytic, research-y frames discussed in this post.)

If certain actors held certain attitudes, they would make better decisions. One way to affect attitudes is to spread [memes](https://www.lesswrong.com/tag/memetics). A meme could be *explicit agreement with a specific proposition*; *the attitude that certain organizations, projects, or goals are (seen as) shameful*; *the attitude that certain ideas are sensible and respectable or not*; or merely *a tendency to pay more attention to something*. The goal of meme research is finding good memes—memes that would improve decisions if widely accepted (or accepted by a particular set of actors[10](https://aiimpacts.org/framing-ai-strategy/#easy-footnote-bottom-10-3406 "What it would be great for national security people to believe is distinct from what it would be great for machine learning people to believe. These differences are mostly due to relevance: for example, memes about particular US government interventions are very relevant to policy people and have little relevance to ML people. Differences are also due in part to it being optimal for different actors to have different attitudes; for example, perhaps it would be good if ML people believed AI is totally terrifying and bad if policy people believed AI is totally terrifying. Note also that how well a meme spreads, and how it could be better spread or stifled, differs by audience too.")) and are tractable to spread—and figuring out how to spread them. Meme research is complemented by work actually causing those memes to spread.

For example, potential good memes in AI safety include things like *AI is powerful but not robust, and in particular [specification gaming or Goodhart or distributional shift or adversarial attack] is a big deal*. Perhaps *misalignment as catastrophic accidents* is easier to understand than *misalignment as powerseeking agents*, or vice versa. And perhaps *misuse risk* is easy to understand and unlikely to be catastrophically misunderstood, but less valuable-if-spread.

A frame tells people what to notice and how to make sense of an aspect of the world. Frames can be internalized by a person or contained in a text. Frames for AI might include frames related to consciousness, Silicon Valley, AI racism, national security, or specific kinds of applications such as chatbots or weapons.

Higher-level research could also be valuable. This would involve topics like *how to communicate ideas about AI safety* or even *how to communicate ideas* and *how groups form beliefs*.

This approach to strategy could also involve researching how to stifle harmful memes, like perhaps “powerful actors are incentivized to race for highly capable AI” or “we need a Manhattan Project for AI.”

**Exploration, world-modeling, and forecasting**
------------------------------------------------

Sometimes strategy greatly depends on particular questions about the world and the future.

More generally, you can reasonably expect that increasing clarity about important-seeming aspects of the world and the future will inform strategy and interventions, even without thinking about specific goals, actors, or interventions. For AI strategy, exploration includes central questions about the future of AI and relevant actors, understanding the effects of possible actions, and perhaps also topics like decision theory, acausal trade, digital minds, and anthropics.

*Constructing a map* is part of many different approaches to strategy. This roughly involves understanding the landscape and discovering analytically useful concepts, like reframing *victory means causing AI systems to be aligned* to *it’s necessary and sufficient to cause the alignment tax to be paid, so it’s necessary and sufficient to reduce the alignment tax and increase the amount-of-tax-that-would-be-paid such that the latter is greater*.

One exploratory, world-model-y goal is a high-level understanding of the strategic landscape. One possible approach to this goal is creating a map of relevant possible events, phenomena, actions, propositions, uncertainties, variables, and/or analytic nodes.

**Nearcasting**
---------------

*Discussing nearcasting, see Holden Karnofsky’s* [*AI strategy nearcasting*](https://www.lesswrong.com/posts/Qo2EkG3dEMv8GnX8d/ai-strategy-nearcasting) *(2022). Illustrating nearcasting, see Karnofsky’s* [*Nearcast-based “deployment problem” analysis*](https://www.lesswrong.com/posts/vZzg8NS7wBtqcwhoJ/nearcast-based-deployment-problem-analysis) *(2022).*

Holden Karnofsky defines “AI strategy nearcasting” as

trying to answer key strategic questions about transformative AI, under the assumption that key events (e.g., the development of transformative AI) will happen in a world that is otherwise relatively similar to today’s. One (but not the only) version of this assumption would be “Transformative AI will be developed soon, using methods like what AI labs focus on today.”

When I think about AI strategy nearcasting, I ask:

* What would a near future where powerful AI could be developed look like?
* In this possible world, what goals should we have?
* In this possible world, what important actions could relevant actors take?
	+ And what facts about the world make those actions possible? (For example, some actions would require that a lab has certain AI capabilities, or most people believe a certain thing about AI capabilities, or all major labs believe in AI risk.)
* In this possible world, what interventions are available?
* Relative to this possible world, how should we expect the real world to be different?[11](https://aiimpacts.org/framing-ai-strategy/#easy-footnote-bottom-11-3406 "Karnofsky says:

“nearcasting can serve as a jumping-off point. If we have an idea of what the best actions to take would be if transformative AI were developed in a world otherwise similar to today’s, we can then start asking “Are there particular ways in which we expect the future to be different from the nearer term, that should change our picture of which actions would be most helpful?”")

* And how do those differences affect the goals we should have, and the interventions that are available to us?

Nearcasting seems to be a useful tool for

* predicting relevant events concretely and
* forcing you to notice how you think the world will be different in the future and how that matters.

**Leverage**
------------

*I’m not aware of other public writeups on leverage. See also Daniel Kokotajlo’s* [*What considerations influence whether I have more influence over short or long timelines?*](https://www.lesswrong.com/posts/pTK2cDnXBB5tpoP74/what-considerations-influence-whether-i-have-more-influence) *(2020). Related concept:* [*crunch time*](https://www.lesswrong.com/posts/E7rhL9aij7yCCz9AR/what-s-going-on-with-crunch-time)*.*

When doing strategy and planning interventions, what should you focus on?

A major subquestion is: how should you prioritize focus between possible worlds?[12](https://aiimpacts.org/framing-ai-strategy/#easy-footnote-bottom-12-3406 "Central examples of a “possible world” are the possible worlds described by the conditions human-level AI appears around 2030 or AI takeoff is fast.") Ideally you would prioritize working on *the worlds that working on has highest expected value*, or something like *the worlds that have the greatest product of probability and how much better they would go if you worked on them*. But how can you guess which worlds are high-leverage for you to work on? There are various reasons to prioritize certain possible worlds, both for reasoning about strategy and for evaluating possible interventions. For example, it seems higher-leverage to work on making AI go well conditional on human-level AI appearing in 2050 than in 3000: the former is more foreseeable, more affectable, and more neglected.

We currently lack a good account of leverage, so (going less meta) I’ll begin one for AI strategy here. Given a baseline of weighting possible worlds by their probability, all else equal, you should generally:

* Upweight worlds that you have more control over and that you can better plan for
	+ Upweight worlds with short-ish [timelines](https://www.lesswrong.com/tag/ai-timelines) (since others will exert more influence over AI in long-timelines worlds, and since we have more clarity about the nearer future, and since we can revise strategies in long-timelines worlds)
	+ Take into account future strategy research
		- For example, if you focus on the world in 2030 (or assume that human-level AI is developed in 2030) you can be deferring, not neglecting, some work on 2040
		- For example, if you focus on worlds in which important events happen without much advance warning or clearsightedness, you can be deferring, not neglecting, some work on worlds in which important events happen foreseeably
	+ Focus on what you can better plan for and influence; for AI, perhaps this means:
		- Short timelines
		- The deep learning paradigm continues
		- Powerful AI is resource-intensive
		- Maybe some propositions about risk awareness, warning shots, and world-craziness
	+ Upweight worlds where the probability of victory is relatively close to 50%[13](https://aiimpacts.org/framing-ai-strategy/#easy-footnote-bottom-13-3406 "In particular, perhaps the tractability of a world with P(victory) = p is 4p(1-p) times the tractability of a world with P(victory) = 50%. Consider a binary between victory and doom, and assume that work making victory more probable in a world (monolithically and) linearly increases how prepared that world is, and a world’s P(victory) as a function of preparedness is logistic. The derivative of a logistic distribution’s CDF is proportional to p(1-p) at the point where the CDF’s value is p.

I use a logistic distribution because it’s simple, seems roughly reasonable, and it has a very nice relationship with the log odds ratio: using a logistic distribution is equivalent to assuming that work making victory more probable in a world (monolithically and) linearly increases log-odds of victory in that world. But a different distribution may be more principled or realistic. I weakly intuit that the true distribution is heavier-tailed than the logistic distribution, roughly speaking.

Note that this only works for ‘nice’ worlds– worlds where a logistic distribution is appropriate. We must reason differently about arbitrary sets of possible futures because a combination of logistics isn’t logistic. A combination of a 1%-doomed and a 99%-doomed world is just as intractable as each world individually. (And so we have to be careful about the definition of “world.” It may not be theoretically sound to treat the union of “AGI in 2030” worlds as a single world, for the purpose of the logistic distribution. Also note that this whole frame is flawed insofar as it assumes that a “world” like “AGI in 2030” has a certain probability and that your interventions on this world don’t affect other “worlds”– even though they should be almost as good for “AGI in 2031” as directly working on “AGI in 2031” would be. How to account for the relationship between P(victory) and tractability is an open question.)

A background assumption or approximation here is that there is a binary between victory and doom. But similar conclusions are correct given the messier reality, I think.")
	+ Upweight more neglected worlds (think on the margin)
* Upweight short-timelines worlds insofar as there is more non-AI existential risk in long-timelines worlds
* Upweight analysis that better generalizes to or improves other worlds
* Notice the possibility that you live in a simulation (if that is decision-relevant; unfortunately, the practical implications of living in a simulation are currently unclear)
* Upweight worlds that you have better personal fit for analyzing
	+ Upweight worlds where you have more influence, if relevant
* Consider side effects of doing strategy, including what you gain knowledge about, testing fit, and gaining [credible signals of fit](https://forum.effectivealtruism.org/posts/FwzHrWMZzCWNrEedQ/goals-we-might-have-when-taking-actions-to-improve-the-ea#Gaining_credible_signals_of_fit)

In practice, I tentatively think the biggest (analytically useful) considerations for weighting worlds beyond probability are generally:

1. Short timelines
	1. More foreseeable[14](https://aiimpacts.org/framing-ai-strategy/#easy-footnote-bottom-14-3406 "Sidenote: strategy research and particularly “intermediate goals” and “exploration, world-modeling, and forecasting” increases leverage over long-timelines worlds.")
	2. More affectable
	3. More neglected (by the AI strategy community)
		1. Future people can work on the further future
			1. The AI strategy field is likely to be bigger in the future
	4. Less planning or influence exerted from outside the AI strategy community
2. [Fast takeoff](https://www.lesswrong.com/tag/ai-takeoff)[15](https://aiimpacts.org/framing-ai-strategy/#easy-footnote-bottom-15-3406 "See also Daniel Kokotajlo’s definition.")
	1. Shorter, less foreseeable a certain time in advance, and less salient to the world in advance
		1. More neglected by the AI strategy community; the community would have a longer clear-sighted period to work on slow takeoff
		2. Less planning or influence exerted from outside the AI strategy community

(But there are presumably diminishing returns to focusing on particular worlds, at least at the community level, so the community should diversify the worlds it analyzes.) And I’m most confused about

1. Upweighting worlds where probability of victory is closer to 50% (I’m confused about what the probability of victory is in various possible worlds),
2. How leverage relates to variables like *total influence exerted to affect AI* (the rest of the world exerting influence means that you have less relative influence insofar as you’re [pulling the rope](https://www.overcomingbias.com/2007/05/policy_tugowar.html) along similar axes, but some interventions are amplified by something like *greater attention on AI*) (and related variables like *attention on AI* and *general craziness due to AI*), and
3. The probability and implications of living in a simulation.

A background assumption or approximation in this section is that you allocate research toward a world and the research is effective just if that world obtains. This assumption is somewhat crude: the impact of most research isn’t so binary, being fully effective in some possible futures and totally ineffective in the rest.[16](https://aiimpacts.org/framing-ai-strategy/#easy-footnote-bottom-16-3406 "Related concept: serial research requirements or non-parallelizable research tasks.") And thinking in terms of *influence over a world* is crude: influence depends on the person and on the intervention. Nevertheless, reasoning about leverage in terms of *worlds to allocate research toward* might sometimes be useful for prioritization. And we might discover a better account of leverage.

Leverage considerations should include not just *prioritizing between possible worlds* but also *prioritizing within a world*. For example, it seems high-leverage to focus on important actors’ blind spots and on certain important decisions or “crunchy” periods. And for AI strategy, it might be high-leverage to focus on the first few deployments of powerful AI systems.

---

Strategy work is complemented by

1. actually executing interventions, especially causing actors to make better decisions,
2. gaining resources to better execute interventions and improve strategy, and
3. field-building to better execute interventions and improve strategy.

An individual’s strategy work is complemented by informing the relevant community of their findings (e.g., for AI strategy, the AI strategy community).

In this post, I don’t try to make an ontology of AI strategy frames, or do comparative analysis of frames, or argue about the AI strategy community’s prioritization between frames.[17](https://aiimpacts.org/framing-ai-strategy/#easy-footnote-bottom-17-3406 "Regarding prioritization, briefly, I believe that the community currently does too much unfocused exploration (see e.g. most work by AI Impacts and GovAI) and strategy/governance/forecasting researchers would often benefit from focusing on a theory of change.") But these all seem like reasonable things for someone to do.

Related sources are linked above as relevant; see also Sam Clarke’s [The longtermist AI governance landscape](https://forum.effectivealtruism.org/posts/ydpo7LcJWhrr2GJrx/the-longtermist-ai-governance-landscape-a-basic-overview) (2022), Allan Dafoe’s [AI Governance: Opportunity and Theory of Impact](https://forum.effectivealtruism.org/posts/42reWndoTEhFqu6T8/ai-governance-opportunity-and-theory-of-impact) (2020), and Matthijs Maas’s [Strategic Perspectives on Long-term AI Governance](https://forum.effectivealtruism.org/s/xTkejiJHFsidZ9hMo) (2022).

If I wrote a post on “Framing AI governance,” it would substantially overlap with this list, and it would substantially draw on [The longtermist AI governance landscape](https://forum.effectivealtruism.org/posts/ydpo7LcJWhrr2GJrx/the-longtermist-ai-governance-landscape-a-basic-overview). See also Allan Dafoe’s [AI Governance: A Research Agenda](https://www.governance.ai/research-paper/agenda) (2018) and hanadulset and Caroline Jeanmaire’s [A Map to Navigate AI Governance](https://forum.effectivealtruism.org/posts/tmxkRFx6HyhhvHdz4/a-map-to-navigate-ai-governance) (2022). I don’t know whether an analogous “Framing technical AI safety” would make sense; if so, I would be excited about such a post.

Many thanks to Alex Gray. Thanks also to Linch Zhang for discussion of leverage and to Katja Grace, Eli Lifland, Rick Korzekwa, and Jeffrey Heninger for comments on a draft.

**Footnotes**
-------------