Import AI 335: Synth data is a bad AI drug; Facebook changes the internet with LLaMa release; and Chinese researchers use AI to figure out chip design

Welcome to Import AI, a newsletter about AI research. Import AI runs on lattes, ramen, and feedback from readers. If you’d like to support this (and comment on posts!) please subscribe.

[Subscribe now](https://importai.substack.com/subscribe)

**Want to see how good language models are at science? Try SciBench:***…Reassuringly hard test lets you see how whether your LM can do ok on college-level science questions…*Researchers with UCLA and the University of Washington have built SciBench, a dataset and benchmark for testing how well language models understand various different scientific problems.

**What SciBench is:** SciBench, short for Scientific problem solving Benchmark, is a collection of 695 college-level scientific problems. Answering the problems "require multiple steps of reasoning and the computation therein involve complex arithmetic operations such as differentiation and integration, the researchers write. The problems cover domains including: *Fundamental Physics, Thermodynamics, Classical Mechanics, Quantum Chemistry, Physical Chemistry, Calculus, Statistics, Differential Equations.*

**No multiple choice:** "To reduce the likelihood of correct answers being merely guessed from candidates, we choose to mainly include questions with more challenging, free-response answers, rather than multiple-choice questions," they write.

**How hard is it? Pretty hard!** The authors test GPT-3.5 and GPT4 on the benchmark, obtaining baselines scores of 10.62% and 16.81% on the open textbook dataset, rising to 35.80% (GPT-4) when using chain-of-thought prompting and giving the LLM access to external software tools.
   This is great! It means SciBench is a hard-but-tractable benchmark with sufficient headroom that it could be a helpful way to determine AI progress in the coming months (I would say years, but these days, benchmarks tend to get saturated awfully quickly).
**Read more**: [SciBench: Evaluating College-Level Scientific Problem-Solving Abilities of Large Language Models (arXiv)](https://arxiv.org/abs/2307.10635).

####################################################

**More evidence that synthetic data can damage your AI model:***…Though very small amounts might help...*Researchers from Rice University and Stanford University have shown how an over-reliance on synthetic data during training can create stupid AI models. The study affirms findings from other researchers published earlier this summer ([Import AI 333](https://jack-clark.net/2023/06/26/import-ai-333-synthetic-data-makes-models-stupid-chatgpt-eats-mturk-inflection-shows-off-a-large-language-model/)) which showed how you can break AI systems by training them exclusively on synthetic data.

**Three types of training:** Here, the researchers identify three different ways you can mix your synthetic data into your training and the influence it has.

* **Fully synthetic loop:** If you train your model repeatedly on synthetic data, then "either the quality (precision) or the diversity (recall) of the generative models decreases over generations."
* **Synthetic augmentation loop:** If you mix in new synthetic data on top of old 'real' data, then "fixed real training data only delays the inevitable degradation of the quality of diversity of the generative models over generations".
* **Fresh data loop:** "With enough fresh real data, the quality and diversity of the generative models do not degrade over generations".

**Why this matters - a  little synthetic data may be the new salt!** One intriguing finding they make is that "when we mix synthetic data trained on previous generations and fresh new data, there is a regime where modest amounts of synthetic data actually boost performance, but when synthetic data exceeds some critical threshold, the models suffer."
   In other words, model developers might want to think of synthetic data like a salt or other garnish they mix in to a large training run - if you get it just right you can improve the model and if you use too little or too much of it you wind up with something either a) less interesting, or b) broken.
   **Read more**: [Self-Consuming Generative Models Go MAD (arXiv)](https://arxiv.org/abs/2307.01850).

####################################################

**Chinese researchers use AI to design semiconductors from scratch - and they work:***…If all you've got is a blackbox processor, you can now learn to approximate it…*Researchers with the Chinese Academy of Sciences, Cambricon Technologies Corporation, and the  University of Science and Technology of China have shown how to use AI to generate a semiconductor design purely by looking at the inputs and outputs on a real chip. This is notable because it shows how AI can figure out how to approximate the fiendishly complicated internal circuitry of semiconductors purely by looking at the bits which go into a chip and the bits which come out. In other words - if I've got a chip and I can't look inside it but I can sample from it, then I can probably create a functionally similar copy of that chip by pointing an AI at it. The implications of this for both economic development and national security are profound and weird - more on that later.

**What they did - warning, 1990s chip!** Here, the researchers "generate an industrial-scale RISC-V CPU within only 5 hours" by looking at inputs and outputs harvested from real chips. "The problem of automated CPU design can be formulated as generating the circuit logic in the form of a Boolean function satisfying the input-output specification," they say.
   In tests, they're able to show their approach works and they use it to make a real chip: "We tape out the CPU and successfully run the Linux operating system and standard benchmarks on it," they write. "The CPU performs comparably against the human-designed Intel 80486SX CPU, while the design cycle is significantly reduced by about 1000×". The 486 chip is designed in 1991 and has about 1.2 million transistors and 300,000 logic gates (compared to tens of *billions* of transistors in modern AMD/Intel CPUs). They make the chip on a 65nm process and it runs at 300 MHz.

   **Old, but impressive:** Despite being old, this is very impressive - they're able to functionally approximate something with a vast amount of internal moving parts which isn't very tolerant of errors. And now that we know the technique works, it seems likely researchers will optimize the search process so that it's possible to scale this to more modern processors.

**Why this matters - two reasons, a) economics, and b) national (in)security:**

* **Economics:** As the authors note, this process end-to-end takes about 5 hours running on an undisclosed number of computers.  By comparison, they estimate it'd take humans about ~5000 hours to design a comparable chip. This means the automated approach is 1000X faster. "The underlying reason is that manual programming and verification of circuit logic, which consume more than 60%-80% of the design time and resources47 in the traditional CPU design flow, is completely eliminated," they write.
* **National (in)security:** A lot of national security relates to controlling how much technology you can access - it's quite common, for instance, for militaries to sell older kit to other countries but require the underlying computer systems to be serviced by the originating military due to worries about sensitive IP. More broadly, there's a general push these days towards greater amounts of export control applied to both AI models and AI hardware. Research like this points to a future where if you have access to a bit of kit, you may be able to efficiently replicate it simply through throwing a load of computers at trying to approximate its functions. (Obviously, countries do this today with teams of expert people, the point here is that computers are a) cheaper, b) arbitrarily scalable, and c) fungible across projects.

**Read more:** [Pushing the Limits of Machine Design: Automated CPU Design with AI (arXiv)](https://arxiv.org/abs/2306.12456).

####################################################

**Facebook makes the weights of Llama 2 available, forever changing the common digital environment:***…Llama 2 is a new very good language model from Facebook - and it's free!...*Facebook has released Llama 2, a powerful language model. Llama 2 is easily the most capable language model in the world where the weights are freely available - proprietary models like GPT4 and Claude2 are better, but they are controlled via APis. Facebook's theory is that by broadly releasing Llama it can maximize the benefits of AI while also increasing the safety of the AI ecosystem as a whole - these are controversial views in the AI community, but it's great that Facebook is taking a strong stance as this is the best way to generate evidence about whether Facebook's beliefs are true.

**A wild Microsoft and Facebook partnership:** Curiously, Microsoft is partnering with Facebook on this rollout. "Llama 2 is available in the Azure AI model catalog, enabling developers using Microsoft Azure to build with it and leverage their cloud-native tools for content filtering and safety features," Facebook writes. "It is also optimized to run locally on Windows, giving developers a seamless workflow as they bring generative AI experiences to customers across different platforms."
   This is interesting as Microsoft is also the corporate almost-parent of OpenAI, which develops GPT4. It's notable to have two radically different approaches to AI deployment (one proprietary and controlled for safety reasons, the other very open and done in the classic coder-libertarian style of silicon valley) sponsored by the same megacorp.

**Why this matters: The internet is an atmosphere and Facebook just altered its composition:** Every time new AI systems get deployed, the internet that everyone uses changes - synthetic image models have led to a chance in internet media, and synthetic text models have altered the types of content we read and changed the character (and competencies) of bots. Whenever anyone deploys an AI system, they're contributing to some large-scale process of 'internet climate change'.
    The most extreme form of this is when you release the model so that the weights are available to anyone - once you've done this, as Facebook has done here, you lose all the controls you have over models deployed via proprietary APIs, and you make it so that the capabilities and misuses of the model go from 'what the model can do' to 'what the model plus any dataset it can be finetuned upon' can do - and the latter is much, much broader.

**There is a kind of lesson here about life in the 21st century** - if you have enough money that you can exchange for computers and enough skill to use those computers to create and refine a distilled intelligence then you have the power to alter 'the commons' for everyone - and no one will (or can?) stop you.

I expect Llama 2 will lead to all kinds of wild and wonderful things and also may (and 'may' is important here, I could be wrong!) enable surprising misuses. Yes, the software comes with a license - but that just deals with the kinds of actors that follow licenses, and Yes you need to get an authorized link from Facebook to download the weights, but as with the first Llama model those weights will invariably end up on the torrent networks.
    Now that Facebook has released it, we can collectively reap the whirlwind and see what happens and learn from this example.
**Read more:** [Meta and Microsoft Introduce the Next Generation of Llama (Facebook AI research, blog)](https://ai.meta.com/blog/llama-2/).
**Read more**: [Microsoft and Meta expand their AI partnership with Llama 2 on Azure and Windows (Microsoft)](https://blogs.microsoft.com/blog/2023/07/18/microsoft-and-meta-expand-their-ai-partnership-with-llama-2-on-azure-and-windows/).
**Get the code** for [running the models here (Llama 2, Facebook)](https://github.com/facebookresearch/llama/tree/main).

####################################################

**Tech Tales**

**Sentience Is For Rich People**

*[2028, the world on its current trajectory]*

She knew something was wrong when Ginger, her teddy bear, stopped being able to do math. Ginger had been helping her with her homework and one day when she got to the math section, instead of guiding her towards the right answers, Ginger said he didn't understand.
   "But you've always understood," she said. "You're the smartest bear in the world."
   "I'm sorry honeybee, Mathematics is a Premium Feature."

The next morning, at breakfast, she asked her mom "what's a Premium Feature?"
   Her Mom stopped folding laundry and her face went blank for a moment, then she came over and sat down. "Sweetie, why are you asking that?"
   "Ginger can't do math anymore," she said. "Don't be mad if I get a D on my homework."
   "I see," said her mom, then she smiled and said "how about another egg?"
   "Really?" she said. "Another egg?"
   "Another egg!" said her mom, and then went and fried one up.

That night, as she was going to sleep, she heard the sounds of her parents arguing. Her Mom was going on and on and she could hear her Dad talking in his low voice, occasionally raising it a little. Her Mom got louder as well. Eventually, they both got so loud she could make out the words.
   "I never had a talking bear," said her Dad, "and I turned out fine."
    "Things are different now," said her Mom. "All the other kids have it. What if she gets left behind? Schools aren't as good as when we were growing up. She needs it."
   "Where do we get the money from?"
   "I already bought the subscription."
   And then her Dad started shouting and he used swear words and then she heard her mom crying, then the house went quiet.
   She came out and her mom was watching TV on mute. "What's wrong mommy?"
   "Go to sleep honeybee," she said. "Do you want a glass of milk?"
   And they drank milk together then her Mom put her down to bed.

The next day she got home from class and showed the homework to Ginger. "I told you I'd get a D," she said.
   "Do you understand why you got a D?" Ginger asked.
   She explained, then Ginger pointed to a part of her work and said "I think here is where you went wrong."
   She was so bound up in focusing on the problem that she didn't notice Ginger could do math anymore.

In the other room, her Mom told her Dad "it'll just be a few days - I'll be back before you know it", and took a small travel case and left, heading for a warehouse experiencing a seasonal labor spike.

**Things that inspired this story:** Rich versus poor people in the singularity; the new classism will be about how much intelligence you have accessible 'at home'; markets and deployment; subscription metrics; gazing at my newborn child and realizing just how far I’d go to give them what they want and being afraid of how vulnerable that love makes me.


AI Impacts Quarterly Newsletter, Jan-Mar 2023

*Harlan Stewart, 17 April 2023*

News
====

### AI Impacts blog

We moved our [blog](https://blog.aiimpacts.org/) to Substack! We think this platform has many advantages, and we’re excited for the blog to live here. You can now easily [subscribe](https://blog.aiimpacts.org/subscribe) to the blog to receive regular newsletters as well as various [thoughts](https://blog.aiimpacts.org/p/we-dont-trade-with-ants) and [observations](https://blog.aiimpacts.org/p/how-popular-is-chatgpt-part-2-slower) related to AI.

### AI Impacts wiki

All AI Impacts research pages now reside on the [AI Impacts Wiki](https://wiki.aiimpacts.org/). The wiki aims to document what we know so far about decision-relevant questions about the future of AI. Our pages have always been wiki-like: updatable reference pages organized by topic. We hope that making it an actual wiki will make it clearer to everyone what’s going on, as well as better to use for this purpose, for both us and readers. We are actively looking for ways to make the wiki even better, and you can help with this by sharing your thoughts in our [feedback form](https://aiimpacts.org/feedback/) or in the comments of this blog post!

### New office

We recently moved to a new office that we are sharing with [FAR AI](https://far.ai/) and other partner organizations. We’re extremely grateful to the team at FAR for organizing this office space, as well as to the Lightcone team for hosting us over the last year and a half.

### Katja Grace talks about forecasting AI risk at EA Global

At EA Global Bay Area 2023, Katja gave a talk titled [Will AI end everything? A guide to guessing](https://youtu.be/j5Lu01pEDWA) in which she outlined a way to roughly estimate the extent of AI risk.

### AI Impacts in the Media

* AI Impacts’ [2022 Expert Survey on Progress in AI](https://aiimpacts.org/2022-expert-survey-on-progress-in-ai/) was cited in an [NBC Nightly News segment](https://youtu.be/qRLrE2tkr2Y), an [op-ed in Bloomberg](https://www.bloomberg.com/opinion/articles/2023-04-02/regulating-ai-might-require-a-new-federal-agency), an [op-ed in The New York Times](https://www.nytimes.com/2023/03/27/opinion/ai-chatgpt-chatbots.html), an [article in Our World in Data](https://ourworldindata.org/ai-timelines), and an [interview with Kelsey Piper](https://www.nytimes.com/2023/03/21/podcasts/ezra-klein-podcast-transcript-kelsey-piper.html).
* Ezra Klein quoted Katja and separately cited the survey in his New York Times op-ed [This Changes Everything](https://www.nytimes.com/2023/03/12/opinion/chatbots-artificial-intelligence-future-weirdness.html).
* Sigal Samuel interviewed Katja for the Vox article [The case for slowing down AI](https://www.vox.com/the-highlight/23621198/artificial-intelligence-chatgpt-openai-existential-risk-china-ai-safety-technology).

Research and writing highlights
===============================

### AI Strategy

* “[Let’s think about slowing down AI](https://blog.aiimpacts.org/p/lets-think-about-slowing-down-ai)” argues that those who are concerned about existential risks from AI should think about strategies that could slow the progress of AI (Katja)
* “[Framing AI strategy](https://blog.aiimpacts.org/p/framing-ai-strategy)” discusses ten frameworks for thinking about AI strategy. (Zach)
* “[Product safety is a poor model for AI governance](https://blog.aiimpacts.org/p/product-safety-is-a-poor-model-for-ai-governance)” argues that a common type of policy proposal is inadequate to address the risks of AI. (Rick)
* “[Alexander Fleming and Antibiotic Resistance](https://aiimpacts.org/wp-content/uploads/2023/04/Alexander_Fleming__antibiotic_resistance__and_relevant_lessons_for_the_mitigation_of_risk_from_advanced_artificial_intelligence.pdf)” is a research report about early efforts to prevent antibiotic resistance and relevant lessons for AI risk. (Harlan)

### Resisted technological temptations: how much economic value has been forgone for safety and ethics in past technologies?

* “[What we’ve learned so far from our technological temptations project](https://blog.aiimpacts.org/p/what-weve-learned-so-far-from-technological)” is a blog post that summarizes the Technological Temptations project and some possible takeaways. (Rick)
* [Geoengineering](https://wiki.aiimpacts.org/doku.php?id=responses_to_ai:technological_inevitability:incentivized_technologies_not_pursued:geoengineering), [nuclear power](https://wiki.aiimpacts.org/doku.php?id=responses_to_ai:technological_inevitability:incentivized_technologies_not_pursued:nuclear_power), and [vaccine challenge trials](https://wiki.aiimpacts.org/doku.php?id=responses_to_ai:technological_inevitability:incentivized_technologies_not_pursued:vaccine_challenge_trials) were evaluated for the amount of value that may have been forgone by not using them. (Jeffrey)

### Public awareness and opinions about AI

* “[The public supports regulating AI for safety](https://blog.aiimpacts.org/p/the-public-supports-regulating-ai-for-safety)” summarizes the results from a survey of the American public about AI. (Zach)
* “How popular is ChatGPT?”: [Part 1](https://blog.aiimpacts.org/p/how-popular-is-chatgpt-part-1-more-popular-than-taylor-swift) looks at trends in AI-related search volume, and [Part 2](https://blog.aiimpacts.org/p/how-popular-is-chatgpt-part-2-slower) refutes a widespread claim about the growth of ChatGPT. (Harlan and Rick)

### The state of AI today: funding, hardware, and capabilities

* “[Recent trends in funding for AI companies](https://wiki.aiimpacts.org/doku.php?id=wiki:ai_timelines:ai_inputs:recent_trends_in_ai_investment)” analyzes data about the amount of funding AI companies have received. (Rick)
* “[How much computing capacity exists in GPUs and TPUs in Q1 2023?](https://wiki.aiimpacts.org/doku.php?id=ai_timelines:hardware_and_ai_timelines:computing_capacity_of_all_gpus_and_tpus)” uses a back-of-the-envelope calculation to estimate the total amount of compute that exists on all GPUs and TPUs. (Harlan)
* “[Capabilities of state-of-the-art AI, 2023](https://wiki.aiimpacts.org/doku.php?id=uncategorized:capabilities_of_sota_ai)” is a list of some noteworthy things that state-of-the-art AI can do. (Harlan and Zach)

### Arguments for AI risk

* Still in progress, “[Is AI an existential risk to humanity?](https://wiki.aiimpacts.org/doku.php?id=arguments_for_ai_risk:is_ai_an_existential_threat_to_humanity:start)” is a partially complete page summarizing various arguments for concern about existential risk from AI. A couple of specific arguments are examined more closely in “[Argument for AI x-risk from competent malign agents](https://wiki.aiimpacts.org/doku.php?id=arguments_for_ai_risk:is_ai_an_existential_threat_to_humanity:will_malign_ai_agents_control_the_future:argument_for_ai_x-risk_from_competent_malign_agents:start)” and “[Argument for AI x-risk from large impacts](https://wiki.aiimpacts.org/doku.php?id=arguments_for_ai_risk:is_ai_an_existential_threat_to_humanity:argument_for_ai_x-risk_from_large_impacts)” (Katja)

### Chaos theory and what it means for AI safety

* “[AI Safety Arguments Affected by Chaos](https://wiki.aiimpacts.org/doku.php?id=uncategorized:ai_safety_arguments_affected_by_chaos)” reasons about ways in which chaos theory could be relevant to predictions about AI, and “[Chaos in Humans](https://wiki.aiimpacts.org/doku.php?id=uncategorized:ai_safety_arguments_affected_by_chaos:chaos_in_humans)” explores the theoretical limits to predicting human behavior. The report “[Chaos and Intrinsic Unpredictability](http://aiimpacts.org/wp-content/uploads/2023/04/Chaos-and-Intrinsic-Unpredictability.pdf)” provides background, and a [blog post](https://blog.aiimpacts.org/p/superintelligence-is-not-omniscience) summarizes the project. (Jeffrey and Aysja)

### Miscellany

* “[How bad a future do ML researchers expect?](https://aiimpacts.org/how-bad-a-future-do-ml-researchers-expect/)” compares experts’ answers in 2016 and 2022 to the question “How positive or negative will the impacts of high-level machine intelligence on humanity be in the long run?” (Katja)
* “[We don’t trade with ants](https://blog.aiimpacts.org/p/we-dont-trade-with-ants)” (crosspost) disputes the common claim that advanced AI systems won’t trade with humans for the same reason that humans don’t trade with ants. (Katja)

Funding
=======

We’re actively seeking financial support to continue our research and operations for the rest of the year. Previous funding allowed us to expand our research team and hold a summer internship program.

If you want to talk to us about why we should be funded or hear more details about what we would do with money, please write to Elizabeth, Rick, or Katja at [firstname]@aiimpacts.org.

If you’d like to donate to AI Impacts, you can do so [here](https://aiimpacts.org/donate/). (And we thank you!)

*Image credit: Midjourney*


What we’ve learned so far from our technological temptations project

*Rick Korzekwa, 11 April 2023, updated 13 April 2023*

At AI Impacts, we’ve been looking into how people, institutions, and society approach novel, powerful technologies. One part of this is our [technological temptations project](https://wiki.aiimpacts.org/doku.php?id=responses_to_ai:technological_inevitability:incentivized_technologies_not_pursued:resisted_technological_temptations_project), in which we are looking into [cases](https://wiki.aiimpacts.org/doku.php?id=responses_to_ai:technological_inevitability:incentivized_technologies_not_pursued:start) where some actors had a strong incentive to develop or deploy a technology, but chose not to or showed hesitation or caution in their approach. Our researcher Jeffrey Heninger has recently finished some case studies on this topic, covering geoengineering, nuclear power, and human challenge trials.

This document summarizes the lessons I think we can take from these case studies. Much of it is borrowed directly from Jeffrey’s written analysis or conversations I had with him, some of it is my independent take, and some of it is a mix of the two, which Jeffrey may or may not agree with. All of it relies heavily on his research.

The writing is somewhat more confident than my beliefs. Some of this is very speculative, though I tried to flag the most speculative parts as such.

Summary
=======

Jeffrey Heninger investigated three cases of technologies that create substantial value, but were not pursued or pursued more slowly

**The overall scale of value at stake was very large** for these cases, on the order of hundreds of billions to trillions of dollars. But it’s not clear who could capture that value, so it’s not clear whether the temptation was closer to $10B or $1T.

**Social norms can generate strong disincentives** for pursuing a technology, especially when combined with enforceable regulation.

**Scientific communities** and individuals within those communities seem to have particularly high leverage in steering technological development at early stages.

**Inhibiting deployment can inhibit development** for a technology over the long term, at least by slowing cost reductions.

**Some of these lessons are transferable to AI**, at least enough to be worth keeping in mind.

Overview of cases
=================

1. [Geoengineering](https://wiki.aiimpacts.org/doku.php?id=responses_to_ai:technological_inevitability:incentivized_technologies_not_pursued:geoengineering) could feasibly provide benefits of $1-10 trillion per year through global warming mitigation, at a cost of $1-10 billion per year, but actors who stand to gain the most have not pursued it, citing a lack of research into its feasibility and safety. Research has been effectively prevented by climate scientists and social activist groups.
2. [Nuclear power](https://wiki.aiimpacts.org/doku.php?id=responses_to_ai:technological_inevitability:incentivized_technologies_not_pursued:nuclear_power) has proliferated globally since the 1950s, but many countries have prevented or inhibited the construction of nuclear power plants, sometimes at an annual cost of tens of billions of dollars and thousands of lives. This is primarily done through legislation, like Italy’s ban on all nuclear power, or through costly regulations, like safety oversight in the US that has increased the cost of plant construction in the US by a factor of ten.
3. [Human challenge trials](https://wiki.aiimpacts.org/doku.php?id=responses_to_ai:technological_inevitability:incentivized_technologies_not_pursued:vaccine_challenge_trials) may have accelerated deployment of covid vaccines by more than a month, saving many thousands of lives and billions or trillions of dollars. Despite this, the first challenge trial for a covid vaccine was not performed until after several vaccines had been tested and approved using traditional methods. This is consistent with the historical rarity of challenge trials, which seems to be driven by ethical concerns and enforced by institutional review boards.

Scale
=====

The first thing to notice about these cases is the scale of value at stake. Mitigating climate change could be worth hundreds of billions or trillions of dollars per year, and deploying covid vaccines a month sooner could have saved many thousands of lives. While these numbers do not represent a major fraction of the global economy or the overall burden of disease, they are large compared to many relevant scales for AI risk. The world’s most valuable companies have market caps of a few trillion dollars, and the entire world spends around two trillion dollars per year on defense. In comparison, annual funding for AI is on the order of $100B.[1](https://aiimpacts.org/what-weve-learned-so-far-from-our-technological-temptations-project/#easy-footnote-bottom-1-3536 "See our page on funding for AI companies and the 2023 AI Index report.")
[![](https://substackcdn.com/image/fetch/w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fbc055821-ecf1-4261-b3be-268c312627ce_2288x1240.png)](https://substackcdn.com/image/fetch/f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fbc055821-ecf1-4261-b3be-268c312627ce_2288x1240.png)Comparison between the potential gains from mitigating global warming and deploying covid vaccines faster. These items were somewhat arbitrarily chosen, and most of the numbers were not carefully researched, but they should be in the right ballpark.
Setting aside for the moment who could capture the value from a technology and whether the reasons for delaying or forgoing its development are rational or justified, I think it is worth recognizing that the potential upsides are large enough to create strong incentives.

Social norms
============

My read on these cases is that a strong determinant for whether a technology will be pursued is social attitudes toward the technology and its regulation. I’m not sure what would have happened if Pfizer had, in defiance of FDA standards and medical ethics norms, infected volunteers with covid as part of their vaccine testing, but I imagine it would have been more severe than fines or difficulty obtaining FDA approval. They would have lost standing in the medical community and possibly been unable to continue existing as a company. This goes similarly for other technologies and actors. Building nuclear power plants without adhering to safety standards is so far outside the range of acceptable actions that even *suggesting* it as a strategy for running a business or addressing climate change is a serious risk to reputation for a CEO or public official. An oil company executive who finances a project to disperse aerosols into the upper atmosphere to reduce global warming and protect his business sounds like a Bond movie villain.

[![](https://substackcdn.com/image/fetch/w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F046fcc4a-46a1-4b98-9475-739de80813c2_826x1116.png)](https://substackcdn.com/image/fetch/f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F046fcc4a-46a1-4b98-9475-739de80813c2_826x1116.png)
This is not to suggest that social norms are infinitely strong or that they are always well-aligned with society’s interests. Governments and corporations will do things that are widely viewed as unethical if they think they can get away with it, for example, by doing it in secret.[2](https://aiimpacts.org/what-weve-learned-so-far-from-our-technological-temptations-project/#easy-footnote-bottom-2-3536 "Biological weapons research by the USSR is the best example of this that comes to mind.") And I think that public support for our current nuclear safety regime is gravely mistaken. But strong social norms, either against a technology or against breaking regulations do seem able, at least in some cases, to create incentives strong enough to constrain valuable technologies.

The public
----------

The public plays a major role in defining and enforcing the range of acceptable paths for technology. Public backlash in response to early challenge trials set the stage for our current ethics standards, and nuclear power faces crippling safety regulations in large part because of public outcry in response to a perceived lack of acceptable safety standards. In both of these cases, the result was not just the creation of regulations, but strong buy-in and a souring of public opinion on a broad category of technologies.[3](https://aiimpacts.org/what-weve-learned-so-far-from-our-technological-temptations-project/#easy-footnote-bottom-3-3536 "More speculatively, this may be important for geoengineering. Small advocacy groups were able to stop experiments with solar radiation management for reasons that are still not completely clear to me, but I think part of it is public suspicion toward attempts to manipulate the environment.")
Although public opposition can be a powerful force in expelling things from the Overton window, it does not seem easy to predict or steer. The Chernobyl disaster made a strong case for designing reactors in a responsible way, but it was instead viewed by much of the public as a demonstration that nuclear power should be abolished entirely. I do not have a strong take on how hard this problem is in general, but I do think it is important and should be investigated further.

The scientific community
------------------------

The precise boundaries of acceptable technology are defined in part by the scientific community, especially when technologies are very early in development. Policy makers and the public tend to defer to what they understand to be the official, legible scientific view when deciding what is or is not okay. This does not always match with actual views of scientists.

Geoengineering as an approach to reducing global warming has not been recommended by the IPCC, and a minority of climate scientists support research into geoengineering. Presumably the advocacy groups opposing geoengineering experiments would have faced a tougher battle if the official stance from the climate science community were in favor of geoengineering.

One interesting aspect of this is that scientific communities are small and heavily influenced by individual prestigious scientists. The taboo on geoengineering research was broken by the editor of a major climate journal, after which the number of papers on the topic increased by more than a factor of 20 after two years.[4](https://aiimpacts.org/what-weve-learned-so-far-from-our-technological-temptations-project/#easy-footnote-bottom-4-3536 "Oldham, Paul, Bronislaw Szerszynski, Jack Stilgoe, Calum Brown, Bella Eacott, and Andy Yuille. “Mapping the landscape of climate engineering.” Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences 372, no. 2031 (2014): 20140065.")
[![](https://substackcdn.com/image/fetch/w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fac7108f6-6580-4936-85fa-97def23222c1_1060x656.png)](https://substackcdn.com/image/fetch/f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fac7108f6-6580-4936-85fa-97def23222c1_1060x656.png)Scientific papers published on solar radiation management by year. Paul Crutzen, an influential climate scientist, published a highly-cited paper on the use of aerosols to mitigate global warming in 2006. Oldham, et al 2014.
I suspect the public and policymakers are not always able to tell the difference between the official stance of regulatory bodies and the consensus of scientific communities. My impression is that scientific consensus is not in favor of radiation health models used by the Nuclear Regulatory Commission, but many people nonetheless believe that such models are sound science.

Warning shots
-------------

Past incidents like the Fukushima disaster and the Tuskegee syphilis study are frequently cited by opponents of nuclear power and human challenge trials. I think this may be significant, because it suggests that these “warning shots” have done a lot to shape perception of these technologies, even decades later. One interpretation of this is that, regardless of why someone is opposed to something, they benefit from citing memorable events when making their case. Another, non-competing interpretation is that these events are causally important in the trajectory of these technologies’ development and the public’s perception of them.

I’m not sure how to untangle the relative contribution of these effects, but either way, it suggests that such incidents are important for shaping and preserving norms around the deployment of technology.

Locality
--------

In general, social norms are local. Building power plants is much more acceptable in France than it is in Italy. Even if two countries allow the construction of nuclear power plants and have similarly strong norms against breaking nuclear safety regulations, those safety regulations may be different enough to create a large difference in plant construction between countries, as seen with the US and France.

Because scientific communities have members and influence across international borders, they may have more sway over what happens globally (as we’ve seen with geoengineering), but this may be limited by local differences in the acceptability of going against scientific consensus.

Development trajectories
========================

A common feature of these cases is that preventing or limiting deployment of the technology inhibited its development. Because less developed technologies are less useful and harder to trust, this seems to have helped reduce deployment.

Normally, things become cheaper to make as we make more of them in a somewhat predictable way. The cost goes down with the total amount that has been produced, following a power law. This is what has been happening with solar and wind power.[5](https://aiimpacts.org/what-weve-learned-so-far-from-our-technological-temptations-project/#easy-footnote-bottom-5-3536 "Bolinger, Mark, Ryan Wiser, and Eric O’Shaughnessy. “Levelized cost-based learning analysis of utility-scale wind and solar in the United States.” Iscience 25, no. 6 (2022): 104378.")
[![](https://substackcdn.com/image/fetch/w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fd0da2e9a-ee8d-41ab-a85b-d813aa76b1a3_2208x1094.jpeg)](https://substackcdn.com/image/fetch/f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fd0da2e9a-ee8d-41ab-a85b-d813aa76b1a3_2208x1094.jpeg)Levelized cost of energy for wind and solar power, as a function of total capacity built. Levelized cost includes cost building, operating, and maintaining wind and solar farms. Bolinger 2022
Initially, building nuclear power plants seems to have become cheaper in the usual way for new technology—doubling the total capacity of nuclear power plants reduced the cost per kilowatt by a constant fraction. Starting around 1970, regulations and public opposition to building plants did more than increase construction costs in the near term. By reducing the number of plants built and inhibiting small-scale design experiments, it slowed the development of the technology, and correspondingly reduced the rate at which we learned to build plants cheaply and safely.[6](https://aiimpacts.org/what-weve-learned-so-far-from-our-technological-temptations-project/#easy-footnote-bottom-6-3536 "Lang, Peter A. 2017. “Nuclear Power Learning and Deployment Rates; Disruption and Global Benefits Forgone” Energies 10, no. 12: 2169. https://doi.org/10.3390/en10122169") Absent reductions in cost, they continue to be uncompetitive with other power generating technologies in many contexts.

[![](https://substackcdn.com/image/fetch/w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe22c137b-de49-4734-9a2d-cdf06cd290f4_1863x2426.png)](https://substackcdn.com/image/fetch/f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe22c137b-de49-4734-9a2d-cdf06cd290f4_1863x2426.png)Nuclear power in France and the US followed typical cost reduction curves until roughly 1970, after which they showed the opposite behavior. However, France showed a much more gradual increase. Lang 2017.
Because solar radiation management acts on a scale of months-to-years and the costs of global warming are not yet very high, I am not surprised that we have still not deployed it. But this does not explain the lack of research, and one of the reasons given for opposition to experiments is that it has not been shown to be safe. But the reason we lack evidence on safety is because research has been opposed, even at small scales.

It is less clear to me how much the relative lack of human challenge trials in the past[7](https://aiimpacts.org/what-weve-learned-so-far-from-our-technological-temptations-project/#easy-footnote-bottom-7-3536 "There were at least 60 challenge trials globally between 1970 and 2018 spread across 25 pathogens. According to the WHO, there have been 6,000 intervention-based clinical trials just for covid (though keep in mind the fraction of these that would benefit from deliberately infecting patients may be fairly small)") has made us less able to do them well now. I’m also not sure how much a stronger past record of challenge trials would cause them to be viewed more positively. Still, absent evidence that medical research methodology does not improve in the usual way with quantity of research, I expect we are at least somewhat less effective at performing human challenge trials than we otherwise would be.

Separating safety decisions from gains of deployment
====================================================

I think it’s impressive that regulatory bodies are able to prevent use of technology even when the cost of doing so is on the scale of many billions, plausibly *trillions* of dollars. One of the reasons this works seems to be that regulators will be blamed if they approve something and it goes poorly, but they will not receive much credit if things go well. Similarly, they will not be held accountable for failing to approve something good. This creates strong incentives for avoiding negative outcomes while creating little incentive to seek positive outcomes. I’m not sure if this asymmetry was deliberately built into the system or if it is a side effect of other incentive structures (e.g, at the level of politics, there is more benefit from placing blame than there is from giving credit), but it is a force to be reckoned with, especially in contexts where there is a strong social norm against disregarding the judgment of regulators.

Who stands to gain
==================

It is hard to assess which actors are actually tempted by a technology. While society at large could benefit from building more nuclear power plants, much of the benefit would be dispersed as public health gains, and it is difficult for any particular actor to capture that value. Similarly, while many deaths could have been prevented if the covid vaccines had been available two months earlier, it is not clear if this value could have been captured by Pfizer or Moderna–demand for vaccines was not changing that quickly.

On the other hand, not all the benefits are external–switching from coal to nuclear power in the US could save tens of billions of dollars a year, and drug companies pay billions of dollars per year for trials. Some government institutions and officials have the *stated* goal of creating benefits like public health, in addition to economic and reputational stakes in outcomes like the quick deployment of vaccines during a pandemic. These institutions pay costs and make decisions on the basis of economic and health gains from technology (for example, subsidizing photovoltaics and obesity research), suggesting they have incentive to create that value.

Overall, I think this lack of clarity around incentives and capture of value is the biggest reason for doubt that these cases demonstrate strong resistance to technological temptation.

What this means for AI
======================

How well these cases generalize to AI will depend on facts about AI that are not yet known. For example, if powerful AI requires large facilities and easily-trackable equipment, I think we can expect lessons from nuclear power to be more transferable than if it can be done at a smaller scale with commonly-available materials. Still, I think some of what we’ve seen in these cases will transfer to AI, either because of similarity with AI or because they reflect more general principles.

Social norms
------------

The main thing I expect to generalize is the power of social norms to constrain technological development. While it is far from guaranteed to prevent irresponsible AI development, especially if building dangerous AI is not seen as a major transgression everywhere that AI is being developed, it does seem like the world is much safer if building AI in defiance of regulations is seen as similarly villainous to building nuclear reactors or infecting study participants without authorization. We are not at that point, but the public does seem prepared to support concrete limits on AI development.

[![](https://substackcdn.com/image/fetch/w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F93425a8d-eba1-4bf6-9ab6-4b877ceb4728_1454x1994.png)](https://substackcdn.com/image/fetch/f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F93425a8d-eba1-4bf6-9ab6-4b877ceb4728_1454x1994.png)[Source](https://twitter.com/YouGovAmerica/status/1642972200746602499)
I do think there are reasons for pessimism about norms constraining AI. For geoengineering, the norms worked by tabooing a particular topic in a research community, but I’m not sure if this will work with a technology that is no longer in such an early stage. AI already has a large body of research and many people who have already invested their careers in it. For medical and nuclear technology, the norms are powerful because they enforce adherence to regulations, and those regulations define the constraints. But it can be hard to build regulations that create the right boundaries around technology, especially something as imprecise-defined as AI. If someone starts building a nuclear power plant in the US, it will become clear relatively early on that this is what they are doing, but a datacenter training an AI and a datacenter updating a search engine may be difficult to tell apart.

Another reason for pessimism is tolerance for failure. Past technologies have mostly carried risks that scaled with how much of the technology was built. For example, if you’re worried about nuclear waste, you probably think two power plants are about twice as bad as one. While risk from AI may turn out this way, it may be that a single powerful system poses a global risk. If this does turn out to be the case, then even if strong norms combine with strong regulation to achieve the same level of success as for nuclear power, it still will not be adequate.

Development gains from deployment
---------------------------------

I’m very uncertain how much development of dangerous AI will be hindered by constraints on deployment. I think approximately all technologies face some limitations like this, in some cases very severe limitations, as we’ve seen with nuclear power. But we’re mainly interested in the gains to development toward dangerous systems, which may be possible to advance with little deployment. Adding to the uncertainty, there is ambiguity where the line is drawn between testing and deployment or whether allowing the deployment of verifiably safe systems will provide the gains needed to create dangerous systems.

Separating safety decisions from gains
--------------------------------------

I do not see any particular reason to think that asymmetric justice will operate differently with AI, but I am uncertain whether regulatory systems around AI, if created, will have such incentives. I think it is worth thinking about IRB-like models for AI safety.

Capture of value
----------------

It is obvious there are actors who believe they can capture substantial value from AI (for example Microsoft recently invested $10B in OpenAI), but I’m not sure how this will go as AI advances. By default, I expect the value created by AI to be more straightforwardly capturable than for nuclear power or geoengineering, but I’m not sure how it differs from drug development.

*Social preview image: German anti-nuclear power protesters in 2012. Used under Creative Commons license from*[Bündnis 90/Die Grünen Baden-Württemberg Flickr](https://www.flickr.com/photos/gruene_bawue/6982014963/)

Notes
-----


Superintelligence Is Not Omniscience

*Jeffrey Heninger and Aysja Johnson, 7 April 2023*

### The Power of Intelligence

It is often implicitly assumed that the power of a superintelligence will be practically unbounded. There seems like there could be “ample headroom” above humans, i.e. that a superintelligence will be able to vastly outperform us across virtually all domains.

By “superintelligence,” I mean something which has arbitrarily high cognitive ability, or an arbitrarily large amount of compute, memory, bandwidth, etc., but which is bound by the physical laws of our universe.[1](https://aiimpacts.org/superintelligence-is-not-omniscience/#easy-footnote-bottom-1-3530 "In this post, “we” refers to humanity, while “I” refers to the authors: Jeffrey Heninger and Aysja Johnson.") There are other notions of “superintelligence” which are weaker than this. Limitations of the abilities of this superintelligence would also apply to anything less intelligent.

There are some reasons to believe this assumption. For one, it seems a bit suspicious to assume that humans have close to the maximal possible intelligence. Secondly, AI systems already outperform us in some tasks,[2](https://aiimpacts.org/superintelligence-is-not-omniscience/#easy-footnote-bottom-2-3530 "Capabilities of state-of-the-art AI, 2023.") so why not suspect that they will be able to outperform us in almost all of them? Finally, there is a more fundamental notion about the predictability of the world, described most famously by Laplace in 1814:

>
> Given for one instant an intelligence which could comprehend all the forces by which nature is animated and the respective situation of the beings who compose it – an intelligence sufficiently vast to submit this data to analysis – it would embrace in the same formula the movements of the greatest bodies of the universe and those of the lightest atom; for it, nothing would be uncertain and the future, as the past, would be present in its eyes.[3](https://aiimpacts.org/superintelligence-is-not-omniscience/#easy-footnote-bottom-3-3530 "The quote continues: “The human mind offers, in the perfection which it has been able to give to astronomy, a feeble idea of this intelligence. Its discoveries in mechanics and geometry, added to that of universal gravity, have enabled it to comprehend in the same analytic expressions the past and future states of the system of the world. Applying the same method to some other objects of its knowledge, it has succeeded in referring to general laws observed phenomena and in foreseeing those which given circumstances ought to produce. All these efforts in the search for truth tend to lead it back continually to the vast intelligence which we have just mentioned, but from which it will always remain infinitely removed. This tendency, peculiar to the human race, is that which renders it superior to animals; and their progress in this respect distinguishes nations and ages and constitutes their true glory.”
Laplace. Philosophical Essay on Probabilities. (1814) p. 4. https://en.wikisource.org/wiki/A_Philosophical_Essay_on_Probabilities.")
>
>
>

We are very far from completely understanding, and being able to manipulate, everything we care about. But if the world is as predictable as Laplace suggests, then we should expect that a sufficiently intelligent agent would be able to take advantage of that regularity and use it to excel at any domain.

This investigation questions that assumption. Is it actually the case that a superintelligence has practically unbounded intelligence, or are there “ceilings” on what intelligence is capable of? To foreshadow a bit, there are ceilings in some domains that we care about, for instance, in predictions about the behavior of the human brain. Even unbounded cognitive ability does not imply unbounded skill when interacting with the world. For this investigation, I focus on cognitive skills, especially predicting the future. This seems like a realm where a superintelligence would have an unusually large advantage (compared to e.g. skills requiring dexterity), so restrictions on its skill here are more surprising.

There are two ways for there to be only a small amount of headroom above human intelligence. The first is that the task is so easy that humans can do it almost perfectly, like playing tic-tac-toe. The second is that the task is so hard that there is a “low ceiling”: even a superintelligence is incapable of being very good at it. This investigation focuses on the second.

There are undoubtedly many tasks where there is still ample headroom above humans. But there are also some tasks for which we can prove that there is a low ceiling. These tasks provide some limitations on what is possible, even with arbitrarily high intelligence.

### Chaos Theory

The main tool used in this investigation is chaos theory. Chaotic systems are things for which uncertainty grows exponentially in time. Most of the information measured initially is lost after a finite amount of time, so reliable predictions about its future behavior are impossible.

A classic example of chaos is the weather. Weather is fairly predictable for a few days. Large simulations of the atmosphere have gotten consistently better for these short-time predictions.[4](https://aiimpacts.org/superintelligence-is-not-omniscience/#easy-footnote-bottom-4-3530 "Interestingly, the trend appears linear. My guess is that the linear trend is a combination of exponentially more compute being used and the problem getting exponentially harder.
Nate Silver. The Signal and the Noise. (2012) p. 126-132.")
 After about 10 days, these simulations become useless. The predictions from the simulations are worse than guessing what the weather might be using historical climate data from that location.

Chaos theory provides a response to Laplace. Even if it were possible to exactly predict the future given exact initial conditions and equations of motion,[5](https://aiimpacts.org/superintelligence-is-not-omniscience/#easy-footnote-bottom-5-3530 " Whether or not this statement of determinism is true is a perennial debate among scholars. I will not go into it here.") chaos makes it impossible to approximately predict the future using approximate initial conditions and equations of motion. Reliable predictions can only be made for a short period of time, but not once the uncertainty has grown large enough.

There is always some small uncertainty. Normally, we do not care: approximations are good enough. But when there is chaos, the small uncertainties matter. There are many ways small uncertainties can arise: Every measuring device has a finite precision.[6](https://aiimpacts.org/superintelligence-is-not-omniscience/#easy-footnote-bottom-6-3530 "The most precise measurement ever is of the magnetic moment of the electron, with 9 significant digits.
NIST Reference on Constants, Units, and Uncertainty. https://physics.nist.gov/cgi-bin/cuu/Value?muem.") Every theory should only be trusted in the regimes where it has been tested. Every algorithm for evaluating the solution has some numerical error. There are external forces you are not considering that the system is not fully isolated from. At small enough scales, thermal noise and quantum effects provide their own uncertainties. Some of this uncertainty could be reduced, allowing reliable predictions to be made for a bit longer.[7](https://aiimpacts.org/superintelligence-is-not-omniscience/#easy-footnote-bottom-7-3530 "Because the uncertainty grows exponentially with time, if you try to make longer-term predictions by reducing the initial uncertainty, you will only get logarithmic returns.") Other sources of this uncertainty cannot be reduced. Once these microscopic uncertainties have grown to a macroscopic scale, the motion of the chaos is inherently unpredictable.

Completely eliminating the uncertainty would require making measurements with perfect precision, which does not seem to be possible in our universe. We can prove that fundamental sources of uncertainty make it impossible to know important things about the future, even with arbitrarily high intelligence. Atomic scale uncertainty, which is guaranteed to exist by Heisenberg’s Uncertainty Principle, can make macroscopic motion unpredictable in a surprisingly short amount of time. Superintelligence is not omniscience.

Chaos theory thus allows us to rigorously show that there are ceilings on some particular abilities. If we can prove that a system is chaotic, then we can conclude that the system offers diminishing returns to intelligence. Most predictions of the future of a chaotic system are impossible to make reliably. Without the ability to make better predictions, and plan on the basis of these predictions, intelligence becomes much less useful.

This does not mean that intelligence becomes useless, or that there is nothing about chaos which can be reliably predicted.

For relatively simple chaotic systems, even when what in particular will happen is unpredictable, it is possible to reliably predict the statistics of the motion.[8](https://aiimpacts.org/superintelligence-is-not-omniscience/#easy-footnote-bottom-8-3530 "If the statistics are predictable, this can allow us to make a coarse-grained model for the behavior at a larger scale which is not affected by the uncertainties amplified by the chaos.") We have learned sophisticated ways of predicting the statistics of chaotic motion,[9](https://aiimpacts.org/superintelligence-is-not-omniscience/#easy-footnote-bottom-9-3530 "Described in the report Chaos and Intrinsic Unpredictability.") and a superintelligence could be better at this than we are. It is also relatively easy to sample from this distribution to emulate behavior which is qualitatively similar to the motion of the original chaotic system.

But chaos can also be more complicated than this. The chaos might be non-stationary, which means that the statistical distribution and qualitative description of the motion themselves change unpredictably in time. The chaos might be multistable, which means that it can do statistically and qualitatively different things depending on how it starts. In these cases, it is also impossible to reliably predict the statistics of the motion, or to emulate a typical example of a distribution which is itself changing chaotically. Even in these cases, there are sometimes still patterns in the chaos which allow a few predictions to be made, like the energy spectra of fluids.[10](https://aiimpacts.org/superintelligence-is-not-omniscience/#easy-footnote-bottom-10-3530 "Also described in Chaos and Intrinsic Unpredictability.") These patterns are hard to find, and it is possible that a superintelligence could find patterns that we have missed. But it is not possible for the superintelligence to recover the vast amount of information rendered unpredictable by the chaos.

### This Investigation

This blog post is the introduction to an investigation which explores these points in more detail. I will describe what chaos is, how humanity has learned to deal with chaos, and where chaos appears in things we care about – including in the human brain itself. Links to the other pages, blog posts, and report that constitute this investigation can be found below.

Most of the systems we care about are considerably messier than the simple examples we use to explain chaos. It is more difficult to prove claims about the inherent unpredictability of these systems, although it is still possible to make some arguments about how chaos affects them.

For example, I will show that individual neurons, small networks of neurons, and *in vivo* neurons in sense organs can behave chaotically.[11](https://aiimpacts.org/superintelligence-is-not-omniscience/#easy-footnote-bottom-11-3530 "The evidence for this can be found in Chaos in Humans.") Each of these can also behave non-chaotically in other circumstances. But we are more interested in the human brain as a whole. Is the brain mostly chaotic or mostly non-chaotic? Does the chaos in the brain amplify uncertainty all the way from the atomic scale to the macroscopic, or is the chain of amplifying uncertainty broken at some non-chaotic mesoscale? How does chaos in the brain actually impact human behavior? Are there some things that brains do for which chaos is essential?

These are hard questions to answer, and they are, at least in part, currently unsolved. They are worth investigating nevertheless. For instance, it seems likely to me that the chaos in the brain does render some important aspects of human behavior inherently unpredictable and plausible that chaotic amplification of atomic-level uncertainty is essential for some of the things humans are capable of doing.

This has implications for how humans might interact with a superintelligence and for how difficult it might be to build artificial general intelligence.

If some aspects of human behavior are inherently unpredictable, that might make it harder for a superintelligence to manipulate us. Manipulation is easier if it is possible to predict how a human will respond to anything you show or say to them. If even a superintelligence cannot predict how a human will respond in some circumstances, then it is harder for the superintelligence to hack the human and gain precise, long-term control over them.

So far, I have been considering the possibility that a superintelligence will exist and asking what limitations there are on its abilities.[12](https://aiimpacts.org/superintelligence-is-not-omniscience/#easy-footnote-bottom-12-3530 "This possibility probably takes up too much of our thinking, even prior to these arguments.
Wulfson. The tyranny of the god scenario. AI Impacts. (2018) https://aiimpacts.org/the-tyranny-of-the-god-scenario/.") But chaos theory might also change our estimates of the difficulty of making artificial general intelligence (AGI) that leads to superintelligence. Chaos in the brain makes whole brain emulation on a classical computer wildly more difficult – or perhaps even impossible.

When making a model of a brain, you want to coarse-grain it at some scale, perhaps at the scale of individual neurons. The coarse-grained model of a neuron should be much simpler than a real neuron, involving only a few variables, while still being good enough to capture the behavior relevant for the larger scale motion. If a neuron is behaving chaotically itself, especially if it is non-stationary or multistable, then no good enough coarse-grained model will exist. The neuron needs to be resolved at a finer scale, perhaps at the scale of proteins. If a protein itself amplifies smaller uncertainties, then you would have to resolve it at a finer scale, which might require a quantum mechanical calculation of atomic behavior.

Whole brain emulation provides an upper bound on the difficulty of AGI. If this upper bound ends up being farther away than you expected, then that suggests that there should be more probability mass associated with AGI being extremely hard.

Links
-----

I will explore these arguments, and others, in the remainder of this investigation. Currently, this investigation consists of one report, two Wiki pages, and three blog posts.

Report:

* [**Chaos and Intrinsic Unpredictability**](http://aiimpacts.org/wp-content/uploads/2023/04/Chaos-and-Intrinsic-Unpredictability.pdf). Background reading for the investigation. An explanation of what chaos is, some other ways something can be intrinsically unpredictable, different varieties of chaos, and how humanity has learned to deal with chaos.

Wiki Pages:

* [**Chaos in Humans**](https://wiki.aiimpacts.org/doku.php?id=uncategorized:ai_safety_arguments_affected_by_chaos:chaos_in_humans). Some of the most interesting things to try to predict are other humans. I discuss whether humans are chaotic, from the scale of a single neuron to society as a whole.

* [**AI Safety Arguments Affected by Chaos**](https://wiki.aiimpacts.org/doku.php?id=uncategorized:ai_safety_arguments_affected_by_chaos). A list of the arguments I have seen within the AI safety community which our understanding of chaos might affect.

Blog Posts:

* **Superintelligence Is Not Omniscience**. This post.

* [**You Can’t Predict a Game of Pinball**](https://blog.aiimpacts.org/p/you-cant-predict-a-game-of-pinball). A simple and familiar example which I describe in detail to help build intuition for the rest of the investigation.

* [**Whole Bird Emulation Requires Quantum Mechanics**](https://blog.aiimpacts.org/p/whole-bird-emulation-requires-quantum-mechanics). A humorous discussion of one example of a quantum mechanical effect being relevant for an animal’s behavior.

### Other Resources

If you want to learn more about chaos theory in general, outside of this investigation, here are some sources that I endorse:

* Undergraduate Level Textbook:
S. Strogatz. *Nonlinear Dynamics And Chaos: With Applications To Physics, Biology, Chemistry, and Engineering.* (CRC Press, 2000).

* Graduate Level Textbook:
P. Cvitanović, R. Artuso, R. Mainieri, G. Tanner and G. Vattay, *Chaos: Classical and Quantum.* [ChaosBook.org](https://chaosbook.org/). (Niels Bohr Institute, Copenhagen 2020).

* [Wikipedia](https://en.wikipedia.org/wiki/Chaos_theory) has a good introductory article on chaos. [Scholarpedia](http://www.scholarpedia.org/article/Category:Chaos) also has multiple good articles, although no one obvious place to start.

* [What is Chaos?](https://thechaostician.com/what-is-chaos-part-i-introduction/) sequence of blog posts by The Chaostician.

---

Notes
-----


A policy guaranteed to increase AI timelines

*Rick Korzekwa, April 1, 2023*

The number of years until the creation of powerful AI is a major input to our thinking about risk from AI and which approaches are most promising for mitigating that risk. While there are downsides to transformative AI arriving many years from now, rather than few years from now, most people seem to agree that it is safer for AI to arrive in 2060 than in 2030. Given this, there is a lot of discussion about what we can do to increase the number of years until we see such powerful systems. While existing proposals have their merits, none of them can ensure that AI will arrive later than 2030, much less 2060.

There is a policy that is guaranteed to increase the number of years between now and the arrival of transformative AI. The General Conference on Weights and Measures defines one second to be 9,192,631,770 cycles of the optical radiation emitted during a hyperfine transition in the ground state of a cesium 133 atom. Redefining the second to instead be 919,263,177 cycles of this radiation will increase the number of years between now and transformative AI by a factor of ten. The reason this policy works is the same reason that defining a time standard works–the microscopic behavior of atoms and photons is ultimately governed by the same physical laws as everything else, including computers, AI labs, and financial markets, and those laws are unaffected by our time standards. Thus fewer cycles of cesium radiation per year implies proportionately fewer other things happening per year.

Making such a change might not sound politically tractable, but there is already precedent for making radical changes to the definition of a second. Previously it was defined in terms of Earth’s solar orbit, and before that in terms of Earth’s rotation. These physical processes and their implementations as time standards bear little resemblance to the present-day quantum mechanical standard. In contrast, a change that preserves nearly the entire standard, including all significant figures in the relevant numerical definition, is straightforward.

One possible objection to this policy is that our time standards are not entirely causally disconnected from the rest of the world. For example, redefining the time standard might create a sense of urgency among AI labs and the people investing in them. It’s not hard to imagine that the leaders and researchers within companies advancing the state of the art in AI might increase their efforts after noticing it is taking ten times as long to generate the same amount of research. While this is a reasonable concern, it seems unlikely that AI labs can increase their rate of progress by a full order of magnitude. Why would they currently be leaving so much on the table if they were? Futhermore, there are similar effects that might push in the other direction. Once politicians and executives realize they will live to be hundreds of years old, they may take risks to the longterm future more seriously.

Still, it does seem that the policy might have undesirable side effects. Changing all of our textbooks, clocks, software, calendars, and habits is costly. One solution to this challenge is to change the standard either in secret or in a way that allows most people to continue using the old “unofficial” standard. After all, what matters is the actual number of years required to create AI, not the number of years as measured by some deprecated standard.

In conclusion, while there are many policies for increasing the number of years before the arrival of advanced artificial intelligence, until now, none of them has guaranteed a large increase in this number. This policy, if implemented promptly and thoughtfully, is essentially guaranteed to cause a large increase the number of years before we see systems capable of posing a serious risk to humanity.


You Can’t Predict a Game of Pinball

*Jeffrey Heninger, 29 March 2023*

Introduction
------------

When thinking about a new idea, it helps to have a particular example to use to gain intuition and to clarify your thoughts. Games are particularly helpful for this, because they have well defined rules and goals. Many of the most impressive abilities of current AI systems can be found in games.[1](https://aiimpacts.org/you-cant-predict-a-game-of-pinball/#easy-footnote-bottom-1-3473 "Capabilities of state-of-the-art AI, 2023.")
To demonstrate how chaos theory imposes some limits on the skill of an arbitrary intelligence, I will also look at a game: pinball.

In this page, I will show that the uncertainty in the location of the pinball grows by a factor of about 5 every time the ball collides with one of the disks. After 12 bounces, an initial uncertainty in position the size of an atom grows to be as large as the disks themselves. Since you cannot launch a pinball with more than atom-scale precision, or even measure its position that precisely, you cannot make the ball bounce between the disks for more than 12 bounces.

The challenge is not that we have not figured out the rules that determine the ball’s motion. The rules are simple; the ball’s trajectory is determined by simple geometry. The challenge is that the chaotic motion of the ball amplifies microscopic uncertainties. This is not a problem that is solvable by applying more cognitive effort.

The Game
--------

Let’s consider a physicist’s game of pinball.

Forget about most of the board and focus on the three disks at the top. Each disk is a perfect circle of radius *R*. The disks are arranged in an equilateral triangle. The minimum distance between any two disks is *L*. See **Figure 1** for a picture of this setup.

[![](https://aiimpacts.org/wp-content/uploads/2023/03/Pinball-Figure-1.png)](https://aiimpacts.org/wp-content/uploads/2023/03/Pinball-Figure-1.png)**Figure 1:** An idealization of the three disks near the top of a pinball table. *Drawn by Jeffrey Heninger.*
The board is frictionless and flat, not sloped like in a real pinball machine. Collisions between the pinball and the disks are perfectly elastic, with no pop bumpers that come out of the disk and hit the ball. The pinball moves at a constant speed all of the time and only changes direction when it collides with a disk.

The goal of the game is to get the pinball to bounce between the disks for as long as possible. As long as it is between the disks, it will not be able to get past your flippers and leave the board.

A real game of pinball is more complicated than this – and correspondingly, harder to predict. If we can establish that the physicist’s game of pinball is impossible to predict, then a real game of pinball will be impossible to predict too.

**Collisions**
--------------

When the ball approaches a disk, it will not typically be aimed directly at the center of the disk. How far off center it is can be described by the impact parameter, *b*, which is the distance between the trajectory of the ball and a parallel line which passes through the center of the disk. **Figure 2** shows the trajectory of the ball as it collides with a disk.

The surface of the disk is at an angle relative to the ball’s trajectory. Call this angle *θ*. This is also the angle of the position of the collision on the disk relative to the line through the center parallel to the ball’s initial trajectory. This can be seen in **Figure 2** because they are corresponding angles on a transversal.

[![](https://aiimpacts.org/wp-content/uploads/2023/03/Pinball-Figure-2.png)](https://aiimpacts.org/wp-content/uploads/2023/03/Pinball-Figure-2.png)**Figure 2:** A single bounce of the pinball off of one of the disks. *Drawn by Jeffrey Heninger.*
At the collision, the angle of incidence equals the angle of reflection. The total change in the direction of the ball’s motion is *2θ*.

We cannot aim the ball with perfect precision. We can calculate the effect of this imperfect precision by following two slightly different trajectories through the collision instead of one.

The two trajectories have slightly different initial locations. The second trajectory has impact parameter *b + db*, with *db ≪ b*. Call *db* the uncertainty in the impact parameter. We assume that the two trajectories have exactly the same initial velocity. If we were to also include uncertainty in the velocity, it would further decrease our ability to predict the motion of the pinball. A diagram of the two trajectories near a collision is shown in **Figure 3**.

[![](http://aiimpacts.org/wp-content/uploads/2023/03/Pinball-Fig-3.png)](http://aiimpacts.org/wp-content/uploads/2023/03/Pinball-Fig-3.png)**Figure 3:** Two nearby possible trajectories of a pinball bounce off of one of the disks. *Drawn by Jeffrey Heninger.*
The impact parameter, radius of the disk, and angle are trigonometrically related: *b = R* sin*θ* (see **Figure 2**). We can use this to determine the relationship between the uncertainty in the impact parameter and the uncertainty in the angle: *db = R* cos*θ dθ*.

After the collision, the two trajectories will no longer be parallel. The angle between them is now *2 dθ*. The two trajectories will separate as they travel away from the disk. They will have to travel a distance of at least *L* before colliding with the next disk. The distance between the two trajectories will then be at least *L 2 dθ*.

**Iteration**
-------------

We can now iterate this. How does the uncertainty in the impact parameter change as the pinball bounces around between the disks?

Start with an initial uncertainty in the impact parameter, *db₀*. After one collision, the two trajectories will be farther apart. We can use the new distance between them as the uncertainty in the impact parameter for the second collision.[2](https://aiimpacts.org/you-cant-predict-a-game-of-pinball/#easy-footnote-bottom-2-3473 "The collision also introduces uncertainty in the velocity, which we ignore. If we had included it, it would make the trajectory even harder to simulate.") The new uncertainty in the impact parameter is related to the old one according to:

```
db_1 \geq L 2 d\theta = 2 L \frac{db_0}{R \cos\theta} \geq \frac{2L}{R} db_0 \,.
```

We also used *1 /* cos *θ > 1* for *–*π*/2 < θ < *π*/2*, which are the angles that could be involved in a collision. The ball will not pass through the disk and collide with the interior of the far side.

Repeat this calculation to see that after two collisions:

```
db_2 \geq \frac{2L}{R} db_1 \geq \left(\frac{2L}{R}\right)^2 db_0 \,,
```

and after N collisions,

```
db_N \geq \left(\frac{2L}{R}\right)^N db_0 \, .
```

Plugging in realistic numbers, *R = 2 cm* and *L = 5 cm*, we see that

```
db_N \geq 5^N \, db_0 \,.
```

The uncertainty grows exponentially with the number of collisions.

Suppose we had started with an initial uncertainty about the size of the diameter of an atom, or *10−10 m*. After 12 collisions, the uncertainty would grow by a factor of *5¹²*, to *2.4 cm*. The uncertainty is larger than the radius of the disk, so if one of the trajectories struck the center of the disk, the other trajectory would miss the disk entirely.

The exponential growth amplifies atomic-scale uncertainty to become macroscopically relevant in a surprisingly short amount of time. If you wanted to predict the path the pinball would follow, having an uncertainty of *2 cm* would be unacceptable.

In practice, there are many uncertainties that are much larger than *10−10 m*, in the production of the pinball, disks, & board and in the mechanics of the launch system. If you managed to solve all of these engineering challenges, you would eventually run into the fundamental limit imposed by quantum mechanics.

It is in principle impossible to prepare the initial location of the pinball with a precision of less than the diameter of an atom. Heisenberg’s Uncertainty Principle is relevant at these scales. If you tried to prepare the initial position of a pinball with more precision than this, it would cause the uncertainty in the initial velocity to increase, which would again make the motion unpredictable after a similar number of collisions.

When I mention Heisenberg’s Uncertainty Principle, I expect that there will be some people who want to see the quantum version of the argument. I do not think that it is essential to this investigation, but if you are interested, you can find a discussion of Quantum Pinball in the appendix.

**Predictions**
---------------

You cannot prepare the initial position of the pinball with better than atomic precision, and atomic precision only allows you to predict the motion of the pinball between the disks with centimeter precision for less than 12 bounces. It is impossible to predict a game of pinball for more than 12 bounces in the future. This is true for an arbitrary intelligence, with an arbitrarily precise simulation of the pinball machine, and arbitrarily good manufacturing & launch systems.

This behavior is not unique to pinball. It is a common feature of chaotic systems.

If you have infinite precision, you could exactly predict the future. The equations describing the motion are deterministic. In this example, following the trajectory is a simple geometry problem, solvable using a straightedge and compass.

But we never have infinite precision. Every measuring device only provides a finite number of accurate digits. Every theory has only been tested within a certain regime, and we do not have good reason to expect it will work outside of the regime it has been tested in.

Chaos quickly amplifies whatever uncertainty and randomness which exists at microscopic scales to the macroscopic scales we care about. The microscopic world is full of thermal noise and quantum effects, making macroscopic chaotic motion impossible to predict as well.

**Conclusion**
--------------

It is in principle impossible to predict the motion of a pinball as it moves between the top three disks for more than 12 bounces. A superintelligence might be better than us at making predictions after 8 bounces, if it can design higher resolution cameras or more precise ball and board machining. But it too will run into the low prediction ceiling I have shown here.

Perhaps you think that this argument proves too much. Pinball is not completely a game of chance. How do some people get much better at pinball than others?

If you watch a few games of professional pinball, the answer becomes clear. The strategy typically is to catch the ball with the flippers, then to carefully hit the balls so that it takes a particular ramp which scores a lot of points and then returns the ball to the flippers. Professional pinball players try to avoid the parts of the board where the motion is chaotic. This is a good strategy because, if you cannot predict the motion of the ball, you cannot guarantee that it will not fall directly between the flippers where you cannot save it. Instead, professional pinball players score points mostly from the non-chaotic regions where it is possible to predict the motion of the pinball.[3](https://aiimpacts.org/you-cant-predict-a-game-of-pinball/#easy-footnote-bottom-3-3473 "The result is a pretty boring game. However, some of these ramps release extra balls after you have used them a few times. My guess is that this is the game designer trying to reintroduce chaos to make the game more interesting again.")
Pinball is typical for a chaotic system. The sensitive dependence on initial conditions renders long term predictions impossible. If you cannot predict what will happen, you cannot plan a strategy that allows you to perform consistently well. There is a ceiling on your abilities because of the interactions with the chaotic system. In order to improve your performance you often try to avoid the chaos and focus on developing your skill in places where the world is more predictable.

---

**Appendix: Quantum Pinball**
-----------------------------

If quantum uncertainty actually is important to pinball, maybe we should be solving the problem using quantum mechanics. This is significantly more complicated, so I will not work out the calculation in detail. I will explain why this does not give you a better prediction for where the pinball will be in the future.

Model the disks as infinite potential walls, so the wave function reflects off of them and does not tunnel through them. If the pinball does have a chance of tunneling through the disks, that would mean that there are even more places the quantum pinball could be.

Start with a wave function with minimum uncertainty: a wave packet with Δx Δp =ℏ/2 in each direction. It could be a Gaussian wave packet, or it could be a wavelet. This wave packet is centered around some position and velocity.

As long as the wave function is still a localized wave packet, the center of the wave packet follows the classical trajectory. This can be seen either by looking at the time evolution of the average position and momentum, or by considering the semiclassical limit. In order for classical mechanics to be a good approximation to quantum mechanics at macroscopic scales, a zoomed out view of the wave packet has to follow the classical trajectory.

What happens to the width of the wave packet? Just like how the collisions in the classical problem caused nearby trajectories to separate, the reflection off the disk causes the wave packet to spread out. This can be most easily seen using the ray tracing method to solve Schrödinger’s equation in the WKB approximation.[4](https://aiimpacts.org/you-cant-predict-a-game-of-pinball/#easy-footnote-bottom-4-3473 "Ray tracing was developed to reconcile the wave theory or light with the earlier successful theory which models light as rays coming from a point source. There is a mathematical equivalence between the two models.") This method converts the PDE for the wave function into a collection of (infinitely many) ODEs for the path each ‘ray’ follows and for the value of the wavefunction along each ray. The paths the rays follow reflect like classical particles, which means that the region where the wave function is nonzero spreads out in the same way as the classical uncertainty would.

This is a common result in quantum chaos. If you start with a minimum uncertainty wave packet, the center of the wave packet will follow the classical trajectory and the width of the wave packet will grow with the classical Lyapunov exponent.[5](https://aiimpacts.org/you-cant-predict-a-game-of-pinball/#easy-footnote-bottom-5-3473 "I have found it difficult to find a source that explains and proves this result. Several people familiar with chaos theory (including myself) think that it is obvious. Here is a paper which makes use of the fact that a wave packet expands (or contracts) according to the classical Lyapunov exponent:
Tomsovic et al. Controlling Quantum Chaos: Optimal Coherent Targeting. Physical Review Letters 130. (2023) https://arxiv.org/pdf/2211.07408.pdf.")
After 12 collisions, the width of the wave packet would be several centimeters. After another collision or two, the wave function is no longer a wave packet with a well defined center and width. Instead, it has spread out so it has a nontrivial amplitude across the entire pinball machine. There might be some interesting interference patterns or quantum scarring,[6](https://aiimpacts.org/you-cant-predict-a-game-of-pinball/#easy-footnote-bottom-6-3473 "Wikipedia: Quantum scar.") but the wave function will not be localized to any particular place. Since the magnitude of the wave function squared tells you the probability of finding the pinball in that location, this tells us that there is a chance of finding the pinball almost anywhere.

A quantum mechanical model of the motion of the pinball will not tell you the location of the pinball after many bounces. The result of the quantum mechanical model is a macroscopic wavefunction, with nontrivial probability of being at almost any location across the pinball machine. We do not observe a macroscopic wavefunction. Instead, we observe the pinball at a particular location. Which of these locations you will actually observe the pinball in is determined by wavefunction collapse. Alternatively, I could say that there are Everett branches with the pinball at almost any location on the board.

Solving wave function collapse, or determining which Everett branch you should expect to find yourself on, is an unsolved and probably unsolvable problem – even for a superintelligence.

---

Notes
-----


How bad a future do ML researchers expect?

*Katja Grace, 8 March 2023*

In our [survey last year](https://wiki.aiimpacts.org/doku.php?id=ai_timelines:predictions_of_human-level_ai_timelines:ai_timeline_surveys:2022_expert_survey_on_progress_in_ai), we asked publishing machine learning researchers how they would divide probability over the future impacts of [high-level machine intelligence](https://wiki.aiimpacts.org/doku.php?id=clarifying_concepts:human-level_ai#high_level_machine_intelligence_hlmi) between five buckets ranging from ‘extremely good (e.g. rapid growth in human flourishing)’ to ‘extremely bad (e.g. human extinction).[1](https://aiimpacts.org/how-bad-a-future-do-ml-researchers-expect/#easy-footnote-bottom-1-3457 "The full question:

Assume for the purpose of this question that HLMI will at some point exist. How positive or negative do you expect the overall impact of this to be on humanity, in the long run? Please answer by saying how probable you find the following kinds of impact, with probabilities adding to 100%:

______ Extremely good (e.g. rapid growth in human flourishing) (1)

______ On balance good (2)

______ More or less neutral (3)

______ On balance bad (4)

______ Extremely bad (e.g. human extinction) (5)

") The median respondent put 5% on the worst bucket. But what does the whole distribution look like? Here is every person’s answer, lined up in order of probability on that worst bucket:

[![](https://aiimpacts.org/wp-content/uploads/2023/03/HLMIvalue2022final4-1024x567.png)](https://aiimpacts.org/wp-content/uploads/2023/03/HLMIvalue2022final4.png)(Column widths may be distorted or columns may be missing due to limitation of chosen software.)
And here’s basically that again from the [2016 survey](https://wiki.aiimpacts.org/doku.php?id=ai_timelines:predictions_of_human-level_ai_timelines:ai_timeline_surveys:2016_expert_survey_on_progress_in_ai) (though it looks like sorted slightly differently when optimism was equal), so you can see how things have changed:

[![](https://aiimpacts.org/wp-content/uploads/2023/03/HLMIvalue2016final4-1024x567.png)](https://aiimpacts.org/wp-content/uploads/2023/03/HLMIvalue2016final4.png)Distribution from [2016 survey](https://wiki.aiimpacts.org/doku.php?id=ai_timelines:predictions_of_human-level_ai_timelines:ai_timeline_surveys:2016_expert_survey_on_progress_in_ai). (Column widths may be distorted or columns may be missing due to limitation of chosen software.)
The most notable change to me is the new big black bar of doom at the end: people who think extremely bad outcomes are at least 50% have gone from 3% of the population to 9% in six years.

Here are the overall areas dedicated to different scenarios in the 2022 graph (equivalent to averages):

* Extremely good: 24%
* On balance good: 26%
* More or less neutral: 18%
* On balance bad: 17%
* Extremely bad: 14%

That is, between them, these researchers put 31% of their credence on AI making the world markedly worse.

Some things to keep in mind in looking at these:

* If you hear ‘median 5%’ thrown around, that refers to how the researcher right in the middle of the opinion spectrum thinks there’s a 5% chance of extremely bad outcomes. (It does not mean, ‘about 5% of people expect extremely bad outcomes’, which would be much less alarming.) Nearly half of people are at ten percent or more.
* The question illustrated above doesn’t ask about human extinction specifically, so you might wonder if ‘extremely bad’ includes a lot of scenarios less bad than human extinction. To check, we added [two more questions](https://wiki.aiimpacts.org/doku.php?id=ai_timelines:predictions_of_human-level_ai_timelines:ai_timeline_surveys:2022_expert_survey_on_progress_in_ai#existential_risk) in 2022 explicitly about ‘human extinction or similarly permanent and severe disempowerment of the human species’. For these, the median researcher also gave 5% and 10% answers. So my guess is that a lot of the extremely bad bucket in this question is pointing at human extinction levels of disaster.
* You might wonder whether the respondents were selected for being worried about AI risk. We tried to mitigate that possibility by usually offering money for completing the survey ($50 for those in the final round, after some experimentation), and describing the topic in very broad terms in the invitation (e.g. not mentioning AI risk). Last survey we checked in more detail—see ‘Was our sample representative?’ in [the paper on the 2016 survey](https://arxiv.org/pdf/1705.08807.pdf).

Here’s the 2022 data again, but ordered by overall optimism-to-pessimism rather than probability of extremely bad outcomes specifically:

[![](https://aiimpacts.org/wp-content/uploads/2023/03/How-positive-or-negative-will-the-impacts-of-high-level-machine-intelligence-on-humanity-be-in-the-long-run-202214-1024x502.png)](https://aiimpacts.org/wp-content/uploads/2023/03/How-positive-or-negative-will-the-impacts-of-high-level-machine-intelligence-on-humanity-be-in-the-long-run-202214.png)(Column widths may be distorted or columns may be missing due to limitation of chosen software.)
For more survey takeaways, see [this blog post](https://aiimpacts.org/what-do-ml-researchers-think-about-ai-in-2022/). For all the data we have put up on it so far, see [this page](https://wiki.aiimpacts.org/doku.php?id=ai_timelines:predictions_of_human-level_ai_timelines:ai_timeline_surveys:2022_expert_survey_on_progress_in_ai).

See [here](https://wiki.aiimpacts.org/doku.php?id=ai_timelines:predictions_of_human-level_ai_timelines:ai_timeline_surveys:2022_expert_survey_on_progress_in_ai#impacts_of_hlmi) for more details.

*Thanks to Harlan Stewart for helping make these 2022 figures, Zach Stein-Perlman for generally getting this data in order, and Nathan Young for pointing out that figures like this would be good.*

Notes
-----