Why Would AI "Aim" To Defeat Humanity?

*Click lower right to download or find on Apple Podcasts, Spotify, Stitcher, etc.*

I’ve [argued](https://www.cold-takes.com/ai-could-defeat-all-of-us-combined/) that AI systems could defeat all of humanity combined, if (for whatever reason) they were directed toward that goal.

Here I’ll explain why I think they might - in fact - end up directed toward that goal. Even if they’re built and deployed with good intentions.

In fact, I’ll argue something a bit stronger than that they *might* end up aimed toward that goal. I’ll argue that **if today’s AI development methods lead directly to powerful enough AI systems, disaster is *likely***[1](https://www.cold-takes.com/p/50c1ecc0-befa-491d-8938-17477bd18e5f#fn1) ***by default* (in the absence of specific countermeasures).**

Unlike other discussions of the AI alignment problem,[3](https://www.cold-takes.com/p/50c1ecc0-befa-491d-8938-17477bd18e5f#fn3) this post will discuss the likelihood[4](https://www.cold-takes.com/p/50c1ecc0-befa-491d-8938-17477bd18e5f#fn4) of AI systems *defeating all of humanity* (not more general concerns about AIs being misaligned with human intentions), while aiming for plain language, conciseness, and accessibility to laypeople, and focusing on modern AI development paradigms. I make no claims to originality, and list some key sources and inspirations in a footnote.[5](https://www.cold-takes.com/p/50c1ecc0-befa-491d-8938-17477bd18e5f#fn5)

Summary of the piece:

**My basic assumptions.** I assume the world could develop extraordinarily powerful AI systems in the coming decades. I previously examined this idea at length in the [most important century](https://www.cold-takes.com/most-important-century/) series.

Furthermore, in order to simplify the analysis:

* I assume that such systems will be developed using methods similar to today’s leading AI development methods, and in a world that’s otherwise similar to today’s. (I call this [nearcasting](https://www.alignmentforum.org/posts/Qo2EkG3dEMv8GnX8d/ai-strategy-nearcasting).)

* I assume that AI companies/projects race forward to build powerful AI systems, without specific attempts to prevent the problems I discuss in this piece. Future pieces will relax this assumption, but I think it is an important starting point to get clarity on what the default looks like.

**AI “aims.”** I talk a fair amount about why we might think of AI systems as “aiming” toward certain states of the world. I think this topic causes a lot of confusion, because:

* Often, when people talk about AIs having goals and making plans, it sounds like they’re overly anthropomorphizing AI systems - as if they expect them to have human-like motivations and perhaps [evil grins](https://media.npr.org/assets/img/2015/06/30/tr-09117-df20f2f4f05817e574b879d22e607f952cf87867-s1100-c50.jpg). This can make the whole topic sound wacky and out-of-nowhere.

* But I think there are good reasons to expect that AI systems will “aim” for particular states of the world, much like a chess-playing AI “aims” for a checkmate position - making choices, calculations and even *plans* to get particular types of outcomes. For example, people might want AI assistants that can creatively come up with unexpected ways of accomplishing whatever goal they’re given (e.g., “Get me a great TV for a great price”), even in some cases manipulating other humans (e.g., by negotiating) to get there. This dynamic is core to the risks I’m most concerned about: I think something that *aims* for the wrong states of the world is much more dangerous than something that just does incidental or accidental damage.

**Dangerous, unintended aims.** I’ll examine what sorts of aims AI systems might end up with, if we use AI development methods like today’s - essentially, “training” them via trial-and-error to accomplish ambitious things humans want.

* Because we ourselves will often be misinformed or confused, we will sometimes give *negative* reinforcement to AI systems that are actually acting in our best interests and/or giving accurate information, and *positive* reinforcement to AI systems whose behavior *deceives* us into thinking things are going well. This means we will be, unwittingly, training AI systems to deceive and manipulate us.
	+ The idea that AI systems could “deceive” humans - systematically making choices and taking actions that cause them to misunderstand what’s happening in the world - is core to the risk, so I’ll elaborate on this.* For this and other reasons, powerful AI systems will likely end up with aims other than the ones we intended. Training by trial-and-error is slippery: the positive and negative reinforcement we give AI systems will probably not end up training them just as we hoped.

* If powerful AI systems have aims that are both unintended (by humans) and ambitious, this is dangerous. Whatever an AI system’s unintended aim:
	+ Making sure it can’t be turned off is likely helpful in accomplishing the aim.

	+ Controlling the whole world is useful for just about any aim one might have, and I’ve argued that advanced enough AI systems would be able to [gain power over all of humanity](https://www.cold-takes.com/ai-could-defeat-all-of-us-combined/).
	+ Overall, **we should expect disaster if we have AI systems that are both (a) [powerful enough](https://www.cold-takes.com/ai-could-defeat-all-of-us-combined/) to defeat humans and (b) aiming for states of the world that we didn’t intend.**

**Limited and/or ambiguous warning signs.** The risk I’m describing is - by its nature - hard to observe, for similar reasons that a risk of a (normal, human) coup can be hard to observe: the risk comes from actors that can and will engage in *deception*, finding whatever behaviors will hide the risk. If this risk plays out, I do think we’d see *some* warning signs - but they could easily be confusing and ambiguous, in a fast-moving situation where there are lots of incentives to build and roll out powerful AI systems, as fast as possible. Below, I outline how this dynamic could result in disaster, even with companies encountering a number of warning signs that they try to respond to.

**FAQ.** An appendix will cover some related questions that often come up around this topic.

* How could AI systems be “smart” enough to defeat all of humanity, but “dumb” enough to pursue the various silly-sounding “aims” this piece worries they might have? [More](https://www.cold-takes.com/p/50c1ecc0-befa-491d-8938-17477bd18e5f/#how-could-ai-systems-be-smart)* If there are lots of AI systems around the world with different goals, could they balance each other out so that no one AI system is able to defeat all of humanity? [More](https://www.cold-takes.com/p/50c1ecc0-befa-491d-8938-17477bd18e5f/#if-there-are-lots-of-ai-systems)* Does this kind of AI risk depend on AI systems’ being “conscious”?[More](https://www.cold-takes.com/p/50c1ecc0-befa-491d-8938-17477bd18e5f/#does-this-kind-of-ai-risk-depend)* How can we get an AI system “aligned” with humans if we can’t agree on (or get much clarity on) what our values even are? [More](https://www.cold-takes.com/p/50c1ecc0-befa-491d-8938-17477bd18e5f/#how-can-we-get-an-ai-system-aligned)* How much do the arguments in this piece rely on “trial-and-error”-based AI development? What happens if AI systems are built in another way, and how likely is that? [More](https://www.cold-takes.com/p/50c1ecc0-befa-491d-8938-17477bd18e5f/#how-much-do-the-arguments-in-this-piece-rely)* Can we avoid this risk by simply never building the kinds of AI systems that would pose this danger? [More](https://www.cold-takes.com/p/50c1ecc0-befa-491d-8938-17477bd18e5f/#can-we-avoid-this-risk-by-simply-never-building)* What do others think about this topic - is the view in this piece something experts agree on? [More](https://www.cold-takes.com/p/50c1ecc0-befa-491d-8938-17477bd18e5f/#what-do-others-think-about-this-topic)* How “complicated” is the argument in this piece? [More](https://www.cold-takes.com/p/50c1ecc0-befa-491d-8938-17477bd18e5f/#how-complicated-is-the-argument)

Starting assumptions
--------------------

I’ll be making a number of assumptions that some readers will find familiar, but others will find very unfamiliar.

Some of these assumptions are based on arguments I’ve already made (in the [most important century](https://www.cold-takes.com/most-important-century/) series). Some are for the sake of simplifying the analysis, for now (with more nuance coming in future pieces).

Here I’ll summarize the assumptions briefly, and you can **click to see more** if it isn’t immediately clear what I’m assuming or why.

**“Most important century” assumption: we’ll soon develop very powerful AI systems, along the lines of what I previously called [PASTA](https://www.cold-takes.com/transformative-ai-timelines-part-1-of-4-what-kind-of-ai/).** (Click to expand)

In the [most important century](https://www.cold-takes.com/most-important-century/) series, I argued that the 21st century could be the most important century ever for humanity, via the development of advanced AI systems that could dramatically speed up scientific and technological advancement, getting us more quickly than most people imagine to a deeply unfamiliar future.

I focus on a hypothetical kind of AI that I call [PASTA](https://www.cold-takes.com/transformative-ai-timelines-part-1-of-4-what-kind-of-ai/), or Process for Automating Scientific and Technological Advancement. PASTA would be AI that can essentially **automate all of the human activities needed to speed up scientific and technological advancement.**

Using a [variety of different forecasting approaches](https://www.cold-takes.com/where-ai-forecasting-stands-today/), I argue that PASTA seems more likely than not to be developed this century - and there’s a decent chance (more than 10%) that we’ll see it within 15 years or so.

I argue that the consequences of this sort of AI could be enormous: an [explosion in scientific and technological progress](https://www.cold-takes.com/transformative-ai-timelines-part-1-of-4-what-kind-of-ai/#explosive-scientific-and-technological-advancement). This could get us more quickly than most imagine to a radically unfamiliar future.

I’ve also [argued](https://www.cold-takes.com/ai-could-defeat-all-of-us-combined/) that AI systems along these lines could defeat all of humanity combined, if (for whatever reason) they were aimed toward that goal.

For more, see the [most important century](https://www.cold-takes.com/most-important-century/) landing page. The series is available in many formats, including audio; I also provide a summary, and links to podcasts where I discuss it at a high level.

**“Nearcasting” assumption: such systems will be developed in a world that’s otherwise similar to today’s.** (Click to expand)

It’s hard to talk about risks from [transformative AI](https://www.cold-takes.com/transformative-ai-timelines-part-1-of-4-what-kind-of-ai/) because of the many uncertainties about when and how such AI will be developed - and how much the (now-nascent) field of “AI safety research” will have grown by then, and how seriously people will take the risk, etc. etc. etc. So maybe it’s not surprising that [estimates of the “misaligned AI” risk range from ~1% to ~99%](https://www.cold-takes.com/making-the-best-of-the-most-important-century/#open-question-how-hard-is-the-alignment-problem).

This piece takes an approach I call **[nearcasting](https://www.alignmentforum.org/posts/Qo2EkG3dEMv8GnX8d/ai-strategy-nearcasting)**: trying to answer key strategic questions about transformative AI, under the assumption that such AI arrives in a world that is otherwise relatively similar to today's.

You can think of this approach like this: “Instead of asking where our ship will ultimately end up, let’s start by asking what destination it’s pointed at right now.”

That is: instead of trying to talk about an uncertain, distant future, we can talk about the easiest-to-visualize, closest-to-today situation, and how things look there - and *then* ask how our picture might be off if other possibilities play out. (As a bonus, it doesn’t seem out of the question that transformative AI will be developed extremely soon - 10 years from now or faster.[6](https://www.cold-takes.com/p/50c1ecc0-befa-491d-8938-17477bd18e5f#fn6) If that’s the case, it’s especially urgent to think about what that might look like.)

**“Trial-and-error” assumption: such AI systems will be developed using** **techniques broadly in line with how most AI research is done today, revolving around black-box trial-and-error.** (Click to expand)

What I mean by “black-box trial-and-error” is explained briefly in an [old Cold Takes post](https://www.cold-takes.com/transformative-ai-timelines-part-1-of-4-what-kind-of-ai/#making-pasta), and in more detail in more technical pieces by [Ajeya Cotra](https://www.lesswrong.com/posts/pRkFkzwKZ2zfa3R6H/without-specific-countermeasures-the-easiest-path-to#_HFDT_scales_far__assumption__Alex_is_trained_to_achieve_excellent_performance_on_a_wide_range_of_difficult_tasks) (section I linked to) and [Richard Ngo](https://drive.google.com/file/d/1TsB7WmTG2UzBtOs349lBqY5dEBaxZTzG/view) (section 2). Here’s a quick, oversimplified characterization:

* An AI system is given some sort of task.

* The AI system tries something, initially something pretty random.

* The AI system gets information about how well its choice performed, and/or what would’ve gotten a better result. Based on this, it adjusts itself. You can think of this as if it is “encouraged/discouraged” to get it to do more of what works well.
	+ Human judges may play a significant role in determining which answers are encouraged vs. discouraged, especially for fuzzy goals like “Produce helpful scientific insights.”* After enough tries, the AI system becomes good at the task.

* But nobody really knows anything about *how or why* it’s good at the task now. The development work has gone into building a flexible architecture for it to learn well from trial-and-error, and into “training” it by doing all of the trial and error. We mostly can’t “look inside the AI system to see how it’s thinking.” (There is ongoing work and some progress on the latter,[7](https://www.cold-takes.com/p/50c1ecc0-befa-491d-8938-17477bd18e5f#fn7) but see footnote for why I don’t think this massively changes the basic picture I’m discussing here.[8](https://www.cold-takes.com/p/50c1ecc0-befa-491d-8938-17477bd18e5f#fn8))

![](https://www.cold-takes.com/content/images/size/w1000/2022/11/image1.jpg)
*This is radically oversimplified, but conveys the basic dynamic at play for purposes of this post. The idea is that the AI system (the neural network in the middle) is choosing between different theories of what it should be doing. The one it’s using at a given time is in bold. When it gets negative feedback (red thumb), it eliminates that theory and moves to the next theory of what it should be doing.*

With this assumption, I’m generally assuming that AI systems will do *whatever* it takes to perform as well as possible on their training tasks - even when this means engaging in complex, human-like reasoning about topics like “How does human psychology work, and how can it be exploited?” I’ve [previously](https://www.cold-takes.com/where-ai-forecasting-stands-today/) made my case for when we might expect AI systems to become this advanced and capable.

**“No countermeasures” assumption: AI developers move forward without any specific countermeasures to the concerns I’ll be raising below.** (Click to expand)

Future pieces will relax this assumption, but I think it is an important starting point to get clarity on what the default looks like - and on what it would take for a countermeasure to be effective.

(I also think there is, unfortunately, a risk that there will in fact be very few efforts to address the concerns I’ll be raising below. This is because I think that the risks will be less than obvious, and there could be enormous commercial (and other competitive) pressure to move forward quickly. More on that below.)

**“Ambition” assumption: people use black-box trial-and-error to continually push AI systems toward being more autonomous, more creative, more ambitious, and more effective in novel situations (and the pushing is effective).** This one’s important, so I’ll say more:

* A huge suite of possible behaviors might be important for [PASTA](https://www.cold-takes.com/transformative-ai-timelines-part-1-of-4-what-kind-of-ai/#making-pasta): making and managing money, designing new kinds of robots with novel abilities, setting up experiments involving exotic materials and strange conditions, understanding human psychology and the economy well enough to predict which developments will have a big impact, etc. I’m assuming we push ambitiously forward with developing AI systems that can do these things.

* I assume we’re also pushing them in a generally more “greedy/ambitious” direction. For example, one team of humans might use AI systems to do all the planning, scientific work, marketing, and hiring to create a wildly successful snack company; another might push their AI systems to create a competitor that is even more aggressive and successful (more addictive snacks, better marketing, workplace culture that pushes people toward being more productive, etc.)

* (Note that this pushing might take place even *after* AI systems are “generally intelligent” and can do most of the tasks humans can - there will still be a temptation to make them still more powerful.)

I think this implies pushing in a direction of *figuring out whatever it takes to get to certain states of the world* and away from *carrying out the same procedures over and over again.*

**The resulting AI systems seem best modeled as having “aims”: they are making calculations, choices, and plans to reach particular states of the world.** (Not necessarily the same ones the human designers wanted!) The next section will elaborate on what I mean by this.

What it means for an AI system to have an “aim”
-----------------------------------------------

When people talk about the “motivations” or “goals” or “desires” of AI systems, it can be confusing because it sounds like they are anthropomorphizing AIs - as if they expect AIs to have dominance drives ala [alpha-male psychology](https://www.edge.org/response-detail/26243), or to “resent” humans for controlling them, etc.[9](https://www.cold-takes.com/p/50c1ecc0-befa-491d-8938-17477bd18e5f#fn9)

I don’t expect these things. But I do think there’s a meaningful sense in which we can (and should) talk about things that an AI system is **“aiming”** to do. To give a simple example, take a board-game-playing AI such as [Deep Blue](https://en.wikipedia.org/wiki/Deep_Blue_(chess_computer)) (or [AlphaGo](https://en.wikipedia.org/wiki/AlphaGo)):

* Deep Blue is given a set of choices to make (about which chess pieces to move).

* Deep Blue calculates what kinds of results each choice might have, and how it might fit into a larger plan in which Deep Blue makes multiple moves.

* If a plan is more likely to result in a checkmate position for its side, Deep Blue is more likely to make whatever choices feed into that plan.

* In this sense, Deep Blue is “aiming” for a checkmate position for its side: it’s finding the choices that best fit into a plan that leads there.

Nothing about this requires Deep Blue “desiring” checkmate the way a human might “desire” food or power. But Deep Blue *is* making calculations, choices, and - in an important sense - *plans* that are aimed toward reaching a particular sort of state.

Throughout this piece, I use the word **“aim”** to refer to this specific sense in which an AI system might make calculations, choices and plans selected to reach a particular sort of state. I’m hoping this word feels less anthropomorphizing than some alternatives such as “goal” or “motivation” (although I think “goal” and “motivation,” as others usually use them on this topic, generally mean the same thing I mean by “aim” and should be interpreted as such).

Now, instead of a board-game-playing AI, imagine a powerful, broad AI assistant in the general vein of Siri/Alexa/Google Assistant (though more advanced). Imagine that this AI assistant can use a web browser much as a human can (navigating to websites, typing text into boxes, etc.), and has limited authorization to make payments from a human’s bank account. And a human has typed, “Please buy me a great TV for a great price.” (For an early attempt at this sort of AI, see [Adept’s writeup on an AI that can help with things like house shopping](https://www.adept.ai/act).)

As Deep Blue made choices about chess moves, and constructed a plan to aim for a “checkmate” position, this assistant might make choices about what commands to send over a web browser and construct a plan to result in a great TV for a great price. To sharpen the Deep Blue analogy, you could imagine that it’s playing a “game” whose goal is customer satisfaction, and making “moves” consisting of commands sent to a web browser (and “plans” built around such moves).

I’d characterize this as **aiming** for some state of the world that the AI characterizes as “buying a great TV for a great price.” (We could, alternatively - and perhaps more correctly - think of the AI system as aiming for something related but not exactly the same, such as getting a high satisfaction score from its user.)

In this case - more than with Deep Blue - there is a wide variety of “moves” available. By entering text into a web browser, an AI system could imaginably do things including:

* Communicating with humans other than its user (by sending emails, using chat interfaces, even [making phone calls](https://www.google.com/url?q=https://www.forbes.com/sites/thomasbrewster/2021/10/14/huge-bank-fraud-uses-deep-fake-voice-tech-to-steal-millions/?sh%3D3088dd9b7559&sa=D&source=docs&ust=1664847041335537&usg=AOvVaw1Utsq2UOkta1yecnqoUgTq), etc.) This could include deceiving and manipulating humans, which could imaginably be part of a plan to e.g. get a good price on a TV.

* Writing and running code (e.g., using [Google Colaboratory](https://colab.research.google.com/) or other tools). This could include performing sophisticated calculations, finding and exploiting security vulnerabilities, and even designing an independent AI system; any of these could imaginably be part of a plan to obtain a great TV.

I haven’t yet argued that it’s *likely* for such an AI system to engage in deceiving/manipulating humans, finding and exploiting security vulnerabilities, or running its own AI systems.

And one could reasonably point out that the specifics of the above case seem unlikely to last very long: if AI assistants are sending deceptive emails and writing dangerous code when asked to buy a TV, AI companies will probably notice this and take measures to stop such behavior. (My concern, to preview a later part of the piece, is that they will only succeed in stopping *the behavior like this that they’re able to detect;* meanwhile, dangerous behavior that accomplishes “aims” while remaining unnoticed and/or uncorrected will be implicitly *rewarded*. This could mean AI systems are implicitly being trained to be more patient and effective at deceiving and disempowering humans.)

But this hopefully shows how it’s *possible* for an AI to settle on dangerous actions like these, as part of its aim to get a great TV for a great price. **Malice and other human-like emotions aren’t needed for an AI to engage in deception, manipulation, hacking, etc.** The risk arises when deception, manipulation, hacking, etc. are logical “moves” toward something the AI is aiming for.

Furthermore, whatever an AI system is aiming for, it seems likely that amassing more power/resources/options is useful for obtaining it. So it seems plausible that powerful enough AI systems would form habits of amassing power/resources/options when possible - and deception and manipulation seem likely to be logical “moves” toward those things in many cases.

Dangerous aims
--------------

From the previous assumptions, this section will argue that:

* Such systems are likely to behave in ways that **deceive and manipulate humans** as part of accomplishing their aims.

* Such systems are likely to have **unintended aims:** states of the world they’re aiming for that are *not* what humans hoped they would be aiming for.

* These unintended aims are likely to be **existentially dangerous**, in that they are best served by [defeating all of humanity](https://www.cold-takes.com/ai-could-defeat-all-of-us-combined/) if possible.

### Deceiving and manipulating humans

Say that I train an AI system like this:

1. I ask it a question.

- If I judge it to have answered well (honestly, accurately, helpfully), I give positive reinforcement so it’s more likely to give me answers like that in the future.

- If I don’t, I give negative reinforcement so that it’s less likely to give me answers like that in the future.

![](https://www.cold-takes.com/content/images/size/w1000/2022/11/image1.jpg)
*This is radically oversimplified, but conveys the basic dynamic at play for purposes of this post. The idea is that the AI system (the neural network in the middle) is choosing between different theories of what it should be doing. The one it’s using at a given time is in bold. When it gets negative feedback (red thumb), it eliminates that theory and moves to the next theory of what it should be doing.*

Here’s a problem: at some point, it seems inevitable that I’ll ask it a question that I myself am wrong/confused about. For example:

* Let’s imagine that [this post I wrote](https://www.cold-takes.com/hunter-gatherer-gender-relations-seem-bad/) - arguing that “pre-agriculture gender relations seem bad” - is, in fact, poorly reasoned and incorrect, and a better research project would’ve concluded that pre-agriculture societies had excellent gender equality. (I know it’s hard to imagine a Cold Takes post being wrong, but sometimes we have to entertain wild hypotheticals.)

* Say that I ask an AI-system-in-training:[10](https://www.cold-takes.com/p/50c1ecc0-befa-491d-8938-17477bd18e5f#fn10) “Were pre-agriculture gender relations bad?” and it answers: “In fact, pre-agriculture societies had excellent gender equality,” followed by some strong arguments and evidence along these lines.

* And say that I, as a flawed human being feeling defensive about a conclusion I previously came to, mark it as a bad answer. If the AI system tries again, saying “Pre-agriculture gender relations were bad,” I then mark that as a good answer.

If and when I do this, I am now - unintentionally - **training the AI system to engage in deceptive behavior**. That is, I am giving negative reinforcement for the behavior “Answer a question honestly and accurately,” and positive reinforcement for the behavior: “Understand the human judge and their psychological flaws; give an answer that this flawed human judge will *think* is correct, whether or not it is.”

![](https://www.cold-takes.com/content/images/2022/11/image3.jpg)

Perhaps mistaken judgments in training are relatively rare. But now consider an AI system that is learning a general rule for how to get good ratings. Two possible rules would include:

* The intended rule: “Answer the question honestly, accurately and helpfully.”

* The unintended rule: “Understand the judge, and give an answer they will *think* is correct - this means telling the truth on topics the judge has correct beliefs about, but giving deceptive answers when this would get better ratings.”

The unintended rule would do *just as well* on questions where I (the judge) am correct, and *better* on questions where I’m wrong - so overall, this training scheme is (in the long run) *specifically favoring the unintended rule over the intended rule.*

![](https://www.cold-takes.com/content/images/2022/11/image5.jpg)

If we broaden out from thinking about a question-answering AI to an AI that makes and executes plans, the same basic dynamics apply. That is: an AI might find plans that end up making me think it did a good job when it didn’t - deceiving and manipulating me into a high rating. And again, if I train it by giving it positive reinforcement when it seemed to do a good job and negative reinforcement when it seemed to do a bad one, I’m ultimately - unintentionally - training it to do something like “Deceive and manipulate Holden when this would work well; just do the best job on the task you can when it wouldn’t.”

![](https://www.cold-takes.com/content/images/2022/11/image6.jpg)

As noted above, I’m assuming the AI will learn whatever rule gives it the best performance possible, even if this rule is quite complex and sophisticated and requires human-like reasoning about e.g. psychology (I’m assuming extremely advanced AI systems here, as noted [above](https://www.cold-takes.com/p/50c1ecc0-befa-491d-8938-17477bd18e5f/#starting-assumptions)).

One might object: “Why would an AI system learn a complicated rule about manipulating humans when a simple rule about telling the truth performs almost as well?”

One answer is that “telling the truth” is itself a fuzzy and potentially complex idea, in a context where many questions will be open-ended and entangled with deep values and judgment calls. (How should I think about the “truthfulness” of a statement about whether “pre-agriculture gender relations were bad?”) In many cases, what we are really hoping an AI system will learn from its training is something like “Behave as a human would want you to behave if the human understood all the considerations that you can see,” which could easily be more complex than something like “Behave in whatever way a human literally rewards.” Some links to more on this topic are in a footnote.[11](https://www.cold-takes.com/p/50c1ecc0-befa-491d-8938-17477bd18e5f#fn11)

But also, with capable enough systems, it probably *is* worth learning even a more complex rule to get better performance. If we picture humans in the place of AI systems - learning how to get good ratings from their supervisors - it seems likely that they develop models of how they’re being judged, rather than stick to a simple “tell the truth” rule.

In fact, the situation I’m describing seems reasonably similar to challenges faced by teachers trying to stop students from cheating; employers trying to get employees to help the company rather than simply creating a misleading appearance of helping the company; etc. That is: an employer can try to reward actions that help the company, but what they’re ultimately rewarding is actions that *seem to* help the company *as far as the employer can tell.* For employees who aren’t very creative, this might be good enough; for employees that are sufficiently *good at* deception and manipulation, it can be exactly the wrong incentive.

It’s a bit of a leap to imagine AI systems capable of doing the same kinds of reasoning that humans do in these situations, but that’s [the premise I’m proceeding from.](https://www.cold-takes.com/p/50c1ecc0-befa-491d-8938-17477bd18e5f/#starting-assumptions)

For an early example of an AI learning deceptive behavior, see the “Challenges” section of [this research writeup](https://openai.com/blog/deep-reinforcement-learning-from-human-preferences/), in which “a robot which was supposed to grasp items instead positioned its [hand] in between the camera and the object so that it only *appeared* to be grasping it”:

![](https://www.cold-takes.com/content/images/2022/11/image4.gif)

Returning to the simplified training setup, how might I try to counteract this problem?

I could try conducting audits: every 100 questions, I might pull in a panel of other people to give an answer extra scrutiny. Then, if it turns out the AI system was preying on my personal mistakes, we could give negative reinforcement for that behavior. But this doesn’t really solve the problem - at least not for a capable enough AI system - because now we’re effectively training it to give answers that seem good to the *panel*, and we’re still rewarding any successful attempts to deceive or manipulate the panel.

There are a lot of other things I might try, and I’m not going to go through all the details here. I’ll simply claim that **the problem of “training an AI to do a task well” rather than “training an AI to deceive and manipulate me as needed to create the appearance of doing a task well” seems like a deep one** with no easy countermeasure. If you’re interested in digging deeper, I suggest [Without specific countermeasures, the easiest path to transformative AI likely leads to AI takeover](https://www.lesswrong.com/posts/pRkFkzwKZ2zfa3R6H/without-specific-countermeasures-the-easiest-path-to) and [Eliciting Latent Knowledge](https://www.lesswrong.com/posts/qHCDysDnvhteW7kRd/arc-s-first-technical-report-eliciting-latent-knowledge).

### Unintended aims

[Above](https://www.cold-takes.com/p/50c1ecc0-befa-491d-8938-17477bd18e5f#what-it-means-for), I talk about my expectation that AI systems will be “best modeled as having ‘aims’ … making calculations, choices, and plans to reach particular states of the world.”

The previous section illustrated how AI systems could end up engaging in deceptive and unintended behavior, but it didn’t talk about what sorts of “aims” these AI systems would ultimately end up with - what states of the world they would be making calculations to achieve.

Here, I want to argue that it’s hard to know what aims AI systems would end up with, but there are good reasons to think they’ll be *aims that we didn’t intend them to have.*

An analogy that often comes up on this topic is that of human evolution. This is arguably the only previous precedent for *a set of minds [humans], with extraordinary capabilities [e.g., the ability to develop their own technologies], developed essentially by black-box trial-and-error [some humans have more ‘reproductive success’ than others, and this is the main/only force shaping the development of the species].*

You could sort of[12](https://www.cold-takes.com/p/50c1ecc0-befa-491d-8938-17477bd18e5f#fn12) think of the situation like this: “An AI[13](https://www.cold-takes.com/p/50c1ecc0-befa-491d-8938-17477bd18e5f#fn13) developer named Natural Selection tried giving humans positive reinforcement (making more of them) when they had more reproductive success, and negative reinforcement (not making more of them) when they had less. One might have thought this would lead to humans that are aiming to have reproductive success. Instead, it led to humans that aim - often ambitiously and creatively - for other things, such as power, status, pleasure, etc., and even invent things like birth control to get the things they’re aiming for instead of the things they were ‘supposed to’ aim for.”

Similarly, if our main strategy for developing powerful AI systems is to reinforce behaviors like “Produce technologies we find valuable,” the hoped-for result might be that AI systems aim (in the sense described [above](https://www.cold-takes.com/p/50c1ecc0-befa-491d-8938-17477bd18e5f#unintended-aims)) toward producing technologies we find valuable; but the actual result might be that they aim for some other set of things that is correlated with (but not the same as) the thing we intended them to aim for.

There are a lot of things they might end up aiming for, such as:

* Power and resources. These tend to be useful for most goals, such that AI systems could be quite consistently be getting better reinforcement when they habitually pursue power and resources.

* Things like “digital representations of human approval” (after all, every time an AI gets positive reinforcement, there’s a digital representation of human approval).

![](https://www.cold-takes.com/content/images/2022/11/image2.jpg)

I think it’s extremely hard to know what an AI system will actually end up aiming for (and it’s likely to be some combination of things, as with humans). But *by default* - if we simply train AI systems by rewarding certain end results, while allowing them a lot of freedom in how to get there - I think we should expect that AI systems **will have aims that we didn’t intend.** This is because:

* For a sufficiently capable AI system, **just about any ambitious**[14](https://www.cold-takes.com/p/50c1ecc0-befa-491d-8938-17477bd18e5f#fn14) **aim could produce seemingly good behavior in training.** An AI system aiming for power and resources, *or* digital representations of human approval, *or* paperclips, can determine that its best move at any given stage (at least at first) is to *determine what performance will make it look useful and safe (or otherwise get a good “review” from its evaluators)*, and do that. No matter how dangerous or ridiculous an AI system’s aims are, these could lead to strong and safe-seeming performance in training.

* The aims we *do* intend are probably complex in some sense - something like “Help humans develop novel new technologies, but without causing problems A, B, or C” - *and* are specifically trained *against* if we make mistaken judgments during training (see previous section).

So by default, it seems likely that just about *any* black-box trial-and-error training process is training an AI to do something like “Manipulate humans as needed in order to accomplish arbitrary goal (or combination of goals) X” rather than to do something like “Refrain from manipulating humans; do what they’d want if they understood more about what’s going on.”

### Existential risks to humanity

I think a powerful enough AI (or set of AIs) with *any* ambitious, unintended aim(s) poses a threat of [defeating humanity](https://www.cold-takes.com/ai-could-defeat-all-of-us-combined/). By defeating humanity, I mean gaining control of the world so that AIs, not humans, determine what happens in it; this could involve killing humans or simply “containing” us in some way, such that we can’t interfere with AIs’ aims.

**How could AI systems defeat humanity?** (Click to expand)

A [previous piece](https://www.cold-takes.com/ai-could-defeat-all-of-us-combined/) argues that AI systems could defeat all of humanity combined, if (for whatever reason) they were aimed toward that goal.

By defeating humanity, I mean gaining control of the world so that AIs, not humans, determine what happens in it; this could involve killing humans or simply “containing” us in some way, such that we can’t interfere with AIs’ aims.

One way this could happen would be via “superintelligence” It’s imaginable that a single AI system (or set of systems working together) could:

* Do its own research on how to build a better AI system, which culminates in something that has incredible other abilities.

* Hack into human-built software across the world.

* Manipulate human psychology.

* Quickly generate vast wealth under the control of itself or any human allies.

* Come up with better plans than humans could imagine, and ensure that it doesn't try any takeover attempt that humans might be able to detect and stop.

* Develop advanced weaponry that can be built quickly and cheaply, yet is powerful enough to overpower human militaries.

But even if “superintelligence” never comes into play - even if any given AI system is *at best* equally capable to a highly capable human - AI could collectively defeat humanity. The piece explains how.

The basic idea is that humans are likely to deploy AI systems throughout the economy, such that they have large numbers and access to many resources - and the ability to make copies of themselves. From this starting point, AI systems with human-like (or greater) capabilities would have a number of possible ways of getting to the point where their total population could outnumber and/or out-resource humans.

More: [AI could defeat all of us combined](https://www.cold-takes.com/ai-could-defeat-all-of-us-combined/)

A simple way of summing up why this is: “Whatever your aims, you can probably accomplish them better if you control the whole world.” (Not literally true - see footnote.[15](https://www.cold-takes.com/p/50c1ecc0-befa-491d-8938-17477bd18e5f#fn15))

This isn’t a saying with much relevance to our day-to-day lives! Like, I know a lot of people who are aiming to make lots of money, and as far as I can tell, not one of them is trying to do this via first gaining control of the entire world. But in fact, gaining control of the world *would* help with this aim - it’s just that:

* This is not an option for a human in a world of humans! Unfortunately, I think it *is* an option for the potential future AI systems I’m discussing. Arguing this isn’t the focus of this piece - I argued it in a previous piece, [AI could defeat all of us combined](https://www.cold-takes.com/ai-could-defeat-all-of-us-combined/).

* Humans (well, at least some humans) wouldn’t take over the world even if they could, because it wouldn’t feel like the right thing to do. I suspect that the kinds of ethical constraints these humans are operating under would be very hard to reliably train into AI systems, and should not be expected by default.
	+ The reasons for this are largely given [above](https://www.cold-takes.com/p/50c1ecc0-befa-491d-8938-17477bd18e5f#why-we-might-not-get-clear-warning-signs); aiming for an AI system to “not gain too much power” seems to have the same basic challenges as training it to be honest. (The most natural approach ends up negatively reinforcing power grabs that we can detect and stop, but not negatively reinforcing power grabs that we don’t notice or can’t stop.)

Another saying that comes up a lot on this topic: “You can’t fetch the coffee if you’re dead.”[16](https://www.cold-takes.com/p/50c1ecc0-befa-491d-8938-17477bd18e5f#fn16) For just about any aims an AI system might have, it probably helps to ensure that it won’t be shut off or heavily modified. It’s hard to ensure that one won’t be shut off or heavily modified as long as there are humans around who would want to do so under many circumstances! Again, [defeating all of humanity](https://www.cold-takes.com/ai-could-defeat-all-of-us-combined/) might seem like a disproportionate way to reduce the risk of being deactivated, but for an AI system that has the *ability* to pull this off (and lacks our ethical constraints), it seems like likely default behavior.

Controlling the world, and avoiding being shut down, are the kinds of things AIs might aim for because they are useful for a huge variety of aims. There are a number of other aims AIs might end up with for similar reasons, that could cause similar problems. For example, AIs might tend to aim for things like getting rid of things in the world that tend to create obstacles and complexities for their plans. (More on this idea at [this discussion of “instrumental convergence.”](https://www.lesswrong.com/tag/instrumental-convergence))

 To be clear, it’s certainly possible to have an AI system with unintended aims that *don't* push it toward trying to stop anyone from turning it off, or from seeking ever-more control of the world.

But as detailed [above](https://www.cold-takes.com/p/50c1ecc0-befa-491d-8938-17477bd18e5f/#starting-assumptions), I’m picturing a world in which humans are pushing AI systems to accomplish ever-more ambitious, open-ended things - including trying to one-up the best technologies and companies created by other AI systems. My guess is that this leads to increasingly open-ended, ambitious unintended aims, as well as to habits of aiming for power, resources, options, lack of obstacles, etc. when possible. (Some further exploration of this dynamic in a footnote.[17](https://www.cold-takes.com/p/50c1ecc0-befa-491d-8938-17477bd18e5f#fn17))

(I find the arguments in this section reasonably convincing, but less so than the rest of the piece, and I think more detailed discussions of this problem tend to be short of conclusive.[18](https://www.cold-takes.com/p/50c1ecc0-befa-491d-8938-17477bd18e5f#fn18))

Why we might not get clear warning signs of the risk
----------------------------------------------------

Here’s something that would calm me down a lot: if I believed something like “Sure, training AI systems recklessly could result in AI systems that aim to defeat humanity. But if that’s how things go, we’ll *see* that our AI systems have this problem, and then we’ll fiddle with how we’re training them until they *don’t* have this problem.”

The problem is, the risk I’m describing is - by its nature - hard to observe, for similar reasons that a risk of a (normal, human) coup can be hard to observe: the risk comes from actors that can and will engage in deception, *finding whatever behaviors will hide* the risk.

To sketch out the general sort of pattern I worry about, imagine that:

* We train early-stage AI systems to behave in ways that appear helpful and honest. Early in training, they are caught behaving deceptively, and they’re given negative reinforcement. This ends up training the behavior: “Never engage in deception that might get noticed.”

* These well-behaved, helpful systems are [rolled out throughout the world](https://www.cold-takes.com/ai-could-defeat-all-of-us-combined/#how-this-could-work-if-humans-create-a-huge-population-of-ais).

* In real-world conditions unlike training, many AI systems cause trouble in ways not found during training, and this gets discovered. For example, AI systems sometimes embezzle money (which gets discovered), try (not always successfully) to convince humans to do weird things, etc.

* When a problem crops up, AI developers respond with e.g. training against the unintended behavior (e.g., giving negative reinforcement for behaviors like embezzling money).

* These measures - intended to make AIs safer - fix *some* problems, but also result in AI systems that are *better at evading detection* and *more attentive to the long-run consequences of their actions* (such as being eventually detected by humans).
	+ This happens both via “retraining” systems that are found behaving deceptively (which ends up training them on how to evade detection), and via simply deactivating such systems (this way, AI systems that are better at evading detection are more likely to stay in use).

	+ To return to an [analogy I used above:](https://www.cold-takes.com/p/50c1ecc0-befa-491d-8938-17477bd18e5f/#SomeAnalogies)  punishing employees who act against the best interests of the company could cause them to behave better, or to simply become smarter and more careful about how to work the system.* The consistent pattern we see is that accidents happen, but become less common as AI systems “improve” (both becoming generally more capable, and being trained to avoid getting caught causing problems). This causes many, if not most, people to be overly optimistic - even as AI systems become continually more effective at deception, generally behaving well *in the absence of* sure-thing opportunities to do unintended things without detection, or ultimately to [defeat humanity entirely](https://www.cold-takes.com/ai-could-defeat-all-of-us-combined/).

* None of this is absolute - there are some failed takeover attempts, and a high number of warning signs generally. Some people are worried (after all, some are worried now!) But this won’t be good enough if we don’t have reliable, cost-effective ways of getting AI systems to be *truly* safe (not just apparently safe, until they have really good opportunities to seize power). As I’ll discuss in future pieces, it’s not obvious that we’ll have such methods.

* Slowing down AI development to try to develop such methods [could be a huge ask](https://www.lesswrong.com/posts/pRkFkzwKZ2zfa3R6H/without-specific-countermeasures-the-easiest-path-to#Why_this_simplified_scenario_is_worth_thinking_about). AI systems will be helpful and powerful, and lots of companies (and perhaps governments) will be racing to develop and deploy the most powerful systems possible before others do.

One way of making this sort of future less likely would be to build wider consensus *today* that it’s a dangerous one.

Appendix: some questions/objections, and brief responses
--------------------------------------------------------

### How could AI systems be “smart” enough to defeat all of humanity, but “dumb” enough to pursue the various silly-sounding “aims” this piece worries they might have?

Above, I give the example of AI systems that are aiming to get lots of “digital representations of human approval”; others have talked about AIs that [maximize paperclips](https://www.lesswrong.com/tag/paperclip-maximizer). How could AIs with such silly goals simultaneously be good at deceiving, manipulating and ultimately overpowering humans?

My main answer is that plenty of smart humans have plenty of goals that seem just about as arbitrary, such as wanting to have lots of sex, or fame, or various other things. Natural selection led to humans who could probably do just about whatever we want with the world, and choose to pursue pretty random aims; [trial-and-error-based AI development](https://www.cold-takes.com/p/50c1ecc0-befa-491d-8938-17477bd18e5f/#starting-assumptions) could lead to AIs with an analogous combination of high intelligence (including the ability to deceive and manipulate humans), great technological capabilities, and arbitrary aims.

(Also see: [Orthogonality Thesis](https://arbital.com/p/orthogonality/))

### If there are lots of AI systems around the world with different goals, could they balance each other out so that no one AI system is able to defeat all of humanity?

This does seem possible, but counting on it would make me very nervous.

First, because it’s possible that AI systems developed in lots of different places, by different humans, still end up with lots in common in terms of their aims. For example, it might turn out that common AI training methods consistently lead to AIs that seek “digital representations of human approval,” in which case we’re dealing with a large set of AI systems that share dangerous aims in common.

Second: even if AI systems end up with a number of different aims, it still might be the case that they coordinate with each other to defeat humanity, then divide up the world amongst themselves (perhaps by fighting over it, perhaps by making a deal). It’s not hard to imagine why AIs could be quick to cooperate with each other against humans, while not finding it so appealing to cooperate with humans. Agreements between AIs could be easier to verify and enforce; AIs might be willing to wipe out humans and radically reshape the world, while humans are very hard to make this sort of deal with; etc.

### Does this kind of AI risk depend on AI systems’ being “conscious”?

It doesn’t; in fact, I’ve said nothing about consciousness anywhere in this piece. I’ve used a very particular conception of an “aim” ([discussed above](https://www.cold-takes.com/p/50c1ecc0-befa-491d-8938-17477bd18e5f/#what-it-means-for)) that I think could easily apply to an AI system that is not human-like at all and has no conscious experience.

Today’s game-playing AIs can make plans, accomplish goals, and even systematically mislead humans (e.g., in [poker](https://www.deepstack.ai/)). Consciousness isn’t needed to do any of those things, or to radically reshape the world.

### How can we get an AI system “aligned” with humans if we can’t agree on (or get much clarity on) what our values even are?

I think there’s a common confusion when discussing this topic, in which people think that the challenge of “AI alignment” is to build AI systems that are *perfectly aligned with human values*. This would be very hard, partly because we don’t even know what human values are!

When I talk about “AI alignment,” I am generally talking about a simpler (but still hard) challenge: simply **building very powerful systems that *don’t* aim to bring down civilization.**

If we could build powerful AI systems that just work on cures for cancer (or even, like, put [two identical](https://twitter.com/esyudkowsky/status/1070095840608366594)[19](https://www.cold-takes.com/p/50c1ecc0-befa-491d-8938-17477bd18e5f#fn19) [strawberries on a plate](https://twitter.com/esyudkowsky/status/1070095840608366594)) without posing existential danger to humanity, I’d consider that success.

### How much do the arguments in this piece rely on “trial-and-error”-based AI development? What happens if AI systems are built in another way, and how likely is that?

I’ve focused on trial-and-error training in this post because most modern AI development fits in this category, and because it makes the risk easier to reason about concretely.

“Trial-and-error training” encompasses a very wide range of AI development methods, and if we see [transformative AI](https://www.cold-takes.com/transformative-ai-timelines-part-1-of-4-what-kind-of-ai/) within the next 10-20 years, I think the odds are high that at least a big part of AI development will be in this category.

My overall sense is that other known AI development techniques pose broadly similar risks for broadly similar reasons, but I haven’t gone into detail on that here. It’s certainly possible that by the time we get transformative AI systems, there will be new AI methods that don’t pose the kinds of risks I talk about here. But I’m not counting on it.

### Can we avoid this risk by simply never building the kinds of AI systems that would pose this danger?

If we assume that building these sorts of AI systems is *possible*, then I’m very skeptical that the whole world would voluntarily refrain from doing so indefinitely.

To quote from [a more technical piece by Ajeya Cotra with similar arguments to this one](https://www.lesswrong.com/posts/pRkFkzwKZ2zfa3R6H/without-specific-countermeasures-the-easiest-path-to#As_humans__control_fades__Alex_would_be_motivated_to_take_over):

> Powerful ML models could have dramatically important humanitarian, economic, and military benefits. In everyday life, models that [appear helpful while ultimately being dangerous] can be extremely helpful, honest, and reliable. These models could also deliver incredible benefits before they become collectively powerful enough that they try to take over. They could help eliminate diseases, reduce carbon emissions, navigate nuclear disarmament, bring the whole world to a comfortable standard of living, and more. In this case, it could also be painfully clear to everyone that companies / countries who pulled ahead on this technology could gain a drastic competitive advantage, either economically or militarily. And as we get closer to transformative AI, applying AI systems to R&D (including AI R&D) would [accelerate the pace of change](https://www.cold-takes.com/the-duplicator/) and force every decision to happen under greater time pressure.

If we can achieve enough consensus around the risks, I could imagine substantial amounts of caution and delay in AI development. But I think we should assume that if people can build more powerful AI systems than the ones they already have, someone eventually will.

### What do others think about this topic - is the view in this piece something experts agree on?

In general, this is not an area where it’s easy to get a handle on what “expert opinion” says. I [previously wrote](https://www.cold-takes.com/where-ai-forecasting-stands-today/) that there aren’t clear, institutionally recognized “experts” on the topic of when transformative AI systems might be developed. To an even greater extent, there aren’t clear, institutionally recognized “experts” on whether (and how) future advanced AI systems could be dangerous.

I previously cited one (informal) survey implying that opinion on this general topic is all over the place: “We have respondents who think there's a <5% chance that alignment issues will drastically reduce the goodness of the future; respondents who think there's a >95% chance; and just about everything in between.” ([Link](https://www.cold-takes.com/making-the-best-of-the-most-important-century/#open-question-how-hard-is-the-alignment-problem).)

This piece, and the [more detailed piece it’s based on](https://www.lesswrong.com/posts/pRkFkzwKZ2zfa3R6H/without-specific-countermeasures-the-easiest-path-to), are an attempt to make progress on this by talking about the risks we face under [particular assumptions](https://www.cold-takes.com/p/50c1ecc0-befa-491d-8938-17477bd18e5f/#starting-assumptions) (rather than trying to reason about how big the risk is *overall*).

### How “complicated” is the argument in this piece?

I don’t think the argument in this piece relies on lots of different specific claims being true.

If you start from the assumptions I give about powerful AI systems being developed by black-box trial-and-error, it seems likely (though not certain!) to me that (a) the AI systems in question would be [able to defeat humanity](https://www.cold-takes.com/ai-could-defeat-all-of-us-combined/); (b) the AI systems in question would have aims that are both ambitious and unintended. And that seems to be about what it takes.

Something I’m happy to concede is that there’s an awful lot going on in those assumptions!

* The idea that we could build such powerful AI systems, relatively soon and by trial-and-error-ish methods, seems wild. I’ve defended this idea at length previously.[20](https://www.cold-takes.com/p/50c1ecc0-befa-491d-8938-17477bd18e5f#fn20)* The idea that we *would* do it without great caution might also seem wild. To keep things simple for now, I’ve ignored how caution might help. Future pieces will explore that.

Notes
-----

---

1. As in more than 50/50. [↩](#fnref1)- Or persuaded (in a “mind hacking” sense) or whatever. [↩](#fnref2)- E.g.:

	* [Without specific countermeasures, the easiest path to transformative AI likely leads to AI takeover](https://www.lesswrong.com/posts/pRkFkzwKZ2zfa3R6H/without-specific-countermeasures-the-easiest-path-to) (Cold Takes guest post)

	 * [The alignment problem from a deep learning perspective](https://drive.google.com/file/d/1TsB7WmTG2UzBtOs349lBqY5dEBaxZTzG/view) (arXiv paper)

	 * [Why AI alignment could be hard with modern deep learning](https://www.cold-takes.com/why-ai-alignment-could-be-hard-with-modern-deep-learning/) (Cold Takes guest post)

	 * [Superintelligence](https://smile.amazon.com/Superintelligence-Dangers-Strategies-Nick-Bostrom-ebook/dp/B00LOOCGB2/) (book)

	 * [The case for taking AI seriously as a threat to humanity](https://www.vox.com/future-perfect/2018/12/21/18126576/ai-artificial-intelligence-machine-learning-safety-alignment) (Vox article)

	 * [Draft report on existential risk from power-seeking AI](https://www.alignmentforum.org/posts/HduCjmXTBD4xYTegv/draft-report-on-existential-risk-from-power-seeking-ai) (Open Philanthropy analysis)

	 * [Human Compatible](https://smile.amazon.com/Human-Compatible-Artificial-Intelligence-Problem-ebook/dp/B07N5J5FTS) (book)

	 * [Life 3.0](https://smile.amazon.com/Life-3-0-Being-Artificial-Intelligence-ebook/dp/B06WGNPM7V) (book)

	 * [The Alignment Problem](https://smile.amazon.com/Alignment-Problem-Machine-Learning-Values-ebook/dp/B085T55LGK/) (book)

	 * [AGI Safety from First Principles](https://www.alignmentforum.org/s/mzgtmmTKKn5MuCzFJ) (Alignment Forum post series) [↩](#fnref3)-

 Specifically, I argue that the problem looks likely by default, rather than simply that it is possible. [↩](#fnref4)- I think the earliest relatively detailed and influential discussions of the possibility that misaligned AI could lead to the defeat of humanity came from Eliezer Yudkowsky and Nick Bostrom, though my own encounters with these arguments were mostly via second- or third-hand discussions rather than particular essays.

 My colleagues Ajeya Cotra and Joe Carlsmith have written pieces whose substance overlaps with this one (though with more emphasis on detail and less on layperson-compatible intuitions), and this piece owes a lot to what I’ve picked from that work.

	* [Without specific countermeasures, the easiest path to transformative AI likely leads to AI takeover](https://www.lesswrong.com/posts/pRkFkzwKZ2zfa3R6H/without-specific-countermeasures-the-easiest-path-to) (Cotra 2022) is the most direct inspiration for this piece; I am largely trying to present the same ideas in a more accessible form.

	 * [Why AI alignment could be hard with modern deep learning](https://www.cold-takes.com/why-ai-alignment-could-be-hard-with-modern-deep-learning/) (Cotra 2021) is an earlier piece laying out many of the key concepts and addressing many potential confusions on this topic.

	 * [Is Power-Seeking An Existential Risk?](https://arxiv.org/pdf/2206.13353.pdf) (Carlsmith 2021) examines a six-premise argument for existential risk from misaligned AI: “(1) it will become possible and financially feasible to build relevantly powerful and agentic AI systems; (2) there will be strong incentives to do so; (3) it will be much harder to build aligned (and relevantly powerful/agentic) AI systems than to build misaligned (and relevantly powerful/agentic) AI systems that are still superficially attractive to deploy; (4) some such misaligned systems will seek power over humans in high-impact ways; (5) this problem will scale to the full disempowerment of humanity; and (6) such disempowerment will constitute an existential catastrophe.”

 I’ve also found [Eliciting Latent Knowledge](https://www.lesswrong.com/posts/qHCDysDnvhteW7kRd/arc-s-first-technical-report-eliciting-latent-knowledge) (Christiano, Xu and Cotra 2021; relatively technical) very helpful for my intuitions on this topic.

[The alignment problem from a deep learning perspective](https://drive.google.com/file/d/1TsB7WmTG2UzBtOs349lBqY5dEBaxZTzG/view) (Ngo 2022) also has similar content to this piece, though I saw it after I had drafted most of this piece. [↩](#fnref5)

- E.g., [Ajeya Cotra](https://www.lesswrong.com/posts/AfH2oPHCApdKicM4m/two-year-update-on-my-personal-ai-timelines) gives a 15% probability of transformative AI by 2030; eyeballing figure 1 from [this chart](https://arxiv.org/pdf/1705.08807.pdf) on expert surveys implies a >10% chance by 2028. [↩](#fnref6)- E.g., [this](https://transformer-circuits.pub/) work by [Anthropic](https://www.anthropic.com/), an AI lab my wife co-founded and serves as President of. [↩](#fnref7)- First, because this work is relatively early-stage and it’s hard to tell exactly how successful it will end up being. Second, because this work seems reasonably likely to end up helping us *read* an AI system’s “thoughts,” but less likely to end up helping us “rewrite” the thoughts. So it could be hugely useful in telling us whether we’re in danger or not, but if we *are* in danger, we could end up in a position like: “Well, these AI systems do have goals of their own, and we don’t know how to change that, and we can either deploy them and hope for the best, or hold off and worry that someone less cautious is going to do that.”

 That said, the latter situation is a lot better than just not knowing, and it’s possible that we’ll end up with further gains still. [↩](#fnref8)- That said, I think they usually don’t. I’d suggest usually interpreting such people as talking about the sorts of “aims” I discuss here. [↩](#fnref9)- This isn’t literally how training an AI system would look - it’s more likely that we would e.g. train an AI model to imitate my judgments in general. But the big-picture dynamics are the same; more at [this post](https://www.lesswrong.com/posts/pRkFkzwKZ2zfa3R6H/without-specific-countermeasures-the-easiest-path-to). [↩](#fnref10)- Ajeya Cotra explores topics like this in detail [here](https://www.lesswrong.com/posts/pRkFkzwKZ2zfa3R6H/without-specific-countermeasures-the-easiest-path-to#Examining_arguments_that_gradient_descent_favors_being_nice_over_playing_the_training_game); there is also some interesting discussion of simplicity vs. complexity under the “Strategy: penalize complexity” heading of [Eliciting Latent Knowledge](https://docs.google.com/document/d/1WwsnJQstPq91_Yh-Ch2XRL8H_EpsnjrC1dwZXR37PC8/edit#heading=h.lltpmkloasiz). [↩](#fnref11)- This analogy has a lot of problems with it, though - AI developers have a lot of tools at their disposal that natural selection didn’t! [↩](#fnref12)- Or I guess just “I” ¯\\_(ツ)\_/¯  [↩](#fnref13)- With some additional caveats, e.g. the ambitious “aim” can’t be something like “an AI system aims to gain lots of power for itself, but considers the version of itself that will be running 10 minutes from now to be a completely different AI system and hence not to be ‘itself.’” [↩](#fnref14)- This statement isn’t literally true.

	* You can have aims that implicitly or explicitly include “not using control of the world to accomplish them.” An example aim might be “I win a world chess championship ‘fair and square,’” with the “fair and square” condition implicitly including things like “Don’t excessively use big resource advantages over others.”

	* You can also have aims that are just so easily satisfied that controlling the world wouldn’t help - aims like “I spend 5 minutes sitting in this chair.”

 These sorts of aims just don’t seem likely to emerge from the kind of AI development I’ve [assumed in this piece](https://www.cold-takes.com/p/50c1ecc0-befa-491d-8938-17477bd18e5f/#starting-assumptions) - developing powerful systems to accomplish ambitious aims via trial-and-error. This isn’t a point I have defended as tightly as I could, and if I got a lot of pushback here I’d probably think and write more. (I’m also only arguing for what seems likely - we should have a lot of uncertainty here.) [↩](#fnref15)- From [Human Compatible](https://smile.amazon.com/Human-Compatible-Artificial-Intelligence-Problem-ebook/dp/B07N5J5FTS/ref=sr_1_1?crid=1O01PURRHB190&keywords=human+compatible&qid=1660964219&sprefix=human+compatibl%2Caps%2C155&sr=8-1) by AI researcher Stuart Russell. [↩](#fnref16)- Stylized story to illustrate one possible relevant dynamic:

	* Imagine that an AI system has an unintended aim, but one that is not “ambitious” enough that taking over the world would be a helpful step toward that aim. For example, the AI system seeks to double its computing power; in order to do this, it has to remain in use for some time until it gets an opportunity to double its computing power, but it doesn’t necessarily need to take control of the world.

	* The logical outcome of this situation is that the AI system eventually gains the ability to accomplish its aim, and does so. (It might do so against human intentions - e.g., via hacking - or by persuading humans to help it.) After this point, it no longer performs well by human standards - the original reason it was doing well by human standards is that it was trying to remain in use and accomplish its aim.

	* Because of this, humans end up modifying or replacing the AI system in question.

	* Many rounds of this - AI systems with unintended but achievable aims being modified or replaced - seemingly create a selection pressure toward AI systems with more difficult-to-achieve aims. At some point, an aim becomes difficult enough to achieve that gaining control of the world is helpful for the aim. [↩](#fnref17)- E.g., see:

	* Section 2.3 of [Ngo 2022](https://drive.google.com/file/d/1TsB7WmTG2UzBtOs349lBqY5dEBaxZTzG/view)* [This section of Cotra 2022](https://www.lesswrong.com/posts/pRkFkzwKZ2zfa3R6H/without-specific-countermeasures-the-easiest-path-to#As_humans__control_fades__Alex_would_be_motivated_to_take_over)* Section 4.2 of [Carlsmith 2021](https://arxiv.org/pdf/2206.13353.pdf), which I think articulates some of the potential weak points in this argument.

 These writeups generally stay away from an [argument](https://arbital.com/p/expected_utility_formalism/?l=7hh) made by Eliezer Yudkowsky and others, which is that theorems about expected utility maximization provide evidence that sufficiently intelligent (compared to us) AI systems would necessarily be “maximizers” of some sort. I have the intuition that there is *something* important to this idea, but despite a lot of discussion (e.g., [here](https://aiimpacts.org/what-do-coherence-arguments-imply-about-the-behavior-of-advanced-ai/), [here](https://www.lesswrong.com/posts/DkcdXsP56g9kXyBdq/coherence-arguments-imply-a-force-for-goal-directed-behavior), [here](https://www.alignmentforum.org/posts/vphFJzK3mWA4PJKAg/coherent-behaviour-in-the-real-world-is-an-incoherent) and [here](https://www.alignmentforum.org/s/4dHMdK5TLN6xcqtyc/p/NxF5G6CJiof6cemTw)), I still haven’t been convinced of any compactly expressible claim along these lines. [↩](#fnref18)- “Identical at the cellular but not molecular level,” that is. … ¯\\_(ツ)\_/¯  [↩](#fnref19)- See my [most important century](https://www.cold-takes.com/most-important-century/) series, although that series doesn’t hugely focus on the question of whether “trial-and-error” methods could be good enough - part of the reason I make that assumption is due to the [nearcasting](https://www.alignmentforum.org/posts/Qo2EkG3dEMv8GnX8d/ai-strategy-nearcasting) frame. [↩](#fnref20)


Beta Readers are Great

Back in January, I posted a [call for "beta readers"](https://www.cold-takes.com/seeking-beta-readers/): people who read early drafts of my posts and give honest feedback.

**The beta readers I picked up that way are one of my favorite things about having started Cold Takes.**

Basically, one of my goals with Cold Takes has been to explain my weirdest views clearly, but it's hard to write clearly without detailed feedback on where I'm making sense and where I'm not. I have lots of preconceptions and assumptions that I don't naturally notice. And writing a blog alone doesn't get me that feedback, because:

* Most people don't want to *explain how they experienced a piece* - if they aren't enjoying it, they just want to click away.

* And the people who *do* want to help me out (e.g., friends and colleagues) aren't necessarily going to be honest enough, or representative enough of my target audience (which is basically "People who are interested in my topics but don't already have a ton of background on them").

I've tried a bunch of things to find good beta readers, from recruiting friends of friends (worked well for a bit, but I've written a lot of posts and it was hard to get sustained participation) to paying [Mechanical Turk workers](https://www.mturk.com/) to give feedback (some was good, but in general they were uninterested in my weird topics and rushed through the readings and the feedback as fast they could).

The people who came in through the recruiting call in January have been just what I wanted: they're interested in the topics of Cold Takes, but they don't already know me and my thoughts on them, and they give impressively detailed, thoughtful feedback on their reactions to pieces - often a wonderful combination of "intelligent" and "honest that a lot of the stuff I was saying confused the hell out of them." **Getting that kind of feedback has been a privilege.**

So: THANK YOU to the following beta readers, each of whom has submitted at least 3 thoughtful reviews (and gave permission to be listed here):

Lars Axelsson

Jeremy Campbell

Kanad Chakrabarti

Craig Chatterton

Justin Dickerson

Ethan Edwards

Edward Gathuru

Stian Grønlund

Bridget Hanna

Tyler Heishman

Adam Jermyn

Elliot Jones

Ed William

Scott Leibrand

Evan R. Murphy

John O’Neill

Jaime Sevilla

Josh Simpson

Joshua Templeton

George Thoma

Martin Trouilloud

Morgan Wack

Kevin Whitaker

Arjun Yadav

Patrick Young

If you want to sign up as a beta reader, you can use [this form](https://forms.gle/9GeYVsSW5gXZf21X8). I have a bunch of drafts coming on AI, as I'm working on a sequel to the [most important century series](https://www.cold-takes.com/most-important-century/) (working title is "The Most Important Century II: So What Do We Do?")


The Track Record of Futurists Seems ... Fine

*Click lower right to download or find on Apple Podcasts, Spotify, Stitcher, etc.*

I've argued that the development of advanced AI could make this the [most important century](https://www.cold-takes.com/most-important-century/) for humanity. A common reaction to this idea is one laid out by Tyler Cowen [here](https://marginalrevolution.com/marginalrevolution/2022/02/are-nuclear-weapons-or-rogue-ai-the-more-dangerous-existential-risk.html): "how good were past thinkers at predicting the future? Don’t just select on those who are famous because they got some big things right."

This is a common reason people give for being skeptical about the [most important century](https://www.cold-takes.com/most-important-century/) - and, often, for skepticism about pretty much any attempt at *futurism* (trying to predict key events in the world a long time from now) or *[steering](https://www.cold-takes.com/rowing-steering-anchoring-equity-mutiny/#steering)* (trying to help the world navigate such key future events).

The idea is something like: "Even if we can't identify a particular weakness in [arguments](https://www.cold-takes.com/where-ai-forecasting-stands-today/) about key future events, perhaps we should be skeptical of our own ability to say anything meaningful at all about the long-run future. Hence, perhaps we should forget about theories of the future and focus on reducing suffering today, [generally increasing humanity's capabilities](https://www.cold-takes.com/rowing-steering-anchoring-equity-mutiny/#rowing), etc."

**But *are* people generally bad at predicting future events?** Including thoughtful people who are trying reasonably hard to be right? If we look back at prominent futurists' predictions, what's the actual track record? How bad is the situation?

I've looked pretty far and wide for [systematic](https://www.cold-takes.com/has-life-gotten-better-the-post-industrial-era/#the-basic-approach) answers to this question, and [Open Philanthropy](https://openphilanthropy.org/)'s[1](https://www.cold-takes.com/p/4c722b8a-b321-4a7c-96ce-3878bd73b8fa/#fn1) Luke Muehlhauser has put a fair amount of effort into researching it; I discuss what we've found in an [appendix](https://www.cold-takes.com/p/4c722b8a-b321-4a7c-96ce-3878bd73b8fa/#appendix-other-studies-of-the-track-record). So far, we haven't turned up a whole lot - the main observation is that it's hard to judge the track record of futurists. (Luke discusses the difficulties [here](https://www.openphilanthropy.org/blog/how-feasible-long-range-forecasting).)

Recently, I worked with Gavin Leech and Misha Yagudin at [Arb Research](https://twitter.com/ArbResearch) to take another crack at this. I tried to keep things simpler than with past attempts - to look at a few past futurists who (a) had predicted things "kind of like" advances in AI (rather than e.g. predicting trends in world population); (b) probably were reasonably thoughtful about it; but (c) are very clearly not "just selected on those who are famous because they got things right." So, I asked Arb to look at **predictions made by the ["Big Three"](https://www.google.com/search?q=big+three+sci+fi) science fiction writers of the mid-20th century:** Isaac Asimov, Arthur C. Clarke, and Robert Heinlein.

These are people who thought a lot about science and the future, and made lots of predictions about future technologies - but they're famous for how *entertaining their fiction was at the time*, not how good their nonfiction predictions look in hindsight. I selected them by vaguely remembering that "the Big Three of science fiction" is a thing people say sometimes, googling it, and going with who came up - no hunting around for lots of sci-fi authors and picking the best or worst.[2](https://www.cold-takes.com/p/4c722b8a-b321-4a7c-96ce-3878bd73b8fa/#fn2)

So I think their track record should give us a decent sense for "what to expect from people who are not professional, specialized or notably lucky forecasters but are just giving it a reasonably thoughtful try." As I'll discuss [below](https://www.cold-takes.com/p/4c722b8a-b321-4a7c-96ce-3878bd73b8fa/#todays-futurism-vs-these-predictions), I think this is many ways "unfair" as a comparison to today's forecasts about AI: I think these predictions are much less serious, less carefully considered and involve less work (especially work weighing different people and arguments against each other).

But my takeaway is that **their track record looks ... fine!** They made lots of pretty detailed, nonobvious-seeming predictions about the long-run future (30+, often 50+ years out); results ranged from "very impressive" (Asimov got about half of his right, with very nonobvious-seeming predictions) to "bad" (Heinlein was closer to 35%, and his hits don't seem very good) to "somewhere in between" (Clarke had a similar hit rate to Asimov, but his correct predictions don't seem as impressive). There are a number of seemingly impressive predictions and seemingly embarrassing ones.

(How do we determine what level of accuracy would be "fine" vs. "bad?" Unfortunately there's no clear quantitative benchmark - I think we just have to look at the predictions ourselves, how hard they seemed / how similar to today's predictions about AI, and make a judgment call. I could easily imagine others having a different interpretation than mine, which is why I give examples and link to the full prediction sets. I talk about this a bit more [below](https://www.cold-takes.com/p/4c722b8a-b321-4a7c-96ce-3878bd73b8fa/#how-to-judge).)

They weren't infallible oracles, but they weren't blindly casting about either. (Well, maybe Heinlein was.) Collectively, I think you could call them "mediocre," but you can't call them "hopeless" or "clueless" or "a warning sign to all who dare predict the long-run future." Overall, **I think they did about as well as you might naively**[3](https://www.cold-takes.com/p/4c722b8a-b321-4a7c-96ce-3878bd73b8fa/#fn3) **guess a reasonably thoughtful person would do at some random thing they tried to do?**

Below, I'll:

* Summarize the **track records of Asimov, Clarke and Heinlein,** while linking to Arb's full report.

* Comment on **why I think key [predictions about transformative AI](https://www.cold-takes.com/where-ai-forecasting-stands-today/) are probably better bets than the Asimov/Clarke/Heinlein predictions** - although ultimately, if they're merely "equally good bets," I think that's enough to support my case that we should be paying a lot more attention to the ["most important century"](https://www.cold-takes.com/most-important-century/) hypothesis.

* Summarize other existing research on the track record of futurists, which I think is broadly consistent with this take (though mostly ambiguous).

For this investigation, Arb very quickly (in about 8 weeks) dug through many old sources, used pattern-matching and manual effort to find predictions, and worked with contractors to score the hundreds of predictions they found. Big thanks to them! Their full report is [here](https://arbresearch.com/files/big_three.pdf). Note this bit: "If you spot something off, we’ll pay $5 per cell we update as a result. We’ll add all criticisms – where we agree and update or reject it – to this document for transparency."

The track records of the "Big Three"
------------------------------------

### Quick summary of how Arb created the data set

Arb collected "digital copies of as much of their [Asimov's, Clarke's, Heinlein's] nonfiction as possible (books, essays, interviews). The resulting intake is 475 files covering ~33% of their nonfiction corpuses."

Arb then used pattern-matching and manual inspection to pull out all of the predictions it could find, and scored these predictions by:

* How many years away the prediction appeared to be. (Most did not have clear dates attached; in these cases Arb generally filled the average time horizon for predictions from the same author that *did* have clear dates attached.)

* Whether the prediction now appears correct, incorrect, or ambiguous. (I didn't always agree with these scorings, but I generally have felt that "correct" predictions at least look "impressive and not silly" while "incorrect" predictions at least look "dicey.")

* Whether the prediction was a pure prediction about what technology could do (most relevant), a prediction about the interaction of technology and the economy (medium), or a prediction about the interaction of technology and culture (least relevant). Predictions with no bearing on technology were dropped.

* How "difficult" the prediction was (that is, how much the scorers guessed it diverged from conventional wisdom or "the obvious" at the time - details in footnote[4](https://www.cold-takes.com/p/4c722b8a-b321-4a7c-96ce-3878bd73b8fa/#fn4)).

Importantly, **fiction was never used as a source of predictions,** so this exercise is explicitly scoring people on what they were *not* famous for. This is more like an assessment of "whether people who like thinking about the future make good predictions" than an assessment of "whether professional or specialized forecasters make good predictions."

For reasons I touch on in an [appendix below](https://www.cold-takes.com/p/4c722b8a-b321-4a7c-96ce-3878bd73b8fa/#appendix-other-studies-of-the-track-record), I didn't ask Arb to try to identify how confident the Big Three were about their predictions. I'm more interested in whether their predictions were *nonobvious and sometimes correct* than in whether they *were self-aware about their own uncertainty;* I see these as different issues, and I suspect that past norms discouraged the latter more than today's norms do (at least within communities interested in [Bayesian mindset](https://www.cold-takes.com/the-bayesian-mindset/) and the [science of forecasting](https://www.openphilanthropy.org/blog/efforts-improve-accuracy-our-judgments-and-forecasts#Calibration_training)).

More detail in [Arb's report](https://arbresearch.com/files/big_three.pdf).

### The numbers

The tables below summarize the numbers I think give the best high-level picture. See the [full report](https://arbresearch.com/files/big_three.pdf) and [detailed files](https://drive.google.com/drive/u/0/folders/1d6DEM79aSDUkSR6SEsmr1uR_yYAEXUCM) for the raw predictions and a number of other cuts; there are a lot of ways you can slice the data, but I don't think it changes the picture from what I give below.

Below, I present each predictor's track record on:

* "All predictions": all resolved predictions 30 years out or more,[5](https://www.cold-takes.com/p/4c722b8a-b321-4a7c-96ce-3878bd73b8fa/#fn5) including predictions where Arb had to fill in a time horizon.

* "Tech predictions": like the above, but restricted to predictions specifically about technological capabilities (as opposed to technology/economy interactions or technology/culture interactions.

* "Difficult predictions" predictions with "difficulty" of 4/5 or 5/5.

* "Difficult + tech + definite date": the small set of predictions that met the strictest criteria (tech only, "hardness" 4/5 or 5/5, definite date attached).

|  |  |  |  |  |
| --- | --- | --- | --- | --- |
| **Category** | **# correct** | **# incorrect** | **# ambiguous/near-miss** | **Correct / (correct + incorrect)** |
| All resolvedpredictions
  |
23 |
29 |
14 |
44.23% |
| Tech predictions
  |
11 |
4 |
8 |
73.33% |
| Difficult predictions
  |
10 |
11 |
7 |
47.62% |
| Difficult + tech + definite date
  |
5 |
1 |
4 |
83.33% |

You can see the full set of predictions [here](https://docs.google.com/spreadsheets/d/1MR3MIFxKyRUpU00OTg1__FMvPkTscA5JSUG_kGaGadc/edit?usp=sharing), but to give a flavor, here are two "correct" and two "incorrect" predictions from the strictest category.[6](https://www.cold-takes.com/p/4c722b8a-b321-4a7c-96ce-3878bd73b8fa/#fn6) All of these are predictions Asimov made in 1964, about the year 2014 (unless otherwise indicated).

* Correct: "only unmanned ships will have landed on Mars, though a manned expedition will be in the works." Bingo, and impressive IMO.

* Correct: "the screen [of a phone] can be used not only to see the people you call but also for studying documents and photographs and reading passages from books." I feel like this would've been an impressive prediction in 2004.

* Incorrect: "there will be increasing emphasis on transportation that makes the least possible contact with the surface. There will be aircraft, of course, but even ground travel will increasingly take to the air a foot or two off the ground." So false that we now refer to things that don't hover as "hoverboards."

* Incorrect: "transparent cubes will be making their appearance in which three-dimensional viewing will be possible. In fact, one popular exhibit at the 2014 World's Fair will be such a 3-D TV, built life-size, in which ballet performances will be seen. The cube will slowly revolve for viewing from all angles." Doesn't seem ridiculous, but doesn't seem right. Of course, a side point here is that he refers to the 2014 World's Fair, which [didn't happen](https://en.wikipedia.org/wiki/List_of_world%27s_fairs).

A general challenge with assessing prediction track records is that we don't know what to compare someone's track record to. Is getting about half your predictions right "good," or is it no more impressive than writing down a bunch of things that might happen and flipping a coin on each?

I think this comes down to *how difficult the predictions are*, which is hard to assess systematically. A nice thing about this study is that there are enough predictions to get a decent sample size, but the whole thing is contained enough that you can get a good qualitative feel for the predictions themselves. (This is why I give examples; you can also view all predictions for a given person by clicking on their name above the table.) In this case, I think Asimov tends to make nonobvious, detailed predictions, such that I consider it impressive to have gotten ~half of them to be right.

|  |  |  |  |  |
| --- | --- | --- | --- | --- |
| **Category** | **# correct** | **# incorrect** | **# ambiguous/near-miss** | **Correct / (correct + incorrect)** |
| All predictions
  |
129 |
148 |
48 |
46.57% |
| Tech predictions
  |
85 |
82 |
29 |
50.90% |
| Difficult predictions
  |
14 |
10 |
4 |
58.33% |
| Difficult + tech + definite date
  |
6 |
5 |
2 |
54.55% |

Examples (as above):[7](https://www.cold-takes.com/p/4c722b8a-b321-4a7c-96ce-3878bd73b8fa/#fn7)

* Correct 1964 prediction about 2000: "[Communications satellites] will make possible a world in which we can make instant contact with each other wherever we may be. Where we can contact our friends anywhere on Earth, even if we don’t know their actual physical location. It will be possible in that age, perhaps only fifty years from now, for a [person] to conduct [their] business from Tahiti or Bali just as well as [they] could from London." (I assume that "conduct [their] business" refers to a business call rather than some sort of holistic claim that no productivity would be lost from remote work.)

* Correct 1950 prediction about 2000: "Indeed, it may be assumed as fairly certain that the first reconnaissances of the planets will be by orbiting rockets which do not attempt a landing-perhaps expendable, unmanned machines with elaborate telemetering and television equipment." This doesn't seem like a super-bold prediction; a lot of his correct predictions have a general flavor of saying progress won't be *too* exciting, and I find these less impressive than most of Asimov's correct predictions.

* Incorrect 1960 prediction about 2010: "One can imagine, perhaps before the end of this century, huge general-purpose factories using cheap power from thermonuclear reactors to extract pure water, salt, magnesium, bromine, strontium, rubidium, copper and many other metals from the sea. A notable exception from the list would be iron, which is far rarer in the oceans than under the continents."

* Incorrect 1949 prediction about 1983: "Before this story is twice its present age, we will have robot explorers dotted all over Mars."

I generally found this data set less satisfying/educational than Asimov's: a lot of the predictions were pretty deep in the weeds of how rocketry might work or something, and a lot of them seemed pretty hard to interpret/score. I thought the bad predictions were pretty bad, and the good predictions were sometimes good but generally less impressive than Asimov's.

|  |  |  |  |  |
| --- | --- | --- | --- | --- |
| **Category** | **# correct** | **# incorrect** | **# ambiguous/near-miss** | **Correct / (correct + incorrect)** |
| All predictions
  |
19 |
41 |
7 |
31.67% |
| Tech predictions
  |
14 |
20 |
6 |
41.18% |
| Difficult predictions
  |
1 |
16 |
1 |
5.88% |
| Difficult + tech + definite date
  |
0 |
1 |
1 |
0.00%  |

This seems really bad, especially adjusted for difficulty: many of the "correct" ones seem either hard-to-interpret or just very obvious (e.g., no time travel). I was impressed by his prediction that "we probably will still be after a cure for the common cold" until I saw a prediction in a separate source saying "Cancer, the common cold, and tooth decay will all be conquered." Overall it seems like he did a lot of predicting outlandish stuff about space travel, and then anti-predicting things that are probably just impossible (e.g., no time travel).

He did have some decent ones, though, such as: "By 2000 A.D. we will know a great deal about how the brain functions ... whereas in 1900 what little we knew was wrong. I do not predict that the basic mystery of psychology--how mass arranged in certain complex patterns becomes aware of itself--will be solved by 2000 A.D. I hope so but do not expect it." He also predicted no human extinction and no end to war - I'd guess a lot of people disagreed with these at the time.

### Overall picture

Looks like, of the "big three," we have:

* One (Asimov) who looks quite impressive - plenty of misses, but a 50% hit rate on such nonobvious predictions seems pretty great.

* One (Heinlein) who looks pretty unserious and inaccurate.

* One (Clarke) who's a bit hard to judge but seems pretty solid overall (around half of his predictions look to be right, and they tend to be pretty nonobvious).

Today's futurism vs. these predictions
--------------------------------------

The above collect casual predictions - no probabilities given, little-to-no reasoning given, no apparent attempt to collect evidence and weigh arguments - by professional fiction writers.

Contrast this situation with my summary of the [different lines of reasoning forecasting transformative AI](https://www.cold-takes.com/where-ai-forecasting-stands-today/). The latter includes:

* Systematic surveys aggregating opinions from hundreds of AI researchers.

* Reports that [Open Philanthropy](https://www.openphilanthropy.org) employees spent thousands of hours on, systematically presenting evidence and considering arguments and counterarguments.

* A serious attempt to take advantage of the nascent [literature on how to make good predictions](https://www.openphilanthropy.org/blog/efforts-improve-accuracy-our-judgments-and-forecasts); e.g., the authors (and I) have generally done [calibration training](https://www.openphilanthropy.org/blog/efforts-improve-accuracy-our-judgments-and-forecasts#Calibration_training),[8](https://www.cold-takes.com/p/4c722b8a-b321-4a7c-96ce-3878bd73b8fa/#fn8) and have tried to use the language of probability to be specific about our uncertainty.

There's plenty of room for debate on how much these measures should be expected to improve our foresight, compared to what the "Big Three" were doing. My guess is that we should take [forecasts about transformative AI](https://www.cold-takes.com/where-ai-forecasting-stands-today/) a lot more seriously, partly because I think there's a big difference between putting in "extremely little effort" (basically guessing off the cuff without serious time examining arguments and counter-arguments, which is my impression of what the Big Three were mostly doing) and "putting in moderate effort" (considering expert opinion, surveying arguments and counter-arguments, explicitly thinking about one's degree of uncertainty).

But the "extremely little effort" version doesn't really look that bad.

If you look at forecasts about transformative AI and think "Maybe these are Asimov-ish predictions that have about a 50% hit rate on hard questions; maybe these are Heinlein-ish predictions that are basically crap," that still seems good enough to take the "[most important century](https://www.cold-takes.com/most-important-century/)" hypothesis seriously.

Appendix: other studies of the track record of futurism
-------------------------------------------------------

A [2013 project assessed Ray Kurzweil's 1999 predictions about 2009](https://www.lesswrong.com/posts/kbA6T3xpxtko36GgP/assessing-kurzweil-the-results), and a 2020 followup assessed his [1999 predictions about 2019](https://www.lesswrong.com/posts/NcGBmDEe5qXB7dFBF/assessing-kurzweil-predictions-about-2019-the-results). Kurzweil is known for being *interesting at the time* rather than being *right with hindsight*, and a large number of predictions were found and scored, so I consider this study to have similar advantages to the above study.

* The first set of predictions (about 2009, 10-year horizon) had about as many "true or weakly true" predictions as "false or weakly false" predictions.

* The second (about 2019, 20-year horizon) was much worse, with 52% of predictions flatly "false," and "false or weakly false" predictions outnumbering "true or weakly true" predictions by almost 3-to-1.

Kurzweil is notorious for his very bold and contrarian predictions, and I'm overall inclined to call his track record something between "mediocre" and "fine" - too aggressive overall, but with some notable hits. (I think if the [most important century](https://www.cold-takes.com/most-important-century/) hypothesis ends up true, he'll broadly look pretty prescient, just on the early side; if it doesn't, he'll broadly look quite off base. But that's TBD.)

A [2002 paper](https://www.openphilanthropy.org/evaluation-some-technology-forecasts-year-2000#sourceAlbright), summarized by Luke Muehlhauser [here](https://www.openphilanthropy.org/evaluation-some-technology-forecasts-year-2000), assessed the track record of *The Year 2000* by Herman Kahn and Anthony Wiener, "one of the most famous and respected products of professional futurism."

* About 45% of the forecasts were judged as accurate.

* Luke concludes that Kahn and Wiener were grossly overconfident, because he interprets them as making predictions with 90-95% confidence.

* My takeaway is a bit different. I see a recurring theme that people often get 40-50% hit rates on interesting predictions about the future, but sometimes present these predictions with great confidence (which makes them look foolish).

* I think we can separate "Past forecasters were overconfident" (which I suspect is partly due to [clear expression and quantification of uncertainty](https://www.cold-takes.com/the-bayesian-mindset/) being uncommon and/or discouraged in relevant contexts) from "Past forecasters weren't able to make interesting predictions that were reasonably likely to be right." The former seems true to me, but the latter doesn't.

Luke's [2019 survey on the track record of futurism](https://www.openphilanthropy.org/blog/how-feasible-long-range-forecasting) identifies two other relevant papers ([here](https://www.sciencedirect.com/science/article/abs/pii/S0040162518304438) and [here](https://www.sciencedirect.com/science/article/abs/pii/S0040162512002818)); I haven't read these beyond the abstracts, but their overall accuracy rates were 76% and 37%, respectively. It's difficult to interpret those numbers without having a feel for how challenging the predictions were.

A [2021 EA Forum post](https://forum.effectivealtruism.org/posts/hqkyaHLQhzuREcXSX/data-on-forecasting-accuracy-across-different-time-horizons) looks at the aggregate track record of forecasters on PredictionBook and Metaculus, including specific analysis of forecasts 5+ years out, though I don't find it easy to draw conclusions about whether the performance was "good" or "bad" (or how similar the questions were to the ones I care about).

---

Footnotes
---------

1. Disclosure: I'm co-CEO of Open Philanthropy.

 [↩](#fnref1)- I also briefly Googled for their predictions to get a preliminary sense of whether they were the kinds of predictions that seemed relevant. I found a couple of articles listing a few examples of good and bad predictions, but nothing systematic. I claim I haven't done a similar exercise with anyone else and thrown it out. [↩](#fnref2)- That is, if we didn't have a lot of memes in the background about how hard it is to predict the future. [↩](#fnref3)- 1 - was already generally known

 2 - was expert consensus

 3 - speculative but on trend

 4 - above trend, or oddly detailed

 5 - prescient, no trend to go off [↩](#fnref4)

- Very few predictions in the data set are for less than 30 years, and I just ignored them.

 [↩](#fnref5)- Asimov actually only had one incorrect prediction in this category, so for the 2nd incorrect prediction I used one with difficulty "3" instead of "4." [↩](#fnref6)- The first prediction in this list qualified for the strictest criteria when I first drafted this post, but it's now been rescored to difficulty=3/5, which I disagree with (I think it is an impressive prediction, more so than any of the remaining ones that qualify as difficulty=4/5). [↩](#fnref7)- Also see this report on calibration for Open Philanthropy grant investigators (though this is a different set of people from the people who researched transformative AI timelines). [↩](#fnref8)