High-level hopes for AI alignment

*Click lower right to download or find on Apple Podcasts, Spotify, Stitcher, etc.*

In previous pieces, I argued that there's a real and large risk of AI systems' [aiming](https://www.cold-takes.com/why-would-ai-aim-to-defeat-humanity/) to defeat all of humanity combined - and [succeeding](https://www.cold-takes.com/ai-could-defeat-all-of-us-combined/).

I first argued that this sort of catastrophe would be likely without specific countermeasures to prevent it. I then argued that countermeasures could be challenging, due to some [key difficulties of AI safety research.](https://www.cold-takes.com/ai-safety-seems-hard-to-measure/)

But while I think misalignment risk is serious and presents major challenges, I don’t agree with sentiments[1](https://www.cold-takes.com/p/51b33fd6-2f1e-40bd-9d2c-2cfe2ebd5fc5#fn1) along the lines of “We haven’t figured out how to align an AI, so if transformative AI comes soon, we’re doomed.” Here I’m going to talk about some of my **high-level hopes for how we might end up avoiding this risk.**

I’ll first recap the challenge, using Ajeya Cotra’s [young businessperson](https://www.cold-takes.com/why-ai-alignment-could-be-hard-with-modern-deep-learning/#analogy-the-young-ceo) analogy to give a sense of some of the core difficulties. In a nutshell, once AI systems get capable enough, it could be hard to test whether they’re safe, because they might be able to deceive and manipulate us into getting the wrong read. Thus, trying to determine whether they’re safe might be something like “being an eight-year-old trying to decide between adult job candidates (some of whom are manipulative).”

I’ll then go through what I see as three key possibilities for navigating this situation:

* **Digital neuroscience**: perhaps we’ll be able to read (and/or even rewrite) the “digital brains” of AI systems, so that we can know (and change) what they’re “aiming” to do directly - rather than having to infer it from their behavior. (Perhaps the eight-year-old is a mind-reader, or even a young [Professor X](https://en.wikipedia.org/wiki/Professor_X#Powers_and_abilities).)

* **Limited AI**: perhaps we can make AI systems safe by making them *limited* in various ways - e.g., by leaving certain kinds of information out of their training, designing them to be “myopic” (focused on short-run as opposed to long-run goals), or something along those lines. Maybe we can make “limited AI” that is nonetheless able to carry out particular helpful tasks - such as doing lots more research on how to achieve safety without the limitations. (Perhaps the eight-year-old can limit the authority or knowledge of their hire, and still get the company run successfully.)

* **AI checks and balances**: perhaps we’ll be able to employ some AI systems to critique, supervise, and even rewrite others. Even if no single AI system would be safe on its own, the right “checks and balances” setup could ensure that human interests win out. (Perhaps the eight-year-old is able to get the job candidates to evaluate and critique each other, such that all the eight-year-old needs to do is verify basic factual claims to know who the best candidate is.)

These are some of the main categories of hopes that are pretty easy to picture today. Further work on AI safety research might result in further ideas (and the above are not exhaustive - see my [more detailed piece](https://www.alignmentforum.org/posts/rCJQAkPTEypGjSJ8X/how-might-we-align-transformative-ai-if-it-s-developed-very), posted to the Alignment Forum rather than Cold Takes, for more).

I’ll talk about both challenges and reasons for hope here. I think that for the most part, these hopes look much better if AI projects are moving cautiously rather than racing furiously.

I don’t think we’re at the point of having much sense of how the hopes and challenges net out; the best I can do at this point is to [say](https://www.alignmentforum.org/posts/rCJQAkPTEypGjSJ8X/how-might-we-align-transformative-ai-if-it-s-developed-very#So__would_civilization_survive_): “I don’t currently have much sympathy for someone who’s highly confident that AI takeover would or would not happen (that is, for anyone who thinks the odds of AI takeover … are under 10% or over 90%).”

The challenge
-------------

*This is all recapping previous pieces. If you remember them super well, skip to the [next section](https://www.cold-takes.com/p/51b33fd6-2f1e-40bd-9d2c-2cfe2ebd5fc5/#digital-neuroscience).*

In previous pieces, I argued that:

* The coming decades could see the development of AI systems that could automate - and dramatically speed up - scientific and technological advancement, getting us more quickly than most people imagine to a deeply unfamiliar future. (More: [The Most Important Century](https://www.cold-takes.com/most-important-century/))

* If we develop this sort of AI via ambitious use of the “black-box trial-and-error” common in AI development today, then there’s a substantial risk that:
	+ These AIs will develop **unintended aims** (states of the world they make calculations and plans toward, as a chess-playing AI "aims" for checkmate);

	+ These AIs will deceive, manipulate, and overpower humans as needed to achieve those aims;

	+ Eventually, this could reach the point where AIs [take over the world from humans entirely](https://www.cold-takes.com/ai-could-defeat-all-of-us-combined/).* People today are doing AI safety research to prevent this outcome, but such research has a [number of deep difficulties:](https://www.cold-takes.com/ai-safety-seems-hard-to-measure/)

|  |
| --- |
| **“Great news - I’ve tested this AI and it looks safe.”** Why might we still have a problem?
  |
| *Problem* | *Key question* | *Explanation* |
| The **Lance Armstrong problem** | Did we get the AI to be **actually safe** or **good at hiding its dangerous actions?** | When dealing with an intelligent agent, it’s hard to tell the difference between “behaving well” and “*appearing* to behave well.”

When professional cycling was cracking down on performance-enhancing drugs, Lance Armstrong was very successful and seemed to be unusually “clean.” It later came out that he had been using drugs with an unusually sophisticated operation for concealing them.
  |
| The **King Lear problem** | The AI is **(actually) well-behaved when humans are in control.** Will this transfer to **when AIs are in control?** | It's hard to know how someone will behave when they have power over you, based only on observing how they behave when they don't.

AIs might behave as intended as long as humans are in control - but at some future point, AI systems might be capable and widespread enough to have opportunities to [take control of the world entirely](https://www.cold-takes.com/ai-could-defeat-all-of-us-combined/). It's hard to know whether they'll take these opportunities, and we can't exactly run a clean test of the situation.

Like King Lear trying to decide how much power to give each of his daughters before abdicating the throne.
  |
| The **lab mice problem** | **Today's "subhuman" AIs are safe.**What about **future AIs with more human-like abilities?** | Today's AI systems aren't advanced enough to exhibit the basic behaviors we want to study, such as deceiving and manipulating humans.

Like trying to study medicine in humans by experimenting only on lab mice.
  |
| The **first contact problem** | Imagine that **tomorrow's "human-like" AIs are safe.** How will things go **when AIs have capabilities far beyond humans'?** | AI systems might (collectively) become vastly more capable than humans, and it's ... just really hard to have any idea what that's going to be like. As far as we know, there has never before been anything in the galaxy that's vastly more capable than humans in the relevant ways! No matter what we come up with to solve the first three problems, we can't be too confident that it'll keep working if AI advances (or just proliferates) a lot more.

Like trying to plan for first contact with extraterrestrials (this barely feels like an analogy).
  |

An analogy that incorporates these challenges is Ajeya Cotra’s “young businessperson” [analogy](https://www.cold-takes.com/why-ai-alignment-could-be-hard-with-modern-deep-learning/#analogy-the-young-ceo):

> Imagine you are an eight-year-old whose parents left you a $1 trillion company and no trusted adult to serve as your guide to the world. You must hire a smart adult to run your company as CEO, handle your life the way that a parent would (e.g. decide your school, where you’ll live, when you need to go to the dentist), and administer your vast wealth (e.g. decide where you’ll invest your money).
>
>
>
>
>
>  You have to hire these grownups based on a work trial or interview you come up with -- you don't get to see any resumes, don't get to do reference checks, etc. Because you're so rich, tons of people apply for all sorts of reasons. ([More](https://www.cold-takes.com/why-ai-alignment-could-be-hard-with-modern-deep-learning/#analogy-the-young-ceo))
>
>

If your applicants are a mix of "saints" (people who genuinely want to help), "sycophants" (people who just want to make you happy in the short run, even when this is to your long-term detriment) and "schemers" (people who want to siphon off your wealth and power for themselves), how do you - an eight-year-old - tell the difference?

(Click to expand) More detail on why AI could make this the most important century

In [The Most Important Century](https://www.cold-takes.com/most-important-century/), I argued that the 21st century could be the most important century ever for humanity, via the development of advanced AI systems that could dramatically speed up scientific and technological advancement, getting us more quickly than most people imagine to a deeply unfamiliar future.

[This page](https://www.cold-takes.com/most-important-century/) has a ~10-page summary of the series, as well as links to an audio version, podcasts, and the full series.

The key points I argue for in the series are:

* **The long-run future is radically unfamiliar.** Enough advances in technology could lead to a long-lasting, galaxy-wide civilization that could be a radical utopia, dystopia, or anything in between.
* **The long-run future could come much faster than we think,** due to a possible AI-driven productivity explosion.
* The relevant kind of **AI looks like it will be developed this century** - making this century the one that will initiate, and have the opportunity to shape, a future galaxy-wide civilization.
* These claims seem too "wild" to take seriously. But there are a lot of reasons to think that **we live in a wild time, and should be ready for anything.*** We, the people living in this century, have the chance to have a huge impact on huge numbers of people to come - if we can make sense of the situation enough to find helpful actions. But right now, **we aren't ready for this.**

(Click to expand) Why would AI "aim" to defeat humanity?

A [previous piece](https://www.cold-takes.com/why-would-ai-aim-to-defeat-humanity/) argued that if today’s AI development methods lead directly to powerful enough AI systems, disaster is likely by default (in the absence of specific countermeasures).

In brief:

* Modern AI development is essentially based on “training” via trial-and-error.

* If we move forward incautiously and ambitiously with such training, and if it gets us all the way to very powerful AI systems, then such systems will likely end up *aiming for certain states of the world* (analogously to how a chess-playing AI aims for checkmate).

* And these states will be *other than the ones we intended*, because our trial-and-error training methods won’t be accurate. For example, when we’re confused or misinformed about some question, we’ll reward AI systems for giving the wrong answer to it - unintentionally training deceptive behavior.

* We should expect disaster if we have AI systems that are both (a) [powerful enough](https://www.cold-takes.com/ai-could-defeat-all-of-us-combined/) to defeat humans and (b) aiming for states of the world that we didn’t intend. (“Defeat” means taking control of the world and doing what’s necessary to keep us out of the way; it’s unclear to me whether we’d be literally killed or just forcibly stopped[1](https://www.cold-takes.com/p/51b33fd6-2f1e-40bd-9d2c-2cfe2ebd5fc5#fn1) from changing the world in ways that contradict AI systems’ aims.)

(Click to expand) How could AI defeat humanity?

In a [previous piece](https://www.cold-takes.com/ai-could-defeat-all-of-us-combined/), I argue that AI systems could defeat all of humanity combined, if (for whatever reason) they were aimed toward that goal.

By defeating humanity, I mean gaining control of the world so that AIs, not humans, determine what happens in it; this could involve killing humans or simply “containing” us in some way, such that we can’t interfere with AIs’ aims.

One way this could happen is if AI became extremely advanced, to the point where it had "cognitive superpowers" beyond what humans can do. In this case, a single AI system (or set of systems working together) could imaginably:

* Do its own research on how to build a better AI system, which culminates in something that has incredible other abilities.

* Hack into human-built software across the world.

* Manipulate human psychology.

* Quickly generate vast wealth under the control of itself or any human allies.

* Come up with better plans than humans could imagine, and ensure that it doesn't try any takeover attempt that humans might be able to detect and stop.

* Develop advanced weaponry that can be built quickly and cheaply, yet is powerful enough to overpower human militaries.

However, my piece also explores what things might look like if *each AI system basically has similar capabilities to humans.* In this case:

- Humans are likely to deploy AI systems throughout the economy, such that they have large numbers and access to many resources - and the ability to make copies of themselves.

- From this starting point, AI systems with human-like (or greater) capabilities would have a number of possible ways of getting to the point where their total population could outnumber and/or out-resource humans.

- I address a number of possible objections, such as "How can AIs be dangerous without bodies?"

More: [AI could defeat all of us combined](https://www.cold-takes.com/ai-could-defeat-all-of-us-combined/)

Digital neuroscience
--------------------

I’ve previously argued that it could be inherently [difficult to measure whether AI systems are safe](https://www.cold-takes.com/ai-safety-seems-hard-to-measure/), for reasons such as: AI systems that are *not deceptive* probably look like AI systems that are *so good at deception that they hide all evidence of it*, in any way we can easily measure.

Unless we can “read their minds!”

Currently, today’s leading AI research is in the genre of “black-box trial-and-error.” An AI tries a task; it gets “encouragement” or “discouragement” based on whether it does the task well; it tweaks the wiring of its “digital brain” to improve next time; it improves at the task; but we humans aren’t able to make much sense of its “digital brain” or say much about its “thought process.”

(Click to expand) Why are AI systems "black boxes" that we can't understand the inner workings of?

What I mean by “black-box trial-and-error” is explained briefly in an [old Cold Takes post](https://www.cold-takes.com/transformative-ai-timelines-part-1-of-4-what-kind-of-ai/#making-pasta), and in more detail in more technical pieces by [Ajeya Cotra](https://www.lesswrong.com/posts/pRkFkzwKZ2zfa3R6H/without-specific-countermeasures-the-easiest-path-to#_HFDT_scales_far__assumption__Alex_is_trained_to_achieve_excellent_performance_on_a_wide_range_of_difficult_tasks) (section I linked to) and [Richard Ngo](https://arxiv.org/abs/2209.00626) (section 2). Here’s a quick, oversimplified characterization.

Today, the most common way of building an AI system is by using an "artificial neural network" (ANN), which you might think of sort of like a "digital brain" that starts in an empty (or random) state: it hasn't yet been wired to do specific things. A process something like this is followed:

* The AI system is given some sort of task.

* The AI system tries something, initially something pretty random.

* The AI system gets information about how well its choice performed, and/or what would’ve gotten a better result. Based on this, it “learns” by tweaking the wiring of the ANN (“digital brain”) - literally by strengthening or weakening the connections between some “artificial neurons” and others. The tweaks cause the ANN to form a stronger association between the choice it made and the result it got.

* After enough tries, the AI system becomes good at the task (it was initially terrible).

* But nobody really knows anything about *how or why* it’s good at the task now. The development work has gone into building a flexible architecture for it to learn well from trial-and-error, and into “training” it by doing all of the trial and error. We mostly can’t “look inside the AI system to see how it’s thinking.”

* For example, if we want to know why a chess-playing AI such as AlphaZero made some particular chess move, we can't look inside its code to find ideas like "Control the center of the board" or "Try not to lose my queen." Most of what we see is just a vast set of numbers, denoting the strengths of connections between different artificial neurons. As with a human brain, we can mostly only guess at what the different parts of the "digital brain" are doing.

Some AI research ([example](https://www.transformer-circuits.pub/2022/mech-interp-essay/index.html))[2](https://www.cold-takes.com/p/51b33fd6-2f1e-40bd-9d2c-2cfe2ebd5fc5#fn2) is exploring how to change this - how to decode an AI system’s “digital brain.” This research is in relatively early stages - today, it can “decode” only parts of AI systems (or fully decode very small, deliberately simplified AI systems).

As AI systems advance, it might get harder to decode them - or easier, if we can start to use AI for help decoding AI, and/or change AI design techniques so that AI systems are less “black box”-ish.

I think there is a wide range of possibilities here, e.g.:

**Failure:** “digital brains” keep getting bigger, more complex, and harder to make sense of, and so “digital neuroscience” generally stays about as hard to learn from as human neuroscience. In this world, we wouldn’t have anything like “lie detection” for AI systems engaged in [deceptive behavior](https://www.cold-takes.com/why-would-ai-aim-to-defeat-humanity/#deceiving-and-manipulating).

**Basic mind-reading:** we’re able to get a handle on things like “whether an AI system is behaving [deceptively](https://www.cold-takes.com/why-would-ai-aim-to-defeat-humanity/#deceiving-and-manipulating), e.g. whether it has internal representations of ‘beliefs’ about the world that contradict its statements” and “whether an AI system is [aiming](https://www.cold-takes.com/why-would-ai-aim-to-defeat-humanity/#what-it-means-for) to accomplish some [strange goal we didn’t intend it to](https://www.cold-takes.com/why-would-ai-aim-to-defeat-humanity/).”

* It may be hard to fix things like this by just continuing trial-and-error-based training (perhaps because we worry that AI systems are manipulating their own “digital brains” - see later bullet point).

* But we’d at least be able to get early warnings of potential problems, or early evidence that we *don’t* have a problem, and adjust our level of caution appropriately. This sort of mind-reading could also be helpful with [AI checks and balances](https://www.cold-takes.com/p/51b33fd6-2f1e-40bd-9d2c-2cfe2ebd5fc5/#ai-checks-and-balances) (below).

**Advanced mind-reading:** we’re able to understand an AI system’s “thought process” in detail (what observations and patterns are the main reasons it’s behaving as it is), understand how any worrying aspects of this “thought process” (such as [unintended aims](https://www.cold-takes.com/why-would-ai-aim-to-defeat-humanity/)) came about, and make lots of small adjustments until we can verify that an AI system is free of [unintended aims](https://www.cold-takes.com/why-would-ai-aim-to-defeat-humanity/) or [deception.](https://www.cold-takes.com/why-would-ai-aim-to-defeat-humanity/#deceiving-and-manipulating)

**Mind-*writing* (digital neurosurgery):** we’re able to alter a “digital brain” directly, rather than just via the “trial-and-error” process discussed [earlier.](https://www.cold-takes.com/p/51b33fd6-2f1e-40bd-9d2c-2cfe2ebd5fc5#Box4)

One potential failure mode for digital neuroscience is if AI systems end up able to *manipulate their own “digital brains.*” This could lead “digital neuroscience” to have the same problem as [other AI safety research](https://www.cold-takes.com/ai-safety-seems-hard-to-measure/): if we’re shutting down or negatively reinforcing AI systems that appear to have unsafe “aims” based on our “mind-reading,” we might end up selecting for AI systems whose “digital brains” only *appear* safe.

* This could be a real issue, especially if AI systems end up with far-beyond-human capabilities (more below).

* But naively, an AI system manipulating its own “digital brain” to appear safe seems quite a bit harder than simply *behaving* deceptively.

I should note that I’m lumping in much of the (hard-to-explain) research on the [Eliciting Latent Knowledge](https://www.lesswrong.com/posts/qHCDysDnvhteW7kRd/arc-s-first-technical-report-eliciting-latent-knowledge) (ELK) agenda under this category.[3](https://www.cold-takes.com/p/51b33fd6-2f1e-40bd-9d2c-2cfe2ebd5fc5#fn3) The ELK agenda is largely[4](https://www.cold-takes.com/p/51b33fd6-2f1e-40bd-9d2c-2cfe2ebd5fc5#fn4) about thinking through what kinds of “digital brain” patterns might be associated with honesty vs. deception, and trying to find some impossible-to-fake sign of honesty.

**How likely is this to work?** I think it’s very up-in-the-air right now. I’d say “digital neuroscience” is a young field, tackling a problem that may or may not prove tractable. If we have several decades before transformative AI, then I’d expect to at least succeed at “basic mind-reading,” whereas if we have less than a decade, I think that’s around 50/50. I think it’s less likely that we’ll succeed at some of the more ambitious goals, but definitely possible.

Limited AI
----------

I [previously](https://www.cold-takes.com/why-would-ai-aim-to-defeat-humanity/) discussed why AI systems could end up with “aims,” in the sense that they make calculations, choices and plans selected to reach a particular sort of state of the world. For example, chess-playing AIs “aim” for checkmate game states; a recommendation algorithm might “aim” for high customer engagement or satisfaction. I then argued that AI systems would do “whatever it takes” to get what they’re “aiming” at, even when this means deceiving and disempowering humans.

But AI systems won’t necessarily have the sorts of “aims” that risk trouble. Consider two different tasks you might “train” an AI to do, via trial-and-error (rewarding success at the task):

* “Write whatever code a particular human would write, if they were in your situation.”

* “Write whatever code accomplishes goal X [including coming up with things much better than a human could].”

The second of these seems like a recipe for having the sort of ambitious “aim” I’ve [claimed is dangerous](https://www.cold-takes.com/why-would-ai-aim-to-defeat-humanity/) - it’s an open-ended invitation to do *whatever* leads to good performance on the goal. By contrast, the first is about imitating a particular human. It leaves a lot less scope for creative, unpredictable behavior and for having [“ambitious” goals that lead to conflict with humans.](https://www.cold-takes.com/why-would-ai-aim-to-defeat-humanity/)

(For more on this distinction, see my [discussion of process-based optimization](https://www.alignmentforum.org/posts/rCJQAkPTEypGjSJ8X/how-might-we-align-transformative-ai-if-it-s-developed-very#Limited_AI_systems), although I’m not thrilled with this and hope to write something better later.)

My guess is that in a competitive world, people will be able to get more done, faster, with something like the second approach. But:

* Maybe the first approach will work better *at first*, and/or AI developers will deliberately stick with the first approach as much as they can for safety reasons.

* And maybe that will be enough to build AI systems that can, themselves, [do huge amounts of AI alignment research](https://openai.com/blog/our-approach-to-alignment-research/) applicable to future, less limited systems. Or enough to build AI systems that can do other useful things, such as creating convincing demonstrations of the risks, patching security holes that dangerous AI systems would otherwise exploit, and more. (More on “how safe AIs can protect against dangerous AIs” in a future piece.)

* A risk that would remain: these AI systems might also be able to do huge amounts of research on *making AIs bigger and more capable*. So simply having “AI systems that can do alignment research” isn’t good enough by itself - we would need to then hope that the leading AI developers prioritize safety research rather than racing ahead with building more powerful systems, up until the point where they can make the more powerful systems safe.

There are a number of other ways in which we might “limit” AI systems to make them safe. One can imagine AI systems that are:

* “Short-sighted” or “[myopic](https://www.alignmentforum.org/posts/LCLBnmwdxkkz5fNvH/open-problems-with-myopia)”: they might have “aims” ([see previous post on what I mean by this term](https://www.cold-takes.com/why-would-ai-aim-to-defeat-humanity/#what-it-means-for)) that only apply to their short-run future. So an AI system might be aiming to gain more power, but only over the next few hours; such an AI system wouldn’t exhibit some of the behaviors I worry about, such as [deceptively behaving in “safe” seeming ways in hopes of getting more power later](https://www.cold-takes.com/why-would-ai-aim-to-defeat-humanity/#deceiving-and-manipulating).

* “Narrow”: they might have only a particular set of capabilities, so that e.g. they can help with AI alignment research but don’t understand human psychology and can’t [deceive and manipulate humans](https://www.cold-takes.com/why-would-ai-aim-to-defeat-humanity/#deceiving-and-manipulating).

* “Unambitious”: even if AI systems develop unintended aims, these might be aims they satisfy fairly easily, causing some strange behavior but not aiming to [defeat all of humanity](https://www.cold-takes.com/ai-could-defeat-all-of-us-combined/).

A further source of hope: even if such “limited” systems aren’t very powerful on their own, we might be able to [amplify](https://www.alignmentforum.org/s/EmDuGeRw749sD3GKd/p/HqLxuZ4LhaFhmAHWk#Core_concept__Analogy_to_AlphaGoZero) them by setting up combinations of AIs that work together on difficult tasks. For example:

* One “slow but deep” AI might do lots of analysis on every action it takes - for example, when it writes a line of code, it might consider hundreds of possibilities for that single line.

* Another “fast and shallow” AI might be trained to quickly, efficiently imitate the sorts of actions the “slow but deep” one takes - writing the sorts of lines of code it produces after considering hundreds of possibilities.

* Further AIs might be trained to summarize the analysis of other AIs, assign different parts of tasks to different AIs, etc. The result could be something like a “team” of AIs with different roles, such that a large number of limited AIs ends up quite a lot more powerful (and, depending on the details, also more dangerous) than any of the individual AIs.

I’d guess that in a competitive world, AI systems that are *not* “limited” will - at least eventually - be more powerful, versatile and ultimately useful. But limited AIs might get us pretty far.

**How likely is this to work?** I’d guess that we’ll eventually be able to build very powerful AIs whose limits make them relatively safe. However, I’d also guess that AIs without such limits will eventually be more powerful. So I think a lot of how things go will come down to *how cautious we are*: will we stick with *limited* AIs until the point at which we make *more powerful* AIs safe? And I think it’s very hard to predict how much caution the world will have - it partly depends on how well-understood the issues discussed in this series become over time!

AI checks and balances
----------------------

![](https://www.cold-takes.com/content/images/2022/12/image1.jpg)

Central to my [worries](https://www.cold-takes.com/why-would-ai-aim-to-defeat-humanity/) is the idea that AIs could be good at “deceiving” humans: proficiently choosing courses of action that humans don’t fully understand, and don’t catch the problems with. This is important both for [how AIs could develop unintended, “dangerous” aims in the first place](https://www.cold-takes.com/why-would-ai-aim-to-defeat-humanity/) and [how they could execute on these aims by defeating humanity](https://www.cold-takes.com/ai-could-defeat-all-of-us-combined/).

We could potentially mitigate this issue by **using AIs to supervise and critique each other.**

A simple example: say we’re worried that AI systems might find computer security vulnerabilities, and use them opportunistically to gain power and resources. We could train some AI systems to *specifically seek out, expose and patch* computer security vulnerabilities. (A footnote explains why we might expect such specialized systems to patch most vulnerabilities they find, rather than exploiting the vulnerabilities as often as less specialized systems would.[5](https://www.cold-takes.com/p/51b33fd6-2f1e-40bd-9d2c-2cfe2ebd5fc5#fn5))

* This is a basic example of “using some AIs to identify potential problems other ones could cause.”

* A nice property: there might be some vulnerabilities that are much easier for AI systems to notice than humans; these are probably the ones we should be most worried about, *and* the ones most likely to be found by “vulnerability-finding AIs.”

 Analogously, we could [train](https://www.cold-takes.com/p/51b33fd6-2f1e-40bd-9d2c-2cfe2ebd5fc5#Box4) AI systems to do things like:

* Produce critiques of each others’ proposed plans, and explanations of how these plans could lead to unintended behavior.

* Help out with the “digital neuroscience” goal [above](https://www.cold-takes.com/p/51b33fd6-2f1e-40bd-9d2c-2cfe2ebd5fc5/#digital-neuroscience): analyze each others’ “digital brains” and produce compelling-to-humans discussions of whether there’s evidence there about whether the AI systems have [unintended, dangerous “aims.”](https://www.cold-takes.com/why-would-ai-aim-to-defeat-humanity/)* Conduct randomized audits of each others’ behavior: looking for any way in which another AI system’s behavior might be deceptive or otherwise unintended.
	+ Importantly, auditor AIs would have different “incentives” (they get positive reinforcement for different behaviors), compared to the AIs they’re auditing: the audited AI system might be searching for any answer to a question that humans rate highly, whereas the “auditor” might be *specifically* searching for compelling arguments that an otherwise-highly-rated-answer is actually misleading.

	+ I previously [argued](https://www.cold-takes.com/why-would-ai-aim-to-defeat-humanity/#Audits) that these sorts of audits have limitations - they could end up unwittingly training the audited AI systems to become effective at deceiving human auditors. But in this case, deceiving auditors might be more like deceiving *copies of themselves,* potentially a harder-to-learn behavior.

There are a lot of potential wrinkles here, which I discuss in a more detailed [non-Cold-Takes piece](https://www.alignmentforum.org/posts/rCJQAkPTEypGjSJ8X/how-might-we-align-transformative-ai-if-it-s-developed-very). In brief:

* AI checks and balances might fail if different AI systems end up *coordinating with each other against us.* For example, it might end up being the case that even auditor AI systems are best able to accomplish their [ambitious, unintended aims](https://www.cold-takes.com/why-would-ai-aim-to-defeat-humanity/) by refraining from warning humans about the most important problems they find with other AIs.

* We might try to make this less likely by creating **ensembles of AI systems that “check and balance” each other, while being different enough from each other to make coordination difficult.**
	+ For example, some AI systems might have limitations ([see above](https://www.cold-takes.com/p/51b33fd6-2f1e-40bd-9d2c-2cfe2ebd5fc5/#limited-ai)) that make them less dangerous, while also less capable.

	+ In order to pull off deception successfully, more powerful systems would then have to deceive not just humans but a whole variety of other (less capable, but in different ways) AI systems - some of which could be [amplified](https://www.cold-takes.com/p/51b33fd6-2f1e-40bd-9d2c-2cfe2ebd5fc5#Amplification), as discussed above.* Another way AI checks and balances could fail: when one AI system critiques another, the resulting “debate” could quickly become too hard for humans to referee effectively. For example, imagine that an [eight-year-old](https://www.cold-takes.com/ai-safety-seems-hard-to-measure/#The-young-businessperson) has asked adult A to look for signs that adult B is deceiving them, and adult A is now arguing that this is happening while adult B is arguing that it isn’t. Can the eight-year-old figure out what the truth is? Seems genuinely uncertain (and dependent on the details)!

There is already some research on “using AIs to critique each other.” A recent example is [this paper](https://openai.com/blog/critiques/), which actually does show that an AI trained to critique its own answers can surface helpful critiques that help humans rate its answers more accurately.

Other possibilities
-------------------

I discuss possible hopes in more detail in an [Alignment Forum](https://www.alignmentforum.org/posts/rCJQAkPTEypGjSJ8X/how-might-we-align-transformative-ai-if-it-s-developed-very) piece. And I think there is significant scope for “unknown unknowns”: researchers working on AI safety might come up with approaches that nobody has thought of yet.

High-level fear: things get too weird, too fast
-----------------------------------------------

Rather than end on a positive note, I want to talk about a general dynamic that feels like it could make the situation *very* difficult, and make it hard for any of the above hopes to work out.

To quote from my [previous piece](https://www.cold-takes.com/ai-safety-seems-hard-to-measure/):

Maybe at some point, AI systems will be able to do things like:

* Coordinate with each other incredibly well, such that it's hopeless to use one AI to help supervise another.

* Perfectly understand human thinking and behavior, and know exactly what words to say to make us do what they want - so just letting an AI send emails or write Tumblr posts gives it vast power over the world.

* Manipulate their own "digital brains," so that our [attempts to "read their minds"](https://www.cold-takes.com/p/51b33fd6-2f1e-40bd-9d2c-2cfe2ebd5fc5/#digital-neuroscience) backfire and mislead us.

* Reason about the world (that is, [make plans to accomplish their aims](https://www.cold-takes.com/why-would-ai-aim-to-defeat-humanity/#what-it-means-for)) in completely different ways from humans, with concepts like "glooble"[6](https://www.cold-takes.com/p/51b33fd6-2f1e-40bd-9d2c-2cfe2ebd5fc5#fn6) that are incredibly useful ways of thinking about the world but that humans couldn't understand with centuries of effort.

At this point, whatever methods we've developed for making human-like AI systems safe, honest and restricted could fail - and silently, as such AI systems could go from "being honest and helpful" to "appearing honest and helpful, while setting up opportunities to [defeat humanity](https://www.cold-takes.com/ai-could-defeat-all-of-us-combined/)."

I’m not wedded to any of the details above, but I think the general dynamic in which “AI systems get extremely powerful, strange, and hard to deal with very quickly” could happen for a few different reasons:

* The nature of AI development might just be such that we very quickly go from having very weak AI systems to having “superintelligent” ones. How likely this is has been debated a lot.[7](https://www.cold-takes.com/p/51b33fd6-2f1e-40bd-9d2c-2cfe2ebd5fc5#fn7)* Even if AI improves relatively slowly, we might *initially* have a lot of success with things like “AI checks and balances,” but continually make more and more capable AI systems - such that they eventually become extraordinarily capable and very “alien” to us, at which point previously-effective methods break down. ([More](https://www.alignmentforum.org/posts/rCJQAkPTEypGjSJ8X/how-might-we-align-transformative-ai-if-it-s-developed-very#DecisiveDoesntExist))

* The most likely reason this would happen, in my view, is that **we - humanity - choose to move too fast.** It’s easy to envision a world in which everyone is in a furious race to develop more powerful AI systems than everyone else - focused on “competition” rather than “caution” (more on the distinction [here](https://www.cold-takes.com/making-the-best-of-the-most-important-century/)) - and everything [accelerates dramatically](https://www.cold-takes.com/transformative-ai-timelines-part-1-of-4-what-kind-of-ai/#explosive-scientific-and-technological-advancement) once we’re able to use AI systems to automate scientific and technological advancement.

So … is AI going to defeat humanity or is everything going to be fine?
----------------------------------------------------------------------

I don’t know! There are a number of ways we might be fine, and a number of ways we might not be. I could easily see this century ending in [humans defeated](https://www.cold-takes.com/ai-could-defeat-all-of-us-combined/) or in a glorious [utopia](https://www.cold-takes.com/visualizing-utopia/). You could maybe even think of it as the most important century.

So far, I’ve mostly just talked about the technical challenges of AI alignment: why AI systems might end up misaligned, and how we might design them to avoid that outcome. In future pieces, I’ll go into a bit more depth on some of the political and strategic challenges (e.g., what AI companies and governments might do to reduce the risk of a furious race to deploy dangerous AI systems), and work my way toward the question: “What can we do today to improve the odds that things go well?”

---

Footnotes
---------

1. [E.g.](https://www.lesswrong.com/posts/uMQ3cqWDPHhjtiesc/agi-ruin-a-list-of-lethalities) [↩](#fnref1)- Disclosure: my wife Daniela is President and co-founder of Anthropic, which employs prominent researchers in “mechanistic interpretability” and hosts the site I link to for the term. [↩](#fnref2)- Disclosure: I’m on the board of [ARC](https://alignment.org/), which wrote this document. [↩](#fnref3)- Though not entirely [↩](#fnref4)- The basic idea:

	* A lot of security vulnerabilities might be the kind of thing where it’s clear that there’s some weakness in the system, but it’s not immediately clear how to exploit this for gain. An AI system with an [unintended “aim”](https://www.cold-takes.com/why-would-ai-aim-to-defeat-humanity/) might therefore “save” knowledge about the vulnerability until it encounters enough *other* vulnerabilities, and the right circumstances, to accomplish its aim.

	 * But now imagine an AI system that is trained and rewarded *exclusively* for finding and patching such vulnerabilities. Unlike with the first system, revealing the vulnerability gets more positive reinforcement than just about *anything else it can do* (and an AI that reveals no such vulnerabilities will perform extremely poorly). It thus might be much more likely than the previous system to do so, rather than simply leaving the vulnerability in place in case it’s useful later.

	 * And now imagine that there are *multiple* AI systems trained and rewarded for finding and patching such vulnerabilities, with each one needing to find some vulnerability overlooked by others in order to achieve even moderate performance. These systems might also have enough variation that it’s hard for one such system to confidently predict what another will do, which could further lower the gains to leaving the vulnerability in place.
	 [↩](#fnref5)- This is a concept that only I understand.  [↩](#fnref6)- See [here](https://www.alignmentforum.org/s/v55BhXbpJuaExkpcD/p/GNhMPAWcfBCASy8e6), [here](https://www.alignmentforum.org/s/n945eovrA3oDueqtq/p/hwxj4gieR7FWNwYfa), and [here](https://www.alignmentforum.org/s/n945eovrA3oDueqtq/p/vwLxd6hhFvPbvKmBH). Also see the tail end of [this Wait but Why piece](https://waitbutwhy.com/2015/01/artificial-intelligence-revolution-1.html), which draws on similar intuitions to the longer treatment in [Superintelligence](https://smile.amazon.com/Superintelligence-Dangers-Strategies-Nick-Bostrom-ebook/dp/B00LOOCGB2/) [↩](#fnref7)


AI Safety Seems Hard to Measure

*Click lower right to download or find on Apple Podcasts, Spotify, Stitcher, etc.*

In previous pieces, I argued that there's a real and large risk of AI systems' [developing dangerous goals of their own](https://www.cold-takes.com/why-would-ai-aim-to-defeat-humanity/) and [defeating all of humanity](https://www.cold-takes.com/ai-could-defeat-all-of-us-combined/) - at least in the absence of specific efforts to prevent this from happening.

A young, growing field of **AI safety research** tries to reduce this risk, by finding ways to ensure that AI systems behave as intended (rather than forming ambitious aims of their own and deceiving and manipulating humans as needed to accomplish them).

Maybe we'll succeed in reducing the risk, and maybe we won't. **Unfortunately, I think it could be hard to know either way**. This piece is about four fairly distinct-seeming reasons that this could be the case - and that AI safety could be an unusually difficult sort of science.

This piece is aimed at a broad audience, because I think it's **important for the challenges here to be broadly understood.** I expect powerful, dangerous AI systems to have a lot of benefits (commercial, military, etc.), and to potentially *appear* safer than they are - so I think it will be hard to be as cautious about AI as we should be. I think our odds look better if many people understand, at a high level, some of the challenges in knowing whether AI systems are as safe as they appear.

First, I'll recap the basic challenge of AI safety research, and outline what I *wish* AI safety research could be like. I wish it had this basic form: "Apply a test to the AI system. If the test goes badly, try another AI development method and test that. If the test goes well, we're probably in good shape." I think car safety research mostly looks like this; I think AI *capabilities* research mostly looks like this.

Then, I’ll give four reasons that **apparent success in AI safety can be misleading.**

|  |
| --- |
| **“Great news - I’ve tested this AI and it looks safe.”** Why might we still have a problem?
  |
| *Problem* | *Key question* | *Explanation* |
| The **Lance Armstrong problem** | Did we get the AI to be **actually safe** or **good at hiding its dangerous actions?** | When dealing with an intelligent agent, it’s hard to tell the difference between “behaving well” and “*appearing* to behave well.”

When professional cycling was cracking down on performance-enhancing drugs, Lance Armstrong was very successful and seemed to be unusually “clean.” It later came out that he had been using drugs with an unusually sophisticated operation for concealing them.
  |
| The **King Lear problem** | The AI is **(actually) well-behaved when humans are in control.** Will this transfer to **when AIs are in control?** | It's hard to know how someone will behave when they have power over you, based only on observing how they behave when they don't.

AIs might behave as intended as long as humans are in control - but at some future point, AI systems might be capable and widespread enough to have opportunities to [take control of the world entirely](https://www.cold-takes.com/ai-could-defeat-all-of-us-combined/). It's hard to know whether they'll take these opportunities, and we can't exactly run a clean test of the situation.

Like King Lear trying to decide how much power to give each of his daughters before abdicating the throne.
  |
| The **lab mice problem** | **Today's "subhuman" AIs are safe.**What about **future AIs with more human-like abilities?** | Today's AI systems aren't advanced enough to exhibit the basic behaviors we want to study, such as deceiving and manipulating humans.

Like trying to study medicine in humans by experimenting only on lab mice.
  |
| The **first contact problem** | Imagine that **tomorrow's "human-like" AIs are safe.** How will things go **when AIs have capabilities far beyond humans'?** | AI systems might (collectively) become vastly more capable than humans, and it's ... just really hard to have any idea what that's going to be like. As far as we know, there has never before been anything in the galaxy that's vastly more capable than humans in the relevant ways! No matter what we come up with to solve the first three problems, we can't be too confident that it'll keep working if AI advances (or just proliferates) a lot more.

Like trying to plan for first contact with extraterrestrials (this barely feels like an analogy).
  |

I'll close with Ajeya Cotra's "[young businessperson](https://www.cold-takes.com/why-ai-alignment-could-be-hard-with-modern-deep-learning/#analogy-the-young-ceo)" analogy, which in some sense ties these concerns together. A future piece will discuss some reasons for hope, despite these problems.

Recap of the basic challenge
----------------------------

A [previous piece](https://www.cold-takes.com/why-would-ai-aim-to-defeat-humanity/) laid out the basic case for concern about AI misalignment. In brief: if extremely capable AI systems are developed using methods like the ones AI developers use today, it seems like there's a substantial risk that:

* These AIs will develop **unintended aims** (states of the world they make calculations and plans toward, as a chess-playing AI "aims" for checkmate);

* These AIs will deceive, manipulate, and overpower humans as needed to achieve those aims;

* Eventually, this could reach the point where AIs [take over the world from humans entirely](https://www.cold-takes.com/ai-could-defeat-all-of-us-combined/).

I see **AI safety research** as trying to **design AI systems that won't [aim](https://www.cold-takes.com/why-would-ai-aim-to-defeat-humanity/#what-it-means-for) to deceive, manipulate or defeat humans - even if and when these AI systems are extraordinarily capable** (and would be very effective at deception/manipulation/defeat if they were to aim at it).That is: AI safety research is trying to reduce the risk of the above scenario, *even if* (as I've assumed) humans rush forward with training powerful AIs to do ever-more ambitious things.

(Click to expand) More detail on why AI could make this the most important century

In [The Most Important Century](https://www.cold-takes.com/most-important-century/), I argued that the 21st century could be the most important century ever for humanity, via the development of advanced AI systems that could dramatically speed up scientific and technological advancement, getting us more quickly than most people imagine to a deeply unfamiliar future.

[This page](https://www.cold-takes.com/most-important-century/) has a ~10-page summary of the series, as well as links to an audio version, podcasts, and the full series.

The key points I argue for in the series are:

* **The long-run future is radically unfamiliar.** Enough advances in technology could lead to a long-lasting, galaxy-wide civilization that could be a radical utopia, dystopia, or anything in between.

* **The long-run future could come much faster than we think,** due to a possible AI-driven productivity explosion.

* The relevant kind of **AI looks like it will be developed this century** - making this century the one that will initiate, and have the opportunity to shape, a future galaxy-wide civilization.

* These claims seem too "wild" to take seriously. But there are a lot of reasons to think that **we live in a wild time, and should be ready for anything.*** We, the people living in this century, have the chance to have a huge impact on huge numbers of people to come - if we can make sense of the situation enough to find helpful actions. But right now, **we aren't ready for this.**

(Click to expand) Why would AI "aim" to defeat humanity?

A [previous piece](https://www.cold-takes.com/why-would-ai-aim-to-defeat-humanity/) argued that if today’s AI development methods lead directly to powerful enough AI systems, disaster is likely by default (in the absence of specific countermeasures).

In brief:

* Modern AI development is essentially based on “training” via trial-and-error.

* If we move forward incautiously and ambitiously with such training, and if it gets us all the way to very powerful AI systems, then such systems will likely end up *aiming for certain states of the world* (analogously to how a chess-playing AI aims for checkmate)*.** And these states will be *other than the ones we intended*, because our trial-and-error training methods won’t be accurate. For example, when we’re confused or misinformed about some question, we’ll reward AI systems for giving the wrong answer to it - unintentionally training deceptive behavior.

* We should expect disaster if we have AI systems that are both (a) [powerful enough](https://www.cold-takes.com/ai-could-defeat-all-of-us-combined/) to defeat humans and (b) aiming for states of the world that we didn’t intend. (“Defeat” means taking control of the world and doing what’s necessary to keep us out of the way; it’s unclear to me whether we’d be literally killed or just forcibly stopped[1](https://www.cold-takes.com/p/4d63edc6-4be6-4c77-ae5b-c70e730acb58#fn1) from changing the world in ways that contradict AI systems’ aims.)

(Click to expand) *How* could AI defeat humanity?

In a [previous piece](https://www.cold-takes.com/ai-could-defeat-all-of-us-combined/), I argue that AI systems could defeat all of humanity combined, if (for whatever reason) they were aimed toward that goal.

By defeating humanity, I mean gaining control of the world so that AIs, not humans, determine what happens in it; this could involve killing humans or simply “containing” us in some way, such that we can’t interfere with AIs’ aims.

One way this could happen is if AI became extremely advanced, to the point where it had "cognitive superpowers" beyond what humans can do. In this case, a single AI system (or set of systems working together) could imaginably:

* Do its own research on how to build a better AI system, which culminates in something that has incredible other abilities.

* Hack into human-built software across the world.

* Manipulate human psychology.

* Quickly generate vast wealth under the control of itself or any human allies.

* Come up with better plans than humans could imagine, and ensure that it doesn't try any takeover attempt that humans might be able to detect and stop.

* Develop advanced weaponry that can be built quickly and cheaply, yet is powerful enough to overpower human militaries.

However, my piece also explores what things might look like if *each AI system basically has similar capabilities to humans.* In this case:

* Humans are likely to deploy AI systems throughout the economy, such that they have large numbers and access to many resources - and the ability to make copies of themselves.

* From this starting point, AI systems with human-like (or greater) capabilities would have a number of possible ways of getting to the point where their total population could outnumber and/or out-resource humans.

* I address a number of possible objections, such as "How can AIs be dangerous without bodies?"

More: [AI could defeat all of us combined](https://www.cold-takes.com/ai-could-defeat-all-of-us-combined/)

I wish AI safety research were straightforward
----------------------------------------------

I wish AI safety research were like car safety research.[2](https://www.cold-takes.com/p/4d63edc6-4be6-4c77-ae5b-c70e730acb58#fn2)

While I'm sure this is an oversimplification, I think a lot of car safety research looks basically like this:

* Companies carry out test crashes with test cars. The results give a pretty good (not perfect) indication of what would happen in a real crash.

* Drivers try driving the cars in low-stakes areas without a lot of traffic. Things like steering wheel malfunctions will probably show up here; if they don't and drivers are able to drive normally in low-stakes areas, it's probably safe to drive the car in traffic.

* None of this is perfect, but the occasional problem isn't, so to speak, [the end of the world](https://www.cold-takes.com/ai-could-defeat-all-of-us-combined/). The worst case tends to be a handful of accidents, followed by a recall and some changes to the car's design validated by further testing.

Overall, **if we have problems with car safety, we'll probably be able to observe them relatively straightforwardly under relatively low-stakes circumstances.**

In important respects, many types of research and development have this basic property: we can observe how things are going during testing to get good evidence about how they'll go in the real world. Further examples include medical research,[3](https://www.cold-takes.com/p/4d63edc6-4be6-4c77-ae5b-c70e730acb58#fn3) chemistry research,[4](https://www.cold-takes.com/p/4d63edc6-4be6-4c77-ae5b-c70e730acb58#fn4) software development,[5](https://www.cold-takes.com/p/4d63edc6-4be6-4c77-ae5b-c70e730acb58#fn5) etc.

**Most AI research looks like this as well.** People can test out what an AI system is capable of reliably doing (e.g., translating speech to text), before integrating it into some high-stakes commercial product like Siri. This works both for ensuring that the AI system is *capable* (e.g., that it does a good job with its tasks) and that it's *safe in certain ways* (for example, if we're worried about toxic language, testing for this is relatively straightforward).

The rest of this piece will be about some of the ways in which "testing" for AI safety **fails to give us straightforward observations about whether, once AI systems are deployed in the real world, the world will actually be safe.**

While all research has to deal with *some* differences between testing and the real world, I think the challenges I'll be going through are unusual ones.

Four problems
-------------

### (1) The Lance Armstrong problem: is the AI *actually safe* or *good at hiding its dangerous actions*?

First, let's imagine that:

* We have AI systems available that can do roughly everything a human can, with some different strengths and weaknesses but no huge difference in "overall capabilities" or economic value per hour of work.

* We're observing early signs that AI systems behave in unintended, deceptive ways, such as giving wrong answers to questions we ask, or writing software that falsifies metrics instead of doing the things the metrics were supposed to measure (e.g., software meant to make a website run faster might instead falsify metrics about its loading time).

We theorize that modifying the AI training in some way[6](https://www.cold-takes.com/p/4d63edc6-4be6-4c77-ae5b-c70e730acb58#fn6) will make AI systems less likely to behave deceptively. We try it out, and find that, in fact, our AI systems seem to be behaving better than before - we are finding fewer incidents in which they behaved in unintended or deceptive ways.

But that's just a statement about *what we're noticing*. Which of the following just happened:

* Did we just train our AI systems to be less deceptive?

* Did we just train our AI systems to be *better at* deception, and so to make us *think* they became less deceptive?

* Did we just train our AI systems to be better at calculating when they might get caught in deception, and so to be less deceptive only when the deception would otherwise be caught?
	+ This one could be useful! Especially if we're able to set up auditing systems in many real-world situations, such that we *could* expect deception to be caught a lot of the time. But it does leave open the [King Lear problem](https://www.cold-takes.com/p/4d63edc6-4be6-4c77-ae5b-c70e730acb58#The-King-Lear-problem) covered next.

(...Or some combination of the three?)

We're hoping to be able to deploy AI systems throughout the economy, so - just like human specialists - they will almost certainly have some opportunities to be deceptive without being caught. The fact that they *appear honest in our testing* is not clear comfort against this risk.

The analogy here is to competitive cyclist [Lance Armstrong](https://en.wikipedia.org/wiki/Lance_Armstrong). Armstrong won the Tour de France race 7 times in a row, while many of his competitors were caught using performance-enhancing drugs and disqualified. But more than 5 years after his last win, an investigation "concluded that Armstrong had used performance-enhancing drugs over the course of his career[5] and named him as the ringleader of 'the most sophisticated, professionalized and successful doping program that sport has ever seen'." Now the list of Tour de France winners looks like this:

![alt_text](https://www.cold-takes.com/content/images/size/w1600/2022/12/image1.png "image_tooltip")

A broader issue here is that **when AI systems become capable enough, AI safety research starts to look more like social sciences (studying human beings) than like natural sciences.** Social sciences are generally less rigorous and harder to get clean results from, and one factor in this is that it can be hard to study someone who's aware they're being studied.[7](https://www.cold-takes.com/p/4d63edc6-4be6-4c77-ae5b-c70e730acb58#fn7)

Two broad categories of research that might help with the Lance Armstrong problem:

* [Mechanistic interpretability](https://www.transformer-circuits.pub/2022/mech-interp-essay/index.html)[8](https://www.cold-takes.com/p/4d63edc6-4be6-4c77-ae5b-c70e730acb58#fn8) can be thought of analyzing the "digital brains" of AI systems (not just analyzing their behavior and performance.) Currently, AI systems are [black boxes](https://www.cold-takes.com/p/4d63edc6-4be6-4c77-ae5b-c70e730acb58#Box4) in the sense that they perform well on tasks, but we can't say much about *how* they are doing it; mechanistic interpretability aims to change this, which could give us the ability to "mind-read" AIs and detect deception. (There could still be a risk that AI systems are arranging their own "digital brains" in misleading ways, but this seems quite a bit harder than simply *behaving* deceptively.)
* Some researchers work on "scalable supervision" or "competitive supervision." The idea is that if we are training an AI system that might become deceptive, we set up some supervision process for it that we expect to reliably catch any attempts at deception. This could be because the supervision process itself uses AI systems with more resources than the one being supervised, or because it uses a system of randomized audits where extra effort is put into catching deception.

(Click to expand) Why are AI systems "black boxes" that we can't understand the inner workings of?

I explain this briefly in an [old Cold Takes post](https://www.cold-takes.com/transformative-ai-timelines-part-1-of-4-what-kind-of-ai/#making-pasta); it's explained in more detail in more technical pieces by [Ajeya Cotra](https://www.lesswrong.com/posts/pRkFkzwKZ2zfa3R6H/without-specific-countermeasures-the-easiest-path-to#_HFDT_scales_far__assumption__Alex_is_trained_to_achieve_excellent_performance_on_a_wide_range_of_difficult_tasks) (section I linked to) and [Richard Ngo](https://drive.google.com/file/d/1TsB7WmTG2UzBtOs349lBqY5dEBaxZTzG/view) (section 2).

What I mean by “black-box trial-and-error” is explained briefly in an [old Cold Takes post](https://www.cold-takes.com/transformative-ai-timelines-part-1-of-4-what-kind-of-ai/#making-pasta), and in more detail in more technical pieces by [Ajeya Cotra](https://www.lesswrong.com/posts/pRkFkzwKZ2zfa3R6H/without-specific-countermeasures-the-easiest-path-to#_HFDT_scales_far__assumption__Alex_is_trained_to_achieve_excellent_performance_on_a_wide_range_of_difficult_tasks) (section I linked to) and [Richard Ngo](https://drive.google.com/file/d/1TsB7WmTG2UzBtOs349lBqY5dEBaxZTzG/view) (section 2). Here’s a quick, oversimplified characterization.

Today, the most common way of building an AI system is by using an "artificial neural network" (ANN), which you might think of sort of like a "digital brain" that starts in an empty (or random) state: it hasn't yet been wired to do specific things. A process something like this is followed:

* The AI system is given some sort of task.

* The AI system tries something, initially something pretty random.

* The AI system gets information about how well its choice performed, and/or what would’ve gotten a better result. Based on this, it “learns” by tweaking the wiring of the ANN (“digital brain”) - literally by strengthening or weakening the connections between some “artificial neurons” and others. The tweaks cause the ANN to form a stronger association between the choice it made and the result it got.

* After enough tries, the AI system becomes good at the task (it was initially terrible).

* But nobody really knows anything about *how or why* it’s good at the task now. The development work has gone into building a flexible architecture for it to learn well from trial-and-error, and into “training” it by doing all of the trial and error. We mostly can’t “look inside the AI system to see how it’s thinking.”

* For example, if we want to know why a chess-playing AI such as AlphaZero made some particular chess move, we can't look inside its code to find ideas like "Control the center of the board" or "Try not to lose my queen." Most of what we see is just a vast set of numbers, denoting the strengths of connections between different artificial neurons. As with a human brain, we can mostly only guess at what the different parts of the "digital brain" are doing.

### (2) The King Lear problem: how do you test what will happen when it's no longer a test?

The Shakespeare play [King Lear](https://en.wikipedia.org/wiki/King_Lear) opens with the King (Lear) stepping down from the throne, and immediately learning that he has left his kingdom to the wrong two daughters. Loving and obsequious while he was deciding on their fate,[9](https://www.cold-takes.com/p/4d63edc6-4be6-4c77-ae5b-c70e730acb58#fn9) they reveal their contempt for him as soon as he's out of power and they're in it.

If we're building AI systems that can reason like humans, dynamics like this become a potential issue.

I [previously](https://www.cold-takes.com/why-would-ai-aim-to-defeat-humanity/#existential-risks-to-humanity) noted that an AI with *any* ambitious [aim](https://www.cold-takes.com/why-would-ai-aim-to-defeat-humanity/#what-it-means-for) - or just an AI that wants to avoid being shut down or modified - might calculate that the best way to do this is by behaving helpfully and safely in all "tests" humans can devise. But once there is a real-world opportunity to disempower humans for good, that same aim [could cause the AI to disempower humans.](https://www.cold-takes.com/why-would-ai-aim-to-defeat-humanity/#existential-risks-to-humanity)

In other words:

* (A) When we're developing and testing AI systems, we have the power to decide which systems will be modified or shut down and which will be deployed into the real world. (Like King Lear deciding who will inherit his kingdom.)

* (B) But at some later point, these systems could be operating in the economy, in high numbers with a lot of autonomy. (This possibility is spelled out/visualized a bit more [here](https://www.cold-takes.com/ai-could-defeat-all-of-us-combined/#how-this-could-work-if-humans-create-a-huge-population-of-ais) and [here](https://www.lesswrong.com/posts/pRkFkzwKZ2zfa3R6H/without-specific-countermeasures-the-easiest-path-to#As_humans__control_fades__Alex_would_be_motivated_to_take_over).) At that point, they may have opportunities to [defeat all of humanity](https://www.cold-takes.com/ai-could-defeat-all-of-us-combined/) such that we never make decisions about them again. (Like King Lear's daughters after they've taken control.)

(Click to expand) How could AI defeat humanity?

In a [previous piece](https://www.cold-takes.com/ai-could-defeat-all-of-us-combined/), I argue that AI systems could defeat all of humanity combined, if (for whatever reason) they were aimed toward that goal.

By defeating humanity, I mean gaining control of the world so that AIs, not humans, determine what happens in it; this could involve killing humans or simply “containing” us in some way, such that we can’t interfere with AIs’ aims.

One way this could happen is if AI became extremely advanced, to the point where it had "cognitive superpowers" beyond what humans can do. In this case, a single AI system (or set of systems working together) could imaginably:

* Do its own research on how to build a better AI system, which culminates in something that has incredible other abilities.

* Hack into human-built software across the world.

* Manipulate human psychology.

* Quickly generate vast wealth under the control of itself or any human allies.

* Come up with better plans than humans could imagine, and ensure that it doesn't try any takeover attempt that humans might be able to detect and stop.

* Develop advanced weaponry that can be built quickly and cheaply, yet is powerful enough to overpower human militaries.

However, my piece also explores what things might look like if *each AI system basically has similar capabilities to humans.* In this case:

* Humans are likely to deploy AI systems throughout the economy, such that they have large numbers and access to many resources - and the ability to make copies of themselves.

* From this starting point, AI systems with human-like (or greater) capabilities would have a number of possible ways of getting to the point where their total population could outnumber and/or out-resource humans.

* I address a number of possible objections, such as "How can AIs be dangerous without bodies?"

More: [AI could defeat all of us combined](https://www.cold-takes.com/ai-could-defeat-all-of-us-combined/)

**If AI systems can detect the difference between (A) and (B) above, then their behavior might systematically change from one to the other - and there's no clear way to *test* their behavior in (B).**

(Click to expand) The Volkswagen emissions scandal

In 2015, the US Environmental Protection Agency "found that Volkswagen had intentionally programmed ... engines to activate their emissions controls only during laboratory emissions testing, which caused the vehicles' [nitrogen oxide] output to meet US standards during regulatory testing. However, the vehicles emitted up to 40 times more [nitrogen oxide] in real-world driving." (From [Wikipedia](https://en.wikipedia.org/wiki/Volkswagen_emissions_scandal))

This is an example of the "King Lear problem": there was a deliberate distinction made where the engines were behaving differently during tests and during real-world use. I don't know much about this incident, but I'm guessing that regulators were able to detect the issue because of (a) [independent tests](https://en.wikipedia.org/wiki/Volkswagen_emissions_scandal#European_discrepancies,_2014) under road conditions; (b) being able to understand the "[defeat device](https://en.wikipedia.org/wiki/Volkswagen_emissions_scandal#EPA_Notice_of_Violation,_2015)" that the Volkswagen cars used to create the misleading readings.

Neither of these would necessarily apply to the King Lear problem as described here: (a) you can test a car under road conditions to see its emissions, but you can't (or at least, shouldn't) test how an AI behaves when it has an opportunity to disempower humans entirely; (b) AI systems may be [black boxes](https://www.cold-takes.com/p/4d63edc6-4be6-4c77-ae5b-c70e730acb58#Box4) such that it's hard to understand what's going on inside them.

In general, modern machine learning researchers consider it challenging to handle what's called "distributional shift": systematic differences between situations AIs were [trained](https://www.cold-takes.com/transformative-ai-timelines-part-1-of-4-what-kind-of-ai/#making-pasta) on and situations they're now in. To me, the King Lear problem looks like **arguably the most inconvenient possible distributional shift:** AI systems risk behaving in unexpected ways *just as* (and in fact, because) they're now able to defeat humanity, rather than being in a controlled test environment.

Some lines of research that might help here:

* If we could solve the [Lance Armstrong problem](https://www.cold-takes.com/p/4d63edc6-4be6-4c77-ae5b-c70e730acb58#The-Lance-Armstrong-Problem) robustly enough - such that we could be confident AIs were never behaving deceptively - we could simply prompt AIs to answer questions like "Would AI system X disempower humans given an opportunity to do so?" The [digital-brain-based](https://www.cold-takes.com/p/4d63edc6-4be6-4c77-ae5b-c70e730acb58#DigitalNeuroscience) techniques noted above could imaginably get us here.

* There might be ways of specifically trying to target the *worst-case* behavior of AI systems, so that they are nearly guaranteed not to behave in certain ways *regardless of their situation*. This could look something roughly like "simulating cases where an AI system has an opportunity to disempower humans, and giving it negative reinforcement for choosing to do so." More on this sort of approach, along with some preliminary ongoing work, [here](https://www.lesswrong.com/posts/pXLqpguHJzxSjDdx7/why-i-m-excited-about-redwood-research-s-current-project).

### (3) The lab mice problem: the AI systems we'd like to study don't exist today

Above, I said: "when AI systems become capable enough, AI safety research starts to look more like social sciences (studying human beings) than like natural sciences." But today, AI systems *aren't* capable enough, which makes it especially hard to have a meaningful test bed and make meaningful progress.

Specifically, we don't have much in the way of AI systems that seem to *deceive and manipulate* their supervisors,[10](https://www.cold-takes.com/p/4d63edc6-4be6-4c77-ae5b-c70e730acb58#fn10) the way I worry that  [they might when they become capable enough](https://www.cold-takes.com/why-would-ai-aim-to-defeat-humanity/).

In fact, it's not 100% clear that AI systems could learn to deceive and manipulate supervisors even if we deliberately tried to train them to do it. This makes it hard to even get started on things like discouraging and detecting deceptive behavior.

I think AI safety research is a bit unusual in this respect: most fields of research aren't explicitly about "solving problems that don't exist yet." (Though a lot of research *ends up* useful for more important problems than the original ones it's studying.) As a result, doing AI safety research today is a bit like **trying to study medicine in humans by experimenting only on lab mice** (no human subjects available).

This does *not* mean there's no productive AI safety research to be done! (See the previous sections.) It just means that the research being done today is somewhat analogous to research on lab mice: informative and important up to a point, but only up to a point.

How bad is this problem? I mean, I do think it's a temporary one: by the time we're facing the [problems I worry about](https://www.cold-takes.com/why-would-ai-aim-to-defeat-humanity/), we'll be able to study them more directly. The concern is that [things could be moving very quickly by that point](https://www.cold-takes.com/transformative-ai-timelines-part-1-of-4-what-kind-of-ai/#explosive-scientific-and-technological-advancement): by the time we have AIs with human-ish capabilities, companies might be furiously making copies of those AIs and using them for all kinds of things (including both AI safety research and further research on making AI systems faster, cheaper and more capable).

So I do worry about the lab mice problem. And I'd be excited to see more effort on making "better model organisms": AI systems that show early versions of the properties we'd most like to study, such as deceiving their supervisors. (I even think it would be worth training AIs specifically to do this;[11](https://www.cold-takes.com/p/4d63edc6-4be6-4c77-ae5b-c70e730acb58#fn11) if such behaviors are going to emerge eventually, I think it's best for them to emerge early while there's relatively little risk of AIs' actually [defeating humanity](https://www.cold-takes.com/ai-could-defeat-all-of-us-combined/).)

### (4) The "first contact" problem: how do we prepare for a world where AIs have capabilities vastly beyond those of humans?

All of this piece so far has been about trying to make safe "human-like" AI systems.

What about AI systems with capabilities *far* beyond humans - what Nick Bostrom calls [superintelligent](https://smile.amazon.com/Superintelligence-Dangers-Strategies-Nick-Bostrom-ebook/dp/B00LOOCGB2/) AI systems?

Maybe at some point, AI systems will be able to do things like:

* Coordinate with each other incredibly well, such that it's hopeless to use one AI to help supervise another.

* Perfectly understand human thinking and behavior, and know exactly what words to say to make us do what they want - so just letting an AI send emails or write tweets gives it vast power over the world.

* Manipulate their own "digital brains," so that our  [attempts to "read their minds"](https://www.cold-takes.com/p/4d63edc6-4be6-4c77-ae5b-c70e730acb58#DigitalNeuroscience) backfire and mislead us.

* Reason about the world (that is,  [make plans to accomplish their aims](https://www.cold-takes.com/why-would-ai-aim-to-defeat-humanity/#what-it-means-for)) in completely different ways from humans, with concepts like "glooble"[12](https://www.cold-takes.com/p/4d63edc6-4be6-4c77-ae5b-c70e730acb58#fn12) that are incredibly useful ways of thinking about the world but that humans couldn't understand with centuries of effort.

At this point, whatever methods we've developed for making human-like AI systems safe, honest, and restricted could fail - and silently, as such AI systems could go from "behaving in honest and helpful ways" to "appearing honest and helpful, while setting up opportunities to [defeat humanity](https://www.cold-takes.com/ai-could-defeat-all-of-us-combined/)."

Some people think this sort of concern about "superintelligent" systems is ridiculous; some[13](https://www.cold-takes.com/p/4d63edc6-4be6-4c77-ae5b-c70e730acb58#fn13) seem to consider it extremely likely. I'm not personally sympathetic to having high confidence either way.

But additionally, a world with huge numbers of human-like AI systems could be strange and foreign and fast-moving enough to have a lot of this quality.

Trying to prepare for futures like these could be like trying to **prepare for first contact with extaterrestrials** - it's hard to have any idea what kinds of challenges we might be dealing with, and the challenges might arise quickly enough that we have little time to learn and adapt.

The young businessperson
------------------------

For one more analogy, I'll return to the one used by Ajeya Cotra [here](https://www.cold-takes.com/why-ai-alignment-could-be-hard-with-modern-deep-learning/#analogy-the-young-ceo):

> Imagine you are an eight-year-old whose parents left you a $1 trillion company and no trusted adult to serve as your guide to the world. You must hire a smart adult to run your company as CEO, handle your life the way that a parent would (e.g. decide your school, where you’ll live, when you need to go to the dentist), and administer your vast wealth (e.g. decide where you’ll invest your money).
>
>
>
>
>
>  You have to hire these grownups based on a work trial or interview you come up with -- you don't get to see any resumes, don't get to do reference checks, etc. Because you're so rich, tons of people apply for all sorts of reasons. ([More](https://www.cold-takes.com/why-ai-alignment-could-be-hard-with-modern-deep-learning/#analogy-the-young-ceo))
>
>

If your applicants are a mix of "saints" (people who genuinely want to help), "sycophants" (people who just want to make you happy in the short run, even when this is to your long-term detriment) and "schemers" (people who want to siphon off your wealth and power for themselves), how do you - an eight-year-old - tell the difference?

This analogy combines most of the worries above.

* The young businessperson has trouble knowing whether candidates are truthful in interviews, and trouble knowing whether any work trial *actually* went well or just *seemed* to go well due to deliberate deception. (The [Lance Armstrong problem](https://www.cold-takes.com/p/4d63edc6-4be6-4c77-ae5b-c70e730acb58#The-Lance-Armstrong-Problem).)

* Job candidates could have bad intentions that don't show up until they're in power (the [King Lear Problem)](https://www.cold-takes.com/p/4d63edc6-4be6-4c77-ae5b-c70e730acb58#The-King-Lear-problem).

* If the young businessperson were trying to prepare for this situation before actually being in charge of the company, they could have a lot of trouble simulating it (the [lab mice problem)](https://www.cold-takes.com/p/4d63edc6-4be6-4c77-ae5b-c70e730acb58#The-Lab-mice-problem).
* And it's generally just hard for an eight-year-old to have much grasp *at all* on the world of adults - to even think about all the things they should be thinking about (the [first contact problem](https://www.cold-takes.com/p/4d63edc6-4be6-4c77-ae5b-c70e730acb58#The-first-contact-problem)).

Seems like a tough situation.

 [Previously](https://www.cold-takes.com/why-would-ai-aim-to-defeat-humanity/), I talked about the dangers of AI *if* AI developers don't take specific countermeasures. This piece has tried to give a sense of why, even if they *are* trying to take countermeasures, doing so could be hard. The next piece will talk about some ways we might succeed anyway.

---

Footnotes
---------

1. Or persuaded (in a “mind hacking” sense) or whatever. [↩](#fnref1)- Research? Testing. Whatever. [↩](#fnref2)- Drugs can be tested in vitro, then in animals, then in humans. At each stage, we can make relatively straightforward observations about whether the drugs are working, and these are reasonably predictive of how they'll do at the next stage. [↩](#fnref3)- You can generally see how different compounds interact in a controlled environment, before rolling out any sort of large-scale processes or products, and the former will tell you most of what you need to know about the latter. [↩](#fnref4)- New software can be tested by a small number of users before being rolled out to a large number, and the initial tests will probably find most (not all) of the bugs and hiccups. [↩](#fnref5)- Such as:

	* Being more careful to avoid [wrong answers that can incentivize deception](https://www.cold-takes.com/why-would-ai-aim-to-defeat-humanity/#deceiving-and-manipulating)* Conducting randomized "audits" where we try extra hard to figure out the right answer to a question, and give an AI extra negative reinforcement if it gives an answer that we *would have* believed if not for the audit (this is "extra negative reinforcement for wrong answers that superficially look right")

	* Using methods along the lines of ["AI safety via debate"](https://openai.com/blog/debate/) [↩](#fnref6)- Though there are other reasons social sciences are especially hard, such as the fact that there are often big limits to what kinds of experiments are ethical, and the fact that it's often [hard to make clean comparisons between differing populations](https://www.cold-takes.com/how-digital-people-could-change-the-world/#social-science). [↩](#fnref7)- This paper is from Anthropic, a company that my wife serves as President of. [↩](#fnref8)- Like, he actually asks them to talk about their love for him just before he decides on what share of the realm they'll get. Smh [↩](#fnref9)- [This paper](https://arxiv.org/pdf/2109.07958.pdf) is a potential example, but its results [seem pretty brittle](https://www.cold-takes.com/ai-alignment-research-links/#helpful-honest-harmless). [↩](#fnref10)- E.g., I think it would be interesting to train AI [coding systems](https://github.com/features/copilot) to write [underhanded C](http://www.underhanded-c.org/): code that looks benign to a human inspector, but does unexpected things when run. They could be given negative reinforcement when humans can correctly identify that the code will do unintended things, and positive reinforcement when the code achieves the particular things that humans are attempting to stop. This would be challenging with today's AI systems, but not necessarily impossible. [↩](#fnref11)- This is a concept that only I understand. [↩](#fnref12)- E.g., see the discussion of the "hard left turn" [here](https://www.alignmentforum.org/posts/3pinFH3jerMzAvmza/on-how-various-plans-miss-the-hard-bits-of-the-alignment) by Nate Soares, head of [MIRI](https://intelligence.org/). My impression is that others at MIRI, including Eliezer Yudkowsky, have a similar picture. [↩](#fnref13)