Ngo’s view on alignment difficulty

This post features a write-up by Richard Ngo on his views, with inline comments.

Color key:

|  |  |  |
| --- | --- | --- |
|   Chat   |   Google Doc content   |   Inline comments   |

13. Follow-ups to the Ngo/Yudkowsky conversation
------------------------------------------------

### 13.1. Alignment difficulty debate: Richard Ngo’s case

[Ngo][9:31]  (Sep. 25)
As promised, here’s a write-up of some thoughts from my end. In particular, since I’ve spent a lot of the debate poking Eliezer about his views, I’ve tried here to put forward more positive beliefs of my own in this doc (along with some more specific claims): [GDocs link]

|  |
| --- |
| [Soares: ✨]  |

[Ngo]  (Sep. 25 Google Doc)
We take as a starting observation that a number of “grand challenges” in AI have been solved by AIs that are very far from the level of generality which people expected would be needed. Chess, once considered to be the pinnacle of human reasoning, was solved by an algorithm that’s essentially useless for real-world tasks. Go required more flexible learning algorithms, but policies which beat human performance are still nowhere near generalising to anything else; the same for StarCraft, DOTA, and the protein folding problem. Now it seems very plausible that AIs will even be able to pass (many versions of) the Turing Test while still being a long way from AGI.

[Yudkowsky][11:26]  (Sep. 25 comment)

>
> Now it seems very plausible that AIs will even be able to pass (many versions of) the Turing Test while still being a long way from AGI.
>
>
>

I remark:  Restricted versions of the Turing Test.  Unrestricted passing of the Turing Test happens after the world ends.  Consider how smart you’d have to be to pose as an AGI to an AGI; you’d need all the cognitive powers of an AGI as well as all of your human powers.

[Ngo][11:24]  (Sep. 29 comment)
Perhaps we can quantify the Turing test by asking something like:

* What percentile of competence is the judge?
* What percentile of competence are the humans who the AI is meant to pass as?
* How much effort does the judge put in (measured in, say, hours of strategic preparation)?

Does this framing seem reasonable to you? And if so, what are the highest numbers for each of these metrics that correspond to a Turing test which an AI could plausibly pass before the world ends?

[Ngo]  (Sep. 25 Google Doc)
I expect this trend to continue until after we have AIs which are superhuman at mathematical theorem-proving, programming, many other white-collar jobs, and many types of scientific research. It seems like Eliezer doesn’t. I’ll highlight two specific disagreements which seem to play into this.

[Yudkowsky][11:28]  (Sep. 25 comment)

>
> doesn’t
>
>
>

Eh?  I’m pretty fine with something proving the Riemann Hypothesis before the world ends.  It came up during my recent debate with Paul, in fact.

Not so fine with something designing nanomachinery that can be built by factories built by proteins.  They’re legitimately different orders of problem, and it’s no coincidence that the second one has a path to pivotal impact, and the first does not.

[Ngo]  (Sep. 25 Google Doc)
A first disagreement is related to Eliezer’s characterisation of GPT-3 as a shallow pattern-memoriser. I think there’s a continuous spectrum between pattern-memorisation and general intelligence. In order to memorise more and more patterns, you need to start understanding them at a high level of abstraction, draw inferences about parts of the patterns based on other parts, and so on. When those patterns are drawn from the real world, then this process leads to the gradual development of a world-model.

This position seems more consistent with the success of deep learning so far than Eliezer’s position (although my advocacy of it loses points for being post-hoc; I was closer to Eliezer’s position before the GPTs). It also predicts that deep learning will lead to agents which can reason about the world in increasingly impressive ways (although I don’t have a strong position on the extent to which new architectures and algorithms will be required for that). I think that the spectrum from less to more intelligent animals (excluding humans) is a good example of what it looks like to gradually move from pattern-memorisation to increasingly sophisticated world-models and abstraction capabilities.

[Yudkowsky][11:30]  (Sep. 25 comment)

>
> In order to memorise more and more patterns, you need to start understanding them at a high level of abstraction, draw inferences about parts of the patterns based on other parts, and so on.
>
>
>

Correct.  You can believe this and *not* believe that exactly GPT-like architectures can keep going deeper until their overlap of a greater number of patterns achieves the same level of depth and generalization as human depth and generalization from fewer patterns, just like pre-transformer architectures ran into trouble in memorizing deeper patterns than the shallower ones those earlier systems could memorize.

[Ngo]  (Sep. 25 Google Doc)
I expect that Eliezer won’t claim that pattern-memorisation is *unrelated* to general intelligence, but will claim that a pattern-memoriser needs to undergo a sharp transition in its cognitive algorithms before it can reason reliably about novel domains (like open scientific problems) – with his main argument for that being the example of the sharp transition undergone by humans.

However, it seems unlikely to me that humans underwent a major transition in our underlying cognitive algorithms since diverging from chimpanzees, because our brains are so similar to those of chimps, and because our evolution from chimps didn’t take very long. This evidence suggests that we should favour explanations for our success which don’t need to appeal to big algorithmic changes, if we have any such explanations; and I think we do. More specifically, I’d characterise the three key differences between humans and chimps as:

1. Humans have bigger brains.
2. Humans have a range of small adaptations primarily related to motivation and attention, such as infant focus on language and mimicry, that make us much better at cultural learning.
3. Humans grow up in a rich cultural environment.

[Ngo][9:13]  (Sep. 23 comment on earlier draft)

>
> bigger brains
>
>
>

I recall a 3-4x difference; but this paper says 5-6x for frontal cortex: [https://www.nature.com/articles/nn814](https://www.google.com/url?q=https://www.nature.com/articles/nn814&sa=D&source=editors&ust=1633068529191000&usg=AOvVaw2kXhQODWG2jBW_xyHO7R_U)

[Tallinn][3:24]  (Sep. 26 comment)

>
> language and mimicry
>
>
>

“apes are unable to ape sounds” claims david deutsch in “the beginning of infinity”

[Barnes][8:09]  (Sep. 23 comment on earlier draft)

>
> [Humans grow up in a rich cultural environment.]
>
>
>

much richer cultural environment including deliberate teaching

[Ngo]  (Sep. 25 Google Doc)
I claim that the discontinuity between the capabilities of humans and chimps is mainly explained by the general intelligence of chimps not being aimed in the direction of learning the skills required for economically valuable tasks, which in turn is mainly due to chimps lacking the “range of small adaptations” mentioned above.

My argument is a more specific version of Paul’s claim that chimp evolution was not primarily selecting for doing things like technological development. In particular, it was not selecting for them because no cumulative cultural environment existed while chimps were evolving, and selection for the application of general intelligence to technological development is much stronger in a cultural environment. (I claim that the cultural environment was so limited before humans mainly because cultural accumulation is [very sensitive to transmission fidelity](https://royalsocietypublishing.org/doi/10.1098/rstb.2012.0119).)

By contrast, AIs will be trained in a cultural environment (including extensive language use) from the beginning, so this won’t be a source of large gains for later systems.

[Ngo][6:01]  (Sep. 22 comment on earlier draft)

>
> more specific version of Paul’s claim
>
>
>

Based on some of Paul’s recent comments, this may be what he intended all along; though I don’t recall his original writings on takeoff speeds making this specific argument.

[Shulman][14:23]  (Sep. 25 comment)

>
> (I claim that the cultural environment was so limited before humans mainly because cultural accumulation is [very sensitive to transmission fidelity](https://royalsocietypublishing.org/doi/10.1098/rstb.2012.0119).)
>
>
>

There can be other areas with superlinear effects from repeated application of  a skill. There’s reason to think that the most productive complex industries tend to have that character.

Making individual minds able to correctly execute long chains of reasoning by reducing per-step error rate could plausibly have very superlinear effects in programming, engineering, management, strategy, persuasion, etc. And you could have new forms of ‘super-culture’ that don’t work with humans.

<https://ideas.repec.org/a/eee/jeborg/v85y2013icp1-10.htmlhttps://ideas.repec.org/a/eee/jeborg/v85y2013icp1-10.html>

[Ngo]  (Sep. 25 Google Doc)
If true, this argument would weigh against Eliezer’s claims about agents which possess a core of general intelligence being able to easily apply that intelligence to a wide range of tasks. And I don’t think that Eliezer has a compelling alternative explanation of the key cognitive differences between chimps and humans (the closest I’ve seen in his writings is the brainstorming [at the end of this post](https://www.lesswrong.com/posts/XQirei3crsLxsCQoi/surprised-by-brains)).

If this is the case, I notice an analogy between Eliezer’s argument against Kurzweil, and my argument against Eliezer. Eliezer attempted to put microfoundations underneath the trend line of Moore’s law, which led to a different prediction than Kurzweil’s straightforward extrapolation. Similarly, my proposed microfoundational explanation of the chimp-human gap gives rise to a different prediction than Eliezer’s more straightforward, non-microfoundational extrapolation.

[Yudkowsky][11:39]  (Sep. 25 comment)

>
> Similarly, my proposed microfoundational explanation of the chimp-human gap gives rise to a different prediction than Eliezer’s more straightforward, non-microfoundational extrapolation.
>
>
>

Eliezer does not use “non-microfoundational extrapolations” for very much of anything, but there are obvious reasons why the greater Earth does not benefit from me winning debates through convincingly and correctly listing all the particular capabilities you need to add over and above what GPToid architectures can achieve, in order to achieve AGI.  Nobody else with a good model of larger reality will publicly describe such things in a way they believe is correct.  I prefer not to argue convincingly but wrongly.  But, no, it is not Eliezer’s way to sound confident about anything unless he thinks he has a more detailed picture of the microfoundations than the one you are currently using yourself.

[Ngo][11:40]  (Sep. 29 comment)
Good to know; apologies for the incorrect inference.

Given that this seems like a big sticking point in the debate overall, do you have any ideas about how to move forward while avoiding infohazards?

[Ngo]  (Sep. 25 Google Doc)
My position makes some predictions about hypothetical cases:

1. If chimpanzees had the same motivational and attention-guiding adaptations towards cultural learning and cooperation that humans do, and were raised in equally culturally-rich environments, then they could become economically productive workers in a range of jobs (primarily as manual laborers, but plausibly also for operating machinery, etc).
	1. Results from chimps raised in human families, like [Washoe](https://en.wikipedia.org/wiki/Washoe_(chimpanzee)), seem moderately impressive, although still very uncertain. There’s probably a lot of bias towards positive findings – but on the other hand, it’s only been done a handful of times, and I expect that more practice at it would lead to much better results.
	2. Comparisons between humans and chimps which *aren’t* raised in similar ways to humans are massively biased towards humans. For the purposes of evaluating general intelligence, comparisons between chimpanzees and [feral children](https://en.wikipedia.org/wiki/Feral_child) seem fairer (although it’s very hard to know how much the latter were affected by non-linguistic childhoods as opposed to abuse or pre-existing disabilities).
2. Consider a hypothetical species which has the same level of “general intelligence” that chimpanzees currently have, but is as well-adapted to the domains of abstract reasoning and technological development as chimpanzee behaviour is to the domain of physical survival (e.g. because they evolved in an artificial environment where their fitness was primarily determined by their intellectual contributions). I claim that this species would have superhuman scientific research capabilities, and would be able to make progress in novel areas of science (analogously to how chimpanzees can currently learn to navigate novel physical landscapes).
	1. Insofar as Eliezer doubts this, but *does* believe that this species could outperform a society of village idiots at scientific research, then he needs to explain why the village-idiot-to-Einstein gap is so significant in this context but not in others.
	2. However, this is a pretty weird thought experiment, and maybe doesn’t add much to our existing intuitions about AIs. My main intention here is to point at how animal behaviour is *really really well-adapted* to physical environments, in a way which makes people wonder what it would be like to be *really really well-adapted* to intellectual environments.
3. ~~I claim that the difficulty of human-level oracle AGIs matching humans~~ Consider an AI which has been trained only to answer questions, and is now human-level at doing so. I claim that the difficulty of this AI matching humans at a range of real-world tasks (without being specifically trained to do so) would be much closer to the difficulty of teaching chimps to do science, than the difficulty of teaching adult humans to do abstract reasoning about a new domain.
	1. The analogy here is: chimps have reasonably general intelligence, but it’s hard for them to apply it to science because they weren’t trained to apply intelligence to that. Likewise, human-level oracle AGIs have general intelligence, but it’ll be hard for them to apply it to influencing the world because they weren’t trained to apply intelligence to that.

[Barnes][8:21]  (Sep. 23 comment on earlier draft)

>
> village-idiot-to-Einstein gap
>
>
>

I wonder to what extent you can model within-species intelligence differences partly just as something like hyperparameter search – if you have a billion humans with random variation in their neural/cognitive traits, the top human will be a lot better than average. Then you could say something like:

* humans are the dumbest species you could have where the distribution of intelligence in each generation is sufficient for cultural accumulation
* that by itself might not imply a big gap from chimps
* but human society has much larger population, so the smartest individuals are much smarter

[Ngo][9:05]  (Sep. 23 comment on earlier draft)
I think Eliezer’s response (which I’d agree with) would be that the cognitive difference between the best humans and normal humans is strongly constrained by the fact that we’re all one species who can interbreed with each other. And so our cognitive variation can’t be very big compared with inter-species variation (at the top end at least; although it could at the bottom end via things breaking).

[Barnes][9:35]  (Sep. 23 comment on earlier draft)
I think that’s not obviously true – it’s definitely possible that there’s a lot of random variation due to developmental variation etc. If that’s the case then population size could create large within-species differences

[Yudkowsky][11:46]  (Sep. 25 comment)

>
> oracle AGIs
>
>
>

Remind me of what this is?  Surely you don’t just mean the AI that produces plans it doesn’t implement itself, because that AI becomes an agent by adding an external switch that routes its outputs to a motor; it can hardly be much cognitively different from an agent.  Then what do you mean, “oracle AGI”?

(People tend to produce shallow specs of what they mean by “oracle” that make no sense in my microfoundations, a la “Just drive red cars but not blue cars!”, leading to my frequent reply, “Sorry, still AGI-complete in terms of the machinery you have to build to do that.”)

[Ngo][11:44]  (Sep. 29 comment)
Edited to clarify what I meant in this context (and remove the word “oracle” altogether).

[Yudkowsky][12:01]  (Sep. 29 comment)
My reply holds just as much to “AIs that answer questions”; what restricted question set do you imagine suffices to save the world without dangerously generalizing internal engines?

[Barnes][8:15]  (Sep. 23 comment on earlier draft)

>
> The analogy here is: chimps have reasonably general intelligence, but it’s hard for them to apply it to science because they weren’t trained to apply intelligence to that. Likewise, human-level oracle AGIs have general intelligence, but it’ll be hard for them to apply it to influencing the world because they weren’t trained to apply intelligence to that.
>
>
>

this is not intuitive to me; it seems pretty plausible that the subtasks of predicting the world and of influencing the world are much more similar than the subtasks of surviving in a chimp society are to the subtasks of doing science

[Ngo][8:59]  (Sep. 23 comment on earlier draft)
I think Eliezer’s position is that all of these tasks are fairly similar *if you have general intelligence*. E.g. he argued that the difference between very good theorem-proving and influencing the world is significantly smaller than people expect. So even if you’re right, I think his position is too strong for your claim to help him. (I expect him to say that I’m significantly overestimating the extent to which chimps are running general cognitive algorithms).

[Barnes][9:33]  (Sep. 23 comment on earlier draft)
I wasn’t trying to defend his position, just disagreeing with you ![😛](https://s.w.org/images/core/emoji/14.0.0/72x72/1f61b.png)

[Ngo]  (Sep. 25 Google Doc)
*More specific details*

Here are three training regimes which I expect to contribute to AGI:

* Self-supervised training – e.g. on internet text, code, books, videos, etc.
* Task-based RL – agents are rewarded (likely via human feedback, and some version of iterated amplification) for doing well on bounded tasks.
* Open-ended RL – agents are rewarded for achieving long-term goals in rich environments.

[Yudkowsky][11:56]  (Sep. 25 comment)

>
> bounded tasks
>
>
>

There’s an interpretation of this I’d agree with, but all of the work is being carried by the *boundedness* of the tasks, little or none via the “human feedback” part which I shrug at, and none by the “iterated amplification” part since I consider that tech unlikely to exist before the world ends.

[Ngo]  (Sep. 25 Google Doc)
Most of my probability of catastrophe comes from AGIs trained primarily via open-ended RL. Although IA makes these scenarios less likely by making task-based RL more powerful, it doesn’t seem to me that IA tackles the hardest case (of aligning agents trained via open-ended RL) head-on. But disaster from open-ended RL also seems a long way away – mainly because getting long-term real-world feedback is very slow, and I expect it to be hard to create sufficiently rich artificial environments. By that point I do expect the strategic landscape to be significantly different, because of the impact of task-based RL.

[Yudkowsky][11:57]  (Sep. 25 comment)

>
> a long way away
>
>
>

Oh, definitely, at the present rates of progress we’ve got years, plural.

The [history of futurism](https://intelligence.org/2017/10/13/fire-alarm/) says that even saying that tends to be unreliable in the general case (people keep saying it right up until the Big Thing actually happens) and also that it’s rather a difficult form of knowledge to obtain more than a few years out.

[Yudkowsky][12;01]  (Sep. 25 comment)

>
> hard to create sufficiently rich artificial environments
>
>
>

Disagree; I don’t think that making environments more difficult in a way that challenges the environment inside will prove to be a significant AI development bottleneck.  Making simulations easy enough for current AIs to do interesting things in them, but hard enough that the things they do are not completely trivial, takes some work relevant to current levels of AI intelligence.  I think that making those environments more tractably challenging for smarter AIs is not likely to be nearly a bottleneck in progress, compared to making the AIs smarter and able to solve the environment.  It’s a one-way-hash, P-vs-NP style thing – not literally, just that general relationship between it taking a lower amount of effort to pose a problem such that solving it requires a higher amount of effort.

[Ngo]  (Sep. 25 Google Doc)
Perhaps the best way to pin down disagreements in our expectations about the effects of the strategic landscape is to identify some measures that could help to reduce AGI risk, and ask how seriously key decision-makers would need to take AGI risk for each measure to be plausible, and how powerful and competent they would need to be for that measure to make a significant difference. Actually, let’s lump these metrics together into a measure of “amount of competent power applied”. Some benchmarks, roughly in order (and focusing on the effort applied by the US):

* Banning chemical/biological weapons
* COVID
	+ Key points: mRNA vaccines, lockdowns, mask mandates
* Nuclear non-proliferation
	+ Key points: [Nunn-Lugar Act](https://www.openphilanthropy.org/blog/ai-governance-grantmaking), [stuxnet](https://en.wikipedia.org/wiki/Stuxnet), various treaties
* The International Space Station
	+ Cost to US: ~$75 billion
* Climate change
	+ US expenditure: >$154 billion (but not very effectively)
* Project Apollo
	+ Wikipedia says that Project Apollo “was the largest commitment of resources ($156 billion in 2019 US dollars) ever made by any nation in peacetime. At its peak, the Apollo program employed 400,000 people and required the support of over 20,000 industrial firms and universities.”
* WW1
* WW2

[Yudkowsky][12:02]  (Sep. 25 comment)

>
> WW2
>
>
>

This level of effort starts to buy significant amounts of time.  This level will not be reached, nor approached, before the world ends.

[Ngo]  (Sep. 25 Google Doc)
Here are some wild speculations (I just came up with this framework, and haven’t thought about these claims very much):

1. The US and China preventing any other country from becoming a leader in AI requires about as much competent power as banning chemical/biological weapons.
2. The US and China enforcing a ban on AIs above a certain level of autonomy requires about as much competent power as the fight against climate change.
	1. In this scenario, all the standard forces which make other types of technological development illegal have pushed towards making autonomous AGI illegal too.
3. Launching a good-faith joint US-China AGI project requires about as much competent power as launching Project Apollo.
	1. According to [this article](https://www.spacedaily.com/news/russia-97h.html), Kennedy (and later Johnson) made several offers (some of which were public) of a joint US-USSR Moon mission, which Khrushchev reportedly came close to accepting. Of course this is a long way from actually doing a joint project (and it’s not clear how reliable the source is), but it still surprised me a lot, given that I viewed the “space race” as basically a zero-sum prestige project. If your model predicted this, I’d be interested to hear why.

[Yudkowsky][12:07]  (Sep. 25 comment)

>
> The US and China preventing any other country from becoming a leader in AI requires about as much competent power as banning chemical/biological weapons.
>
>
>

I believe this is wholly false.  On my model it requires closer to WW1 levels of effort.  I don’t think you’re going to get it without credible threats of military action leveled at previously allied countries.

AI is easier and more profitable to build than chemical / biological weapons, and correspondingly harder to ban.  Existing GPU factories need to be shut down and existing GPU clusters need to be banned and no duplicate of them can be allowed to arise, across many profiting countries that were previously military allies of the United States, which – barring some vast shift in world popular *and* elite opinion against AI, which is also not going to happen – those countries would be extremely disinclined to sign, especially if the treaty terms permitted the USA and China to forge ahead.

The reason why chem weapons bans were much easier was that people did not like chem weapons.  They were awful.  There was a perceived common public interest in nobody having chem weapons.  It was understood popularly and by elites to be a Prisoner’s Dilemma situation requiring enforcement to get to the Pareto optimum.  Nobody was profiting tons off the infrastructure that private parties could use to make chem weapons.

An AI ban is about as easy as banning advanced metal-forging techniques in current use so nobody can get ahead of the USA and China in making airplanes.  That would be HARD and likewise require credible threats of military action against former allies.

“AI ban is as easy as a chem weapons ban” seems to me like politically crazy talk.  I’d expect a more politically habited person to confirm this.

[Shulman][14:32]  (Sep. 25 comment)
AI ban much, much harder than chemical weapons ban. Indeed chemical weapons were low military utility, that was central to the deal, and they have still been used subsequently.

>
> An AI ban is about as easy as banning advanced metal-forging techniques in current use so nobody can get ahead of the USA and China in making airplanes. That would be HARD and likewise require credible threats of military action against former allies.
>
>
>

If large amounts of compute relative to today are needed (and presumably Eliezer rejects this), the fact that there is only a single global leading node chip supply chain makes it vastly easier than metal forging, which exists throughout the world and is vastly cheaper.

Sharing with allies (and at least embedding allies to monitor US compliance) also reduces the conflict side.

OTOH, if compute requirements were super low then it gets a lot worse.

And the biological weapons ban failed completely: the Soviets built an enormous bioweapons program, the largest ever, after agreeing to the ban, and the US couldn’t even tell for sure they were doing so.

[Yudkowsky][18:15]  (Oct. 4 comment)
I’ve updated somewhat off of Carl Shulman’s argument that there’s only one chip supply chain which goes through eg a single manufacturer of lithography machines (ASML), which could maybe make a lock on AI chips possible with only WW1 levels of cooperation instead of WW2.

That said, I worry that, barring WW2 levels, this might not last very long if other countries started duplicating the supply chain, even if they had to go back one or two process nodes on the chips?  There’s a difference between the proposition “ASML has a lock on the lithography market right now” and “if aliens landed and seized ASML, Earth would forever after be unable to build another lithography plant”.  I mean, maybe that’s just true because we lost technology and can’t rebuild old bridges either, but it’s at least less obvious.

Launching Tomahawk cruise missiles at any attempt anywhere to build a new ASML, is getting back into “military threats against former military allies” territory and hence what I termed WW2 levels of cooperation.

[Shulman][18:30]  (Oct. 4 comment)
China has been trying for some time to build its own and has failed with tens of billions of dollars (but has captured some lagging node share), but would be substantially more likely to succeed with a trillion dollar investment. That said, it is hard to throw money at these things and the tons of tacit knowledge/culture/supply chain networks are tough to replicate. Also many ripoffs of the semiconductor subsidies have occurred. Getting more NASA/Boeing and less SpaceX is a plausible outcome even with huge investment.

They are trying to hire people away from the existing supply chain to take its expertise and building domestic skills with the lagging nodes.

[Yudkowsky][19:14]  (Oct. 4 comment)
Does that same theory predict that if aliens land and grab some but not all of the current ASML personnel, Earth is thereby successfully taken hostage for years, because Earth has trouble rebuilding ASML, which had the irreproducible lineage of masters and apprentices dating back to the era of Lost Civilization?  Or would Earth be much better at this than China, on your model?

[Shulman][19:31]  (Oct. 4 comment)
I’ll read that as including the many suppliers of ASML (one EUV machine has over 100,000 parts, many incredibly fancy or unique). It’s just a matter of how many years it takes. I think Earth fails to rebuild that capacity in 2 years but succeeds in 10.

“A study this spring by Boston Consulting Group and the Semiconductor Industry Association estimated that creating a self-sufficient chip supply chain would take at least $1 trillion and sharply increase prices for chips and products made with them…The situation underscores the crucial role played by ASML, a once obscure company whose market value now exceeds $285 billion. It is “the most important company you never heard of,” said C.J. Muse, an analyst at Evercore ISI.”

[https://www.nytimes.com/2021/07/04/technology/tech-cold-war-chips.html](https://www.google.com/url?q=https://www.nytimes.com/2021/07/04/technology/tech-cold-war-chips.html&sa=D&source=docs&ust=1638426593582000&usg=AOvVaw3sdQnMg5JFsRGyBxhJraVp)

[Yudkowsky][19:59]  (Oct. 4 comment)
No in 2 years, yes in 10 years sounds reasonable to me for this hypothetical scenario, as far as I know in my limited knowledge.

[Yudkowsky][12:10]  (Sep. 25 comment)

>
> Launching a good-faith joint US-China AGI project requires about as much competent power as launching Project Apollo.
>
>
>

It’s really weird, relative to my own model, that you put the item that the US and China can bilaterally decide to do all by themselves, without threats of military action against their former allies, as more difficult than the items that require conditions imposed on other developed countries that don’t want them.

Political coordination is hard.  No, seriously, it’s hard.  It comes with a difficulty penalty that scales with the number of countries, how complete the buy-in has to be, and how much their elites and population don’t want to do what you want them to do relative to how much elites and population agree that it needs doing (where this very rapidly goes to “impossible” or “WW1/WW2” as they don’t particularly want to do your thing).

[Ngo]  (Sep. 25 Google Doc)
So far I haven’t talked about how much competent power I actually expect people to apply to AI governance. I don’t think it’s useful for Eliezer and me to debate this directly, since it’s largely downstream from most of the other disagreements we’ve had. In particular, I model him as believing that there’ll be very little competent power applied to prevent AI risk from governments and wider society, partly because he expects a faster takeoff than I do, and partly because he has a lower opinion of governmental competence than I do. But for the record, it seems likely to me that there’ll be as much competent effort put into reducing AI risk by governments and wider society as there has been into fighting COVID; and plausibly (but not likely) as much as fighting climate change.

One key factor is my expectation that arguments about the importance of alignment will become much stronger as we discover more compelling examples of misalignment. I don’t currently have strong opinions about how compelling the worst examples of misalignment before catastrophe are likely to be; but identifying and publicising them seems like a particularly effective form of advocacy, and one which we should prepare for in advance.

The predictable accumulation of easily-accessible evidence that AI risk is important is one example of a more general principle: that it’s much easier to understand, publicise, and solve problems as those problems get closer and more concrete. This seems like a strong effect to me, and a key reason why so many predictions of doom throughout history have failed to come true, even when they seemed compelling at the time they were made.

Upon reflection, however, I think that even taking this effect into account, the levels of competent power required for the interventions mentioned above are too high to justify the level of optimism about AI governance that I started our debate with. On the other hand, I found Eliezer’s arguments about consequentialism less convincing than I expected. Overall I’ve updated that AI risk is higher than I previously believed; though I expect my views to be quite unsettled while I think more, and talk to more people, about specific governance interventions and scenarios.

The post [Ngo’s view on alignment difficulty](https://intelligence.org/2021/12/14/ngos-view-on-alignment-difficulty/) appeared first on [Machine Intelligence Research Institute](https://intelligence.org).


This post is a transcript of a multi-day discussion between Paul Christiano, Richard Ngo, Eliezer Yudkowsky, Rob Bensinger, Holden Karnofsky, Rohin Shah, Carl Shulman, Nate Soares, and Jaan Tallinn, following up on the Yudkowsky/Christiano debate in [1](https://www.lesswrong.com/posts/vwLxd6hhFvPbvKmBH/yudkowsky-and-christiano-discuss-takeoff-speeds), [2](https://www.lesswrong.com/posts/7MCqRnZzvszsxgtJi/christiano-cotra-shulman-and-yudkowsky-on-ai-progress), [3](https://www.lesswrong.com/posts/sCCdCLPN9E3YvdZhj/shulman-and-yudkowsky-on-ai-progress), and [4](https://www.lesswrong.com/posts/fS7Zdj2e2xMqE6qja/more-christiano-cotra-and-yudkowsky-on-ai-progress).

Color key:

|  |  |
| --- | --- |
|  Chat by Paul, Richard, and Eliezer  |  Other chat  |

12. Follow-ups to the Christiano/Yudkowsky conversation
-------------------------------------------------------

### 12.1. Bensinger and Shah on prototypes and technological forecasting

[Bensinger][16:22]
Quoth Paul:

>
> seems like you have to make the wright flyer much better before it’s important, and that it becomes more like an industry as that happens, and that this is intimately related to why so few people were working on it
>
>
>

Is this basically saying ‘the Wright brothers didn’t personally capture much value by inventing heavier-than-air flying machines, and this was foreseeable, which is why there wasn’t a huge industry effort already underway to try to build such machines as fast as possible.’ ?

My maybe-wrong model of Eliezer says here ‘the Wright brothers knew a (Thielian) secret’, while my maybe-wrong model of Paul instead says:

* They didn’t know a secret — it was obvious to tons of people that you could do something sorta like what the Wright brothers did and thereby invent airplanes; the Wright brothers just had unusual non-monetary goals that made them passionate to do a thing most people didn’t care about.
* Or maybe it’s better to say: they knew some specific secrets about physics/engineering, but only because other people *correctly* saw ‘there are secrets to be found here, but they’re stamp-collecting secrets of little economic value to me, so I won’t bother to learn the secrets’. ~Everyone knows where the treasure is located, and ~everyone knows the treasure won’t make you rich.

[Yudkowsky][17:24]
My model of Paul says there could be a secret, but only because the industry was tiny and the invention was nearly worthless directly.

|  |
| --- |
| [Cotra: ➕] |

[Christiano][17:53]
I mean, I think they knew a bit of stuff, but it generally takes a lot of stuff to make something valuable, and the more people have been looking around in an area the more confident you can be that it’s going to take a lot of stuff to do much better, and it starts to look like an extremely strong regularity for big industries like ML or semiconductors

it’s pretty rare to find small ideas that don’t take a bunch of work to have big impacts

I don’t know exactly what a thielian secret is (haven’t read the reference and just have a vibe)

straightening it out a bit, I have 2 beliefs that combine disjunctively: (i) generally it takes a lot of work to do stuff, as a strong empirical fact about technology, (ii) generally if the returns are bigger there are more people working on it, as a slightly-less-strong fact about sociology

[Bensinger][18:09]
secrets = important undiscovered information (or information that’s been discovered but isn’t widely known), that you can use to get an edge in something. <https://www.lesswrong.com/posts/ReB7yoF22GuerNfhH/thiel-on-secrets-and-indefiniteness>

There seems to be a Paul/Eliezer disagreement about how common these are in general. And maybe a disagreement about how much more efficiently humanity discovers and propagates secrets as you scale up the secret’s value?

[Yudkowsky][18:35]
Many times it has taken much work to do stuff; there’s further key assertions here about “It takes $100 billion” and “Multiple parties will invest $10B first” and “$10B gets you a lot of benefit first because scaling is smooth and without really large thresholds”.

Eliezer is like “ah, yes, sometimes it takes 20 or even 200 people to do stuff, but core researchers often don’t scale well past 50, and there aren’t always predecessors that could do a bunch of the same stuff” even though Eliezer agrees with “it often takes a lot of work to do stuff”. More premises are needed for the conclusion, that one alone does not distinguish Eliezer and Paul by enough.

[Bensinger][20:03]
My guess is that everyone agrees with claims 1, 2, and 3 here (please let me know if I’m wrong!):

1. The history of humanity looks less like **Long Series of Cheat Codes World**, and more like **Well-Designed Game World**.

In Long Series of Cheat Codes World, human history looks like this, over and over: Some guy found a cheat code that totally outclasses everyone else and makes him God or Emperor, until everyone else starts using the cheat code too (if the Emperor allows it). After which things are maybe normal for another 50 years, until a new Cheat Code arises that makes its first adopters invincible gods relative to the previous tech generation, and then the cycle repeats.

In Well-Designed Game World, you can sometimes eke out a small advantage, and the balance isn’t *perfect*, but it’s pretty good and the leveling-up tends to be gradual. A level 100 character totally outclasses a level 1 character, and some level transitions are a bigger deal than others, but there’s no level that makes you a god relative to the people one level below you.

2. General intelligence took over the world once. Someone who updated on that fact but otherwise hasn’t thought much about the topic should not consider it ‘bonkers’ that machine general intelligence could take over the world too, even though they should still consider it ‘bonkers’ that eg a coffee startup could take over the world.

(Because beverages have never taken over the world before, whereas general intelligence has; and because our inside-view models of coffee and of general intelligence make it a lot harder to imagine plausible mechanisms by which coffee could make someone emperor, kill all humans, etc., compared to general intelligence.)

(In the game analogy, the situation is a bit like ‘I’ve never found a crazy cheat code or exploit in this game, but I haven’t ruled out that there is one, and I heard of a character once who did a lot of crazy stuff that’s at least *suggestive* that she might have had a cheat code.’)

3. AGI is arising in a world where agents with science and civilization already exist, whereas humans didn’t arise in such a world. This is one reason to think AGI might not take over the world, but it’s *not* a strong enough consideration on its own to make the scenario ‘bonkers’ (because AGIs are likely to differ from humans in many respects, and it wouldn’t obviously be bonkers if the first AGIs turned out to be qualitatively way smarter, cheaper to run, etc.).

—

If folks agree with the above, then I’m confused about how one updates from the above epistemic state to ‘bonkers’.

It was to a large extent physics facts that determined how easy it was to understand the feasibility of nukes without (say) decades of very niche specialized study. Likewise, it was physics facts that determined you need rare materials, many scientists, and a large engineering+infrastructure project to build a nuke. In a world where the *physics* of nukes resulted in it being some PhD’s quiet ‘nobody thinks this will work’ project like Andrew Wiles secretly working on a proof of Fermat’s Last Theorem for seven years, that would have *happened*.

If an alien came to me in 1800 and told me that totally new physics would let future humans build city-destroying superbombs, then I don’t see why I should have considered it bonkers that it might be lone mad scientists rather than nations who built the first superbomb. The ‘lone mad scientist’ scenario sounds more conjunctive to me (assumes the mad scientist knows something that isn’t widely known, AND has the ability to act on that knowledge without tons of resources), so I guess it should have gotten less probability, but maybe not dramatically less?

‘Mad scientist builds city-destroying weapon in basement’ sounds wild to me, but I feel like almost all of the actual unlikeliness comes from the ‘city-destroying weapons exist at all’ part, and then the other parts only moderately lower the probability.

Likewise, I feel like the prima-facie craziness of basement AGI mostly comes from ‘generally intelligence is a crazy thing, it’s wild that anything could be that high-impact’, and a much smaller amount comes from ‘it’s wild that something important could happen in some person’s basement’.

—

It *does* structurally make sense to me that Paul might know things I don’t about GPT-3 and/or humans that make it obvious to him that we roughly know the roadmap to AGI and it’s this.

If the entire ‘it’s bonkers that some niche part of ML could crack open AGI in 2026 and reveal that GPT-3 (and the mainstream-in-2026 stuff) was on a very different part of the tech tree’ view is coming from a detailed inside-view model of intelligence like this, then that immediately ends my confusion about the argument structure.

I don’t understand why you think you have the roadmap, and given a high-confidence roadmap I’m guessing I’d still put more probability than you on someone finding a very different, shorter path that works too. But the *argument structure* “roadmap therefore bonkers” makes sense to me.

If there are meant to be *other* arguments against ‘high-impact AGI via niche ideas/techniques’ that are strong enough to make it bonkers, then I remain confused about the argument structure and how it can carry that much weight.

I can imagine an inside-view model of human cognition, GPT-3 cognition, etc. that tells you ‘AGI coming from nowhere in 3 years is bonkers’; I can’t imagine an ML-is-a-reasonably-efficient-market argument that does the same, because even a perfectly efficient market isn’t *omniscient* and can still be surprised by undiscovered physics facts that tell you ‘nukes are relatively easy to build’ and ‘the fastest path to nukes is relatively hard to figure out’.

(Caveat: I’m using the ‘basement nukes’ and ‘Fermat’s last theorem’ analogy because it helps clarify the principles involved, not because I think AGI will be that extreme on the spectrum.)

|  |
| --- |
| [Yudkowsky: +1] |

Oh, I also wouldn’t be confused by a view like “I think it’s 25% likely we’ll see a more Eliezer-ish world. But it sounds like Eliezer is, like, 90% confident that will happen, and *that level of confidence* (and/or the weak reasoning he’s provided for that confidence) seems bonkers to me.”

The thing I’d be confused by is e.g. “ML is efficient-ish, therefore *the out-of-the-blue-AGI scenario itself* is bonkers and gets, like, 5% probability.”

[Shah][1:58]
(I’m unclear on whether this is acceptable for this channel, please let me know if not)

>
> I can’t imagine an ML-is-a-reasonably-efficient-market argument that does the same, because even a perfectly efficient market isn’t omniscient and can still be surprised by undiscovered physics facts
>
>
>

I think this seems right as a first pass.

Suppose we then make the empirical observation that in tons and tons of other fields, it is extremely rare that people discover new facts that lead to immediate impact. (Set aside for now whether or not that’s true; assume that it is.) Two ways you could react to this:

1. Different fields are different fields. It’s not like there’s a common generative process that outputs a distribution of facts and how hard they are to find that is common across fields. Since there’s no common generative process, facts about field X shouldn’t be expected to transfer to make predictions about field Y.

2. There’s some latent reason, that we don’t currently know, that makes it so that it is rare for newly discovered facts to lead to immediate impact.

It seems like you’re saying that (2) is not a reasonable reaction (i.e. “not a valid argument structure”), and I don’t know why. There are lots of things we don’t know, is it really so bad to posit one more?

(Once we agree on the argument structure, we should then talk about e.g. reasons why such a latent reason can’t exist, or possible guesses as to what the latent reason is, etc, but fundamentally I feel generally okay with starting out with “there’s probably some reason for this empirical observation, and absent additional information, I should expect that reason to continue to hold”.)

[Bensinger][3:15]
I think 2 is a valid argument structure, but I didn’t mention it because I’d be surprised if it had enough evidential weight (in this case) to produce an ‘update to bonkers’. I’d love to hear more about this if anyone thinks I’m under-weighting this factor. (Or any others I left out!)

[Shah][23:57]
Idk if it gets all the way to “bonkers”, but (2) seems pretty strong to me, and is how I would interpret Paul-style arguments on timelines/takeoff if I were taking on what-I-believe-to-be your framework

[Bensinger][11:06]
Well, I’d love to hear more about that!

Another way of getting at my intuition: I feel like a view that assigns very small probability to ‘suddenly vastly superhuman AI, because something that high-impact hasn’t happened before’

(which still seems weird to me, because physics doesn’t know what ‘impact’ is and I don’t see what physical mechanism could forbid it that strongly and generally, short of simulation hypotheses)

… would also assign very small probability in 1800 to ‘given an alien prediction that totally new physics will let us build superbombs at least powerful enough to level cities, the superbomb in question will ignite the atmosphere or otherwise destroy the Earth’.

But this seems flatly wrong to me — if you buy that the bomb works by a totally different mechanism (and exploits a different physics regime) than eg gunpowder, then the output of the bomb is a *physics* question, and I don’t see how we can concentrate our probability mass much without probing the relevant physics. The history of boat and building sizes is a negligible input to ‘given a totally new kind of bomb that suddenly lets us (at least) destroy cities, what is the total destructive power of the bomb?’.

|  |
| --- |
| [Yudkowsky: +1] |

(Obviously the bomb *didn’t* destroy the Earth, and I wouldn’t be surprised if there’s some Bayesian evidence or method-for-picking-a-prior that could have validly helped you suspect as much in 1800? But it would be a suspicion, not a confident claim.)

[Shah][1:45]

>
> would also assign very small probability in 1800 to ‘given an alien prediction that totally new physics will let us build superbombs at least powerful enough to level cities, the superbomb in question will ignite the atmosphere or otherwise destroy the Earth’
>
>
>

(As phrased you also have to take into account the question of whether humans would deploy the resulting superbomb, but I’ll ignore that effect for now.)

I think this isn’t exactly right. The “totally new physics” part seems important to update on.

Let’s suppose that, in the reference class we built of boat and building sizes, empirically nukes were the 1 technology out of 20 that had property X. (Maybe X is something like “discontinuous jump in things humans care about” or “immediate large impact on the world” or so on.) Then, I think in 1800 you assign ~5% to ‘the first superbomb at least powerful enough to level cities will ignite the atmosphere or otherwise destroy the Earth’.

Once you know more details about how the bomb works, you should be able to update away from 5%. Specifically, “entirely new physics” is an important detail that causes you to update away from 5%. I wouldn’t go as far as you in throwing out reference classes entirely at that point — there can still be unknown latent factors that apply at the level of physics — but I agree reference classes look harder to use in this case.

With AI, I start from ~5% and then I don’t really see any particular detail for AI that I think I should strongly update on. My impression is that Eliezer thinks that “general intelligence” is a qualitatively different sort of thing than that-which-neural-nets-are-doing, and maybe that’s what’s analogous to “entirely new physics”. I’m pretty unconvinced of this, but something in this genre feels quite crux-y for me.

Actually, I think I’ve lost the point of this analogy. What’s the claim for AI that’s analogous to

>
> ‘given an alien prediction that totally new physics will let us build superbombs at least powerful enough to level cities, the superbomb in question will ignite the atmosphere or otherwise destroy the Earth’
>
>
>

?

Like, it seems like this is saying “We figure out how to build a new technology that does X. What’s the chance it has side effect Y?” Where X and Y are basically unrelated.

I was previously interpreting the argument as “if we know there’s a new superbomb based on totally new physics, and we know that the first such superbomb is at least capable of leveling cities, what’s the probability it would have enough destructive force to also destroy the world”, but upon rereading that doesn’t actually seem to be what you were gesturing at.

[Bensinger][3:08]
I’m basically responding to this thing Ajeya wrote:

>
> I think Paul’s view would say:
>
>
> * Things certainly happen for the first time
> * When they do, they happen at small scale in shitty prototypes, like the Wright Flyer or GPT-1 or AlphaGo or the Atari bots or whatever
> * When they’re making a big impact on the world, it’s after a lot of investment and research, like commercial aircrafts in the decades after Kitty Hawk or like the investments people are in the middle of making now with AI that can assist with coding
>
>
>

To which my reply is: I agree that the first AGI systems will be shitty compared to *later* AGI systems. But Ajeya’s Paul-argument seems to additionally require that AGI systems be relatively unimpressive at cognition compared to preceding AI systems that weren’t AGI.

If this is because of some general law that things are shitty / low-impact when they “happen for the first time”, then I don’t understand what physical mechanism could produce such a general law that holds with such force.

As I see it, physics ‘doesn’t care’ about human conceptions of impactfulness, and will instead produce AGI prototypes, aircraft prototypes, and nuke prototypes that have as much impact as is implied by the detailed case-specific workings of general intelligence, flight, and nuclear chain reactions respectively.

We could frame the analogy as:

* ‘If there’s a year where AI goes from being unable to do competitive par-human reasoning in the hard sciences, to being able to do such reasoning, we should estimate the impact of the first such systems by drawing on our beliefs about par-human scientific reasoning itself.’
* Likewise: ‘If there’s a year where explosives go from being unable to destroy cities to being able to destroy cities, we should estimate the impact of the first such explosives by drawing on our beliefs about how (current or future) physics might allow a city to be destroyed, and what other effects or side-effects such a process might have. We should spend little or no time thinking about the impactfulness of the first steam engine or the first telescope.’

[Shah][3:14]
Seems like your argument is something like “when there’s a zero-to-one transition, then you have to make predictions based on reasoning about the technology itself”. I think in that case I’d say this thing from above:

>
> My impression is that Eliezer thinks that “general intelligence” is a qualitatively different sort of thing than that-which-neural-nets-are-doing, and maybe that’s what’s analogous to “entirely new physics”. I’m pretty unconvinced of this, but something in this genre feels quite crux-y for me.
>
>
>

(Like, you wouldn’t a priori expect anything special to happen once conventional bombs become big enough to demolish a football stadium for the first time. It’s because nukes are based on “totally new physics” that you might expect unprecedented new impacts from nukes. What’s the analogous thing for AGI? Why isn’t AGI just regular AI but scaled up in a way that’s pretty continuous?)

I’m curious if you’d change your mind if you were convinced that AGI is just regular AI scaled up, with no qualitatively new methods — I expect you wouldn’t but idk why

[Bensinger][4:03]
In my own head, the way I think of ‘AGI’ is basically: “Something happened that allows humans to do biochemistry, materials science, particle physics, etc., even though none of those things were present in our environment of evolutionary adaptedness. Eventually, AI will similarly be able to generalize to biochemistry, materials science, particle physics, etc. We can call that kind of AI ‘AGI’.”

There might be facts I’m unaware of that justify conclusions like ‘AGI is mostly just a bigger version of current ML systems like GPT-3’, and there might be facts that justify conclusions like ‘AGI will be preceded by a long chain of predecessors, each slightly less general and slightly less capable than its successor’.

But if so, I’m assuming those will be facts about CS, human cognition, etc., not at all a list of a hundred facts like ‘the first steam engine didn’t take over the world’, ‘the first telescope didn’t take over the world’…. Because the physics of brains doesn’t care about those things, and because in discussing brains we’re already in ‘things that have been known to take over the world’ territory.

(I think that paying much attention *at all* to the technology-wide base rate for ‘does this allow you to take over the world?’, once you already know you’re doing something like ‘inventing a new human’, doesn’t really make sense at all? It sounds to me like going to a bookstore and then repeatedly worrying ‘What if they don’t have the book I’m looking for? Most stores don’t sell books at all, so this one might not have the one I want.’ If you know it’s a *book* store, then you shouldn’t be thinking at that level of generality at all; the base rate just goes out the window.)

|  |
| --- |
| [Yudkowsky:] +1 |

My way of thinking about AGI is pretty different from saying AGI follows ‘totally new mystery physics’ — I’m explicitly anchoring to a known phenomenon, humans.

The analogous thing for nukes might be ‘we’re going to build a bomb that uses processes kind of like the ones found in the Sun in order to produce enough energy to destroy (at least) a city’.

[Shah][0:44]

>
> The analogous thing for nukes might be ‘we’re going to build a bomb that uses processes kind of like the ones found in the Sun in order to produce enough energy to destroy (at least) a city’.
>
>
>

(And I assume the contentious claim is “that bomb would then ignite the atmosphere, destroy the world, or otherwise have hugely more impact than just destroying a city”.)

In 1800, we say “well, we’ll probably just make existing fires / bombs bigger and bigger until they can destroy a city, so we shouldn’t expect anything particularly novel or crazy to happen”, and assign (say) 5% to the claim.

There is a wrinkle: you said it was processes like the ones found in the Sun. Idk what the state of knowledge was like in 1800, but maybe they knew that the Sun couldn’t be a conventional fire. If so, then they could update to a higher probability.

(You could also infer that since someone bothered to mention “processes like the ones found in the Sun”, those processes must be ones we don’t know yet, which also allows you to make that update. I’m going to ignore that effect, but I’ll note that this is one way in which the phrasing of the claim is incorrectly pushing you in the direction of “assign higher probability”, and I think a similar thing happens for AI when saying “processes like those in the human brain”.)

With AI I don’t see why the human brain is a different kind of thing than (say) convnets. So I feel more inclined to just take the starting prior of 5%.

Presumably you think that assigning 5% to the nukes claim in 1800 was incorrect, even if that perspective doesn’t know that the Sun is not just a very big conventional fire. I’m not sure why this is. According to me this is just the natural thing to do because things are usually continuous and so in the absence of detailed knowledge that’s what your prior should be. (If I had to justify this, I’d point to facts about bridges and buildings and materials science and so on.)

>
> there might be facts that justify conclusions like ‘AGI will be preceded by a long chain of slightly-less-general, slightly-less-capable successors’.
>
>
>

The frame of “justify[ing] conclusions” seems to ask for more confidence than I expect to get. Rather I feel like I’m setting an initial prior that could then be changed radically by engaging with details of the technology. And then I’m further saying that I don’t see any particular details that should cause me to update away significantly (but they could arise in the future).

For example, suppose I have a random sentence generator, and I take the first well-formed claim it spits out. (I’m using a random sentence generator so that we don’t update on the process by which the claim was generated.) This claim turns out to be “Alice has a fake skeleton hidden inside her home”. Let’s say we know nothing about Alice except that she is a real person somewhere in the US who has a home. You can still assign < 10% probability to the claim, and take 10:1 bets with people who don’t know any additional details about Alice. Nonetheless, as you learn more about Alice, you could update towards higher probability, e.g. if you learn that she loves Halloween, that’s a modest update; if you learn she runs a haunted house at Halloween every year, that’s a large update; if you go to her house and see the fake skeleton you can update to ~100%. That’s the sort of situation I feel like we’re in with AI.

If you asked me what facts justify the conclusion that Alice probably doesn’t have a fake skeleton hidden inside her house, I could only point to reference classes, and all the other people I’ve met who don’t have such skeletons. This is not engaging with the details of Alice’s situation, and I could similarly say “if I wanted to know about Alice, surely I should spend most of my time learning about Alice, rather than looking at what Bob and Carol did”. Nonetheless, it is still correct to assign < 10% to the claim.

It really does seem to come down to — why is human-level intelligence such a special turning point that should receive special treatment? Just as you wouldn’t give special treatment to “the first time bridges were longer than 10m”, it doesn’t seem obvious that there’s anything all that special at the point where AIs reach human-level intelligence (at least for the topics we’re discussing; there are obvious reasons that’s an important point when talking about the economic impact of AI)

[Tallinn][7:04]
FWIW, my current 1-paragraph compression of the debate positions is something like:

**catastrophists**: when evolution was gradually improving hominid brains, suddenly something clicked – it stumbled upon the core of general reasoning – and hominids went from banana classifiers to spaceship builders. hence we should expect a similar (but much sharper, given the process speeds) discontinuity with AI.

**gradualists**: no, there was no discontinuity with hominids per se; human brains merely reached a threshold that enabled cultural accumulation (and in a meaningul sense it was *culture* that built those spaceships). similarly, we should not expect sudden discontinuities with AI per se, just an accelerating (and possibly unfavorable to humans) cultural changes as human contributions will be automated away.

—

one possible crux to explore is “how thick is culture”: is it something that AGI will quickly decouple from (dropping directly to physics-based ontology instead) OR will culture remain AGI’s main environment/ontology for at least a decade.

[Ngo][11:18]

>
> FWIW, my current 1-paragraph compression of the debate positions is something like:
>
>
> **catastrophists**: when evolution was gradually improving hominid brains, suddenly something clicked – it stumbled upon the core of general reasoning – and hominids went from banana classifiers to spaceship builders. hence we should expect a similar (but much sharper, given the process speeds) discontinuity with AI.
>
>
> **gradualists**: no, there was no discontinuity with hominids per se; human brains merely reached a threshold that enabled cultural accumulation (and in a meaningul sense it was *culture* that built those spaceships). similarly, we should not expect sudden discontinuities with AI per se, just an accelerating (and possibly unfavorable to humans) cultural changes as human contributions will be automated away.
>
>
> —
>
>
> one possible crux to explore is “how thick is culture”: is it something that AGI will quickly decouple from (dropping directly to physics-based ontology instead) OR will culture remain AGI’s main environment/ontology for at least a decade.
>
>
>

Clarification: in the sentence “just an accelerating (and possibly unfavorable to humans) cultural changes as human contributions will be automated away”, what work is “cultural changes” doing? Could we just say “changes” (including economic, cultural, etc) instead?

>
> In my own head, the way I think of ‘AGI’ is basically: “Something happened that allows humans to do biochemistry, materials science, particle physics, etc., even though none of those things were present in our environment of evolutionary adaptedness. Eventually, AI will similarly be able to generalize to biochemistry, materials science, particle physics, etc. We can call that kind of AI ‘AGI’.”
>
>
> There might be facts I’m unaware of that justify conclusions like ‘AGI is mostly just a bigger version of current ML systems like GPT-3’, and there might be facts that justify conclusions like ‘AGI will be preceded by a long chain of predecessors, each slightly less general and slightly less capable than its successor’.
>
>
> But if so, I’m assuming those will be facts about CS, human cognition, etc., not at all a list of a hundred facts like ‘the first steam engine didn’t take over the world’, ‘the first telescope didn’t take over the world’…. Because the physics of brains doesn’t care about those things, and because in discussing brains we’re already in ‘things that have been known to take over the world’ territory.
>
>
> (I think that paying much attention *at all* to the technology-wide base rate for ‘does this allow you to take over the world?’, once you already know you’re doing something like ‘inventing a new human’, doesn’t really make sense at all? It sounds to me like going to a bookstore and then repeatedly worrying ‘What if they don’t have the book I’m looking for? Most stores don’t sell books at all, so this one might not have the one I want.’ If you know it’s a *book* store, then you shouldn’t be thinking at that level of generality at all; the base rate just goes out the window.)
>
>
>

I’m broadly sympathetic to the idea that claims about AI cognition should be weighted more highly than claims about historical examples. But I think you’re underrating historical examples. There are at least three ways those examples can be informative – by telling us about:

1. Domain similarities

2. Human effort and insight

3. Human predictive biases

You’re mainly arguing against 1, by saying that there are facts about physics, and facts about intelligence, and they’re not very related to each other. This argument is fairly compelling to me (although it still seems plausible that there are deep similarities which we don’t understand yet – e.g. the laws of statistics, which apply to many different domains).

But historical examples can also tell us about #2 – for instance, by giving evidence that great leaps of insight are rare, and so if there exists a path to AGI which doesn’t require great leaps of insight, that path is more likely than one which does.

And they can also tell us about #3 – for instance, by giving evidence that we usually overestimate the differences between old and new technologies, and so therefore those same biases might be relevant to our expectations about AGI.

[Bensinger][12:31]
In the ‘alien warns about nukes’ example, my intuition is that ‘great leaps of insight are rare’ and ‘a random person is likely to overestimate the importance of the first steam engines and telescopes’ tell me practically nothing, compared to what even a small amount of high-uncertainty physics reasoning tells me.

The ‘great leap of insight’ part tells me ~nothing because even if there’s an easy low-insight path to nukes and a hard high-insight path, I don’t thereby know the explosive yield of a bomb on either path (either absolutely or relatively); it depends on how nukes work.

Likewise, I don’t think ‘a random person is likely to overestimate the first steam engine’ really helps with estimating the power of nuclear explosions. I could *imagine* a world where this bias exists and is so powerful and inescapable it ends up being a big weight on the scales, but I don’t think we live in that world?

I’m not even sure that a random person *would* overestimate the importance of prototypes in general. Probably, I guess? But my intuition is still that you’re better off in 1800 focusing on physics calculations rather than the tug-of-war ‘maybe X is cognitively biasing me in *this* way, no wait maybe Y is cognitively biasing me in this other way, no wait…’

Our situation might not be analogous to the 1800-nukes scenario (e.g., maybe we know by observation that current ML systems are basically scaled-down humans). But if it *is* analogous, then I think the history-of-technology argument is not very useful here.

[Tallinn][13:00]
re “cultural changes”: yeah, sorry, i meant “culture” in very general “substrate of human society” sense. “cultural changes” would then include things like changes in power structures and division of labour, but *not* things like “diamondoid bacteria killing all humans in 1 second” (that would be a change in humans, not in the culture)

[Shah][13:09]
I want to note that I agree with your (Rob’s) latest response, but I continue to think most of the action is in whether AGI involves something analogous to “totally new physics”, where I would guess “no” (and would do so particularly strongly for shorter timelines).

(And I would still point to historical examples for “many new technologies don’t involve something analogous to ‘totally new physics'”, and I’ll note that Richard’s #2 about human effort and insight still applies)

### 12.2. Yudkowsky on Steve Jobs and gradualism

[Yudkowsky][15:26]
So recently I was talking with various people about the question of why, for example, Steve Jobs could not find somebody else with UI taste 90% as good as his own, to take over Apple, even while being able to pay infinite money. A successful founder I was talking to was like, “Yep, I sure would pay $100 million to hire somebody who could do 80% of what I can do, in fact, people have earned more than that for doing less.”

I wondered if OpenPhil was an exception to this rule, and people with more contact with OpenPhil seemed to think that OpenPhil did not have 80% of a Holden Karnofsky (besides Holden).

And of course, what sparked this whole thought process in me, was that I’d staked all the effort I put into the Less Wrong sequences, into the belief that if I’d managed to bring myself into existence, then there ought to be lots of young near-Eliezers in Earth’s personspace including some with more math talent or physical stamina not so unusually low, who could be started down the path to being Eliezer by being given a much larger dose of concentrated hints than I got, starting off the compounding cascade of skill formations that I saw as having been responsible for producing me, “on purpose instead of by accident”.

I see my gambit as having largely failed, just like the successful founder couldn’t pay $100 million to find somebody 80% similar in capabilities to himself, and just like Steve Jobs could not find anyone to take over Apple for presumably much larger amounts of money and status and power. Nick Beckstead had some interesting stories about various ways that Steve Jobs had tried to locate successors (which I wasn’t even aware of).

I see a plausible generalization as being a “Sparse World Hypothesis”: The shadow of an Earth with eight billion people, projected into some dimensions, is much sparser than plausible arguments might lead you to believe. Interesting people have few neighbors, even when their properties are collapsed and projected onto lower-dimensional tests of output production. The process of forming an interesting person passes through enough 0-1 critical thresholds that all have to be passed simultaneously in order to start a process of gaining compound interest in various skills, that they then cannot find other people who are 80% as good as what they *do* (never mind being 80% similar to them as people).

I would expect human beings to start out much denser in a space of origins than AI projects, and for the thresholds and compounding cascades of our mental lives to be much less sharp than chimpanzee-human gaps.

Gradualism about humans sure sounds totally reasonable! It is in fact much more plausible-sounding a priori than the corresponding proposition about AI projects! I staked years of my own life on the incredibly reasoning-sounding theory that if one actual Eliezer existed then there should be lots of neighbors near myself that I could catalyze into existence by removing some of the accidental steps from the process that had accidentally produced me.

But it didn’t work in real life because plausible-sounding gradualist arguments just… plain don’t work in real life even though they sure sound plausible. I spent a lot of time arguing with Robin Hanson, who was more gradualist than I was, and was taken by surprise when reality itself was much less gradualist than I was.

My model has Paul or Carl coming back with some story about how, why, no, it is totally reasonable that Steve Jobs couldn’t find a human who was 90% as good at a problem class as Steve Jobs to take over Apple for billions of dollars despite looking, and, why, no, this is not at all a falsified retroprediction of the same gradualist reasoning that says a leading AI project should be inside a dense space of AI projects that projects onto a dense space of capabilities such that it has near neighbors.

If so, I was not able to use this hypothetical model of *selective* gradualist reasoning to deduce in advance that replacements for myself would be sparse in the same sort of space and I’d end up unable to replace myself.

I do not really believe that, without benefits of hindsight, the advance predictions of gradualism would differ between the two cases.

I think if you don’t peek at the answer book in advance, the same sort of person who finds it totally reasonable to expect successful AI projects to have close lesser earlier neighbors, would also find it totally reasonable to think that Steve Jobs definitely ought to be able to find somebody 90% as good to take over his job – and should actually be able to find somebody *much* better because Jobs gets to run a wider search and offer more incentive than when Jobs was wandering into early involvement in Apple.

It’s completely reasonable-sounding! Totally plausible to a human ear! Reality disagrees. Jobs tried to find a successor, couldn’t, and now the largest company in the world by market cap seems no longer capable of sending the iPhones back to the designers and asking them to do something important differently.

This is part of the story for why I put gradualism into a mental class of “arguments that sound plausible and just fail in real life to be binding on reality; reality says ‘so what’ and goes off to do something else”.

[Christiano][17:46]  (Sep. 28)
It feels to me like a common pattern is: I say that ML in particular, and most technologies in general, seem to improve quite gradually on metrics that people care about or track. You say that some kind of “gradualism” worldview predicts a bunch of other stuff (some claim about markets or about steve jobs or whatever that feels closely related on your view but not mine). But it feels to me like there are just a ton of technologies, and a ton of AI benchmarks, and those are just *much* more analogous to “future AI progress.” I know that to you this feels like reference class tennis, but I think I legitimately don’t understand what kind of approach to forecasting you are using that lets you just make (what I see as) the obvious boring prediction about all of the non-AGI technologies.

Perhaps you are saying that symmetrically you don’t understand what approach to forecasting I’m using, that would lead me to predict that technologies improve gradually yet people vary greatly in their abilities. To me it feels like the simplest thing in the world: I expect future technological progress in domain X to be like past progress in domain X, and future technological progress to be like past technological progress, and future market moves to be like past market moves, and future elections to be like past elections.

And it seems like you *must* be doing something that ends up making almost the same predictions as that almost all the time, which is why you don’t get incredibly surprised every single year by continuing boring and unsurprising progress in batteries or solar panels or robots or ML or computers or microscopes or whatever. Like it’s fine if you say “Yes, those areas have trend breaks sometimes” but there are *so many* boring years that you must somehow be doing something like having the baseline “this year is probably going to be boring.”

Such that intuitively it feels to me like the disagreement between us *must* be in the part where AGI feels to me like it is similar to AI-to-date and feels to you like it is very different and better compared to evolution of life or humans.

It has to be the kind of argument that you can make about progress-of-AI-on-metrics-people-care-about, but *not* progress-of-other-technologies-on-metrics-people-care-about, otherwise it seems like you are getting hammered every boring year for every boring technology.

I’m glad we have the disagreement on record where I expect ML progress to continue to get less jumpy as the field grows, and maybe the thing to do is just poke more at that since it is definitely a place where I gut level expect to win bayes points and so could legitimately change my mind on the “which kinds of epistemic practices work better?” question. But it feels like it’s not the main action, the main action has got to be about you thinking that there is a really impactful change somewhere between {modern AI, lower animals} and {AGI, humans} that doesn’t look like ongoing progress in AI.

I think “would GPT-3 + 5 person-years of engineering effort foom?” feels closer to core to me.

(That said, the way AI could be different need not feel like “progress is lumpier,” could totally be more like “Progress is always kind of lumpy, which Paul calls ‘pretty smooth’ and Eliezer calls ‘pretty lumpy’ and doesn’t lead to any disagreements; but Eliezer thinks AGI is different in that kind-of-lumpy progress leads to fast takeoff, while Paul thinks it just leads to kind-of-lumpy increases in the metrics people care about or track.”)

[Yudkowsky][7:46]  (Sep. 29)

>
> I think “would GPT-3 + 5 person-years of engineering effort foom?” feels closer to core to me.
>
>
>

I truly and legitimately cannot tell which side of this you think we should respectively be on. My guess is you’re against GPT-3 fooming because it’s too low-effort and a short timeline, even though I’m the one who thinks GPT-3 isn’t on a smooth continuum with AGI??

With that said, the rest of this feels on-target to me; I sure do feel like {natural selection, humans, AGI} form an obvious set with each other, though even there the internal differences are too vast and the data too scarce for legit outside viewing.

>
> I truly and legitimately cannot tell which side of this you think we should respectively be on. My guess is you’re against GPT-3 fooming because it’s too low-effort and a short timeline, even though I’m the one who thinks GPT-3 isn’t on a smooth continuum with AGI??
>
>
>

I mean I obviously think you can foom starting from an empty Python file with 5 person-years of effort if you’ve got the Textbook From The Future; you wouldn’t use the GPT code or model for anything in that, the Textbook says to throw it out and start over.

[Christiano][9:45]  (Sep. 29)
I think GPT-3 will foom given very little engineering effort, it will just be much slower than the human foom

and then that timeline will get faster and faster over time

it’s also fair to say that it wouldn’t foom because the computers would break before it figured out how to repair them (and it would run out of metal before it figured out how to mine it, etc.), depending on exactly how you define “foom,” but the point is that “you can repair the computers faster than they break” happens much before you can outrun human civilization

so the relevant threshold you cross is the one where you are outrunning civilization

(and my best guess about human evolution is pretty similar, it looks like humans are smart enough to foom over a few hundred thousand years, and that we were the ones to foom because that is also roughly how long it was taking evolution to meaningfully improve our cognition—if we foomed slower it would have instead been a smarter successor who overtook us, if we foomed faster it would have instead been a dumber predecessor, though this is *much* less of a sure-thing than the AI case because natural selection is not trying to make something that fooms)

and regarding {natural selection, humans, AGI} the main question is why modern AI and homo erectus (or even chimps) aren’t in the set

it feels like the core disagreement is that I mostly see a difference in degree between the various animals, and between modern AI and future AI, a difference that is likely to be covered by gradual improvements that are pretty analogous to contemporary improvements, and so as the AI community making contemporary improvements grows I get more and more confident that TAI will be a giant industry rather than an innovation

[Ngo][5:45]
Do you have a source on Jobs having looked hard for a successor who wasn’t Tim Cook?

Also, I don’t have strong opinions about how well Apple is doing now, so I default to looking at the share price, which seems very healthy.

(Although I note in advance that this doesn’t feel like a particularly important point, roughly for the same reason that Paul mentioned: gradualism about Steve Jobs doesn’t seem like a central example of the type of gradualism that informs beliefs about AI development.)

[Yudkowsky][10:40]
My source is literally “my memory of stuff that Nick Beckstead just said to me in person”, maybe he can say more if we invite him.

I’m not quite sure what to do with the notion that “gradualism about Steve Jobs” is somehow less to be expected than gradualism about AGI projects. Humans are GIs. They are *extremely* similar to each other design-wise. There are a *lot* of humans, billions of them, many many many more humans than I expect AGI projects. Despite this the leading edge of human-GIs is sparse enough in the capability space that there is no 90%-of-Steve-Jobs that Jobs can locate, and there is no 90%-of-von-Neumann known to 20th century history. If we are not to take any evidence about this to A-GIs, then I do not understand the rules you’re using to apply gradualism to some domains but not others.

And to be explicit, a skeptic who doesn’t find these divisions intuitive, might well ask, “Is gradualism perhaps isomorphic to ‘The coin always comes up heads on Heady occasions’, where ‘Heady’ occasions are determined by an obscure intuitive method going through some complicated nonverbalizable steps one of which is unfortunately ‘check whether the coin actually came up heads’?”

(As for my own theory, it’s always been that AGIs are mostly like AGIs and not very much like humans or the airplane-manufacturing industry, and I do not, on my own account of things, appeal much to supposed outside viewing or base rates.)

[Shulman][11:11]
I think the way to apply it is to use observable data (drawn widely) and math.

Steve Jobs does look like a (high) draw (selected for its height, in the sparsest tail of the CEO distribution) out of the economic and psychometric literature (using the same kind of approach I use in other areas like estimating effects of introducing slightly superhuman abilities on science, the genetics of height, or wealth distributions). You have roughly normal or log-normal distributions on some measures of ability (with fatter tails when there are some big factors present, e.g. super-tall people are enriched for normal common variants for height but are more frequent than a Gaussian estimated from the middle range because of some weird disease/hormonal large effects). And we have lots of empirical data about the thickness and gaps there. Then you have a couple effects that can make returns in wealth/output created larger.

You get amplification from winner-take-all markets, IT, and scale that let higher ability add value to more places. This is the same effect that lets top modern musicians make so much money. Better CEOs get allocated to bigger companies because multiplicative management decisions are worth more in big companies. Software engineering becomes more valuable as the market for software grows.

Wealth effects are amplified by multiplicative growth (noise in a given period multiplies wealth for the rest of the series, and systematic biases from abilities can grow exponentially or superexponentially over a lifetime), and there are some versions of that in gaining expensive-to-acquire human capital (like fame for Hollywood actors, or experience using incredibly expensive machinery or companies).

And we can read off the distributions of income, wealth, market share, lead time in innovations, scientometrics, etc.

That sort of data lead you to expect cutting edge tech to be months to a few years ahead of followers, winner-take-all tech markets to a few leading firms and often a clearly dominant one (but not driving an expectation of being able to safely rest on laurels for years while others innovate without a moat like network effects). That’s one of my longstanding arguments with Robin Hanson, that his model has more even capabilities and market share for AGI/WBE than typically observed (he says that AGI software will have to be more diverse requiring more specialized companies, to contribute so much GDP).

It is tough to sample for extreme values on multiple traits at once, superexponentially tough as you go out or have more criteria. CEOs of big companies are smarter than average, taller than average, have better social skills on average, but you can’t find people who are near the top on several of those.

<https://www.hbs.edu/ris/Publication%20Files/16-044_9c05278e-9d11-4315-a744-de008edf4d80.pdf>

Correlations between the things help, but it’s tough. E.g. if you have thousands of people in a class on a measure of cognitive skill, and you select on only partially correlated matters of personality, interest, motivation, prior experience, etc, the math says it gets thin and you’ll find different combos (and today we see more representation of different profiles of abilities, including rare and valuable ones, in this community)

I think the bigger update for me from trying to expand high-quality save the world efforts has been on the funny personality traits/habits of mind that need to be selected and their scarcity.

[Karnofsky][11:30]
A cpl comments, without commitment to respond to responses:

1. Something in the zone of “context / experience / obsession” seems important for explaining the Steve Jobs type thing. It seems to me that people who enter an area early tend to maintain an edge even over more talented people who enter later – examples are not just founder/CEO types but also early employees of some companies who are more experienced with higher-level stuff (and often know the history of how they got there) better than later-entering people.

2. I’m not sure if I am just rephrasing something Carl or Paul has said, but something that bugs me a lot about the Rob/Eliezer arguments is that I feel like if I accept >5% probability for the kind of jump they’re talking about, I don’t have a great understanding of how I avoid giving >5% to a kajillion other claims from various startups that they’re about to revolutionize their industry, in ways that seem inside-view plausible and seem to equally “depend on facts about some physical domain rather than facts about reference classes.”

The thing that actually most comes to mind here is Thiel – he has been a phenomenal investor financially, but he has also invested by now in a lot of “atoms” startups with big stories about what they might do, and I don’t think any have come close to reaching those visions (though they have sometimes made $ by doing something orders of magnitude less exciting).

If a big crux here is “whether Thielian secrets exist” this track record could be significant.

I think I might update if I had a cleaner sense of how I could take on this kind of “Well, if it is just a fact about physics that I have no idea about, it can’t be that unlikely” view without then betting on a lot of other inside-view-plausible breakthroughs that haven’t happened. Right now all I can say to imitate this lens is “General intelligence is ‘different'”

I don’t feel the same way about “AI might take over the world” – I feel like I have good reasons this applies to AI and not a bunch of other stuff

[Soares][11:11]
Ok, a few notes from me (feel free to ignore):

1. It seems to me like the convo here is half attempting-to-crux and half attempting-to-distill-out-a-bet. I’m interested in focusing explicitly on cruxing for the time being, for whatever that’s worth. (It seems to me like y’all’re already trending in that direction.)

2. It seems to me that one big revealed difference between the Eliezerverse and the Paulverse is something like:

* In the Paulverse, we already have basically all the fundamental insights we need for AGI, and now it’s just a matter of painstaking scaling.
* In the Eliezerverse, there are large insights yet missing (and once they’re found we have plenty of reason to expect things to go quickly).

For instance, in Eliezerverse they say “The Wright flyer didn’t need to have historical precedents, it was allowed to just start flying. Similarly, the AI systems of tomorrow are allowed to just start GIing without historical precedent.”, and in the Paulverse they say “The analog of the Wright flyer has already happened, it was Alexnet, we are now in the phase analogous to the slow grinding transition from human flight to commercially viable human flight.”

(This seems to me like basically what Ajeya articulated [upthread](https://www.lesswrong.com/posts/fS7Zdj2e2xMqE6qja/more-christiano-cotra-and-yudkowsky-on-ai-progress).)

3. It seems to me that another revealed intuition-difference is in the difficulty that people have operating each other’s models. This is evidenced by, eg, Eliezer/Rob saying things like “I don’t know how to operate the gradualness model without making a bunch of bad predictions about Steve Jobs”, and Paul/Holden responding with things like “I don’t know how to operate the secrets-exist model without making a bunch of bad predictions about material startups”.

I’m not sure whether this is a shallower or deeper disagreement than (2). I’d be interested in further attempts to dig into the questions of how to operate the models, in hopes that the disagreement looks interestingly different once both parties can at least operate the other model.

|  |
| --- |
| [Tallinn: ➕] |

The post [Conversation on technology forecasting and gradualism](https://intelligence.org/2021/12/09/conversation-on-technology-forecasting-and-gradualism/) appeared first on [Machine Intelligence Research Institute](https://intelligence.org).