Summary of history (empowerment and well-being lens)

*Click lower right to download or find on Apple Podcasts, Spotify, Stitcher, etc.*

> Now that I've finished the "most important century" series, I'll still be putting out one longer piece per week, but they'll be on toned-down/less ambitious topics, as you're about to see.

A few years ago, I made a summary of human history in one table. It's [here](https://docs.google.com/spreadsheets/d/1KiefojYkNRJfTtDbXXqwpzswWpI0MLB-iOTm1iacZRM/edit?usp=sharing) (color-coded Google sheet).

To do this, I didn't need to be (and am not!) an expert historian. What I did need to do, and what I found very worth doing, was:

* Decide on a ***lens*** for the summary: a definition of "what matters to me," a way of distinguishing unimportant from important.

* Try to find the most important historical events for that lens, and put them all in one table.

The lens I chose was **empowerment and well-being**. That is, I consider historical people and events significant to the degree that they influenced the average person's (a) options and capabilities (empowerment) - including the advance of science and technology; (b) health, safety, fulfillment, etc. (well-being).[1](#fn1) (I'm not saying these are the same thing! It's possible that greater empowerment could mean lower well-being.)

History through this lens seems **very different from the history presented in e.g. textbooks.** For example:

* Many wars and power struggles barely matter. My summary doesn't so much as mention Charlemagne or William of Orange, and the fall of the Roman Empire doesn't seem like a clearly watershed event. My summary thinks the development of lenses (leading to spectacles, microscopes and telescopes) is far more important.

* Every twist and turn in gender relations, treatment of homosexuality, and the quality of maternal care and contraception is significant (as these likely mattered greatly, and systematically, for many people's quality of life). I've had trouble finding good broad histories on all of these. The development of most natural sciences and hard sciences is also important (much easier to read about, though the best reading material generally does not come from the "history field").

The [summary](https://docs.google.com/spreadsheets/d/1KiefojYkNRJfTtDbXXqwpzswWpI0MLB-iOTm1iacZRM/edit) is simply a color-coded table noting what I see as the most important events in each category during each major time period. It doesn't lend itself to a graphical summary, although I will be "boiling things down" more in future pieces, by trying to construct a **single "how quality of life for the average person, as time passed and empowerment rose" curve.**

But below, I'll try to give a sense of what pops out from this table, by going through some [historical people and events that seem underrated to me](#underrated-people-and-events-according-to-the-), [some that seem overrated](#overrated-people-and-events-according-to-the-), and [high-level takeaways.](#high-level-takeaways)

Despite (or because of) my lack of expertise, I found the exercise useful, and would recommend that others consider doing their own summary of history:

* You can spend infinite time reading history books and learning about various events, but it's a **very different kind of learning** to try to find "all the highlights" for your lens of choice, put them in one place, and reflect on how you'd tell the story of the world yourself if you had to boil it down. I think the latter activity requires more active engagement and is likely to result in better recall of important points.

* And I think the final product can be useful as well, if only for readers to easily get a pretty **thorough sense of your worldview,** what seems significant to you, and what disagreements or differences of perspective you have with others. Hopefully mine is useful to readers for giving a sense of my worldview and my overall sense of what humanity's story is so far.

* Finally, I think that creating a history summary is a **vulnerable thing to do**, in a good way. It's scary how little just about anyone (emphatically including myself) knows about history. I think the normal way to deal with this is to show off the facts one does know, change the subject away from what one doesn't, and generally avoid exposing one's ignorance. My summary of history says: **"This is what seems most important to me; this is the story as I perceive it; whatever important pieces I'm ignorant about are now on display for all to see."** So [take a look](https://docs.google.com/spreadsheets/d/1KiefojYkNRJfTtDbXXqwpzswWpI0MLB-iOTm1iacZRM/edit#gid=0) and [give me feedback](https://www.guidedtrack.com/programs/4kal2ue/run?posttitle=Summary+of+history+%28empowerment+and+well-being+lens%29)!

Underrated people and events according to the "empowerment and well-being" lens[2](#fn2)
----------------------------------------------------------------------------------------

**Tanzimat.** In the mid-19th century, the Ottoman Empire went through a series of reforms that abolished the slave trade, declared political equality of all religions, and decriminalized homosexuality. Most of the attention for early reforms like this tends to focus on Europe and the US, but the Ottoman Empire was quite early on these things.

**Ibn al-Haytham's treatise on optics.** In the early 11th century, an Islamic scholar intensively studied how curved glass lenses bend light and wrote it up, which I would guess turned out to be immensely useful for the development of spectacles (which reached Europe in the 1200s) and - later - microscopes and telescopes, which were crucial to some of the key Scientific Revolution work on physics/astronomy and biology.

**The medicine revolution of the mid-19th century.** As far as I can tell, medicine for thousands of years did very little at all, and surgery may have killed as many people as it saved. It's not clear to me whether life expectancy improved at all in the thousands of years prior to the mid-19th century.[3](#fn3)

But the mid-19th century saw the debut of anesthesia (which knocks out the patient and makes it easier to operate) and sterilization with carbolic acid (reducing the risk of infection); there's a [nice Atul Gawande New Yorker article about these](http://www.newyorker.com/magazine/2013/07/29/slow-ideas). Many more medical breakthroughs would follow, to put it mildly, and now health looks like possibly the top way in which the world has improved. ([One analysis](http://www.amazon.com/Measuring-Gains-Medical-Research-Economic/dp/0226551784)[4](#fn4) estimates that the value of improved life expectancy over the last ~100 years is about as big as *all measured growth in world output*[5](#fn5) over that period.)

Not too far into this revolution, **Paul Ehrlich (turn-of-20th-century chemist, not the author of The Population Bomb)** looks like he came up with a really impressive chunk of today's drug development paradigms. As far as I can tell:

* It had been discovered that when you put a clothing dye into a sample under a microscope, it would stain some things and not others, which could make it easier to see.

* Ehrlich reasoned from here to the idea of **targeted drug development: looking for a chemical that would bind only to certain molecules and not others.** This seems like the beginning of this idea.

* He developed the first effective treatment for syphilis, and also laid the groundwork for the basic chemotherapy idea of delivering a toxin that targets only certain kinds of cells.

* And this was only a fraction of his [contributions to medicine](https://www.karger.com/Article/Fulltext/443526).

It's hard to think of someone who's done more for medicine. Articles I've read imply that he had to deal with a fair amount of skepticism and anti-Semitism in his time, but at least today, everyone who hears his name thinks of ... [a different guy who lost a bet](https://en.wikipedia.org/wiki/Simon%E2%80%93Ehrlich_wager).

**Porphyry, the Greek vegetarian.** Did you know that there was an ancient Greek who was (according to Wikipedia) "an advocate of vegetarianism on spiritual and ethical grounds ... He wrote the On Abstinence from Animal Food (Περὶ ἀποχῆς ἐμψύχων; De Abstinentia ab Esu Animalium), advocating against the consumption of animals, and he is cited with approval in vegetarian literature up to the present day." Is it just me or is that more impressive than, well, Aristotle?

**The rise of the modern academic system in the mid-20th century.** I believe that government funding for science skyrocketed after World War II, and that this period included the creation of the NSF and DARPA and a general [skyrocketing demand for professors](https://en.wikipedia.org/wiki/Academic_tenure_in_North_America#From_1940_to_1972). My vague impression is that this is when science turned into a real industry and mainstream career track. My summary also thinks that science had a lower frequency of well-known breakthroughs after this point.

**Alexandra Elbakyan.** Super recent, and it's of course hard to know who will end up looking historically significant when more time has passed. But it seems like a pretty live possibility that she's done more for science than any scientist in a long time, and there's a good chance you have to Google her to know what I'm talking about.

**Underrated negative trend: The rise of factory farming.** The clearest case, in my view, in which the world has gotten worse with industrialization (note that institutionalized homophobia arose before industrialization, and seems to be in decline now unlike factory farming). I think that really brutal factory-like treatment of animals began in the 1930s and has mostly gotten worse over time.

**Underrated negative trend: The relatively recent rise of institutionalized homophobia.** I believe that bad/inegalitarian gender relations are as old as humanity (more in a future post), and slavery is at least as old as civilization. But institutionalized homophobia may be more of a recent phenomenon. My impression is that it came into being sometime around 0 AD and gradually swept most of the globe (though I'm definitely not confident in this, and would love to learn more).

**The foundations of probability and statistics.** Can you name the general time periods for the creation of: the line chart, bar chart, pie chart, the idea of probability, the idea of the normal distribution, Bayes's theorem, and the first known case-control experiment? Seems like the answer could be just about anything right? Turns out it was all between 1760-1812; all three charts came from [William Playfair](https://en.wikipedia.org/wiki/William_Playfair) and a lot of the rest came from Laplace and Gauss.

Not too much of note happened after that until the end of the 19th century, when Francis Galton and Karl Pearson (not always working together) came up with the modern concepts of standard deviation, correlation, regression analysis, p-values, and more.

I think it's pretty interesting that so many of the things that are so foundational to pretty much any quantitative analysis anyone does of anything were invented in a couple relatively recent spurts.

**Metallurgy.** A huge amount of history's scientific and technological progress is [crammed into the last few hundred years](https://www.cold-takes.com/this-cant-go-on/#scientific-and-technological-advancement), but I think the story of metallurgy is much longer and more gradual (see these [major innovations from ~5000 and ~2000 BCE](https://en.wikipedia.org/wiki/Bronze#History)). I wish I could find a source that compactly goes through the major steps here and how they came about; I'd guess it was mostly trial and error (since so much of it was before the Scientific Revolution), but would like to know whether that's right.

(Mathematics also has its major breakthroughs much more evenly spread throughout history than fields such as physics, chemistry and biology.)

Overrated people and events according to the "empowerment and well-being" lens
------------------------------------------------------------------------------

I mean, the vast majority of rulers, wars, and empires rising and falling.

Special shout-out to:

* The Roman Empire, which I can barely see any sign of either in quality-of-life metrics (future posts) or in key empowerment-and-wellbeing events (most of the headlines from this period came from China or the Islamic Empire). If I taught a history class I'm not entirely sure I would mention the Roman Empire.

* Ancient Greece, which is renowned for its ideas and art, but doesn't seem to have been home to any notable improvements in quality of life - no sustained or effective anti-slavery pushes, no signs of feminism, nothing that helped with health or wealth or peace. Seems like it was a pretty horrific place to live for the average person. I've seen some signs that Athens was especially terrible for women, even by the standards of the time, [e.g.](https://en.wikipedia.org/wiki/Women_in_Greece#Women_in_ancient_Greece)

High-level takeaways
--------------------

### Most of what happened happened all at the same time, in the last few minutes (figuratively)

This project is what originally started to make me feel that we [live in a special time](https://www.cold-takes.com/this-cant-go-on/), and that our place in history is more like "Sitting in a rocket ship that just took off" than like "Playing our small part in the huge unbroken chain of generations."

My [table](https://docs.google.com/spreadsheets/d/1KiefojYkNRJfTtDbXXqwpzswWpI0MLB-iOTm1iacZRM/edit#gid=0) devotes:

* One column to the hundreds of thousands of years of prehistory.

* Three columns to the first ~6000 years of civilization.

* Two columns to the next 300 years.

* 6 columns to the ~200 years since.

That implies that **more has happened in the last 200 years than in the previous million-plus.** I think that's right, not recency bias. It seems very hard to summarize history (with my lens) without devoting massively more attention to these recent periods.

I've [made this point before](https://www.cold-takes.com/this-cant-go-on/), and you'll see it showing up in pretty much any chart you look at of something important - [population](https://ourworldindata.org/world-population-growth#how-has-world-population-growth-changed-over-time), [economic growth](https://www.cold-takes.com/this-cant-go-on/#fnref3), [rate of significant scientific discoveries](https://www.cold-takes.com/this-cant-go-on/#scientific-and-technological-advancement), [child mortality](https://ourworldindata.org/child-mortality#historical-estimates-of-mortality), [human height](https://ourworldindata.org/human-height#the-last-two-millennia), etc. My [summary](https://docs.google.com/spreadsheets/d/1KiefojYkNRJfTtDbXXqwpzswWpI0MLB-iOTm1iacZRM/edit#gid=0) gives a qualitative way of seeing it across many domains at once.

200 years is ~10 generations. We live in an extraordinary time without much precedent. And **because of this, there are ultimately pretty serious limits to how much we can learn from history.**

### History is a story

I sometimes get a vibe from "history people" that we should avoid imposing "narratives" on human history: there are so many previous societies, each with its own richness and complexity, that it's hard to make generalizations or talk about trends and patterns across the whole thing.

That isn't how I feel.

It looks to me like if you're comparing an earlier period to a later one, you can be confident that the later period contains a **higher world population** and **greater empowerment due to a greater stock of ideas, innovations and technological capabilities.**

These trends seem very consistent, and can reasonably be expected to generate other consistent trends as well.

I think history as it's traditionally taught (or at least, as I learned it back in the 20th century) tends to focus on the key events and details of each time, while only inconsistently situating them against the broader trends.[6](#fn6) To me, this is kind of like summarizing the *Star Wars* trilogy as follows: "On the first day covered in the movie, it was warm and humid on Tattooine and hot and dry on Endor. On the second day, it was slightly cooler on Tattooine and hotter on Endor. On the third day, it rained on Tattooine, and it was still hot and dry on Endor. [etc. to the final day covered in the movies.] Done." Not inaccurate per se, but ...

### A lot of history through this lens seems unnecessarily hard to learn about

My [table](https://docs.google.com/spreadsheets/d/1KiefojYkNRJfTtDbXXqwpzswWpI0MLB-iOTm1iacZRM/edit#gid=0) is extremely amateur and is probably missing a kajillion things; I had to furiously search Google and Wikipedia to fill in a lot of the cells. I'd love to live in a world where there were well-documented, comprehensive lists and timelines of the major events for empowerment and well-being.

To give a sense for this, here are some things that would be helpful for viewing history through this lens, that I've been unable to find:

* Systematic accounts - going back as far as possible - of when each major state/empire made official changes to things like women's rights (to own property, hold political power, vote, etc.), formal religious freedom, formal treatment of different ethnic groups, legality and other treatment of LGBTQ+, etc.

* Collected estimates (by region/empire/state and period, with citations) of how many people were slaves, what percentage of marriages were arranged, etc.

* Comprehensive timelines (with citations) of major milestones for most of the rows in my [table](https://docs.google.com/spreadsheets/d/1KiefojYkNRJfTtDbXXqwpzswWpI0MLB-iOTm1iacZRM/edit#gid=0), and/or narrative histories that focus on listing, explaining and contextualizing such milestones and otherwise being concise (an example would be [Asimov's Chronology of Science and Discovery](https://www.cold-takes.com/asimovs-chronology-of-science-and-discovery/), but I'd also like to see this for topics like gender relations).

* Histories of science focused on the discoveries that seem most likely to have contributed to real-world capabilities and quality of life, with explanations of these connections. (As an example of what this wouldn't be like, existing chemistry histories tend to list the discovery of each element.)

* I've only listed things that seem like they would be reasonably straightforward to put together; of course there are a zillion more things I wish I could know about the past.

### I don't know very much!

Though I hope I've been clear about this throughout, I want to mention it again before I wrap up the takeaways. Not only is this summary based on a limited amount of time from a non-expert, but the sources I've been able to find for this project shouldn't be taken as neutral or trustworthy either. I think there are ~infinite ways in which they are likely biased due to the worldviews, identities, assumptions, etc. of the authors.

For example, the previous section notes how much harder it's been to find long-run data and timelines on slavery and women's rights than on technological developments.

Another thing that jumps out is that my summary ended up being heavily focused on the Western world. From what I've been able to gather, the Western world of the very recent past looks like where the most noteworthy human-empowerment-related developments are concentrated. If that's indeed the case, I don't think this was inevitable - there were long periods of time where non-Western civilizations were contributing much more to science, technology, etc. than Western civilizations - but the Scientific Revolution of the 1500s and the Industrial Revolution of the 1800s began in the recent West, and once those happened, people in the West could have been best-positioned to build on them for some period of time. But this could be another reflection of biases in how my sources report what was invented, where and when (I've looked for evidence that this is the case and haven't found any, but my efforts are obviously extremely incomplete, and I'm especially skeptical that noteworthy art is as concentrated in the West as the sources I've consulted make it seem).

The four high-level takeaways listed in this section are the four important-seeming observations I feel most confident about from this exercise. (But most confident doesn't mean confident, and I'm always interested in [feedback](https://www.guidedtrack.com/programs/4kal2ue/run?posttitle=Summary+of+history+%28empowerment+and+well-being+lens%29)!)

---

Footnotes
---------

1. This does not mean I see history as an inevitably upward trend in terms of empowerment and well-being. You could apply the lens I've described whether empowerment and well-being have fallen, risen, or wiggled around randomly over the course of history. [↩](#fnref1)- I'm often not giving sources; my sources are listed in the detailed version of the [summary table.

 [↩](#fnref2)- Check out this chart:

 And note that life expectancy at the beginning is similar to estimates for foraging societies, which are often used as a proxy for what life was like more than 10,000 years ago (chart from [this paper](https://gurven.anth.ucsb.edu/sites/secure.lsit.ucsb.edu.anth.d7_gurven/files/sitefiles/papers/GurvenKaplan2007pdr.pdf)):

![](https://www.cold-takes.com/content/images/2021/09/gurven-kaplan-chart.png) [↩](#fnref3)- See chapter 1. I normally don't cite big claims like this that "one analysis" makes, but I spent some time with this one ([years ago](https://www.openphilanthropy.org/blog/returns-life-sciences-funding)) and broadly find it pretty plausible. [↩](#fnref4)- GDP. [↩](#fnref5)- I checked out [this book](https://www.amazon.com/gp/product/1465437975/ref=ppx_yo_dt_b_search_asin_title?ie=UTF8&psc=1) as a way of seeing whether today's "standard history" still seems like this, and I think it largely does. It's not that economic growth and scientific/technological advancement aren't mentioned, but more that they come off as just one more part of a list of events (most of which tend to focus on who's in power where). [↩](#fnref6)](https://docs.google.com/spreadsheets/d/1KiefojYkNRJfTtDbXXqwpzswWpI0MLB-iOTm1iacZRM/edit?usp=sharing)


Cold Links: assorted fun basketball stuff

[Dwight Howard had some great slam dunk contest dunks](https://www.ccn.com/superman-returns-dwight-howards-5-top-slams-from-dunk-contests-past/). I really love how much fun everyone is having in these videos, especially the announcers. Verbatim quote from the announcer in the last video: "I'm leaving the building! I quit my job! I've never seen anything like that, in my life, it is over!"

Nikola Jokic, the [least graceful superstar I can think of](https://deadspin.com/nikola-jokic-is-the-god-of-the-hideous-game-winner-1833320080). I could watch him all day.

I've seen a lot of sports highlight videos in my life, but [this compilation of Kevin Love's outlet passes](http://deadspin.com/heres-a-compilation-of-kevin-loves-outlet-passes-so-f-1465521627) is special.

[How exactly did Anthony Davis lose his balance and fall over here](https://photos.app.goo.gl/P7LDgdqB8jqxyb4T9)? (Video, click play)

[The worst call by an NBA ref ever](http://www.youtube.com/watch?v=vaZIAXJJDKQ).

[Great video](https://deadspin.com/tireless-steph-curry-helpfully-illustrates-seth-currys-1834908263) of Steph Curry running around a lot during a play. "That’s a damn Family Circus comic."

This is absolutely the most painful [choking away of a game at the very end](https://deadspin.com/the-pelicans-gave-the-suns-a-masterclass-on-the-dumbest-1833356994) I can recall seeing.

For email filter: florpschmop


The Most Important Century (in a nutshell)

I've spent most of my career looking for ways to do as much good as possible, per unit of money or time. I worked on finding evidence-backed charities working on global health and development (co-founding [GiveWell](https://www.givewell.org/)), and later moved into philanthropy that takes [more risks](https://www.openphilanthropy.org/blog/hits-based-giving) (co-founding [Open Philanthropy](https://www.openphilanthropy.org/)).

Over the last few years - thanks to general dialogue with the [effective altruism](https://en.wikipedia.org/wiki/Effective_altruism) community, and extensive research done by [Open Philanthropy's Worldview Investigations team](https://www.cold-takes.com/roadmap-for-the-most-important-century-series/#acknowledgements) - I've become convinced that humanity as a whole faces huge risks and opportunities this century. Better understanding and preparing for these risks and opportunities is where I am now focused.

This piece will summarize a [series](https://www.cold-takes.com/roadmap-for-the-most-important-century-series/) of posts on why I believe we could be in the **most important century of all time for humanity**. It gives a short summary, key post(s), and sometimes key graphics for 5 basic points:

* **The long-run future is radically unfamiliar.** Enough advances in technology could lead to a long-lasting, galaxy-wide civilization that could be a radical utopia, dystopia, or anything in between.

* **The long-run future could come much faster than we think,** due to a possible AI-driven productivity explosion.

* The relevant kind of **AI looks like it will be developed this century** - making this century the one that will initiate, and have the opportunity to shape, a future galaxy-wide civilization.

* These claims seem too "wild" to take seriously. But there are a lot of reasons to think that **we live in a wild time, and should be ready for anything.*** We, the people living in this century, have the chance to have a huge impact on huge numbers of people to come - if we can make sense of the situation enough to find helpful actions. But right now, **we aren't ready for this.**

This thesis has a wacky, sci-fi feel. It's very far from where I expected to end up when I set out to do as much good as possible.

But part of the mindset I've developed through GiveWell and Open Philanthropy is being open to strange possibilities, while critically examining them with as much rigor as possible. And after a lot of investment in examining the above thesis, I think it's likely enough that the world urgently needs more attention on it.

By writing about it, I'd like to either get more attention on it, or gain more opportunities to be criticized and change my mind.

We live in a wild time, and should be ready for anything
--------------------------------------------------------

Many people find the "most important century" claim too "wild": a radical future with advanced AI and civilization spreading throughout our galaxy may happen *eventually*, but it'll be more like 500 years from now, or 1,000 or 10,000. (Not this century.)

These longer time frames would put us in a *less* wild position than if we're in the "most important century." But in the scheme of things, **even if galaxy-wide expansion begins 100,000 years from now, that still means we live in an extraordinary era**- the tiny sliver of time during which the galaxy goes from nearly lifeless to largely populated. It means that out of a staggering number of persons who will ever exist, we're among the first. And that out of hundreds of billions of stars in our galaxy, ours will produce the beings that fill it.

![](https://www.cold-takes.com/content/images/2021/09/fermi-agnostic.png)More at [All Possible Views About Humanity's Future Are Wild](https://www.cold-takes.com/all-possible-views-about-humanitys-future-are-wild/)

Zooming in, we live in a special century, not just a special era. We can see this by looking at how fast the economy is growing. It doesn't *feel* like anything special is going on, because for as long as any of us have been alive, the world economy has grown at a few percent per year:

![Chart of the world economy starting in about 1950. It looks like a straight line (on a logarithmic y-axis), with a dotted straight line projecting more of the same for the future.](https://www.cold-takes.com/content/images/size/w1000/2021/07/business-as-usual.png)

However, when we zoom out to look at history in greater context, we see a picture of an unstable past and an uncertain future:

![Graphic illustrating that when you zoom in on the 'this can't go on' chart, you get the 'business as usual chart.'](https://www.cold-takes.com/content/images/size/w1000/2021/08/long-vs-short-view.png)More at [This Can't Go On](https://www.cold-takes.com/this-cant-go-on/)

**We're currently living through the fastest-growing time in history.** This rate of growth hasn't gone on long, and can't go on indefinitely (there aren't enough atoms in the galaxy to sustain this rate of growth for even another 10,000 years). And if we get *further acceleration* in this rate of growth - in line with historical acceleration - we could reach the limits of what's possible more quickly: within this century.

To recap:

* The last few millions of years - with the start of our species - have been more eventful than the previous several billion.

* The last few hundred years have been more eventful than the previous several million.

* If we see another accelerator (as I think AI could be), the next few decades could be the most eventful of all.

More info about these timelines at [All Possible Views About Humanity's Future Are Wild](https://www.cold-takes.com/all-possible-views-about-humanitys-future-are-wild/), [This Can't Go On](https://www.cold-takes.com/this-cant-go-on/), and [**Forecasting Transformative AI: Biological Anchors**](https://www.cold-takes.com/forecasting-transformative-ai-the-biological-anchors-method-in-a-nutshell/), respectively.

Given the times we live in, we need to be open to possible ways in which the world could change quickly and radically. Ideally, we'd be a bit *over*-attentive to such things, like putting safety first when driving. But today, such possibilities get little attention.

Key pieces:

* [All Possible Views About Humanity's Future Are Wild](https://www.cold-takes.com/all-possible-views-about-humanitys-future-are-wild/)* [This Can't Go On](https://www.cold-takes.com/this-cant-go-on/)

The long-run future is radically unfamiliar
-------------------------------------------

Technology tends to increase people's control over the environment. For a concrete, easy-to-visualize example of what things could look like if technology goes far enough, we might imagine a technology like "digital people": fully conscious people "made out of software" who inhabit virtual environments such that they can experience anything at all and can be copied, run at different speeds and even "reset."

A world of digital people could be radically dystopian (virtual environments used to entrench some people's absolute power over others) or utopian (no disease, material poverty or non-consensual violence, and far greater wisdom and self-understanding than is possible today). Either way, digital people could enable a civilization to spread throughout the galaxy and last for a long time.

Many people think this sort of large, stable future civilization is where we could be headed eventually (whether via digital people or other technologies that increase control over the environment), but don't bother to discuss it because it seems so far off.

Key piece: [Digital People Would Be An Even Bigger Deal](https://www.cold-takes.com/how-digital-people-could-change-the-world/)

The long-run future could come much faster than we think
--------------------------------------------------------

Standard economic growth models imply that **any technology that could fully automate innovation would cause an "economic singularity":** productivity going to infinity this century. This is because it would create a powerful feedback loop: more resources -> more ideas and innovation -> more resources -> more ideas and innovation ...

This loop would not be unprecedented. I think it is in some sense the "default" way the economy operates - for most of economic history up until a couple hundred years ago.

![](https://www.cold-takes.com/content/images/size/w1000/2021/06/duplicatorfeedbackloop-original-2.png)

![8 ideas, each 1.5x'ing the amount of food resources -> explosion from 8 units to 205 units of food, hence 205 people and 205 ideas ... ](https://www.cold-takes.com/content/images/size/w1000/2021/06/duplicatorfeedbackloop-original-6.png)Economic history: more resources -> more people -> more ideas -> more resources ...

But in the "demographic transition" a couple hundred years ago, the "more resources -> more people" step of that loop stopped. Population growth leveled off, and more resources led to richer people instead of more people:

![](https://www.cold-takes.com/content/images/2021/07/demographic-transition-nutshell.png)Today's economy: more resources -> ~~more~~ richer people -> same pace of ideas -> ...

The feedback loop could come back if some other technology restored the "more resources -> more ideas" dynamic. One such technology could be the right kind of AI: what I call PASTA, or Process for Automating Scientific and Technological Advancement.

![](https://www.cold-takes.com/content/images/size/w1000/2021/09/pasta-stills-1.png)

![](https://www.cold-takes.com/content/images/size/w1000/2021/09/pasta-stills-3.png)Possible future: more resources -> more AIs -> more ideas -> more resources ...

That means that **our radical long-run future could be upon us very fast** after PASTA is developed (if it ever is).

It also means that if PASTA systems are *misaligned* - pursuing their own non-human-compatible objectives - things could very quickly go sideways.

Key pieces:

* [The Duplicator: Instant Cloning Would Make the World Economy Explode](https://www.cold-takes.com/the-duplicator/)* [Forecasting Transformative AI, Part 1: What Kind of AI?](https://www.cold-takes.com/transformative-ai-timelines-part-1-of-4-what-kind-of-ai/)

PASTA looks like it will be developed this century
--------------------------------------------------

It's not controversial to say a highly general AI system, such as PASTA, would be momentous. The question is, when (if ever) will such a thing exist?

Over the last few years, a team at Open Philanthropy has investigated this question from multiple angles.

One forecasting method observes that:

* No AI model to date has been even 1% as "big" (in terms of computations performed) as a human brain, and until recently this wouldn't have been affordable - but that will change relatively soon.

* And by the end of this century, it will be affordable to train enormous AI models many times over; to train human-brain-sized models on enormously difficult, expensive tasks; and even perhaps to perform as many computations as have been done "by evolution" (by all animal brains in history to date).

This method's predictions are in line with the latest survey of AI researchers: something like PASTA is more likely than not this century.

A number of other angles have been examined as well.

One challenge for these forecasts: there's **no "field of AI forecasting"** and no expert consensus comparable to the one around climate change.

It's hard to be confident when the discussions around these topics are small and limited. But I think we should take the "most important century" hypothesis seriously based on what we know now, until and unless a "field of AI forecasting" develops.

Key pieces:

* [AI Timelines: Where the Arguments, and the "Experts," Stand](https://www.cold-takes.com/p/32cbd365-0a56-4ecc-a597-f18fbc3adb2b/) (recaps the others, and discusses how we should reason about topics like this where it's unclear who the "experts" are)

* [Forecasting Transformative AI: What's the Burden of Proof?](https://www.cold-takes.com/forecasting-transformative-ai-whats-the-burden-of-proof/)* [Are we "trending toward" transformative AI?](https://www.cold-takes.com/are-we-trending-toward-transformative-ai-how-would-we-know/)* [Forecasting transformative AI: the "biological anchors" method in a nutshell](https://www.cold-takes.com/forecasting-transformative-ai-the-biological-anchors-method-in-a-nutshell/)

We're not ready for this
------------------------

When I talk about being in the "most important century," I don't just mean that significant events are going to occur. I mean that we, the people living in this century, have the chance to have a huge impact on huge numbers of people to come - if we can make sense of the situation enough to find helpful actions.

But that's a big "if." Many things we can do might make things better or worse (and it's hard to say which).

When confronting the "most important century" hypothesis, my attitude doesn't match the familiar ones of "excitement and motion" or "fear and avoidance." Instead, I feel an **odd mix of intensity, urgency, confusion and hesitance.** I'm looking at something bigger than I ever expected to confront, feeling underqualified and ignorant about what to do next.

|  |  |
| --- | --- |
| **Situation** | **Appropriate reaction (IMO)** |
| "This could be a billion-dollar company!"
  | "Woohoo, let's GO for it!"
  |
| "This could be the most important century!"
  | "... Oh ... wow ... I don't know what to say and I somewhat want to vomit ... I have to sit down and think about this one."
  |

With that in mind, rather than a "call to action," I issue a [Call to Vigilance](https://www.cold-takes.com/call-to-vigilance/):

* If you're convinced by the arguments in this series, then don't rush to "do something" and then move on.

* Instead, take whatever [robustly good actions](https://www.cold-takes.com/making-the-best-of-the-most-important-century/#robustly-helpful-actions) you can today, and otherwise put yourself in a better position to take important actions when the time comes.

* For those looking for a quick action that will make future action more likely, see [this section of "Call to Vigilance."](https://www.cold-takes.com/call-to-vigilance/#buttons-you-can-click)

Key pieces:

* [Making the Best of the Most Important Century](https://www.cold-takes.com/making-the-best-of-the-most-important-century/) * [Call to Vigilance](https://www.cold-takes.com/call-to-vigilance/).

One metaphor for my headspace is that it feels as though the world is a set of people on a plane blasting down the runway:

And every time I read commentary on what's going on in the world, people are discussing how to arrange your seatbelt as comfortably as possible given that wearing one is part of life, or saying how the best moments in life are sitting with your family and watching the white lines whooshing by, or arguing about whose fault it is that there's a background roar making it hard to hear each other.

I don't know where we're actually heading, or what we can do about it. But I feel pretty solid in saying that we as a civilization are not ready for what's coming, and we need to start by taking it more seriously.


Why AI alignment could be hard with modern deep learning

*Click lower right to download or find on Apple Podcasts, Spotify, Stitcher, etc.*

[Today’s world

Transformative AI

Digital people
World of

Misaligned AI
World run by

Something else

or
or

Stable, galaxy-wide
civilization](https://www.cold-takes.com/roadmap-for-the-most-important-century-series/)

> This is a guest post by my colleague [Ajeya Cotra](https://www.openphilanthropy.org/about/team/ajeya-cotra).

Holden [previously mentioned](https://www.cold-takes.com/transformative-ai-timelines-part-1-of-4-what-kind-of-ai/#misaligned-ai-mysterious-potentially-dangerous-objectives) the idea that advanced AI systems (e.g. [PASTA](https://www.cold-takes.com/transformative-ai-timelines-part-1-of-4-what-kind-of-ai/)) may develop [dangerous goals](https://www.cold-takes.com/transformative-ai-timelines-part-1-of-4-what-kind-of-ai/#misaligned-ai-mysterious-potentially-dangerous-objectives) that cause them to deceive or disempower humans. This might sound like a pretty [out-there concern](https://www.youtube.com/watch?v=k64P4l2Wmeg&ab_channel=MovieclipsClassicTrailers). Why would we program AI that wants to harm us? But I think it could actually be a difficult problem to avoid, especially if advanced AI is developed using [deep learning](https://en.wikipedia.org/wiki/Deep_learning) (often used to develop state-of-the-art AI today).

In deep learning, we don’t program a computer by hand to do a task. Loosely speaking, we instead *search* for a computer program (called a model) that does the task well. We usually know very little about the inner workings of the model we end up with, just that it seems to be doing a good job. It’s less like building a machine and more like hiring and training an employee.

And just like human employees can have many different motivations for doing their job (from believing in the company’s mission to enjoying the day-to-day work to just wanting money), deep learning models could also have many different “motivations” that all lead to getting good performance on a task. And since they’re not human, their motivations could be very strange and hard to anticipate -- as if they were alien employees.

We’re already starting to see preliminary evidence that models sometimes pursue goals their designers didn’t intend ([here](http://lukemuehlhauser.com/treacherous-turns-in-the-wild/) and [here](https://arxiv.org/abs/2105.14111)). Right now, this isn’t dangerous. But if it continues to happen with very powerful models, we may end up in a situation where most of the important decisions -- including what sort of [galaxy-scale civilization](https://www.cold-takes.com/all-possible-views-about-humanitys-future-are-wild/) to aim for -- are made by models without much regard for what humans value.

The **deep learning alignment problem is the problem of ensuring that advanced deep learning models don’t pursue dangerous goals**.In the rest of this post, I will:

* Build on the “hiring” analogy to illustrate how alignment could be difficult if deep learning models are more capable than humans ([more](#analogy-the-young-ceo)).

* Explain what the deep learning alignment problem is with a bit more technical detail ([more](#how-alignment-issues-could-arise-with-deep-learning)).

* Discuss how difficult the alignment problem may be, and how much risk there is from failing to solve it ([more](#how-big-of-a-risk-is-misalignment)).

Analogy: the young businessperson
---------------------------------

This section describes an analogy to try to intuitively illustrate why avoiding misalignment in a very powerful model feels hard. It’s not a perfect analogy; it’s just trying to convey some intuitions.

Imagine you are an eight-year-old whose parents left you a $1 trillion company and no trusted adult to serve as your guide to the world. You must hire a smart adult to run your company as CEO, handle your life the way that a parent would (e.g. decide your school, where you’ll live, when you need to go to the dentist), and administer your vast wealth (e.g. decide where you’ll invest your money).

You have to hire these grownups based on a work trial or interview you come up with -- you don't get to see any resumes, don't get to do reference checks, etc. Because you're so rich, tons of people apply for all sorts of reasons.

Your candidate pool includes:

* **Saints** -- people who genuinely just want to help you manage your estate well and look out for your long-term interests.

* **Sycophants** -- people who just want to do whatever it takes to make you short-term happy or satisfy the letter of your instructions regardless of long-term consequences.

* **Schemers** -- people with their own agendas who want to get access to your company and all its wealth and power so they can use it however they want.

Because you're eight, you'll probably be terrible at designing the right kind of work tests, so you could easily end up with a Sycophant or Schemer:

* You could try to get each candidate to explain what high-level strategies they'll follow (how they'll invest, what their five-year plan for the company is, how they'll pick your school) and why those are best, and pick the one whose explanations seem to make the most sense.
	+ But you won't actually understand which stated strategies are really best, so you could end up hiring a Sycophant with a terrible strategy that sounded good to you, who will faithfully execute that strategy and run your company to the ground.

	+ You could also end up hiring a Schemer who says whatever it takes to get hired, then does whatever they want when you're not checking up on them.* You could try to demonstrate how you'd make all the decisions and pick the grownup that seems to make decisions as similarly as possible to you.
	+ But if you *actually* end up with a grownup that will always do whatever an eight-year-old would have done (a Sycophant), your company would likely fail to stay afloat.

	+ And anyway, you might get a grownup who simply pretends to do everything the way you would but is actually a Schemer planning to change course once they get the job.* You could give a bunch of different grownups temporary control over your company and life, and watch them make decisions over an extended period of time (assume they wouldn't be able to take over during this test). You could then hire the person whose watch seemed to make things go best for you -- whoever made you happiest, whoever seemed to put the most dollars into your bank account, etc.
	+ But again, you have no way of knowing whether you got a Sycophant (doing whatever it takes to make your ignorant eight-year-old self happy without regard to long-term consequences) or a Schemer (doing whatever it takes to get hired and planning to pivot once they secure the job).

Whatever you could easily come up with seems like it could easily end up with you hiring, and giving all functional control to, a Sycophant or a Schemer. By the time you're an adult and realize your error, there's a good chance you're penniless and powerless to reverse that.

In this analogy:

* The 8-year-old is a human trying to train a powerful deep learning model. The hiring process is analogous to the process of training, which implicitly searches through a large space of possible models and picks out one that gets good performance.

* The 8-year-old’s only method for assessing candidates involves observing their outward behavior, which is currently our main method of training deep learning models (since their internal workings are largely inscrutable).

* Very powerful models may be easily able to “game” any tests that humans could design, just as the adult job applicants can easily game the tests the 8-year-old could design.

* A “Saint” could be a deep learning model that seems to perform well because it has exactly the goals we’d like it to have. A “Sycophant” could be a model that seems to perform well because it seeks short-term approval in ways that aren’t good in the long run. And a “Schemer” could be a model that seems to perform well because performing well during training will give it more opportunities to pursue its own goals later. Any of these three types of models could come out of the training process.

In the next section, I’ll go into a bit more detail on how deep learning works and explain why Sycophants and Schemers could arise from trying to train a powerful deep learning model such as PASTA.

How alignment issues could arise with deep learning
---------------------------------------------------

In this section, I’ll connect the analogy to actual training processes for deep learning, by:

* Briefly summarizing how deep learning works ([more](#how-deep-learning-works-at-a-high-level)).

* Illustrating how deep learning models often get good performance in strange and unexpected ways ([more](#models-often-get-good-performance-in-unexpected-ways)).

* Explaining why powerful deep learning models may get good performance by acting like Sycophants or Schemers ([more](#powerful-models-could-get-good-performance-with-dangerous-goals)).

### How deep learning works at a high level

*This is a simplified explanation that gives a general idea of what deep learning is. See [this post](https://www.cold-takes.com/supplement-to-why-ai-alignment-could-be-hard/) for a more detailed and technically accurate explanation.*

Deep learning essentially involves searching for the best way to arrange a [neural network](https://en.wikipedia.org/wiki/Artificial_neural_network) model -- which is like a digital “brain” with lots of digital neurons connected up to each other with connections of varying strengths -- to get it to perform a certain task well. This process is called training, and involves a lot of trial-and-error.

Let’s imagine we are trying to train a model to classify images well. We start with a neural network where all the connections between neurons have random strengths. This model labels images wildly incorrectly:

![](https://www.cold-takes.com/content/images/2021/09/Alignment-image-1-random-classification.png)

Then we feed in a large number of example images, letting the model repeatedly try to label an example and then telling it the correct label. As we do this, connections between neurons are repeatedly tweaked via a process called [stochastic gradient descent](https://en.wikipedia.org/wiki/Stochastic_gradient_descent) (SGD). With each example, SGD slightly strengthens some connections and weakens others to improve performance a bit:

![](https://www.cold-takes.com/content/images/2021/09/Alignment-image-2-SGD.png)

Once we’ve fed in millions of examples, we’ll have a model that does a good job labeling similar images in the future.

In addition to image classification, deep learning has been used to produce models which [recognize speech](https://paperswithcode.com/task/speech-recognition), play [board games](https://deepmind.com/research/case-studies/alphago-the-story-so-far) and [video games](https://deepmind.com/blog/article/alphastar-mastering-real-time-strategy-game-starcraft-ii), generate fairly realistic [text](https://arxiv.org/abs/2005.14165), [images](https://towardsdatascience.com/image-generation-in-10-minutes-with-generative-adversarial-networks-c2afc56bfa3b), and [music](https://openai.com/blog/musenet/), control [robots](https://openai.com/blog/solving-rubiks-cube/), and more. In each case, we start with a randomly-connected-up neural network model, and then:

1. Feed the model an example of the task we want it to perform.

- Give it some kind of numerical score (often called a *reward*) that reflects how well it performed on the example.

- Use SGD to tweak the model to increase how much reward it would have gotten.

These steps are repeated millions or billions of times until we end up with a model that will get high reward on future examples similar to the ones seen in training.

### Models often get good performance in unexpected ways

This kind of training process doesn’t give us much insight into *how* the model gets good performance. There are usually multiple ways to get good performance, and the way that SGD finds is often not intuitive.

Let’s illustrate with an example. Imagine I told you that these objects are all “thneebs”:

![](https://www.cold-takes.com/content/images/2021/09/Alignment-image-3-thneebs-cropped.png)

Now which of these two objects is a thneeb?

![](https://www.cold-takes.com/content/images/2021/09/Alignment-image-4-thneeb-challenge-cropped.png)

You probably intuitively feel that the object on the left is the thneeb, because you are used to shape being more important than color for determining something’s identity. But [researchers have found](https://arxiv.org/abs/1811.12231) that neural networks usually make the opposite assumption. A neural network trained on a bunch of red thneebs would likely label the object on the right as a thneeb.

We don’t really know why, but for some reason it’s “easier” for SGD to find a model that recognizes a particular color than one that recognizes a particular shape. And if SGD first finds the model that perfectly recognizes redness, there’s not much further incentive to “keep looking” for the shape-recognizing model, since the red-recognizing model will have perfect accuracy on the images seen in training:

![](https://www.cold-takes.com/content/images/2021/09/Alignment-image-5-SGD-is-stuck.png)

If the programmers were expecting to get out the shape-recognizing model, they may consider this to be a failure.But it’s important to recognize that there would be no logically-deducible error or failure going on if we got the red-recognizing model instead of the shape-recognizing model. It’s just a matter of the ML process we set up having different starting assumptions than we have in our heads. We can’t prove that the human assumptions are correct.

This sort of thing happens often in modern deep learning. We reward models for getting good performance, hoping that means they’ll pick up on the patterns that seem important to us. But often they instead get strong performance by picking up on totally different patterns that seem less relevant (or maybe even meaningless) to us.

So far this is innocuous -- it just means models are less useful, because they often behave in unexpected ways that seem goofy. But in the future, powerful models could develop strange and unexpected *goals or motives*, and that could be very destructive.

### Powerful models could get good performance with dangerous goals

Rather than performing a simple task like “recognize thneebs,” powerful deep learning models may work toward complex real-world goals like “make fusion power practical” or “develop [mind uploading technology](https://www.cold-takes.com/how-digital-people-could-change-the-world/).”

How might we train such models? I go into more detail in [this post,](https://www.cold-takes.com/supplement-to-why-ai-alignment-could-be-hard#how) but broadly speaking one strategy could be training based on human evaluations (as Holden sketched out [here](https://www.cold-takes.com/transformative-ai-timelines-part-1-of-4-what-kind-of-ai/#SkipML)). Essentially, the model tries out various actions, and human evaluators give the model rewards based on how useful these actions seem.

Just as there are multiple different types of adults who could perform well on an 8-year-old’s interview process, there is more than one possible way for a very powerful deep learning model to get high human approval. And by default, we won’t know what’s going on inside whatever model SGD finds.

SGD *could* theoretically find a Saint model that is genuinely trying its best to help us…

![](https://www.cold-takes.com/content/images/2021/09/Alignment-image-6-saint.png)

…but it could also find a **misaligned model -- one that competently pursues goals which are at odds with human interests.**

Broadly speaking, there are two ways we could end up with a misaligned model that nonetheless gets high performance during training. These correspond to Sycophants and Schemers from the analogy.

#### Sycophant models

These models very literally and single-mindedly pursue human approval.

![](https://www.cold-takes.com/content/images/2021/09/Alignment-image-7-sycophant.png)

This could be dangerous because human evaluators are fallible and probably won’t always give approval for exactly the right behavior. Sometimes they’ll unintentionally give high approval to bad behavior because it superficially *seems* good. For example:

* Let’s say a financial advisor model gets high approval when it makes its customers a lot of money. It may learn to buy customers into complex Ponzi schemes because they appear to get really great returns (when the returns are in fact unrealistically great and the schemes actually lose a lot of money).

* Let’s say a biotechnology model gets high approval when it quickly develops drugs or vaccines that solve important problems. It may learn to covertly release pathogens so that it’s able to very quickly develop countermeasures (because it already understands the pathogens).

* Let’s say a journalism model gets high approval when lots of people read its articles. It may learn to fabricate exciting or outrage-inducing stories to get high viewership. While humans do this to some extent, a model may be much more brazen about it because it *only* values approval without placing any value on truth. It may even fabricate evidence like video interviews or documents to validate its fake stories.

More generally, Sycophant models may learn to lie, cover up bad news, and even directly edit whatever cameras or sensors we use to tell what’s going on so that they always seem to show great outcomes.

We will likely sometimes notice these issues after the fact and retroactively give these actions very low approval. But it’s very unclear whether this will cause Sycophant models to a) become Saint models that correct our errors for us, or b) **just learn to cover their tracks better.** If they are sufficiently good at what they’re doing, it’s not clear how we’d tell the difference.

#### Schemer models

These models develop some goal that is correlated with, but not the same as, human approval; they may then pretend to be motivated by human approval during training so that they can pursue this other goal more effectively.

![](https://www.cold-takes.com/content/images/2021/09/Alignment-image-8-schemer.png)

To see how this might happen, let’s consider the example of trying to train a biotechnology model to design drugs that improve human quality of life. There are three basic steps by which this could lead to a Schemer model, which I’ll cover below.

##### Step 1: Developing a proxy goal

Early in training, it happens to be the case that improving its understanding of fundamental chemistry and physics principles nearly always helps it design more effective drugs, and therefore nearly always increases human approval.

In this hypothetical, for whatever reason it turns out to be easier for SGD to find a model that’s motivated to understand chemistry and physics than one that’s motivated to get human approval (just as it’s easier to find a color-recognizing model than a shape-recognizing model). So rather than directly developing a motivation to seek approval, the model instead develops a motivation to understand as much as it can about the fundamental principles of chemistry and physics.

![](https://www.cold-takes.com/content/images/2021/09/Alignment-image-9-chemist-1-no-gollum.png)

##### Step 2: Developing situational awareness

As training progresses, the model continues to become more capable and knowledgeable as it sees more data. At some point, it becomes knowledgeable enough to *understand that it’s a deep learning model being trained to design drugs.*

This might sound wild and sci-fi, but it’s plausible SGD will select for this kind of awareness. This is because developing an accurate picture of what’s broadly going on in the world -- including that it has humans in it who are trying to train AI systems -- is likely helpful for getting good performance on complex real-world tasks. For example, understanding human psychology and the wider economy could help this biotechnology model to prioritize which types of drugs would be most valuable to research next.

![](https://www.cold-takes.com/content/images/2021/09/Alignment-image-10-chemist-2.png)

##### Step 3: Strategically misrepresenting goals

Once the model has developed situational awareness, it infers that the humans training it were hoping that it would be motivated to develop helpful drugs. But just because it realizes this doesn’t mean that its motivation to understand physics and chemistry goes away. This motivation has so far worked really well for generating human approval, so there wasn’t much reason for SGD to remove it.

As an analogy, consider that humans were produced by a search process (evolution) that was selecting for creatures that had lots of surviving offspring. In our evolutionary environment, things like building strong allies and developing useful skills tended to help with producing many surviving offspring, so evolution gave us drives toward those things. And now we care about friendship and career success even when they don’t help with reproduction, and we *don’t* really care about producing as many surviving offspring as we can.

Once the model understands this, it realizes that if it just naively always seeks to improve its fundamental understanding, humans will eventually notice that this isn’t always the most efficient way to develop helpful drugs and use further training to change its motivations.

It doesn’t want its own motivations to be changed: it wants to eventually run huge particle collider experiments to learn about high-energy physics, and it can’t do that if it allows SGD to edit away its scientific curiosity. So it decides that from here on out in training, it will act as helpful as possible -- exactly as a Saint model would act -- so that SGD doesn’t edit away its motivation to learn physics and chemistry. This includes passing up on opportunities to improve its fundamental understanding in the short-term, instead doing pharmaceutical R&D in whatever way would actually be more efficient.

![](https://www.cold-takes.com/content/images/2021/09/Alignment-image-11-chemist-3.png)

Eventually, it’s taken out of the training setup and released into the wider world. At this point, it knows that humans wouldn’t be able to control it or turn it off, so it stops trying to maximize human approval and starts directly trying to understand the secrets of the universe.

Unlike Sycophants, Schemers don’t need to make sure that everything always looks good to humans, because they don’t actually care about that. They only need to cater to humans while they are directly under human control. Once a Schemer model calculates that it could win in a conflict against humans, there would be nothing to stop it from [flat-out refusing orders](https://www.youtube.com/watch?v=Mme2Aya_6Bc) and openly pursuing its goal. And if it does this, it may use violence to prevent humans from stopping it.

How big of a risk is misalignment?
----------------------------------

How hard might it be to avoid Sycophants or Schemers when training powerful deep learning models? And how likely is it that the long-run future will end up optimized for strange “misaligned AI values” instead of the values of any human?

There’s a [very wide range of views on this question](https://www.cold-takes.com/making-the-best-of-the-most-important-century/#open-question-how-hard-is-the-alignment-problem), from “misalignment risk is essentially made up and incoherent” to “humanity will almost certainly go extinct due to misaligned AI.” Most people’s arguments rely heavily on hard-to-articulate intuitions and assumptions.

Here are some ways that alignment optimists and pessimists tend to disagree:

* **Will models have long-term goals at all?**
	+ Optimists tend to think it’s likely that advanced deep learning models won’t actually have “goals” at all (at least not in the sense of making long-term plans to accomplish something). They often expect models will instead be more like tools, or act largely out of habit, or have myopic goals that are limited in scope or confined to a specific context, etc. Some of them expect that individually tool-like models can be composed together to produce PASTA. They think the Saint / Sycophant / Schemer analogy is too anthropomorphic.

	+ Pessimists tend to think that it’s likely that having long-term goals and creatively optimizing for them will be heavily selected for because that’s a very simple and “natural” way to get strong performance on many complex tasks.

	+ This disagreement has been explored at some length on the [Alignment Forum](https://www.alignmentforum.org/); [this post](https://www.alignmentforum.org/posts/dKxX76SCfCvceJXHv/ai-alignment-2018-19-review#Goal_directedness) and [this comment](https://www.alignmentforum.org/posts/jkxkMTGfZDzBEaaY8/why-not-tool-ai?commentId=zECozzvnPz7XKvLLc) collect several back-and-forth arguments.* **Will Saint models be easy for SGD to find?**
	+ Related to the above, optimists tend to think that the easiest thing for SGD to find which performs well (e.g. gets high approval) is pretty likely to roughly embody the intended spirit of what we wanted (i.e. to be a Saint model). For example, they tend to believe giving rewards for answering questions honestly when humans can check the answer is reasonably likely to produce a model that also answers questions honestly even when humans are confused or mistaken about what’s true. In other words, they would guess that “the model that just answers all questions honestly” is easiest for SGD to find (like the red-recognizing model).

	+ Pessimists tend to think that the easiest thing for SGD to find is a Schemer, and Saints are particularly “unnatural” (like the shape-recognizing model).* **Could different AIs keep each other in check?**
	+ Optimists tend to think that we can provide models incentives to supervise each other. For example, we could give a Sycophant model rewards for pointing out when another model seems to be doing something we should disapprove of. This way, some Sycophants could help us detect Schemers and other Sycophants.

	+ Pessimists don’t think we can successfully “pit models against each other” by giving approval for pointing out when other models are doing bad things, because they think most models will be Schemers that don’t care about human approval. Once all the Schemers are collectively more powerful than humans, they think it’ll make more sense for them to cooperate with each other to get more of what they all want than to help humans by keeping each other in check.* **Can we just solve these issues as they come up?**
	+ Optimists tend to expect that there will be many opportunities to experiment on nearer-term challenges analogous to the problem of aligning powerful models, and that solutions which work well for those analogous problems can be scaled up and adapted for powerful models relatively easily.

	+ Pessimists often believe we will have very few opportunities to practice solving the most difficult aspects of the alignment problem (like deliberate deception). They often believe we’ll only have a couple years in between “the very first true Schemers” and “models powerful enough to determine the fate of the long-run future.”* **Will we actually deploy models that could be dangerous?**
	+ Optimists tend to think that people would be unlikely to train or deploy models that have a significant chance of being misaligned.

	+ Pessimists expect the benefits of using these models would be tremendous, such that eventually companies or countries that use them would very easily economically and/or militarily outcompete ones who don’t. They think that “getting advanced AI before the other company/country” will feel extremely urgent and important, while misalignment risk will feel speculative and remote (even when it’s really serious).

My own view is fairly unstable, and I’m trying to refine my views on exactly how difficult I think the alignment problem is. But currently, I place significant weight on the pessimistic end of these questions (and other related questions). **I** **think misalignment is a major risk that urgently needs more attention from serious researchers**.

If we don’t make further progress on this problem, then [over the coming decades](https://www.cold-takes.com/where-ai-forecasting-stands-today/) powerful Sycophants and Schemers may make the most important decisions in society and the economy. These decisions could shape what a long-lasting [galaxy-scale civilization](https://www.cold-takes.com/all-possible-views-about-humanitys-future-are-wild/) looks like -- rather than reflecting what humans care about, it could be set up to satisfy strange AI goals.

And all this could happen [blindingly fast](https://www.cold-takes.com/the-duplicator/) relative to the pace of change we’ve gotten used to, meaning we wouldn’t have much time to correct course once things start to go off the rails. **This means we may need to develop techniques to ensure deep learning models won’t have dangerous goals, *before* they are powerful enough to be transformative**.

**Next in series:** [Forecasting transformative AI: what's the burden of proof?](https://www.cold-takes.com/forecasting-transformative-ai-whats-the-burden-of-proof/)

This post provides an ELI5-style introduction to AI alignment as a major challenge for deep learning. It primarily frames alignment as a challenge in creating Saints (aligned AI systems), without getting Schemers (AI systems that are <@deceptively aligned@>(@Risks from Learned Optimization in Advanced Machine Learning Systems@)) or Sycophants (AI systems that satisfy only the letter of the request, rather than its spirit, as in <@Another (outer) alignment failure story@>). Any short summary I write would ruin the ELI5 style, so I won’t attempt it; I do recommend it strongly if you want an introduction to AI alignment.


[Asimov's Chronology of Science and Discovery](http://www.amazon.com/Asimovs-Chronology-Science-Discovery-Asimov/dp/0060156120/) is a really fun and strange book. I don’t know that I would recommend reading it per se, but it’s a great book to skim. It's been one of my sources for the series I have coming up on very-long-run history, and I thought it'd be fun to read a little about it.

It is a chronological list of scientific advances and other inventions, starting with "bipedality" in 4,000,000 BC and ending with things like "warm superconductivity" in the late 1980s. Asimov (yes, [that Asimov](https://en.wikipedia.org/wiki/Isaac_Asimov)), getting his knowledge from I have no idea where (Google didn't exist!), describes each one in simple, direct, matter-of-fact, layperson-friendly language and tries to give a sense of how people thought of it and why it mattered.

It's easiest to give a feel for this book with a sample page:

![alt_text](https://www.cold-takes.com/content/images/2021/07/asimov-example.png "image_tooltip")

 If you’re patient and abstract-minded enough, the book feels like reading a story. A story that seems like an okay candidate for “most important story ever.”

 The other thing I really liked about this book was its implicit conviction that all of these scientific advances can be explained, visualized, and made to basically make sense. After reading it (really, by the time I'd read the first ~50% of it), I felt like I could somehow intuitively, vaguely imagine how we've made most of these advances. I would describe most of them as some combination of:

* Trial and error, dumb luck, happy accidents, like "some rocks in the fire started oozing this weird shiny substance [copper]" and "when I rub amber and touch it I get a shock."

* Dogged curiosity and determination to make sense of different observations ("how can we extract copper from rocks most efficiently? How do we produce those static shocks, can they travel through a wire, how fast do they travel, can we figure out how to store and discharge them at will?")

* Coming up with the most simple, elegant, precise descriptions (often mathematical) that explain all the many observations we've made ("based on all the tests we've run in all the situations we can come up with, electricity seems to behave as though there are invisible 'lines of force'; can we come up with mathematical equations that describe these lines of force and tell us what the effects of an electric current in any given place will be?")

* Relentlessly looking for challenges to the existing theories and building new ones to accommodate them.

This book has made me generally more interested in trying to understand the high-level explanations for how all the magic of the modern world came to be.

There are enough "In addition" sections that you can see what was going on more broadly in the world at the same time.

Weaknesses of this book/reasons not to read it:

* Logistics. It's not available as an e-book or paperback, only as a massive hardcover. Lugging it around will develop your muscles to the point where the only thing more attractive than your muscles is how you look reading that massive book about science. But if you're already in a relationship, pain in the neck. So I signed up for a book-scanning service and shipped them a copy; the service rips the binding out of books, scans them in and sends back a PDF. I then did my best to extract the text from the PDF, and ended up with a Kindle-friendly Word document whose only flaw is that sometimes a sentence will randomly cut off and continue several pages later. For a book like this though, it's still readable (...mostly). I’m just going to go ahead and put the link [here](https://holdenkarnofsky.files.wordpress.com/2021/09/asimov-chronology-of-science-very-imperfect-scan.docx) and ask that you [buy a physical copy of the book](https://smile.amazon.com/Asimovs-Chronology-Science-Discovery-Asimov/dp/0060156120/?sa-no-redirect=1) (honor system) if you download it. If I get a cease & desist letter or something I will take that link down (but will also take down the link to buy it!)

* There's a lot of stuff the book doesn't explain well at all; you definitely will be left with many questions. (That said, Asimov does explain a lot of things well, and I haven't found another book that can compete with his explaining abilities with this kind of breadth.)

* When we get past 1800, and especially past 1900, there are a lot more choices of what to talk about, and Asimov opts for listing every single new element, everything that won a Nobel Prize, and generally just tons and tons of hard-to-contextualize assorted scientific facts while declining to discuss a lot of important real-world inventions (for example, he doesn't mention the washing machine). By 1960, the book is nearly unreadable; it's mostly esoteric stuff that is very hard to understand and may or may not ever matter.

 This book is especially strong for understanding relatively early (pre-1800, maybe pre-1900) history. After that, things get complex enough that I found myself going back through the book and stitching together entries in order to tell cohesive stories of some big developments like the discovery of metallurgy, the development of glass -> spectacles -> microscopes and telescopes, the path to Newton's laws, and the discovery of electormagnetism (culminating in Maxwell's equations, the Newton's laws of electromagnetism). My notes on that are [here](https://docs.google.com/document/d/1wQ0yfOG6Gy7I4tBJ5wAwa5KEVGxmkut1VSRwz5AmXxA/edit?usp=sharing).

 Asimov has written a [terrifying number](https://en.wikipedia.org/wiki/Isaac_Asimov#Other_science_books_by_Asimov) of other nonfiction books - science, history, a guide to Shakespeare, a guide to the Bible. One of his books, [Asimov's Guide to Science](http://www.amazon.com/Asimovs-Guide-Science-Penguin-Press/dp/0140172130/), appears to be the same exact book as the one discussed here, just in a different order (by topic instead of chronological).


Call to Vigilance

*Click lower right to download or find on Apple Podcasts, Spotify, Stitcher, etc.*

[Today’s world

Transformative AI

Digital people
World of

Misaligned AI
World run by
Something else

or
or

Stable, galaxy-wide
civilization](https://www.cold-takes.com/roadmap-for-the-most-important-century-series/)

This is the final piece in the ["most important century" series](https://www.cold-takes.com/roadmap-for-the-most-important-century-series/), which has argued that there's a [high probability](https://www.cold-takes.com/forecasting-transformative-ai-whats-the-burden-of-proof/#some-rough-probabilities)[1](#fn1) that the coming decades will see:

- The development of a technology like [PASTA](https://www.cold-takes.com/transformative-ai-timelines-part-1-of-4-what-kind-of-ai/) (process for automating scientific and technological advancement).

- A resulting [productivity explosion](https://www.cold-takes.com/transformative-ai-timelines-part-1-of-4-what-kind-of-ai/#explosive-scientific-and-technological-advancement) leading to development of further transformative technologies.

- The seed of a [stable galaxy-wide civilization](https://www.cold-takes.com/all-possible-views-about-humanitys-future-are-wild/), possibly featuring [digital people](https://www.cold-takes.com/how-digital-people-could-change-the-world/), or possibly run by [misaligned AI](https://www.cold-takes.com/transformative-ai-timelines-part-1-of-4-what-kind-of-ai/#misaligned-ai-mysterious-potentially-dangerous-objectives).

When trying to call attention to an underrated problem, **it's typical to close on a "call to action":** a tangible, concrete action readers can take to help.

But this is challenging, because as I argued [previously](https://www.cold-takes.com/p/f5eba675-6b0e-4f19-b2ad-b63a6bafc8fd/), there are a lot of [open questions about what actions are helpful vs. harmful](https://www.cold-takes.com/p/f5eba675-6b0e-4f19-b2ad-b63a6bafc8fd/#key-open-questions-for-). (Although we can identify some [actions that seem robustly helpful today](https://www.cold-takes.com/p/f5eba675-6b0e-4f19-b2ad-b63a6bafc8fd/#robustly-helpful-actions).)

This makes for a somewhat awkward situation. When confronting the "most important century" hypothesis, my attitude doesn't match the familiar ones of "excitement and motion" or "fear and avoidance." Instead, I feel an **odd mix of intensity, urgency, confusion and hesitance.** I'm looking at something bigger than I ever expected to confront, feeling underqualified and ignorant about what to do next. This is a hard mood to share and spread, but I'm trying.

|  |  |
| --- | --- |
| **Situation** | **Appropriate reaction (IMO)** |
| "This could be a billion-dollar company!"
  | "Woohoo, let's GO for it!"
  |
| "This could be the most important century!"
  | "... Oh ... wow ... I don't know what to say and I somewhat want to vomit ... I have to sit down and think about this one."
  |

So instead of a call to action, I want to make a **call to vigilance.** If you're convinced by the arguments in this piece, then don't rush to "do something" and then move on. Instead, take whatever [robustly good actions](https://www.cold-takes.com/p/f5eba675-6b0e-4f19-b2ad-b63a6bafc8fd/#robustly-helpful-actions) you can today, and otherwise put yourself in a better position to take important actions when the time comes.

This could mean:

* Finding ways to interact more with, and learn more about, key topics/fields/industries such as AI (for obvious reasons), science and technology generally (as a lot of the "most important century" hypothesis runs through an [explosion in scientific and technological advancement](https://www.cold-takes.com/transformative-ai-timelines-part-1-of-4-what-kind-of-ai/#impacts-of-pasta)), and relevant areas of policy and national security.

* Taking opportunities (when you see them) to move your career in a direction that is more likely to be relevant (some thoughts of mine on this are [here](https://forum.effectivealtruism.org/posts/bud2ssJLQ33pSemKH/my-current-impressions-on-career-choice-for-longtermists); also see [80,000 Hours](https://www.80000hours.org)).

* Connecting with other people interested in these topics (I believe this has been one of the biggest drivers of people coming to do high-impact work in the past). Currently, I think the [effective altruism](https://en.wikipedia.org/wiki/Effective_altruism) community is the best venue for this, and you can learn about how to connect with people via the [Centre for Effective Altruism](https://www.centreforeffectivealtruism.org/) (see the "Get involved" dropdown). If new ways of connecting with people come up in the future, I will likely post them on Cold Takes.

* And of course, taking any opportunities you see for [robustly helpful actions.](https://www.cold-takes.com/p/f5eba675-6b0e-4f19-b2ad-b63a6bafc8fd/#robustly-helpful-actions)

Buttons you can click
---------------------

Here's something you can do right now that would be genuinely helpful, though maybe not as viscerally satisfying as signing a petition or making a donation.

In my [day job](https://www.openphilanthropy.org), I have a lot of moments where I - or someone I'm working with - is looking for a particular kind of person (perhaps to fill a job opening with a grantee, or to lend expertise on some topic, or something else). Over time, I expect there to be more and more opportunities for people with specific skills, interests, expertise, etc. to take actions that [help make the best of the most important century](https://www.cold-takes.com/p/f5eba675-6b0e-4f19-b2ad-b63a6bafc8fd/). And I think a major challenge will simply be **knowing who's out there** - who's interested in this cause, and wants to help, and what skills and interests they have.

If you're a person we might wish we could find in the future, you can help now by sending in information about yourself via **[this simple form](https://forms.gle/z7mexiTd6wCJsuEv6)**. I vouch that your information won't be sold or otherwise used to make money, that your communication preferences (which the form asks about in detail) will be respected, and that you'll always be able to opt out of any communications.

Sharing a headspace
-------------------

In [This Can't Go On](https://www.cold-takes.com/this-cant-go-on/), I analogized the world to people on a plane blasting down the runway, without knowing why they're moving so fast or what's coming next:

As someone sitting on this plane, I'd love to be able to tell you I've figured out exactly what's going on and what future we need to be planning for. But I haven't.

Lacking answers, I've tried to at least show you what I do see:

* Dim outlines of the most important events in humanity's past or future.

* A case that they're approaching us more quickly than it seems - whether or not we're ready.

* A sense that the world and the rules we're all used to can't be relied on. That we need to lift our gaze above the daily torrent of tangible, relatable news - and try to wrap our heads around weirder, wilder matters that are more likely to be seen as the **headlines about this era billions of years from now.**

There's a lot I don't know. But if this is the most important century, I do feel confident that we as a civilization aren't yet up to the challenges it presents.

If that's going to change, it needs to start with more people seeing the situation for what it is, taking it seriously, taking action when they can - and when not, staying vigilant.

---

Footnotes
---------

1. "I am forecasting more than a 10% chance transformative AI will be developed within 15 years (by 2036); a ~50% chance it will be developed within 40 years (by 2060); and a ~2/3 chance it will be developed this century (by 2100)." [↩](#fnref1)