How to make the best of the most important century?

*Audio also available by searching Stitcher, Spotify, Google Podcasts, etc. for "Cold Takes Audio"*

[Today’s world

Transformative AI

Digital people
World of

Misaligned AI
World run by
Something else

or
or

Stable, galaxy-wide
civilization](https://www.cold-takes.com/roadmap-for-the-most-important-century-series/)

Previously in the ["most important century" series](https://www.cold-takes.com/roadmap-for-the-most-important-century-series/), I've argued that there's a high probability[1](#fn1) that the coming decades will see:

- The development of a technology like [PASTA](https://www.cold-takes.com/transformative-ai-timelines-part-1-of-4-what-kind-of-ai/) (process for automating scientific and technological advancement).

- A resulting [productivity explosion](https://www.cold-takes.com/transformative-ai-timelines-part-1-of-4-what-kind-of-ai/#explosive-scientific-and-technological-advancement) leading to development of further transformative technologies.

- The seed of a [stable galaxy-wide civilization](https://www.cold-takes.com/all-possible-views-about-humanitys-future-are-wild/), possibly featuring [digital people](https://www.cold-takes.com/how-digital-people-could-change-the-world/), or possibly run by [misaligned AI](https://www.cold-takes.com/transformative-ai-timelines-part-1-of-4-what-kind-of-ai/#misaligned-ai-mysterious-potentially-dangerous-objectives).

Is this an optimistic view of the world, or a pessimistic one? To me, it's both and neither, because **this set of events could end up being very good or very bad for the world, depending on the details of how it plays out.**

When I talk about being in the "most important century," I don't just mean that significant events are going to occur. I mean that we, the people living in this century, have the chance to have a huge impact on huge numbers of people to come - if we can make sense of the situation enough to find helpful actions.

But it's also important to understand why that's a big "if" - why the most important century presents a **challenging strategic picture, such that many things we can do might make things better or worse (and it's hard to say which).**

In this post, I will **present two contrasting frames for how to make the best of the most important century:**

* The **"Caution"** frame. In this frame, many of the worst outcomes come from developing something like [PASTA](https://www.cold-takes.com/transformative-ai-timelines-part-1-of-4-what-kind-of-ai/) in a way that is too fast, rushed, or reckless. We may need to achieve (possibly global) coordination in order to mitigate pressures to race, and take appropriate care. ([Caution](#the-caution-frame))

* The **"Competition"** frame. This frame focuses not on *how and when* [PASTA](https://www.cold-takes.com/transformative-ai-timelines-part-1-of-4-what-kind-of-ai/) is developed, but *who* (which governments, which companies, etc.) is first in line to benefit from the resulting productivity explosion. ([Competition](#the-competition-frame))

* People who take the "caution" frame and people who take the "competition" frame often favor **very different, even contradictory** actions. Actions that look important to people in one frame often look actively harmful to people in the other.
	+ I worry that the "competition" frame will be overrated by default, and discuss why below. ([More](#why-i-fear-))

	+ To gain more clarity on how to weigh these frames and what actions are most likely to be helpful, we need more progress on **open questions** about the size of different types of risks from transformative AI. ([Open questions](#key-open-questions-for-))* In the meantime, there are some **robustly helpful actions** that seem likely to improve humanity's prospects regardless. ([Robustly helpful actions](#robustly-helpful-actions))

The "caution" frame
-------------------

I've argued for a good chance that this century will see a transition to a world where [digital people](https://www.cold-takes.com/how-digital-people-could-change-the-world/) or [misaligned AI](https://www.cold-takes.com/transformative-ai-timelines-part-1-of-4-what-kind-of-ai/#misaligned-ai-mysterious-potentially-dangerous-objectives) (or something else very different from today's humans) are the major force in world events.

The "caution" frame emphasizes that **some types of transition seem better than others.** Listed in order from worst to best:

### Worst: Misaligned AI

I discussed this possibility [previously](https://www.cold-takes.com/transformative-ai-timelines-part-1-of-4-what-kind-of-ai/#misaligned-ai-mysterious-potentially-dangerous-objectives), drawing on a number of other and more thorough discussions.[2](#fn2) The basic idea is that AI systems could end up with objectives of their own, and could seek to expand throughout space fulfilling these objectives. Humans, and/or all humans value, could be sidelined (or driven extinct, if we'd otherwise get in the way).

### Next-worst:[3](#fn3) Adversarial Technological Maturity

If we get to the point where there are digital people and/or (non-misaligned) AIs that can copy themselves without limit, and expand throughout space, there might be intense pressure to move - and multiply (via copying) - as fast as possible in order to gain more influence over the world. This might lead to different countries/coalitions furiously trying to outpace each other, and/or to outright military conflict, knowing that a lot could be at stake in a short time.

I would expect this sort of dynamic to risk a lot of the galaxy ending up in a bad state.[4](#fn4)

One such bad state would be "permanently under the control of a single (digital) person (and/or their copies)." Due to the potential of digital people to create [stable civilizations](https://www.cold-takes.com/how-digital-people-could-change-the-world/#lock-in), it seems that a given totalitarian regime could end up permanently entrenched across substantial parts of the galaxy.

People/countries/coalitions who *suspect each other* of posing this sort of danger - of potentially establishing stable civilizations under their control - might compete and/or attack each other early on to prevent this. This could lead to war with difficult-to-predict outcomes (due to the difficult-to-predict technological advancements that PASTA could bring about).

### Second-best: Negotiation and governance

Countries might prevent this sort of [Adversarial Technological Maturity](#next-worst-adversarial-technological-maturity) dynamic by planning ahead and negotiating with each other. For example, perhaps each country - or each person - could be allowed to create a certain number of digital people (subject to human rights protections and other regulations), limited to a certain region of space.

It seems there are a huge range of different potential specifics here, some much more good and just than others.

### Best: Reflection

The world could achieve a high enough level of coordination to *delay* any irreversible steps (including kicking off an [Adversarial Technological Maturity](#next-worst-adversarial-technological-maturity) dynamic).

There could then be something like what Toby Ord (in [The Precipice](https://smile.amazon.com/Precipice-Existential-Risk-Future-Humanity-ebook/dp/B07V9GHKYP/)) calls the "Long Reflection":[5](#fn5) a sustained period in which people could collectively decide upon goals and hopes for the future, ideally representing the most fair available compromise between different perspectives. Advanced technology could imaginably help this go much better than it could today.[6](#fn6)

There are limitless questions about how such a "reflection" would work, and whether there's really any hope that it could reach a reasonably good and fair outcome. Details like "what sorts of digital people are created first" could be enormously important. There is currently little discussion of this sort of topic.[7](#fn7)

### Other

There are probably many possible types of transitions I haven't named here.

### The role of caution

If the above ordering is correct, then the future of the galaxy looks better to the extent that:

* [Misaligned AI](#worst-misaligned-ai) is avoided: powerful AI systems act to help humans, rather than pursuing objectives of their own.

* [Adversarial Technological Maturity](#next-worst-adversarial-technological-maturity) is avoided. This likely means that people do not deploy advanced AI systems, or the technologies they could bring about, in adversarial ways (unless this ends up necessary to prevent something worse).

* Enough coordination is achieved so that key players can "take their time," and [Reflection](#best-reflection) becomes a possibility.

Ideally, everyone with the potential to build something [PASTA](https://www.cold-takes.com/transformative-ai-timelines-part-1-of-4-what-kind-of-ai/)-like would be able to pour energy into building something safe (not misaligned), and carefully planning out (and negotiating with others on) how to roll it out, without a rush or a race. With this in mind, perhaps we should be doing things like:

* Working to improve trust and cooperation between major world powers. Perhaps via AI-centric versions of [Pugwash](https://en.wikipedia.org/wiki/Pugwash_Conferences_on_Science_and_World_Affairs) (an international conference aimed at reducing the risk of military conflict), perhaps by pushing back against hawkish foreign relations moves.

* Discouraging governments and investors from shoveling money into AI research, encouraging AI labs to thoroughly consider the implications of their research before publishing it or scaling it up, etc. Slowing things down in this manner could buy more time to do research on avoiding [misaligned AI](#worst-misaligned-ai), more time to build trust and cooperation mechanisms, more time to generally gain strategic clarity, and a lower likelihood of the [Adversarial Technological Maturity](#next-worst-adversarial-technological-maturity) dynamic.

The "competition" frame
-----------------------

(Note: there's some potential for confusion between the "competition" idea and the [Adversarial Technological Maturity](#next-worst-adversarial-technological-maturity) idea, so I've tried to use very different terms. I spell out the contrast in a footnote.[8](#fn8))

The "competition" frame focuses **less on how the transition to a radically different future happens, and more on who's making the key decisions as it happens.**

* If something like [PASTA](https://www.cold-takes.com/transformative-ai-timelines-part-1-of-4-what-kind-of-ai/) is developed primarily (or first) in country X, then the government of country X could be making a lot of crucial decisions about whether and how to regulate a potential explosion of new technologies.

* In addition, the people and organizations leading the way on AI and other technology advancement at that time could be especially influential in such decisions.

This means it could matter enormously "who leads the way on transformative AI" - which country or countries, which people or organizations.

* Will the governments leading the way on transformative AI be authoritarian regimes?

* Which governments are most likely to (effectively) have a reasonable understanding of the risks and stakes, when making key decisions?

* Which governments are least likely to try to use advanced technology for entrenching the power and dominance of one group? (Unfortunately, I can't say there are any that I feel great about here.) Which are most likely to leave the possibility open for something like "avoiding [locked-in](https://www.cold-takes.com/how-digital-people-could-change-the-world/#lock-in) outcomes, leaving time for general progress worldwide to raise the odds of a good outcome for everyone possible?"

* Similar questions apply to the people and organizations leading the way on transformative AI. Which ones are most likely to push things in a positive direction?

Some people feel that we can make confident statements today about which specific countries, and/or which people and organizations, we should hope lead the way on transformative AI. These people might advocate for actions like:

* Increasing the odds that the first PASTA systems are built in countries that are e.g. less authoritarian, which could mean e.g. pushing for more investment and attention to AI development in these countries.

* Supporting and trying to speed up AI labs run by people who are likely to make wise decisions (about things like how to engage with governments, what AI systems to publish and deploy vs. keep secret, etc.)

Why I fear "competition" being overrated, relative to "caution"
---------------------------------------------------------------

By default, I expect a lot of people to gravitate toward the "competition" frame rather than the "caution" frame - for reasons that I don't think are great, such as:

* I think people naturally get more animated about "helping the good guys beat the bad guys" than about "helping all of us avoid getting a universally bad outcome, for impersonal reasons such as 'we designed sloppy AI systems' or 'we created a dynamic in which haste and aggression are rewarded.'"

* I expect people will tend to be overconfident about which countries, organizations or people they see as the "good guys."

* Embracing the "competition" frame tends to point toward taking actions - such as working to speed up a particular country's or organization's AI development - that are lucrative, exciting and naturally easy to feel energy for. Embracing the "caution" frame is much less this way.

* The biggest concerns that the "caution" frame focuses on - [Misaligned AI](#worst-misaligned-ai) and [Adversarial Technological Maturity](#next-worst-adversarial-technological-maturity) - are a bit abstract and hard to wrap one's head around. In many ways they seem to be the highest-stakes risks, but it's easier to be viscerally scared of "falling behind countries/organizations/people that scare me" than to be viscerally scared of something like "Getting a bad outcome for the long-run future of the galaxy because we rushed things this century."
	+ I think [Misaligned AI](#worst-misaligned-ai) is a particularly hard risk for many to take seriously. It sounds wacky and sci-fi-like; people who worry about it tend to be interpreted as picturing something like The Terminator, and it can be hard for their more detailed concerns to be understood.

	+ I'm hoping to run more posts in the future that help give an intuitive sense for why I think Misaligned AI is a real risk.

So for the avoidance of doubt, I'll state that I think the "caution" frame has an awful lot going for it. In particular, **[Misaligned AI](#worst-misaligned-ai) and [Adversarial Technological Maturity](#next-worst-adversarial-technological-maturity) seem a *lot* worse than other potential transition types,** and both seem like things that have a real chance of making the entire future of our species (and successors) much worse than they could be.

I worry that too much of the "competition" frame will lead to downplaying misalignment risk and rushing to deploy unsafe, unpredictable systems, which could have many negative consequences.

With that said, **I put serious weight on both frames.** I remain quite uncertain overall about which frame is more important and helpful (if either is).

Key open questions for "caution" vs. "competition"
--------------------------------------------------

People who take the "caution" frame and people who take the "competition" frame often favor **very different, even contradictory actions**. Actions that look important to people in one frame often look actively harmful to people in the other.

For example, people in the "competition" frame often favor moving forward as fast as possible on developing more powerful AI systems; for people in the "caution" frame, haste is one of the main things to avoid. People in the "competition" frame often favor adversarial foreign relations, while people in the "caution" frame often want foreign relations to be more cooperative.

(That said, this dichotomy is a simplification. Many people - including myself - resonate with both frames. And either frame could imply actions normally associated with the other; for example, you might take the "caution" frame but feel that haste is needed now in order to establish one country with a clear enough lead in AI that it can then take its time, prioritize avoiding [misaligned AI](#worst-misaligned-ai), etc.)

I wish I could confidently tell you how much weight to put on each frame, and what actions are most likely to be helpful. But I can't. I think we would have more clarity if we had better answers to some key open questions:

### Open question: how hard is the alignment problem?

The path to the future that seems worst is [Misaligned AI](#worst-misaligned-ai), in which AI systems end up with non-human-compatible objectives of their own and seek to fill the galaxy according to those objectives. How seriously should we take this risk - how hard will it be to avoid this outcome? **How hard will it be to solve the "alignment problem,"** which essentially means having the technical ability to build systems that won't do this?[9](#fn9)

* Some people believe that the alignment problem will be formidable; that our only hope of solving it comes in a world where we have enormous amounts of time and aren't in a race to deploy advanced AI; and that avoiding the "Misaligned AI" outcome should be by far the dominant consideration for the most important century. These people tend to heavily favor the "caution" interventions described above: they believe that rushing toward AI development raises our already-substantial risk of the worst possible outcome.

* Some people believe it will be easy, and/or that the whole idea of "misaligned AI" is misguided, silly, or even incoherent - planning for an overly specific future event. These people often are more interested in the "competition" interventions described above: they believe that advanced AI will probably be used effectively by whatever country (or in some cases smaller coalition or company) develops it first, and so the question is who will develop it first.

* And many people are somewhere in between.

The spread here is extreme. For example, see [these results](https://www.lesswrong.com/posts/QvwSr5LsxyDeaPK5s/existential-risk-from-ai-survey-results) from an informal "two-question survey [sent] to ~117 people working on long-term AI risk, asking about the level of existential risk from 'humanity not doing enough technical AI safety research' and from 'AI systems not doing/optimizing what the people deploying them wanted/intended.'" (As the scatterplot shows, people gave similar answers to the two questions.)

![](https://www.cold-takes.com/content/images/size/w1140/2021/09/ai-alignment-views-chart.png)

We have respondents who think there's a <5% chance that alignment issues will drastically reduce the goodness of the future; respondents who think there's a >95% chance; and just about everything in between.[10](#fn10) My sense is that this is a fair representation of the situation: even among the few people who have spent the most time thinking about these matters, there is practically no consensus or convergence on how hard the alignment problem will be.

I hope that over time, the field of people doing research on AI alignment[11](#fn11) will grow, and as both AI and AI alignment research advance, we will gain clarity on the difficulty of the AI alignment problem. This, in turn, could give more clarity on prioritizing "caution" vs. "competition."

### Other open questions

Even if we had clarity on the difficulty of the alignment problem, a lot of thorny questions would remain.

Should we be expecting transformative AI within the next 10-20 years, or much later? Will the leading AI systems go from very limited to very capable quickly ("hard takeoff") or gradually ("slow takeoff")?[12](#fn12) Should we hope that government projects play a major role in AI development, or that transformative AI primarily emerges from the private sector? Are some governments more likely than others to work toward transformative AI being used carefully, inclusively and humanely? What should we hope a government (or company) literally *does* if it gains the ability to dramatically accelerate scientific and technological advancement via AI?

With these questions and others in mind, it's often very hard to look at some action - like starting a new AI lab, advocating for more caution and safeguards in today's AI development, etc. - and say whether it raises the likelihood of good long-run outcomes.

Robustly helpful actions
------------------------

Despite this state of uncertainty, here are a few things that do seem clearly valuable to do today:

**Technical research on the alignment problem.** Some researchers work on building AI systems that can get "better results" (winning more board games, classifying more images correctly, etc.) But a smaller set of researchers works on things like:

* [Training AI systems to incorporate human feedback into how they perform summarization tasks](https://openai.com/blog/learning-to-summarize-with-human-feedback/), so that the AI systems reflect hard-to-define human preferences - something it may be important to be able to do in the future.

* [Figuring out how to understand "what AI systems are thinking and how they're reasoning,"](https://openai.com/blog/microscope/) in order to make them less mysterious.

* [Figuring out how to stop AI systems from making extremely bad judgments on images designed to fool them](https://ai.googleblog.com/2018/09/introducing-unrestricted-adversarial.html), and other work focused on helping avoid the "worst case" behaviors of AI systems.

* [Theoretical work](https://alignmentresearchcenter.org/) on how an AI system might be very advanced, yet not be unpredictable in the wrong ways.

This sort of work could both reduce the risk of the [Misaligned AI](#worst-misaligned-ai) outcome - and/or lead to more clarity on just how big a threat it is. Some takes place in academia, some at AI labs, and some at specialized organizations.

**Pursuit of strategic clarity:** doing research that could address other crucial questions (such as those listed [above](#other-open-questions)), to help clarify what sorts of immediate actions seem most useful.

**Helping governments and societies become, well, nicer.** Helping Country X get ahead of others on AI development could make things better or worse, for reasons given above. But it seems robustly good to work toward a Country X with better, more inclusive values, and a government whose key decision-makers are more likely to make thoughtful, good-values-driven decisions.

**Spreading ideas and building communities.** Today, it seems to me that the world is **extremely short on people who share certain basic expectations and concerns**, such as:

* Believing that AI research could lead to rapid, radical changes of the [extreme kind laid out here](https://www.cold-takes.com/transformative-ai-timelines-part-1-of-4-what-kind-of-ai/#impacts-of-pasta) (well beyond things like e.g. increasing unemployment).

* Believing that the alignment problem (discussed [above](#open-question-how-hard-is-the-alignment-problem)) is at least plausibly a real concern, and taking [the "caution" frame](#the-caution-frame) seriously.

* Looking at the whole situation through a lens of "Let's get the best outcome possible for the whole world over the long future," as opposed to more common lenses such as "Let's try to make money" or "Let's try to ensure that my home country leads the world in AI research."

I think it's very valuable for there to be more people with this basic lens, particularly working for AI labs and governments. If and when we have more strategic clarity about what actions could maximize the odds of the "most important century" going well, I expect such people to be relatively well-positioned to be helpful.

A number of organizations and people have worked to expose people to the lens above, and help them meet others who share it. I think a good amount of progress (in terms of growing communities) has come from this.

**Donating?** One can donate today to places like [this](https://funds.effectivealtruism.org/funds/far-future). But I need to admit that very broadly speaking, there's no easy translation right now between "money" and "improving the odds that the most important century goes well." It's not the case that if one simply sent, say, $1 trillion to the right place, we could all breathe easy about challenges like the alignment problem and [risks of digital dystopias](https://www.cold-takes.com/how-digital-people-could-change-the-world/#would-these-impacts-be-a-good-or-bad-thing).

It seems to me that we - as a species - are currently terribly short on people who are paying *any* attention to the most important challenges ahead of us, and haven't done the work to have good strategic clarity about what tangible actions to take. **We can't solve this problem by throwing money at it.**[13](#fn13) **First, we need to take it more seriously and understand it better.**

**Next (and last) in series:** [Call to Vigilance](https://www.cold-takes.com/call-to-vigilance/)

---

Footnotes
---------

1. From [Forecasting Transformative AI: What's the Burden of Proof?](https://www.cold-takes.com/forecasting-transformative-ai-whats-the-burden-of-proof/#some-rough-probabilities): "I am forecasting more than a 10% chance transformative AI will be developed within 15 years (by 2036); a ~50% chance it will be developed within 40 years (by 2060); and a ~2/3 chance it will be developed this century (by 2100)."

 Also see [Some additional detail on what I mean by "most important century."](https://www.cold-takes.com/some-additional-detail-on-what-i-mean-by-most-important-century/) [↩](#fnref1)- These include the books [Superintelligence](https://smile.amazon.com/Superintelligence-Dangers-Strategies-Nick-Bostrom-ebook/dp/B00LOOCGB2/), [Human Compatible](https://smile.amazon.com/Human-Compatible-Artificial-Intelligence-Problem-ebook/dp/B07N5J5FTS), [Life 3.0](https://smile.amazon.com/Life-3-0-Being-Artificial-Intelligence-ebook/dp/B06WGNPM7V), and [The Alignment Problem](https://smile.amazon.com/Alignment-Problem-Machine-Learning-Values-ebook/dp/B085T55LGK/). The shortest, most accessible presentation I know of is [The case for taking AI seriously as a threat to humanity](https://www.vox.com/future-perfect/2018/12/21/18126576/ai-artificial-intelligence-machine-learning-safety-alignment) (Vox article by Kelsey Piper). This [report on existential risk from power-seeking AI](https://www.alignmentforum.org/posts/HduCjmXTBD4xYTegv/draft-report-on-existential-risk-from-power-seeking-ai), by Open Philanthropy's Joe Carlsmith, lays out a detailed set of premises that would collectively imply the problem is a serious one. [↩](#fnref2)- The order of goodness isn't absolute, of course. There are versions of "Adversarial Technological Maturity" that could be worse than "Misaligned AI" - for example, if the former results in power going to those who deliberately inflict suffering. [↩](#fnref3)- Part of the reason for this is that faster-moving, less-careful parties could end up quickly outnumbering others and determining the future of the galaxy. There is also a longer-run risk discussed in Nick Bostrom's [The Future of Human Evolution](https://www.nickbostrom.com/fut/evolution.html); also see [this discussion](https://slatestarcodex.com/2014/07/13/growing-children-for-bostroms-disneyland/) of Bostrom's ideas on Slate Star Codex, though also see [this piece by Carl Shulman](http://reflectivedisequilibrium.blogspot.com/2012/09/spreading-happiness-to-stars-seems.html) arguing that this dynamic is unlikely to result in total elimination of nice things. [↩](#fnref4)- See page 191. [↩](#fnref5)- E.g., see [this section](https://www.cold-takes.com/how-digital-people-could-change-the-world/#social-science) of [Digital People Would Be An Even Bigger Deal](https://www.cold-takes.com/how-digital-people-could-change-the-world/). [↩](#fnref6)- One relevant paper: [Public Policy and Superintelligent AI: A Vector Field Approach](https://nickbostrom.com/papers/aipolicy.pdf) by Bostrom, Dafoe and Flynn. [↩](#fnref7)- [Adversarial Technological Maturity](#next-worst-adversarial-technological-maturity) refers to a world in which highly advanced technology has **already been developed,** likely with the help of AI, and different coalitions are vying for influence over the world. By contrast, "Competition" refers to a strategy for how to behave ***before* the development of advanced AI**. One might imagine a world in which some government or coalition takes a "competition" frame, develops advanced AI long before others, and then makes a series of good decisions that *prevent* Adversarial Technological Maturity. (Or conversely, a world in which failure to do well at "competition" raises the risks of Adversarial Technological Maturity.) [↩](#fnref8)- See definitions of this problem at [Wikipedia](https://en.wikipedia.org/wiki/AI_control_problem) and [Paul Christiano's Medium](https://ai-alignment.com/clarifying-ai-alignment-cec47cd69dd6). [↩](#fnref9)- A more detailed, private survey done for [this report](https://www.alignmentforum.org/posts/HduCjmXTBD4xYTegv/draft-report-on-existential-risk-from-power-seeking-ai), asking about the probability of "doom" before 2070 due to the type of problem discussed in the report, got answers ranging from <1% to >50%. In my opinion, there are very thoughtful people who have seriously considered these matters at both ends of that range. [↩](#fnref10)- Some example technical topics [here](https://www.openphilanthropy.org/focus/global-catastrophic-risks/potential-risks-advanced-artificial-intelligence/the-open-phil-ai-fellowship#examples). [↩](#fnref11)- Some discussion of this topic here: [Distinguishing definitions of takeoff - AI Alignment Forum](https://www.alignmentforum.org/posts/YgNYA6pj2hPSDQiTE/distinguishing-definitions-of-takeoff) [↩](#fnref12)- Some more thought on "when money isn't enough" at [this old GiveWell post](https://blog.givewell.org/2013/08/29/we-cant-simply-buy-capacity/).. [↩](#fnref13)


One Cold Link: “The Past and Future of Economic Growth: A Semi-Endogenous Perspective”

[The Past and Future of Economic Growth: A Semi-Endogenous Perspective](https://www.nber.org/papers/w29126) is a growth economics paper by Charles I. Jones, asking big questions about what has powered economic growth[1](#fn1) over the last 50+ years, and what the long-run prospects for continued economic growth look like. I think the ideas in it will be unfamiliar to most people, but they make a good amount of intuitive sense; and if true, they seem very important for thinking about the long-run future of the economy.

Key quotes, selected partly for comprehensibility to laypeople and ordered so that you should be able to pick up the gist of the paper by reading them:

“Where do ideas come from? The history of innovation is very clear on this point: new ideas are discovered through the hard work and serendipity of people. Just as more autoworkers will produce more cars, more researchers and innovators will produce more new ideas … The surprise is that we are now done; that is all we need for the semi-endogenous model of economic growth. People produce ideas and ... those ideas raise everyone’s income ... the growth rate of income per person depends on the growth rate of researchers, which is in turn ultimately equal to the growth rate of the population.”

A key idea not explicitly stated in that quote, but emphasized elsewhere in the paper, is that [ideas get harder to find](https://web.stanford.edu/~chadj/IdeaPF.pdf): so if you want to maintain the same rate of innovation, you need more and more researchers over time. This is a simple model that can potentially help explain some otherwise odd-seeming phenomena, such as the fact that [science seems to be “slowing down.”](https://www.theatlantic.com/science/archive/2018/11/diminishing-returns-science/575665/) Basically, it’s possible that how much innovation we get is just a function of how many people are working on innovating - and we need more people over time to keep up the same rate.

So in the short run, you can get more innovation via things like more researcher jobs and better education, but in the long run, the only route is more population.

“Even in this … framework in which population growth is the only potential source of growth in the long run, other factors explain more than 80% of U.S. growth in recent decades: the contribution of population growth is 0.3% out of the 2% growth we observe. In other words, the level effects associated with rising educational attainment, declining misallocation, and rising research intensity have been overwhelmingly important for the past 50+ years.”

“The point to emphasize here is that this framework strongly implies that, unless something dramatic changes, future growth rates will be substantially lower. In particular, all the sources other than population growth are inherently transitory, and once these sources have run their course, all that will remain is the 0.3 percentage point contribution from population growth. In other words … the implication is that long-run growth in living standards will be 0.3% per year rather than 2% per year — an enormous slowdown!”

“if population growth is negative, these idea-driven models predict that living standards stagnate for a population that vanishes! This is a stunningly negative result, especially when compared to the standard result we have been examining throughout the paper. In the usual case with positive population growth, living standards rise exponentially forever for a population that itself rises exponentially. Whether we live in an “expanding cosmos” or an “empty planet” depends, remarkably, on whether the total fertility rate is above or below a number like 2 or 2.1.”

“Peters and Walsh (2021) ... find that declining population growth generates lower entry, reduced creative destruction, increased concentration, rising markups, and lower productivity growth, all facts that we see in the firm-level data.”

So far, the implication is:* In the short run, we’ve had high growth for reasons that can't continue indefinitely. (For example, one such factor is a rising share of the population that has a certain level of education, but that share can't go above 100%. The high-level point is that if we want more researchers, we can only get that via a higher population or a higher % of people who are researchers, and the latter can only go so high.)
 * In the long run, growth (in living standards) basically comes down to population growth.

But the paper also gives two reasons that growth could rise instead of falling.

Reason one:

“The world contains more than 7 billion people. However, according to the OECD’s Main Science and Technology Indicators, the number of full-time equivalent researchers in the world appears to be less than 10 million. In other words something on the order of one or two out of every thousand people in the world is engaged in research ... There is ample scope for substantially increasing the number of researchers over the next century, even if population growth slows or is negative. I see three ways this ‘finding new Einsteins’ can occur …

“**The rise of China, India, and other countries.** The United States, Western Europe, and Japan together have about 1 billion people, or only about 1/7th the world’s population. China and India each have this many people. As economic development proceeds in China, India, and throughout the world, the pool from which we may find new talented inventors will multiply. How many Thomas Edisons and Jennifer Doudnas have we missed out on among these billions of people because they lacked education and opportunity?

“**Finding new Doudnas: women in research.** Another huge pool of underutilized talent is women …. Brouillette (2021) uses patent data to document that in 1976 less than 3 percent of U.S. inventors were women. Even as of 2016 the share was less than 12 percent. He estimates that eliminating the barriers that lead to this misallocation of talent could raise economic growth in the United States by up to 0.3 percentage points per year over the next century.

“**Other sources of within-country talent.** Bell, Chetty, Jaravel, Petkova and Van Reenen (2019) document that the extent to which people are exposed to inventive careers in childhood has a large influence on who becomes an inventor. They show that exposure in childhood is limited for girls, people of certain races, and people in low-income neighborhoods, even conditional on math test scores in grade school, and refer to these missed opportunities as ‘lost Einsteins.’”

The other reason that growth could rise will be familiar to readers of this blog:

“Another potential reason for optimism about future growth prospects is the possibility of automation, both in the production of goods and in the production of ideas … [according to a particular model,] an increase in the automation of tasks in idea production (↑α) causes the growth rate of the economy to increase … if the fraction of tasks that are automated (α) rises to reach the rate at which ideas are getting harder to find (β), we get a singularity! [Caveats follow]”

Oversimplified recap: innovation comes down to the number of researchers; some key recent sources of growth in this can't continue indefinitely; if population growth stagnates, eventually so must innovation and living standards; but we could get more researchers via lowering barriers to entry and/or via AI and automation (and/or via more population growth).

None of these claims are empirical, settled science. They all are implications of what I believe are the leading simple models of economic growth. But to me they all make good sense, and I think the reason they aren’t more "in the water" is because people don’t tend to talk about the drivers of the *long-run past and future* of economic growth (as I have [complained previously](https://www.cold-takes.com/this-cant-go-on/#neglected-possibilities)!)

Here are Leopold Aschenbrenner’s [favorite papers by the same author](https://www.forourposterity.com/best-chad-jones-papers/) (including this one).

---

Footnotes
---------

1. You can try [this short explanation](https://www.cold-takes.com/what-is-economic-growth/) if you don’t know what economic growth is. [↩](#fnref1)
For email filter: florpschmop


Medium Lights

I like this [basketball substack called Medium Lights](https://mediumlights.substack.com/) that explains all kinds of cool basketball plays the announcers wouldn't necessarily comment on. Unlike most sites like this, it usually explains in enough detail that I can actually follow what's cool about the play. It really highlights how much more interesting games could be with good announcers.

A few I've especially enjoyed:

* [LeBron lets people drive past him so he can block their shot from behind](https://mediumlights.substack.com/p/lebron-baits-kirk-hinrich).

* [LeBron also will watch the defense on a failed offensive possession, then start a future possession the same way to get the same behavior and exploit it](https://mediumlights.substack.com/p/lebron-reads-the-defense).

* Daniel Theis's "[paint seal](https://mediumlights.substack.com/p/daniel-theis-paint-seal)," the sort of thing I'd never notice just by watching.

* [Nikola Jokic's water polo pump fake](https://mediumlights.substack.com/p/nikola-jokics-water-polo-pump-fake).

* [A collection of absurdly high-arcing shots](https://mediumlights.substack.com/p/tyler-herros-high-arching-floater).

* "[Dwyane Wade's sneaky baseline cuts: an appreciation for one of the best off-ball players of all time](https://mediumlights.substack.com/p/dwyane-wades-sneaky-baseline-cuts)"

* A walkthrough of [Draymond Green guarding all five opposing players on one possession](https://mediumlights.substack.com/p/draymond-guards-five-players-on-one).

* This one is just silly (but top notch silly): [Robin Lopez makes everyone proud](https://mediumlights.substack.com/p/robin-lopez-makes-everyone-proud)* [Draymond "throwing Steph Curry open."](https://mediumlights.substack.com/p/draymond-green-throws-steph-curry) "honestly, my only reaction when i saw it the first time was 'what...?' ... [Curry's] whole body is turned towards his own basket when the ball is already halfway to him ... i don't really have any words for this. i guess this is what happens when two guys play together for a long time - the chemistry is there and cool shit just happens." (To understand "throwing someone open" you can see [this Reddit post](https://www.reddit.com/r/nfl/comments/96akdt/oc_what_is_throwing_a_guy_open_and_why_does_it/), though I have to say ... watching [this example](https://www.youtube.com/watch?v=UG4DKNasoPU&t=36s), I wondered if it is an "amazing successful pass" that just had a 50-50 chance of being an "embarrassing intercepted pass.")

It's great when someone is able to walk you through why they're this amazed and impressed with something you wouldn't have noticed.


AI Timelines: Where the Arguments, and the "Experts," Stand

*Audio also available by searching Stitcher, Spotify, Google Podcasts, etc. for "Cold Takes Audio"*

[Today’s world

Transformative AI

Digital people
World of

Misaligned AI
World run by
Something else

or
or

Stable, galaxy-wide
civilization](https://www.cold-takes.com/roadmap-for-the-most-important-century-series/)

> This piece starts with a summary of when we should expect transformative AI to be developed, based on the multiple angles covered previously in the series. I think this is useful, even if you've read all of the previous pieces, but if you'd like to skip it, click [here](#Part2).
>
>
>

I then address the question: "Why isn't there a robust expert consensus on this topic, and what does that mean for us?"

I estimate that there is **more than a 10% chance we'll see transformative AI within 15 years (by 2036); a ~50% chance we'll see it within 40 years (by 2060); and a ~2/3 chance we'll see it this century (by 2100).**

(By "transformative AI," I mean "AI powerful enough to bring us into a new, qualitatively different future." I've argued that advanced AI [could](https://www.cold-takes.com/transformative-ai-timelines-part-1-of-4-what-kind-of-ai/#impacts-of-pasta) be sufficient to make this the [most important century](https://www.cold-takes.com/roadmap-for-the-most-important-century-series/).)

This is my overall conclusion based on a number of technical reports approaching AI forecasting from different angles - many of them produced by [Open Philanthropy](https://www.openphilanthropy.org) over the past few years as we've tried to develop a thorough picture of transformative AI forecasting to inform our longtermist grantmaking.

Here's a **one-table summary** of the different angles on forecasting transformative AI that I've discussed, with links to more detailed discussion in [previous posts](https://www.cold-takes.com/roadmap-for-the-most-important-century-series/#forecasting-transformative-ai-this-century) as well as to underlying technical reports:

|  |  |  |
| --- | --- | --- |
| **Forecasting angle** | **Key in-depth pieces (abbreviated titles)** | **My takeaways** |
| *Probability estimates for transformative AI* |
| **[Expert survey.](https://www.cold-takes.com/are-we-trending-toward-transformative-ai-how-would-we-know/#surveying-experts)** What do AI researchers expect?
  | [Evidence from AI Experts](https://arxiv.org/pdf/1705.08807.pdf) | Expert survey implies[1](#fn1) a ~20% probability by 2036; ~50% probability by 2060; ~70% probability by 2100. Slightly differently phrased questions (posed to a minority of respondents) have much later estimates.
  |
| **[Biological anchors framework](https://www.cold-takes.com/forecasting-transformative-ai-the-biological-anchors-method-in-a-nutshell/).** Based on the usual patterns in how much "AI training" costs, how much would it cost to train an AI model as big as a human brain to perform the hardest tasks humans do? And when will this be cheap enough that we can expect someone to do it?
  | [Bio Anchors](https://drive.google.com/drive/u/1/folders/15ArhEPZSTYU8f012bs6ehPS6-xmhtBPP), drawing on [Brain Computation](https://www.openphilanthropy.org/blog/new-report-brain-computation) | >10% probability by 2036; ~50% chance by 2055; ~80% chance by 2100.
  |
| *Angles on the [burden of proof](https://www.cold-takes.com/forecasting-transformative-ai-whats-the-burden-of-proof/)* |
| It's unlikely that any given century would be the "most important" one. ([More](https://www.cold-takes.com/forecasting-transformative-ai-whats-the-burden-of-proof#most-important-century-skepticism))
  | [Hinge](https://static1.squarespace.com/static/5506078de4b02d88372eee4e/t/5f36b015d9a3691ba8e1096b/1597419543571/Are+we+living+at+the+hinge+of+history.pdf); [Response to Hinge](https://forum.effectivealtruism.org/posts/j8afBEAa7Xb2R9AZN/thoughts-on-whether-we-re-living-at-the-most-influential) | We have many reasons to think this century is a "special" one before looking at the details of AI. Many have been covered in previous pieces; another is covered in the next row.
  |
| What would you forecast about transformative AI timelines, based **only** on basic information about (a) how many years people have been trying to build transformative AI; (b) how much they've "invested" in it (in terms of the number of AI researchers and the amount of computation used by them); (c) whether they've done it yet (so far, they haven't)? ([More](https://www.cold-takes.com/forecasting-transformative-ai-whats-the-burden-of-proof#semi-informative-priors))
  | [Semi-informative Priors](https://www.openphilanthropy.org/blog/report-semi-informative-priors) | Central estimates: 8% by 2036; 13% by 2060; 20% by 2100.[2](#fn2) In my view, this report highlights that the history of AI is short, investment in AI is increasing rapidly, and so we shouldn't be too surprised if transformative AI is developed soon.
  |
| Based on analysis of economic models and economic history, how likely is 'explosive growth' - defined as >30% annual growth in the world economy - by 2100? Is this far enough outside of what's "normal" that we should doubt the conclusion? ([More](https://www.cold-takes.com/forecasting-transformative-ai-whats-the-burden-of-proof#most-important-century-skepticism))
  | [Explosive Growth](https://www.openphilanthropy.org/could-advanced-ai-drive-explosive-economic-growth), [Human Trajectory](https://www.openphilanthropy.org/blog/modeling-human-trajectory) | [Human Trajectory](https://www.openphilanthropy.org/blog/modeling-human-trajectory) projects the past forward, implying explosive growth by 2043-2065.

[Explosive Growth](https://www.openphilanthropy.org/could-advanced-ai-drive-explosive-economic-growth) concludes: "I find that economic considerations don’t provide a good reason to dismiss the possibility of TAI being developed in this century. In fact, there is a plausible economic perspective from which sufficiently advanced AI systems are expected to cause explosive growth."
  |
| "How have people predicted AI ... in the past, and should we adjust our own views today to correct for patterns we can observe in earlier predictions? ... We’ve encountered the view that AI has been prone to repeated over-hype in the past, and that we should therefore expect that today’s projections are likely to be over-optimistic." ([More](https://www.cold-takes.com/forecasting-transformative-ai-whats-the-burden-of-proof#history-of-))
  | [Past AI Forecasts](https://www.openphilanthropy.org/focus/global-catastrophic-risks/potential-risks-advanced-artificial-intelligence/what-should-we-learn-past-ai-forecasts) | "The peak of AI hype seems to have been from 1956-1973. Still, the hype implied by some of the best-known AI predictions from this period is commonly exaggerated."
  |

*For transparency, note that many of the technical reports are [Open Philanthropy](https://www.openphilanthropy.org/) analyses, and I am co-CEO of Open Philanthropy.*

Having considered the above, I expect some readers to still feel a sense of unease. Even if they think my arguments make sense, they may be wondering: **if this is true, why isn't it more widely discussed and accepted? What's the state of expert opinion?**

My summary of the state of expert opinion at this time is:

* The claims I'm making do not *contradict* any particular expert consensus. (In fact, the probabilities I've given aren't too far off from what AI researchers seem to predict, as shown in the first row.) But there are some [signs they aren't thinking too hard about the matter](https://www.cold-takes.com/are-we-trending-toward-transformative-ai-how-would-we-know/#surveying-experts).

* The Open Philanthropy technical reports I've relied on have had significant external expert review. Machine learning researchers reviewed [Bio Anchors](https://drive.google.com/drive/u/1/folders/15ArhEPZSTYU8f012bs6ehPS6-xmhtBPP); neuroscientists reviewed [Brain Computation](https://www.openphilanthropy.org/blog/new-report-brain-computation); economists reviewed [Explosive Growth](https://www.openphilanthropy.org/could-advanced-ai-drive-explosive-economic-growth); academics focused on relevant topics in uncertainty and/or probability reviewed [Semi-informative Priors](https://www.openphilanthropy.org/blog/report-semi-informative-priors).[2](#fn2) (Some of these reviews had significant points of disagreement, but none of these points seemed to be cases where the reports contradicted a clear consensus of experts or literature.)

* But there is also no active, robust expert consensus supporting claims like *"There's at least a 10% chance of transformative AI by 2036"* or *"There's a good chance we're in the most important century for humanity,"* the way that there is supporting e.g. the need to take action against climate change.

Ultimately, my claims are about **topics that simply have no "field" of experts devoted to studying them. That, in and of itself, is a scary fact,** and something that I hope will eventually change.

But should we be willing to act on the "most important century" hypothesis in the meantime?

Below, I'll discuss:

* What an "AI forecasting field" might look like.

* A "skeptical view" that says today's discussions around these topics are too small, homogeneous and insular (which I agree with) - and that we therefore shouldn't act on the ["most important century" hypothesis](https://www.cold-takes.com/roadmap-for-the-most-important-century-series/) until there is a mature, robust field (which I don't).

* Why I think we should take the hypothesis seriously in the meantime, until and unless such a field develops:
	+ We don't have time to wait for a robust expert consensus.

	+ If there are good rebuttals out there - or potential future experts who could develop such rebuttals - we haven't found them yet. The more seriously the hypothesis gets taken, the more likely such rebuttals are to appear. (Aka the [Cunningham's Law](https://bigthink.com/david-ryan-polgar/want-the-right-answer-online-dont-ask-questions-just-post-it-wrong) theory: "the best way to get a right answer is to post a wrong answer.")

	+ I think that consistently insisting on a robust expert consensus is a dangerous reasoning pattern. In my view, it's OK to be at some risk of self-delusion and insularity, in exchange for doing the right thing when it counts most.

What kind of expertise is AI forecasting expertise?
---------------------------------------------------

Questions analyzed in the technical reports listed [above](#SummaryTable) include:

* Are AI capabilities getting more impressive over time? (AI, history of AI)

* How can we compare AI models to animal/human brains? (AI, neuroscience)

* How can we compare AI capabilities to animals' capabilities? (AI, ethology)

* How can we estimate the expense of training a large AI system for a difficult task, based on information we have about training past AI systems? (AI, curve-fitting)

* How can we make a minimal-information estimate about transformative AI, based only on how many years/researchers/dollars have gone into the field so far? (Philosophy, probability)

* How likely is explosive economic growth this century, based on theory and historical trends? (Growth economics, economic history)

* What has "AI hype" been like in the past? (History)

When talking about wider implications of transformative AI for the "most important century," I've also discussed things like "How feasible are [digital people](https://www.cold-takes.com/digital-people-faq/#feasibility) and [establishing space settlements throughout the galaxy](https://www.cold-takes.com/how-digital-people-could-change-the-world/#space-expansion)?" These topics touch physics, neuroscience, engineering, philosophy of mind, and more.

**There's no obvious job or credential that makes someone an expert on the question of when we can expect transformative AI, or the question of whether we're in the most important century.**

(I particularly would disagree with any claim that we should be relying exclusively on AI researchers for these forecasts. In addition to the fact that [they don't seem to be thinking very hard about the topic](https://www.cold-takes.com/are-we-trending-toward-transformative-ai-how-would-we-know/#surveying-experts), I think that relying on people who specialize in building ever-more powerful AI models to tell us when transformative AI might come is like relying on solar energy R&D companies - or oil extraction companies, depending on how you look at it - to forecast carbon emissions and climate change. They certainly have part of the picture. But forecasting is a distinct activity from innovating or building state-of-the-art systems.)

And I'm not even sure these questions have the right shape for an academic field. Trying to forecast transformative AI, or determine the odds that we're in the most important century, seems:

* More similar to the [FiveThirtyEight election model](https://projects.fivethirtyeight.com/2020-election-forecast/) ("Who's going to win the election?") than to academic political science ("How do governments and constituents interact?");

* More similar to trading financial markets ("Is this price going up or down in the future?") than to academic economics ("Why do recessions exist?");[3](#fn3)* More similar to [GiveWell's](https://www.givewell.org/) research ("Which charity will help people the most, per dollar?") than to academic development economics ("What causes poverty and what can reduce it?")[4](#fn4)

That is, it's not clear to me what a natural "institutional home" for expertise on transformative AI forecasting, and the "most important century," would look like. But it seems fair to say there aren't large, robust institutions dedicated to this sort of question today.

How should we act in the absence of a robust expert consensus?
--------------------------------------------------------------

### The skeptical view

Lacking a robust expert consensus, I expect some (really, most) people will be skeptical no matter what arguments are presented.

Here's a version of a very general skeptical reaction I have a fair amount of empathy for:

1. *This is all just too [wild](https://www.cold-takes.com/forecasting-transformative-ai-whats-the-burden-of-proof/#formalizing-the-).*- *You're making an over-the-top claim about living in the most important century. This **pattern-matches to self-delusion.***- *You've argued that the [burden of proof](https://www.cold-takes.com/forecasting-transformative-ai-whats-the-burden-of-proof/) shouldn't be so high, because there are lots of ways in which we live in a [remarkable](https://www.cold-takes.com/all-possible-views-about-humanitys-future-are-wild/) and [unstable](https://www.cold-takes.com/this-cant-go-on/) time. But ... I don't trust myself to assess those claims, or your claims about AI, or really anything on these wild topics.*- *I'm worried by how few people seem to be engaging these arguments. About how **small, homogeneous and insular** the discussion seems to be. Overall, this feels more like a story smart people are telling themselves - with lots of charts and numbers to rationalize it - about their place in history. It doesn't feel "real."*- *So call me back when there's a mature field of perhaps hundreds or thousands of experts, critiquing and assessing each other, and they've reached the same sort of consensus that we see for climate change.*

I see how you could feel this way, and I've felt this way myself at times - especially on points #1-#4. But I'll give **three reasons that point #5 doesn't seem right.**

### Reason 1: we don't have time to wait for a robust expert consensus

I worry that the arrival of transformative AI could play out as a kind of slow-motion, higher-stakes version of the COVID-19 pandemic. The case for expecting something big to happen is there, if you look at the best information and analyses available today. But the situation is broadly unfamiliar; it doesn't fit into patterns that our institutions regularly handle. And every extra year of action is valuable.

You could also think of it as a sped-up version of the dynamic with climate change. Imagine if greenhouse gas emissions had only started to rise recently[5](#fn5) (instead of in the [mid-1800s](https://ourworldindata.org/co2-and-other-greenhouse-gas-emissions)), and if there were no established field of climate science. It would be a really bad idea to wait decades for a field to emerge, before seeking to reduce emissions.

### Reason 2: [Cunningham's Law](https://bigthink.com/david-ryan-polgar/want-the-right-answer-online-dont-ask-questions-just-post-it-wrong) ("the best way to get a right answer is to post a wrong answer") may be our best hope for finding the flaw in these arguments

I'm serious, though.

Several years ago, some [colleagues](https://www.cold-takes.com/roadmap-for-the-most-important-century-series/#acknowledgements) and I suspected that the "most important century" hypothesis could be true. But before acting on it too much, we wanted to see whether we could find fatal flaws in it.

One way of interpreting our actions over the last few years is **as if we were doing everything we could to learn that the hypothesis is wrong.**

First, we tried talking to people about the key arguments - AI researchers, economists, etc. But:

* We had vague ideas of the arguments in this series (mostly or perhaps entirely picked up [from other people](https://www.cold-takes.com/roadmap-for-the-most-important-century-series/#acknowledgements)). We weren't able to state them with good crispness and specificity.

* There were a lot of key factual points that we thought would probably check out,[6](#fn6) but hadn't nailed down and couldn't present for critique.

* Overall, we couldn't even really articulate enough of a concrete case to give the others a fair chance to shoot it down.

So we put a lot of work into creating technical reports on many of the key arguments. (These are now public, and included in the table at the top of this piece.) This put us in position to publish the arguments, and potentially encounter fatal counterarguments.

Then, we commissioned external expert reviews.[7](#fn7)

Speaking only for my own views, the "most important century" hypothesis seems to have survived all of this. Indeed, having examined the many angles and gotten more into the details, I believe it more strongly than before.

But let's say that this is just because the *real* experts - people we haven't found yet, with devastating counterarguments - find the whole thing so silly that they're [not bothering to engage](https://philiptrammell.com/blog/46/). Or, let's say that there are people out there today who could *someday* become experts on these topics, and knock these arguments down. What could we do to bring this about?

The best answer I've come up with is: "If this hypothesis became better-known, more widely accepted, and more influential, it would get more critical scrutiny."

This series is an attempted step in that direction - to move toward broader credibility for the "most important century" hypothesis. This would be a good thing if the hypothesis were true; it also seems like the best next step if my only goal were to challenge my beliefs and learn that it is false.

Of course, I'm not saying to accept or promote the "most important century" hypothesis if it doesn't seem correct to you. But I think that if your *only* reservation is about the lack of robust consensus, continuing to ignore the situation seems odd. If people behaved this way generally (ignoring any hypothesis not backed by a robust consensus), I'm not sure I see how any hypothesis - including true ones - would go from fringe to accepted.

### Reason 3: skepticism this general seems like a bad idea

Back when I was focused on [GiveWell](http://www.givewell.org), people would occasionally say something along the lines of: "You know, you can't hold every argument to the standard that GiveWell holds its top charities to - seeking randomized controlled trials, robust empirical data, etc. Some of the best opportunities to do good will be the ones that are less obvious - so this standard risks [ruling out some of your biggest potential opportunities to have impact](https://www.openphilanthropy.org/blog/hits-based-giving#Anti-principles_for_hits-based_giving)."

I think this is right. I think it's important to check one's general approach to reasoning and evidentiary standards and ask: "What are some scenarios in which my approach fails, and in which I'd really prefer that it succeed?" In my view, **it's OK to be at some risk of self-delusion and insularity, in exchange for doing the right thing when it counts most.**

I think the lack of a robust expert consensus - and concerns about self-delusion and insularity - provide good reason to *dig hard* on the "most important century" hypothesis, rather than accepting it immediately. To ask where there might be an undiscovered flaw, to look for some bias toward inflating our own importance, to research the most questionable-seeming parts of the argument, etc.

But if you've investigated the matter as much as is reasonable/practical for you - and haven't found a flaw *other* than considerations like "There's no robust expert consensus" and "I'm worried about self-delusion and insularity" - then I think writing off the hypothesis is the sort of thing that essentially **guarantees you won't be among the earlier people to notice and act on a tremendously important issue, if the opportunity arises.** I think that's too much of a sacrifice, in terms of giving up potential opportunities to do a lot of good.

**Next in series:** [How to make the best of the most important century?](https://www.cold-takes.com/making-the-best-of-the-most-important-century/)

---

Footnotes
---------

1. Technically, these probabilities are for “human-level machine intelligence.” In general, this chart simplifies matters by presenting one unified set of probabilities. In general, all of these probabilities refer to something at *least* as capable as [PASTA](https://www.cold-takes.com/transformative-ai-timelines-part-1-of-4-what-kind-of-ai/), so they directionally should be underestimates of the probability of PASTA (though I don't think this is a major issue). [↩](#fnref1)- Reviews of Bio Anchors are [here](https://drive.google.com/drive/u/1/folders/1XkTYFiZQUT6UAUL2Wyg9wD0qG57KYfjq); reviews of Explosive Growth are [here](https://www.openphilanthropy.org/could-advanced-ai-drive-explosive-economic-growth#AppendixH); reviews of Semi-informative Priors are [here](https://www.openphilanthropy.org/blog/report-semi-informative-priors#LinksToReviewer). Brain Computation was reviewed at an earlier time when we hadn't designed the process to result in publishing reviews, but over 20 conversations with experts that informed the report are available [here](https://www.openphilanthropy.org/research/conversations). Human Trajectory hasn't been reviewed, although a lot of its analysis and conclusions feature in Explosive Growth, which has been. Past AI Forecasts hasn't been reviewed.  [↩](#fnref2)- The academic fields are quite broad, and I'm just giving example questions that they tackle. [↩](#fnref3)- Though climate science is an example of an academic field that invests a lot in forecasting the future. [↩](#fnref4)- The field of AI has existed since [1956](https://en.wikipedia.org/wiki/Dartmouth_workshop), but it's only in the last decade or so that machine learning models have started to get within range of [the size of insect brains](https://www.cold-takes.com/forecasting-transformative-ai-the-biological-anchors-method-in-a-nutshell/#conclusions-of-bio-anchors) and perform well on relatively difficult tasks. [↩](#fnref5)- Often, we were simply going off of our impressions of what others who had thought about the topic a lot thought. [↩](#fnref6)- Reviews of Bio Anchors are [here](https://drive.google.com/drive/u/1/folders/1XkTYFiZQUT6UAUL2Wyg9wD0qG57KYfjq); reviews of Explosive Growth are [here](https://www.openphilanthropy.org/could-advanced-ai-drive-explosive-economic-growth#AppendixH); reviews of Semi-informative Priors are [here](https://www.openphilanthropy.org/blog/report-semi-informative-priors#LinksToReviewer). Brain Computation was reviewed at an earlier time when we hadn't designed the process to result in publishing reviews, but over 20 conversations with experts that informed the report are available [here](https://www.openphilanthropy.org/research/conversations). Human Trajectory hasn't been reviewed, although a lot of its analysis and conclusions feature in Explosive Growth, which has been. Past AI Forecasts hasn't been reviewed.  [↩](#fnref7)


Cold Links: Useful

These are assorted links that I expect readers to maybe find useful on a personal basis. Don't worry, this won't be too frequent an occurrence - I aim for 99% of this blog to be about the last 100-10,000 years and the next 100-10 zillion years.

Ben Kuhn's [tips for the most immersive video calls](https://www.benkuhn.net/vc/) - the most comprehensive guide I've seen to good Zooming.

Some [tips on how to make sure you have good wifi at your hotel/AirBNB](https://thepointsguy.com/2018/03/how-to-find-lodging-with-good-wi-fi/). It shocks me that AirBNB doesn't do more to help (e.g., nudging hosts to verify and list their speeds and networking equipment, allowing guests to filter by this). Some of the tips here are obvious, some I didn't know, but it seems like a good checklist.

I strongly endorse this [advice on PowerPoint presentations](http://sethgodin.typepad.com/seths_blog/2018/04/words-on-slides.html).

Interesting [argument against wearing a bike helmet](https://www.howiechong.com/journal/2014/2/bike-helmets?fbclid=IwAR3KRBESvDwgxEItuRK-0uwo3dEqtjSQXFTpy7Adu_Iu9u41jQdl341eKDY). I wear one because I don't want to look reckless and actively enjoy looking dorky, but ¯\\_(ツ)\_/¯

I recommend traveling with an [octopus plug](https://smile.amazon.com/Leviton-692-W-Triple-Grounding-Adapter/dp/B000P9SXTG/) so you never have to fight over an airport charger (instead, you're a hero!)

[FiveThirtyEight](http://fivethirtyeight.com/features/which-diet-will-help-you-lose-the-most-weight/) looks at a recent review of randomized studies on which diets work best. The effect sizes seem pretty good - ~15lbs avg weight loss after a year - compared to what I expected based on previous coverage of this topic. Atkins doesn't win, Ornish (very traditional approach to dieting) does, though they're all extremely close, consistent with my general view that "any diet you can stick to will probably help because it will be different from eating whatever you feel like." Paleo isn't included in the analysis.

Basic take, but important. WSJ: [The Man Who Wrote Those Password Rules Has a New Tip: N3v$r M1^d!](https://www.wsj.com/articles/the-man-who-wrote-those-password-rules-has-a-new-tip-n3v-r-m1-d-1502124118) "Bill Burr’s 2003 report recommended using numbers, obscure characters and capital letters and updating regularly—he regrets the error." Not surprising at all - it's bad advice that websites still enforce today, and [XKCD's take is spot on](https://xkcd.com/936/). I recommend LastPass and maximal use of two-factor authentication.

Here's an [excruciatingly detailed guide](http://www.kalzumeus.com/2017/09/09/identity-theft-credit-reports/) to filing a complaint with a credit reporting agency that actually gets acted on. Key advice is to stay away from online and phone communications and use certified mail for everything, which shows them you're collecting a paper trail and scares them; key incredible fact is that you're not allowed to use form letters because ... well, because credit reporting agencies don't want you to and they've lobbied to prohibit it. (Even though they use form letters.) I found this whole long thing weirdly fun to read. Just thinking about confronting one of these awful bureaucracies and knowing how to get good results made me smile. Advice may also have applications for dealing with bureaucracies more generally.

[How to recognize when a child is drowning](https://slate.com/technology/2013/06/rescuing-drowning-children-how-to-know-when-someone-is-in-trouble-in-the-water.html). If you donate to effective global health charities to save children's lives, then you need to know how to do this too in order to be consistent.[1](#fn1)

---

Footnotes
---------

1. This is a joke. If you don't get it, don't worry about it. [↩](#fnref1)