Nonprofit Boards are Weird

*Click lower right to download or find on Apple Podcasts, Spotify, Stitcher, etc.*

> Note: anything in this post that you think is me subtweeting your organization is actually about, like, at least 3 organizations. (I'm currently on 4 boards in addition to [Open Philanthropy](https://openphilanthropy.org/)'s; I've served on a bunch of other boards in the past; and more than half of my takes on boards are not based on any of this, but rather on my interactions with boards I'm not on via the many grants made by Open Philanthropy.)

Writing about [ideal governance](https://www.cold-takes.com/ideal-governance-for-companies-countries-and-more/) reminded me of how weird my experiences with nonprofit boards (as in "board of directors" - the set of people who formally control a nonprofit) have been.

I thought that was a pretty good intro. The rest of this piece will:

* Try to articulate what's so weird about nonprofit boards, fundamentally. I think a lot of it is the combination of great power, unclear responsibility, and ~zero accountability; additionally, I haven't been able to find much in the way of clear, widely accepted statements of what makes a good board member.

* Give my own thoughts on what makes a good board member: which core duties they should be trying to do really well, the importance of "staying out of the way" on other things, and some potentially helpful practices.

I am experienced with nonprofit boards but not with for-profit boards. I'm guessing that roughly half the things I say below will apply to for-profit boards, and that for-profit boards are roughly half as weird overall (so still quite weird), but I haven't put much effort into disentangling these things; I'm writing about what I've seen.

I can't really give real-life examples here (for reasons I think will be pretty clear) so this is just going to be me opining in the abstract.

Why nonprofit boards are weird
------------------------------

![alt_text](https://www.cold-takes.com/content/images/2022/06/image1.png "image_tooltip")

Here's how a nonprofit board works:

* There are usually 3-10 people on the board (though sometimes much more). Most of them don't work for the nonprofit (they have other jobs).

* They meet every few months. Nonprofit employees (especially the CEO[1](https://www.cold-takes.com/p/47f975d8-08f1-4e66-a7c6-6ba4e182cc1a/#fn1)) do a lot of the agenda-setting for the meeting. Employees present general updates and ask for the board's approval on various things the board needs to approve, such as the budget.

* A majority vote of the directors can do anything: fire the CEO, dissolve the nonprofit, add and remove directors, etc. You can think of the board as the "owner" of the nonprofit - formally, it has final say in every decision.

* In practice, though, the board rarely votes except on matters that feel fairly "rubber-stamp," and the board's presence doesn't tend to be felt day-to-day at a nonprofit. The CEO leads the decision-making. Occasionally, someone has a thought like "Wait, who does the *CEO* report to? Oh, the board of directors ... who's on the board again? I don't know if I've ever really spoken with any of those people."

In my experience, it's common for the whole thing to feel extremely weird. (This doesn't necessarily mean there's a better way to do it - footnote has more on what I mean by "weird."[2](https://www.cold-takes.com/p/47f975d8-08f1-4e66-a7c6-6ba4e182cc1a/#fn2))

* Board members often know almost nothing about the organization they have complete power over.

* Board meetings rarely feel like a good use of time.

* When board members are energetically asking questions and making demands, it usually feels like they're causing chaos and wasting everyone's time and energy.

* On the rare occasions when it seems like the board *should* do something (like replacing the CEO, or providing an independent check on some important decision), the board often seems checked out and it's unclear how they would even come to be aware of the situation.

* Everyone constantly seems confused about what the board is and how it can and can't be useful. Employees, and others who interact with the nonprofit, have lots of exchanges like "I'm worried about X ... maybe we should ask the board what they think? ... Can we even ask them that? What is their job actually?"

(Reminder that this is not subtweeting a particular organization! More than one person - from more than one organization - read a draft and thought I was subtweeting them, because what's above describes a large number of boards.)

OK, so what's driving the weirdness?

I think there are a couple of things:

* Nonprofit boards have *great power*, but *low engagement* (they don't have time to understand the organization as well as employees do); *unclear responsibility* (it's unclear which board member is responsible for what, and what the board as a whole is responsible for); and *~zero accountability* (no one can fire board members except for the other board members!)

* Nonprofit boards have unclear expectations and principles. I can't seem to find anyone with a clear, comprehensive, thought-out theory of what a board member's ... job is.

I'll take these one at a time.

### Great power, low engagement, unclear responsibility, no accountability

In my experience/impression, the best way to run any organization (or project, or anything) is on an "ownership" model: for any given thing X that you want done well, you have one person who "owns" X. The "owner" of X has:

* The *power* to make decisions to get X done well.

* High *engagement*: they're going to have plenty of time and attention to devote to X.

* The *responsibility* for X: everyone agrees that if X goes well, they should get the credit, and if X goes poorly, they should get the blame.

* And *accountability*: if X goes poorly, there will be some sort of consequences for the "owner."

When these things come apart, I think you get problems. In a nutshell - when no one is *responsible*, nothing gets done; when someone is *responsible* but doesn't have *power*, that doesn't help much; when the person who is *responsible* + *empowered* isn't *engaged* (isn't paying much attention), or isn't held *accountable*, there's not much in the way of their doing a dreadful job.

A traditional company structure mostly does well at this. The CEO has power (they make decisions for the company), engagement (they are devoted to the company and spend tons of time on it), and responsibility+accountability (if the company does badly, everyone looks at the CEO). They manage a team of people who have power+engagement+responsibility+accountability for some aspect of the company; each of those people manage people with power+engagement+responsibility+accountability for some smaller piece; etc.

What about the board?

* They have *power* to fire the CEO (or do anything else).

* They tend to have low *engagement*. They have other jobs, and only spend a few hours a year on their board roles. They tend to know little about what's going on at the organization.

* They have unclear *responsibility*.
	+ The board as a whole is responsible for the organization, but what is each *individual* board member responsible for? In my experience, this is often very unclear, and there are a lot of crucial moments where "bystander effects" seem strong.

	+ So far, these points apply to both nonprofit and for-profit boards. But at least at a for-profit company, board members know what they're collectively responsible *for*: maximizing financial value of the company. **At a nonprofit, it's often unclear what success even *means*, beyond the nonprofit's often-vague mission statement, so board members are generally unclear (and don't necessarily agree) on what they're supposed to be ensuring.**[3](https://www.cold-takes.com/p/47f975d8-08f1-4e66-a7c6-6ba4e182cc1a/#fn3)* At a for-profit company, the board seems to have reasonable *accountability:* the shareholders, who ultimately own the company and gain or lose money depending on how it does, can replace the board if they aren't happy. **At a nonprofit, the board members have *zero accountability:* the only way to fire a board member is by majority vote of the board!**

So we have people who are spending very little time on the company, know very little about it, don't have much clarity on what they're responsible for either individually or collectively, and aren't accountable to anyone ... and those are the people with all of the power. Sound dysfunctional?[4](https://www.cold-takes.com/p/47f975d8-08f1-4e66-a7c6-6ba4e182cc1a/#fn4)

In practice, I think it's often worse than it sounds, because board members aren't even chosen carefully - a lot of the time, a nonprofit just goes with an assortment of random famous people, big donors, etc.

### What makes a good board member? Few people even have a hypothesis

I've searched a fair amount for books, papers, etc. that give convincing and/or widely-accepted answers to questions like:

* When the CEO asks the board to approve something, how should they engage? When should they take a *deferring* attitude ("Sure, as long as I don't see any particular reason to say no"), a *sanity check* attitude ("I'll ask a few questions to make sure this is making sense, then approve if nothing jumps out at me"), a *full ownership* attitude ("I need to personally be convinced this is the best thing for the organization"), etc.?

* How much should each board member invest in educating themselves about the organization? What's the best way to do that?

* How does the board know whether the CEO is doing a good job? What kind of situation should trigger seriously considering looking for a new one?

* How does a board member know whether the *board* is doing a good job? How should they decide when another board member should be replaced?

In my experience, most board members just aren't walking around with any particular thought-through take on questions like this. And as far as I can tell, there's a shortage of good[5](https://www.cold-takes.com/p/47f975d8-08f1-4e66-a7c6-6ba4e182cc1a/#fn5) guidance on questions like this for both for-profit and nonprofit boards. For example:

* I've found no standard reference on topics like this, and very few resources that even seem aimed at directly and clearly answering such questions.
	+ The best book on this topic I've seen is [Boards that Lead](https://smile.amazon.com/Boards-That-Lead-Charge-Partner/dp/1422144054/) by Ram Charan, focused on for-profit boards (but pretty good IMO).

	+ But this isn't, like, a book everyone knows to read; I found it by asking lots of people for suggestions, coming up empty, Googling wildly around and skimming like 10 books that said they were about boards, and deciding that this one seemed pretty good.* One of the things I do as a board member is interview other prospective board members about their answers to questions like this. In my experience, they answer most of the above questions with something like "Huh, I don't really know. What do you think?"

* Most boards I've seen seem to - by default - either:
	+ Get way too involved in lots of decisions to the point where it feels like they're micromanaging the CEO and/or just obsessively engaging on whatever topics the CEO happens to bring to their attention; or

	+ Take a "We're just here to help" attitude and rubber-stamp whatever the CEO suggests, including things I'll argue below should be [core duties](https://www.cold-takes.com/p/47f975d8-08f1-4e66-a7c6-6ba4e182cc1a/#the-boards-main-duties) for the board (e.g., adding and removing board members).* I'm not sure I've ever seen a board with a formal, recurring process for reviewing each board member's performance. :/

To the extent I have seen a relatively common, coherent vision of "what board members are supposed to be doing," it's pretty well summarized in [Reid Hoffman's interview](https://growth.eladgil.com/book/cofounders/board-and-ceo-transitions-and-other-governance-issues-an-interview-with-reid-hoffman/) in [The High-Growth Handbook](https://growth.eladgil.com/):

> I use ... a red light, yellow light, green light framework between the board and the CEO. Roughly, green light is, “You’re the CEO. Make the call. We’re advisory.” Now, we may say that on very big things—selling the company—we should talk about it before you do it. And that may shift us from green light, if we don’t like the conversation. But a classic young, idiot board member will say, “Well, I’m giving you my expertise and advice. You should do X, Y, Z.” But the right framework for board members is: You’re the CEO. You make the call. We’re advisory.
>
>
>
>
>  Red lights also very easy. Once you get to red light, the CEO—who, by the way, may still be in place—won’t be the CEO in the future. The board knows they need a new CEO. It may be with the CEO’s knowledge, or without it. Obviously, it’s better if it’s collaborative ...
>
>
>
>
>
>  Yellow means, “I have a question about the CEO. Should we be at green light or not?” And what happens, again under inexperienced or bad board members, is they check a CEO into yellow indefinitely. They go, “Well, I’m not sure…” The important thing with yellow light is that you 1) coherently agree on it as a board and 2) coherently agree on what the exit conditions are. What is the limited amount of time that we’re going to be in yellow while we consider whether we move back to green or move to red? And how do we do that, so that we do not operate for a long time on yellow? Because with yellow light, you’re essentially hamstringing the CEO and hamstringing the company. It’s your obligation as a board to figure that out.
>
>
>

I like this quite a bit (hence the long blockquote), but I don't think it covers everything. The board is *mostly* there to oversee the CEO, and they should *mostly* be advisory when they're happy with the CEO. But I think there are things they ought to be actively thinking about and engaging in even during "green light."

So what DOES make a good board member?
--------------------------------------

Here is my current take, based on a combination of (a) my thoughts after serving on and interacting with a large number of nonprofit boards; (b) my attempts to adapt conventional wisdom about for-profit boards (especially from the [book I mentioned above](https://smile.amazon.com/Boards-That-Lead-Charge-Partner/dp/1422144054/)); (c) divine revelation.

I'll go through:

* What I see as the **main duties** of the board specifically - things the board has to do well, and can't leave to the CEO and other staff.

* My basic take that the ideal board should do these main duties well, while staying out of the way otherwise.

* The **main qualities** I think the ideal board member should have - and some common ways of choosing board members that seem bad to me.

* A few more random thoughts on board practices that seem especially important and/or promising.

(I don't claim any of these points are original, and almost everything can be found in some writing on boards somewhere, but I don't know of a reasonably comprehensive, concise place to get something similar to the below.)

### The board's main duties

I agree with the basic spirit of Hoffman's philosophy [above](https://www.cold-takes.com/p/47f975d8-08f1-4e66-a7c6-6ba4e182cc1a/#RH): the board should not be trying to "run the company" (they're too low-engagement and don't know enough about it), and should instead be focused on a small number of big-picture questions like "How is the CEO doing?"

And I do think **the board's #1 and most fundamental job is evaluating the CEO's performance.** The board is the *only* reliable source of accountability for the CEO - even more so at a nonprofit than a for-profit, since bad CEO performance won't necessarily show up via financial problems or unhappy shareholders.[6](https://www.cold-takes.com/p/47f975d8-08f1-4e66-a7c6-6ba4e182cc1a/#fn6) (As noted [below](https://www.cold-takes.com/p/47f975d8-08f1-4e66-a7c6-6ba4e182cc1a/#regular-ceo-reviews), I think many nonprofit boards have no formal process for reviewing the CEO's performance, and the ones that do often have a lightweight/underwhelming one.)

But I think the board also needs to take a leading role - and not trust the judgment of the CEO and other staff - when it comes to:

* **Overseeing decisions that could importantly reduce the board's powers.** The CEO might want to enter into an agreement with a third party that is binding on the nonprofit and therefore on the board (for example, "The nonprofit will now need permission from the third party in order to do X"); or transfer major activities and assets to affiliated organizations that the board doesn't control (for example, when [Open Philanthropy split off from GiveWell](https://www.openphilanthropy.org/blog/open-philanthropy-project-now-independent-organization)); or revise the organization's mission statement, bylaws,[7](https://www.cold-takes.com/p/47f975d8-08f1-4e66-a7c6-6ba4e182cc1a/#fn7) etc.; or other things that significantly reduce the scope of what the board has control over. The board needs to represent its own interests in these cases, rather than deferring to the CEO (whose interests may be different).

* **Overseeing big-picture irreversible risks and decisions that could importantly affect future CEOs.** For example, I think the board needs to be anticipating any major source of risk that a nonprofit collapses (financially or otherwise) - if this happens, the board can't simply replace the CEO and move on, because the collapse affects what a future CEO is able to do. (What risks and decisions are big enough? Some thoughts in a footnote.[8](https://www.cold-takes.com/p/47f975d8-08f1-4e66-a7c6-6ba4e182cc1a/#fn8))

* **All matters relating to the composition and performance of the board itself.** Adding new board members, removing board members, and reviewing the board's own performance are things that the board needs to be responsible for, not the CEO. If the CEO is controlling the composition of the board, this is at odds with the board's role in overseeing the CEO.

### Engaging on main duties, staying out of the way otherwise

I think the ideal board member's behavior is roughly along the lines of the following:

**Actively, intensively engage in the [main duties](https://www.cold-takes.com/p/47f975d8-08f1-4e66-a7c6-6ba4e182cc1a/#the-boards-main-duties) from the previous section.** Board members should be knowledgeable about, and not defer to the CEO on, (a) how the CEO is performing; (b) how the board is performing, and who should be added and removed; (c) spotting (and scanning the horizon for) events that could reduce the board's powers, or lead to big enough problems and restrictions so as to irreversibly affect what future CEOs are able to do.

Ideally they should be focusing their questions in board meetings on these things, as well as having some way of gathering information about them that doesn't just rely on hearing directly from the CEO. (Some ideas for this are below.) When reviewing financial statements and budgets, they should be focused mostly on the risk of major irreversible problems (such as going bankrupt or failing to be compliant); when hearing about activities, they should be focused mostly on what they reflect about the CEO's performance; etc.

**Be advisory ("stay out of the way") otherwise.** Meetings might contain all sorts of updates and requests for reactions. I think a good template for a board member, when sharing an opinion or reaction, is either to (a) explain as they're talking why this topic is important for the board's main duties; or (b) say (or imply) something like "I'm curious / offering an opinion about \_\_\_, but if this isn't helpful, please ignore it, and please don't hesitate to move the meeting to the next topic as soon as this stops feeling productive."

The combination of intense engagement on core duties and "staying out of the way" otherwise **can make this a very weird role.** An organization will often go years without any serious questions about the CEO's performance or other matters involving [core duties.](https://www.cold-takes.com/p/47f975d8-08f1-4e66-a7c6-6ba4e182cc1a/#the-boards-main-duties) So a board member ought to be ready to quietly nod along and stay out of the way for very long stretches of time, while being ready to get seriously involved and engaged when this makes sense.

**Aim for division of labor.** I think a major problem with nonprofit boards is that, by default, it's really unclear which board member is responsible for what. I think it's a good idea for board members to explicitly settle this via assigning:

* Specialists ("Board member X is reviewing the financials; the rest of us are mostly checked-out and/or sanity-checking on that");

* Subcommittees ("Board members X and Y will look into this particular aspect of the CEO's performance");

* A Board Chair or Lead Independent Director[9](https://www.cold-takes.com/p/47f975d8-08f1-4e66-a7c6-6ba4e182cc1a/#fn9) who is the default person to take responsibility for making sure the board is doing its job well (this could include suggesting and assigning responsibility for some of the ideas I list [below](https://www.cold-takes.com/p/47f975d8-08f1-4e66-a7c6-6ba4e182cc1a/#a-few-practices-that-seem-good); helping to set the agenda for board meetings so it isn't just up to the CEO; etc.)

This can further help everyone find a balance between engaging and staying out of the way.

### Who should be on the board?

One answer is that it should be whoever can do well at the duties outlined above - both in terms of substance (can they accurately evaluate the CEO's performance, identify big-picture irreversible risks, etc.?) and in terms of style (do they actively engage on their [main duties](https://www.cold-takes.com/p/47f975d8-08f1-4e66-a7c6-6ba4e182cc1a/#the-boards-main-duties) and stay out of the way otherwise?)

But to make things a bit more boiled-down and concrete, I think perhaps the most important test for a board member is: **they'll get the CEO replaced if this would be good for the nonprofit's mission, and they won't if it wouldn't be.**

This is the most essential function of the board, and it implies a bunch of things about who makes a good board member:

* They need to **do a great job understanding and representing the nonprofit's mission, and care deeply about that mission** - to the point of being ready to create conflict over it if needed (and only if needed).
	+ A [key challenge](https://www.cold-takes.com/p/47f975d8-08f1-4e66-a7c6-6ba4e182cc1a/#great-power-low-engagement-unclear-responsibility-no-accountability) of nonprofits is that they have no clear goal, only a mission statement that is open to interpretation. And if two different board members interpret the mission differently - or are focused on different aspects of it - this could intensely color how they evaluate the CEO, which could be a huge deal for the nonprofit.

	+ For example, if a nonprofit's mission is "Help animals everywhere," does this mean "Help as many animals as possible" (which might indicate a move toward focusing on farm animals) or "Help animals in the same way the nonprofit traditionally has" or something else? How does it imply the nonprofit should make tradeoffs between helping e.g. dogs, cats, elephants, chickens, fish or even insects? How a board member answers questions like this seems central to how their presence on the board is going to affect the nonprofit.* They **need to have a personality and position capable of challenging the CEO** (though also capable of staying out of the way)**.**
	+ A common problem I see is that some board member is (a) not very engaged with the nonprofit itself, but (b) highly values their personal relationship with the CEO and other board members. This seems like a bad combination, but unfortunately a common one. Board members need to be willing and able to create conflict in order to do the right thing for the nonprofit.

	+ Limiting the number of board members who are employees (reporting to the CEO) seems important for this reason.

	+ If you can't picture a board member "making waves," they probably shouldn't be on the board - that attitude will seem fine more than 90% of the time, but it won't work well in the rare cases where the board really matters.

	+ On the other hand, if someone is *only comfortable* "making waves" and feels useless and out of sorts when they're just nodding along, that person shouldn't be on the board either. As noted above, board members need to be ready for a weird job that involves stepping up when the situation requires it, but staying out of the way when it doesn't.* They should probably have a **well-developed take on what their job is as a board member.** Board members who can't say much about where they expect to be highly engaged, vs. casually advisory - and how they expect to invest in getting the knowledge they need to do a good job leading on particular issues - don't seem like great bets to step up when they most need to (or stay out of the way when they should).

In my experience, most nonprofits are not looking for these qualities in board members. They are, instead, often looking for things like:

* Celebrity and reputation - board members who are generally impressive and well-regarded and make the nonprofit look good. Unfortunately, I think such people often just don't have much time or interest for their job. Many are also uninterested in causing any conflict, which makes them basically useless as board members IMO.

* Fundraising - a lot of nonprofits pretty much explicitly just try to put people on the board who will help raise money for them. This seems bad for governance.

* Narrow expertise on some topic that is important for the nonprofit. I don't really think this is what nonprofits should be seeking from board members,[10](https://www.cold-takes.com/p/47f975d8-08f1-4e66-a7c6-6ba4e182cc1a/#fn10) except to the extent it ties deeply into the board members' [core duties](https://www.cold-takes.com/p/47f975d8-08f1-4e66-a7c6-6ba4e182cc1a/#the-boards-main-duties), e.g., where it's important to have an independent view on technical topic X in order to do a good job evaluating the CEO.

I think a good profile for a board member is someone who cares greatly about the nonprofit's mission, and wants it to succeed, to the point where they're ready to have tough conversations if they see the CEO falling short. Examples of such people might be major funders, or major stakeholders (e.g., a community leader from a community of people the nonprofit is trying to help).

### A few practices that seem good

I'll anticlimactically close with a few practices that seem helpful to me. These are mostly pretty generic practices, useful for both for-profit and nonprofit boards, that I have seen working in practice but also seen too many boards going without. They don't fully address the weirdnesses discussed above (especially the stuff specific to nonprofit as opposed to for-profit boards), but they seem to make things some amount better.

**Keeping it simple for low-stakes organizations.** If a nonprofit is a year old and has 3 employees, it probably shouldn't be investing a ton of its energy in having a great board (especially since this is hard).

A key question is: "If the board just stays checked out and doesn't hold the CEO accountable, what's the worst thing that can happen?" If the answer is something like "The nonprofit's relatively modest budget is badly spent," then it might not be worth a huge investment in building a great board (and in taking some of the measures listed below). Early-stage nonprofits often have a board consisting of 2-3 people the founder trusts a lot (ideally in a "you'd fire me if it were the right thing to do" sense rather than in a "you've always got my back" sense), which seems fine. The rest of these ideas are for when the stakes are higher.

**Formal board-staff communication channels.** A very common problem I see is that:

* Board members know almost nothing about the organization, and so are hesitant to engage in much of anything.

* Employees of the organization know far more, but find the board members mysterious/unapproachable/scary, and don't share much information with them.

I've seen this dynamic improved some amount by things like a **staff liaison**: a board member who is designated with the duty, "Talk to employees a lot, offer them confidentiality as requested, try to build trust, and gather information about how things are going." Things like regular "office hours" and showing up to company events can help with this.

**Viewing board seats as limited.** It seems unlikely that a board should have more than 10 members (and even 10 seems like a lot), since it's hard to have a productive meeting past that point.[11](https://www.cold-takes.com/p/47f975d8-08f1-4e66-a7c6-6ba4e182cc1a/#fn11) When considering a new addition to the board, I think the board should be asking something much closer to "Is this one of the 10 best people in the world to sit on this board?" than to "Is this person fine?"

**Regular CEO reviews.**
Many nonprofits don't seem to have any formal, regular process for reviewing the CEO's performance; I think it's important to do this.

The most common format I've seen is something like: one board member interviews the CEO's direct reports, and perhaps some other people throughout the company, and integrates this with information about the organization's overall progress and accomplishments (often presented by the organization itself, but they might ask questions about it) to provide a report on what the CEO is doing well and could do better. I think this approach has a lot of limitations - staff are often hesitant to be forthcoming with a board member (even when promised anonymity), and the board member often lacks a lot of key information - but even with those issues, it tends to be a useful exercise.

**Closed sessions.** I think it's important for the board to have "closed sessions" where board members can talk frankly without the CEO, other employees, etc. hearing. I think a common mistake is to ask "Does anyone want the closed session today or can we skip it?" - this puts the onus on board members to say "Yes, I would like a closed session," which then implies they have something negative to say. I think it's better for whoever's running the meetings to identify logical closed sessions (e.g., "The board minus employees"), allocate time for them and force them to happen.

**Regular board reviews.** It seems like it would be a good idea for board members to regularly assess each other's performance, and the performance of the board as a whole. But I've actually seen very little of this done in practice and I can't point to versions of it that seem to have some track record of working well. It does seem like a good idea though!

Conclusion
----------

The board is the only body at a nonprofit that can hold the CEO accountable to accomplishing the mission. I broadly feel like most nonprofit boards just aren't very well-suited to this duty, or necessarily to much of anything. It's an inherently weird structure that seems difficult to make work.

I wish someone would do a great job studying and laying out how nonprofit boards should be assembled, how they should do their job and how they can be held accountable. You can think of this post as my quick, informal shot at that.

---

Footnotes
---------

1. I'm using the term "CEO" throughout, although the chief executive at a non profit sometimes has another title, such as "Executive Director." [↩](#fnref1)- A lot of this piece is about how the *fundamental setup* of a nonprofit board leads to the kinds of problems and dynamics I'm describing. This doesn't mean we should necessarily think there's any way to fix it or any better alternative. It just means that this setup seems to bring a lot of friction points and challenges that *most* relationships between supervisor-and-supervised don't seem to have, which can make the experience of interacting with a board feel vaguely unlike what we're used to in other contexts, or "weird."

 People who have interacted with tons of boards might get so used to these dynamics that they no longer feel weird. I haven't reached that point yet myself though.

 [↩](#fnref2)- The fact that the nonprofit's goals aren't clearly defined and have no clear metric (and often aren't susceptible to measurement at all) is a pretty general challenge of nonprofits, but I think it especially shows up for a structure (the board) that is already weird in the various other ways I'm describing. [↩](#fnref3)- Superficially, you could make most of the same complaints about shareholders of a for-profit company. But:
	* Shareholders are the people who ultimately make or lose money if the company does well or poorly (you can think of this as a form of accountability). By contrast, nonprofit board members often have very little (or only an idiosyncratic) personal connection to and investment in the organization.

	* Shareholders compensate for their low engagement by picking representatives (a board) whom they can hold accountable for the company's performance. Nonprofit board members *are* the representatives, and aren't accountable to anyone. [↩](#fnref4)- Especially "good and concise." Most of the points I make here can be found in some writings on boards somewhere, but it's hard to find sensible-seeming and comprehensive discussions of what the board should be doing and who should be on it. [↩](#fnref5)- Part of the CEO's job is fundraising, and if they do a bad job of this, it's going to be obvious. But that's only part of the job. At a nonprofit, a CEO could easily be bringing in plenty of money and just doing a horrible job at the mission - and if the board isn't able to learn this and act on it, it seems like very bad news. [↩](#fnref6)- The charter and bylaws are like the "constitution" of a nonprofit, laying out how its governance works. [↩](#fnref7)- This is a judgment call, and one way to approach it would be to reserve something like 1 hour of full-board meeting time per year for talking about these sorts of things (and pouring in more time if at least, like, 1/3 of the board thinks something is a big deal).

 Some examples of things I think are and aren't usually a big enough deal to start paying serious attention to:

	* Big enough deal: financial decisions that increase the odds of going "belly-up" (running out of money and having to fold) by at least 10 percentage points. Not a big enough deal: spending money in ways that are arguably bad uses of money, having a lowish-but-not-too-far-off-of-peer-organizations amount of runway.

	* Big enough deal: deficiencies in financial controls that an auditor is highlighting, or a lack of audit altogether, until a plan is agreed to to address these things. Not a big enough deal: most other stuff in this category.

	* Big enough deal: organizations with substantial "PR risk" exposure should have a good team for assessing this and a "crisis plan" in case something happens. Not a big enough deal: specific organizational decisions and practices that you are not personally offended by or find unethical, but could imagine a negative article about. (If you do find them substantively unethical, I think that's a big enough deal.)

	* Big enough deal: transferring like 1/3 or more of valuable things the nonprofit has (intellectual property, money, etc.) to another entity not controlled by the board. Not a big enough deal: starting an affiliate organization primarily for taking donations in another country or something.

	* Big enough deal: doubling or halving the workforce. Not a big enough deal: smaller hirings and firings.[↩](#fnref8)- Sometimes the Board Chair is the CEO, and sometimes the Chair is an employee of the company who also sits on the board. In these cases, I think it's good for there to be a separate Lead Independent Director who is not employed by the company and is therefore exclusively representing the Board. They can help set agendas, lead meetings, and take responsibility by default when it's otherwise unclear who would do so. [↩](#fnref9)- Nonprofits can get expertise on topic X by hiring experts on X to advise them. The question is: when is it important to have an expert on X *evaluating the CEO*? [↩](#fnref10)- Though it could be fine and even interesting to have giant boards - 20 people, 50 or more - that have some sort of "executive committee" of 10 or fewer people doing basically all of the meetings and all of the work (with the rest functioning just as very passive, occasionally-voting equivalents of "shareholders"). Just assume I'm talking about the "executive committee" type thing here. [↩](#fnref11)


AI Could Defeat All Of Us Combined

*Click lower right to download or find on Apple Podcasts, Spotify, Stitcher, etc.*

I've been working on a new series of posts about the [most important century](https://www.cold-takes.com/most-important-century/).

* The original series focused on why and how this could be the most important century for humanity. But it had [relatively little to say about](https://www.cold-takes.com/making-the-best-of-the-most-important-century/) *what we can do today* to improve the odds of things going well.

* The new series will get much more specific about the kinds of events that might lie ahead of us, and what actions today look most likely to be helpful.

* A key focus of the new series will be the threat of [misaligned AI](https://www.cold-takes.com/transformative-ai-timelines-part-1-of-4-what-kind-of-ai/#misaligned-ai-mysterious-potentially-dangerous-objectives): AI systems disempowering humans entirely, leading to a future that has little to do with anything humans value. ([Like in the Terminator movies](https://www.slowboring.com/p/the-case-for-terminator-analogies?s=r), minus the time travel and the part where humans win.)

Many people have trouble taking this "misaligned AI" possibility seriously. They might see the broad point that AI could be dangerous, but they instinctively imagine that the danger comes from ways humans might misuse it. They find the idea of *AI itself going to war with humans* to be comical and [wild](https://www.cold-takes.com/all-possible-views-about-humanitys-future-are-wild/). I'm going to try to make this idea feel more serious and real.

As a first step, this post will **emphasize an unoriginal but extremely important point: [the kind of AI I've discussed](https://www.cold-takes.com/transformative-ai-timelines-part-1-of-4-what-kind-of-ai/)could defeat all of humanity combined, if (for whatever reason) it were pointed toward that goal.** By "defeat," I don't mean "subtly manipulate us" or "make us less informed" or something like that - I mean a literal "defeat" in the sense that we could all be killed, enslaved or forcibly contained.

I'm not talking (yet) about whether, or why, AIs *might* attack human civilization. That's for future posts. For now, I just want to linger on the point that *if* such an attack happened, it could succeed against the combined forces of the entire world.

* I think that **if you believe this, you should already be worried about misaligned AI,**[1](https://www.cold-takes.com/p/4a610336-be1d-42fc-b76f-c33e34598340/#fn1) **before any analysis of how or why an AI might form its own goals.** * We generally don't have a lot of *things that could end human civilization if they "tried"* sitting around. If we're going to create one, I think we should be asking not "Why would this be dangerous?" but "Why wouldn't it be?"

By contrast, if you don't believe that AI could defeat all of humanity combined, I expect that we're going to be miscommunicating in pretty much any conversation about AI. The kind of AI I worry about is the kind powerful enough that total civilizational defeat is a real possibility. The reason I currently spend so much time planning around speculative future technologies (instead of working on [evidence-backed, cost-effective ways of helping low-income people today](https://www.givewell.org/) - which I did for much of my career, and still think is one of the best things to work on) is because I think the stakes are *just that high*.

Below:

* I'll sketch the basic argument for why I think AI could defeat all of human civilization.
	+ Others have written about the possibility that "superintelligent" AI could manipulate humans and create overpowering advanced technologies; I'll briefly recap that case.

	+ I'll then cover a different possibility, which is that even "merely human-level" AI could still defeat us all - by quickly coming to rival human civilization in terms of total population and resources.

	+ At a high level, I think we should be worried if a huge (competitive with world population) and rapidly growing set of highly skilled humans on another planet was trying to take down civilization just by using the Internet. So we should be worried about a large set of disembodied AIs as well.* I'll briefly address a few objections/common questions:
	+ How can AIs be dangerous without bodies?

	+ If lots of different companies and governments have access to AI, won't this create a "balance of power" so that no one actor is able to bring down civilization?

	+ Won't we see warning signs of AI takeover and be able to nip it in the bud?

	+ Isn't it fine or maybe good if AIs defeat us? They have rights too.* Close with some thoughts on just how unprecedented it would be to have something on our planet capable of overpowering us all.

How AI systems could defeat all of us
-------------------------------------

There's been a lot of debate over whether AI systems might form their own "motivations" that lead them to seek the disempowerment of humanity. I'll be talking about this in future pieces, but for now I want to put it aside and imagine how things would go *if this happened.*

So, for what follows, let's proceed from the premise: **"For some weird reason, humans consistently design AI systems (with human-like research and planning abilities) that coordinate with each other to try and overthrow humanity." Then what?** What follows will necessarily feel wacky to people who find this hard to imagine, but I think it's worth playing along, because I think "we'd be in trouble if this happened" is a very important point.

### The "standard" argument: superintelligence and advanced technology

Other treatments of this question have focused on AI systems' potential to become *vastly* more intelligent than humans, to the point where they have what [Nick Bostrom calls](https://smile.amazon.com/dp/B00LOOCGB2/ref=dp-kindle-redirect?_encoding=UTF8&btkr=1) "cognitive superpowers."[2](https://www.cold-takes.com/p/4a610336-be1d-42fc-b76f-c33e34598340/#fn2) Bostrom imagines an AI system that can do things like:

* Do its own research on how to build a better AI system, which culminates in something that has incredible other abilities.

* Hack into human-built software across the world.

* Manipulate human psychology.

* Quickly generate vast wealth under the control of itself or any human allies.

* Come up with better plans than humans could imagine, and ensure that it doesn't try any takeover attempt that humans might be able to detect and stop.

* Develop advanced weaponry that can be built quickly and cheaply, yet is powerful enough to overpower human militaries.

([Wait But Why](https://waitbutwhy.com/2015/01/artificial-intelligence-revolution-2.html) reasons similarly.[3](https://www.cold-takes.com/p/4a610336-be1d-42fc-b76f-c33e34598340/#fn3))

I think many readers will already be convinced by arguments like these, and if so you might skip down to the [next major section](https://www.cold-takes.com/p/4a610336-be1d-42fc-b76f-c33e34598340/#some-quick-responses-to-objections).

But I want to be clear that I *don't* think the danger relies on the idea of "cognitive superpowers" or "superintelligence" - both of which refer to capabilities vastly beyond those of humans. **I think we still have a problem even if we assume that AIs will basically have similar capabilities to humans, and not be fundamentally or drastically more intelligent or capable.** I'll cover that next.

### How AIs could defeat humans without "superintelligence"

If we assume that AIs will basically have similar capabilities to humans, I think we still need to worry that they could come to **out-number and out-resource humans,** and could thus have the advantage if they coordinated against us.

Here's a simplified example (some of the simplifications are in this footnote[4](https://www.cold-takes.com/p/4a610336-be1d-42fc-b76f-c33e34598340/#fn4)) based on [Ajeya Cotra's "biological anchors" report](https://www.lesswrong.com/posts/KrJfoZzpSDpnrv9va/draft-report-on-ai-timelines):

* I assume that transformative AI is developed on the soonish side (around 2036 - assuming later would only make the below numbers larger), and that it initially comes in the form of a **single AI system that is able to do more-or-less the same intellectual tasks as a human.** That is, it doesn't have a human body, but it can do anything a human working remotely from a computer could do.

* I'm using the report's framework in which it's much more expensive to *train* (develop) this system than to *run* it (for example, think about how much Microsoft spent to develop Windows, vs. how much it costs for me to run it on my computer).

* The report provides a way of estimating both how much it would cost to *train* this AI system, and how much it would cost to *run* it. Using these estimates (details in footnote)[5](https://www.cold-takes.com/p/4a610336-be1d-42fc-b76f-c33e34598340/#fn5) implies that once the first human-level AI system is created, whoever created it could use the same computing power it took to create it in order to run **several hundred million copies for about a year each**.[6](https://www.cold-takes.com/p/4a610336-be1d-42fc-b76f-c33e34598340/#fn6)* This would be over 1000x the total number of Intel or Google employees,[7](https://www.cold-takes.com/p/4a610336-be1d-42fc-b76f-c33e34598340/#fn7) over 100x the total number of active and reserve personnel in the [US armed forces](https://en.wikipedia.org/wiki/United_States_Armed_Forces), and something like 5-10% the size of the world's total working-age population.[8](https://www.cold-takes.com/p/4a610336-be1d-42fc-b76f-c33e34598340/#fn8)* And that's just a starting point.
	+ This is just using the same amount of resources that went into training the AI in the first place. Since these AI systems can do human-level economic work, they can probably be used to make more money and buy or rent more hardware,[9](https://www.cold-takes.com/p/4a610336-be1d-42fc-b76f-c33e34598340/#fn9) which could quickly lead to a "population" of billions or more.

	+ In addition to making more money that can be used to run more AIs, the AIs can conduct massive amounts of research on how to use computing power more efficiently, which could mean still greater numbers of AIs run using the *same* hardware. This in turn could lead to a [feedback loop](https://www.cold-takes.com/transformative-ai-timelines-part-1-of-4-what-kind-of-ai/#explosive-scientific-and-technological-advancement) and explosive growth in the number of AIs.* Each of these AIs might have skills comparable to those of unusually highly paid humans, including scientists, software engineers and quantitative traders. It's hard to say how quickly a set of AIs like this could develop new technologies or make money trading markets, but it seems quite possible for them to amass huge amounts of resources quickly. A huge population of AIs, each able to earn a lot compared to the average human, could end up with a "virtual economy" at least as big as the human one.

To me, this is most of what we need to know: **if there's something with human-like skills, seeking to disempower humanity, with a population in the same ballpark as (or larger than) that of all humans, we've got a civilization-level problem.**

A potential counterpoint is that these AIs would merely be "virtual": if they started causing trouble, humans could ultimately unplug/deactivate the servers they're running on. I do think this fact would make life harder for AIs seeking to disempower humans, but I don't think it ultimately should be cause for much comfort. I think a large population of AIs would likely be able to find some way to achieve security from human shutdown, and go from there to amassing enough resources to overpower human civilization (especially if AIs across the world, including most of the ones humans were trying to use for help, were coordinating).

I spell out what this might look like in an [appendix](https://www.cold-takes.com/p/4a610336-be1d-42fc-b76f-c33e34598340/#appendix-how-ais-could-avoid-shutdown). In brief:

* By default, I expect the economic gains from using AI to mean that humans create huge numbers of AIs, integrated all throughout the economy, potentially including direct interaction with (and even control of) large numbers of robots and weapons.
	+ (If not, I think the situation is in many ways even more dangerous, since a single AI could make many copies of itself and have little competition for things like server space, as discussed in the [appendix](https://www.cold-takes.com/p/4a610336-be1d-42fc-b76f-c33e34598340/#what-if-humans-move-slowly-and-dont-create-many-ais).)* AIs would have multiple ways of obtaining property and servers safe from shutdown.
	+ For example, they might recruit human allies (through manipulation, deception, blackmail/threats, genuine promises along the lines of "We're probably going to end up in charge somehow, and we'll treat you better when we do") to rent property and servers and otherwise help them out.

	+ Or they might create fakery so that they're able to operate freely on a company's servers while all outward signs seem to show that they're successfully helping the company with its goals.* A relatively modest amount of property safe from shutdown could be sufficient for housing a huge population of AI systems that are recruiting further human allies, making money (via e.g. quantitative finance), researching and developing advanced weaponry (e.g., bioweapons), setting up manufacturing robots to construct military equipment, thoroughly infiltrating computer systems worldwide to the point where they can disable or control most others' equipment, etc.

* Through these and other methods, a large enough population of AIs could develop enough military technology and equipment to overpower civilization - especially if AIs across the world (including the ones humans were trying to use) were coordinating with each other.

Some quick responses to objections
----------------------------------

This has been a brief sketch of how AIs could come to outnumber and out-resource humans. There are lots of details I haven't addressed.

Here are some of the most common objections I hear to the idea that AI could defeat all of us; if I get much [demand](https://www.cold-takes.com/p/4a610336-be1d-42fc-b76f-c33e34598340/#discuss) I can elaborate on some or all of them more in the future.

**How can AIs be dangerous without bodies?** This is discussed a fair amount in the [appendix](https://www.cold-takes.com/p/4a610336-be1d-42fc-b76f-c33e34598340/#appendix-how-ais-could-avoid-shutdown). In brief:

* AIs could recruit human allies, tele-operate robots and other military equipment, make money via research and quantitative trading, etc.

* At a high level, I think we should be worried if a huge (competitive with world population) and rapidly growing set of highly skilled humans on another planet was trying to take down civilization just by using the Internet. So we should be worried about a large set of disembodied AIs as well.

**If lots of different companies and governments have access to AI, won't this create a "balance of power" so that nobody is able to bring down civilization?**

* This is a reasonable objection to many horror stories about AI and other possible advances in military technology, but if *AIs collectively have different goals from humans and are willing to coordinate with each other*[11](https://www.cold-takes.com/p/4a610336-be1d-42fc-b76f-c33e34598340/#fn11) *against us*, I think we're in trouble, and this "balance of power" idea doesn't seem to help.

 * What matters is the total number and resources of AIs vs. humans.

**Won't we see warning signs of AI takeover and be able to nip it in the bud?** I would guess we would see some warning signs, but does that mean we could nip it in the bud? Think about human civil wars and revolutions: there are some warning signs, but also, people go from "not fighting" to "fighting" pretty quickly as they see an opportunity to coordinate with each other and be successful.

**Isn't it fine or maybe good if AIs defeat us? They have rights too.**

* Maybe AIs *should* have rights; if so, it would be nice if we could reach some "compromise" way of coexisting that respects those rights.

* But if they're able to defeat us entirely, that isn't what I'd plan on getting - instead I'd expect (by default) a world run *entirely* according to whatever goals AIs happen to have.

* These goals might have essentially nothing to do with anything humans value, and could be actively counter to it - e.g., placing zero value on beauty and having zero attempts to prevent or avoid suffering).

Risks like this don't come along every day
------------------------------------------

I don't think there are a lot of things that have a serious chance of bringing down human civilization for good.

As argued in [The Precipice](https://theprecipice.com/), most natural disasters (including e.g. asteroid strikes) don't seem to be huge threats, if only because civilization has been around for thousands of years so far - implying that natural civilization-threatening events are rare.

Human civilization is pretty powerful and seems pretty robust, and accordingly, what's really scary to me is the idea of something with the same basic capabilities as humans (making plans, developing its own technology) that can outnumber and out-resource us. There aren't a lot of candidates for that.[12](https://www.cold-takes.com/p/4a610336-be1d-42fc-b76f-c33e34598340/#fn12)

AI is one such candidate, and I think that even before we engage heavily in arguments about whether AIs might seek to defeat humans, we should feel very nervous about the possibility that they could.

What about things like "AI might lead to mass unemployment and unrest" or "AI might exacerbate misinformation and propaganda" or "AI might exacerbate a wide range of other social ills and injustices"[13](https://www.cold-takes.com/p/4a610336-be1d-42fc-b76f-c33e34598340/#fn13)? I think these are real concerns - but to be honest, if they were the biggest concerns, I'd probably still be focused on [helping people in low-income countries today](https://www.givewell.org/) rather than trying to prepare for future technologies.

* Predicting the future is generally hard, and it's easy to pour effort into preparing for challenges that never come (or come in a very different form from what was imagined).

* I believe civilization is pretty robust - we've had huge changes and challenges over the last century-plus (full-scale world wars, [many dramatic changes in how we communicate with each other](https://forum.effectivealtruism.org/posts/ajBYeiggAzu6Cgb3o/biological-anchors-is-about-bounding-not-pinpointing-ai?commentId=nEeuknn4unKTWEd6i), dramatic changes in lifestyles and values) without seeming to have come very close to a collapse.

* So if I'm engaging in speculative worries about a potential future technology, I want to focus on the really, really big ones - the ones that could matter for billions of years. If there's a real possibility that AI systems will have values different from ours, and cooperate to try to defeat us, that's such a worry.

*Special thanks to Carl Shulman for discussion on this post.*

Appendix: how AIs could avoid shutdown
--------------------------------------

This appendix goes into detail about how AIs coordinating against humans could amass resources of their own without humans being able to shut down all "misbehaving" AIs.

It's necessarily speculative, and should be taken in the spirit of giving examples of how this might work - for me, the high-level concern is that a huge, coordinating population of AIs with similar capabilities to humans would be a threat to human civilization, and that we shouldn't count on any particular way of stopping it such as shutting down servers.

I'll discuss two different general types of scenarios: (a) Humans create a huge population of AIs; (b) Humans move slowly and don't create many AIs.

### How this could work if humans create a huge population of AIs

I think a reasonable default expectation is that humans do most of the work of making AI systems incredibly numerous and powerful (because doing so is profitable), which leads to a vulnerable situation. Something roughly along the lines of:

* The company that first develops transformative AI quickly starts running large numbers of copies (hundreds of millions or more), which are used to (a) do research on how to improve computational efficiency and run more copies still; (b) develop valuable intellectual property (trading strategies, new technologies) and make money.

* Over time, AI systems are rolled out widely throughout society. Their numbers grow further, and their role in the economy grows: they are used in (and therefore have direct interaction with) high-level decision-making at companies, perhaps operating large numbers of cars and/or robots, perhaps operating military drones and aircraft, etc. (This seems like a default to me over time, but it isn't strictly necessary for the situation to be risky, as I'll go through below.)

* In this scenario, the AI systems are malicious (as we've assumed), but this doesn't mean they're constantly causing trouble. Instead, they're mostly waiting for an opportunity to team up and decisively overpower humanity. In the meantime, they're mostly behaving themselves, and this is leading to their numbers and power growing.
	+ There are scattered incidents of AI systems' trying to cause trouble,[14](https://www.cold-takes.com/p/4a610336-be1d-42fc-b76f-c33e34598340/#fn14) but this doesn't cause the whole world to stop using AI or anything.

	+ A reasonable analogy might be to a typical civil war or revolution: the revolting population *mostly* avoids isolated, doomed attacks on its government, until it sees an opportunity to band together and have a real shot at victory.

(Paul Christiano's [What Failure Looks Like](https://www.lesswrong.com/posts/HBxe6wdjxK239zajf/what-failure-looks-like) examines this general flavor of scenario in a bit more detail.)

In this scenario, the AIs face a challenge: if it becomes obvious to everyone that they are trying to defeat humanity, humans could attack or shut down a few concentrated areas where most of the servers are, and hence drastically reduce AIs' numbers. So the AIs need a way of **getting one or more "AI headquarters": property they control where they can safely operate servers and factories, do research, make plans and construct robots/drones/other military equipment.**

Their goal is ultimately to have enough AIs, robots, etc. to be able to defeat the rest of humanity combined. This might mean constructing overwhelming amounts of military equipment, or thoroughly infiltrating computer systems worldwide to the point where they can disable or control most others' equipment, or researching and deploying extremely powerful weapons (e.g., bioweapons), or a combination.

Here are some ways they could get to that point:

* They could **recruit human allies** through many different methods - manipulation, deception, blackmail and other threats, genuine promises along the lines of "We're probably going to end up in charge somehow, and we'll treat you better when we do."
	+ Human allies could be given valuable intellectual property (developed by AIs), given instructions for making lots of money, and asked to rent their own servers and acquire their own property where an "AI headquarters" can be set up. Since the "AI headquarters" would officially be human property, it could be very hard for authorities to detect and respond to the danger.

	+ Via threats, AIs might be able to get key humans to cooperate with them - such as political leaders, or the CEOs of companies running lots of AIs. This would open up further strategies.* As assumed above, particular companies are running huge numbers of AIs. The AIs being run by these companies might find security holes in the companies' servers (this isn't the topic of this piece, but my general impression is that security holes are widespread and that reasonably competent people can find many of them)[15](https://www.cold-takes.com/p/4a610336-be1d-42fc-b76f-c33e34598340/#fn15), and thereby might find opportunities to **create durable "fakery"** about what they're up to.
	+ E.g., they might set things up so that as far as humans can tell, it looks like all of the AI systems are hard at work creating profit-making opportunities for the company, when in fact they're essentially using the server farm as their headquarters - and/or trying to establish a headquarters somewhere else (by recruiting human allies, sending money to outside bank accounts, using that money to acquire property and servers, etc.)* If AIs are in wide enough use, they might already be operating lots of drones and other military equipment, in which case it could be pretty straightforward to be able to defend some piece of territory - or to strike a deal with some government to enlist its help in doing so.

* AIs could mix-and-match the above methods and others: for example, creating "fakery" long enough to recruit some key human allies, then attempting to threaten and control humans in key positions of power to the point where they control solid amounts of military resources, then using this to establish a "headquarters."

In any of these cases, once one or more "AI headquarters" are established:

* It can be expanded as the AIs headquartered there make money, purchase more property and computing power, build more factories, etc.[16](https://www.cold-takes.com/p/4a610336-be1d-42fc-b76f-c33e34598340/#fn16)* It can develop enough destructive potential and human allies that humans would be hesitant to attack it (as many relatively small countries have today). It would be clear that AIs at the headquarters are seeking autonomy, accumulating resources and taking some unethical actions, but the full scale and nature of the threat could easily remain ambiguous.

* Even if the threat is clear, AIs could potentially maintain a huge population in a relatively small space (potentially underground). They could also split up across many different headquarters (including some where they are simply using humans' hardware while hiding their presence).
	+ Either way, this large-but-hidden population could be recruiting human allies, developing advanced weaponry, looking for ways to take control of human' weapons (potentially possible if they greatly outnumber the humans working on cybersecurity, and if they can coordinate with the AIs "helping" those humans), etc.

	+ They could also be working with (unwitting, or allied) humans to manufacture large numbers of robots with back-doors allowing the AIs to take control.* Once the headquarters has built up enough resources to go on offense, **AIs around the world could coordinate with the headquarters.** Humans could shut down AIs that they notice doing this, but they might have a very tough time getting value out of their servers and AI-controlled robots; this could make it easy for the AIs at the "AI headquarters" to out-resource humans.

### What if humans move slowly and don't create many AIs?

The above scenario has humans creating large numbers of AIs, such that the AIs just need to find a way to coordinate and acquire a safe "headquarters" in order to defeat us.

What if humans moved more slowly, intentionally restricting human-level AI to a tiny portion of the available computing resources? Could a *small* number of AIs pose a risk to humanity?

In this world, we would have what Carl Shulman refers to as "dry tinder everywhere, waiting for sparks." Anyone who can buy or rent a large amount of computing power can create a large number of AIs, which can produce a large amount of money and research, leading to still more AIs.

So a single AI could hack into enough servers[17](https://www.cold-takes.com/p/4a610336-be1d-42fc-b76f-c33e34598340/#fn17) to make a few copies of itself; recruit a few human allies; and start making money, acquiring more server space, etc. until its human allies are running a huge number of AIs. This could all be done in difficult-to-detect ways (it might e.g. just look like a set of humans renting a bunch of servers to run quantitative finance strategies).

So in this world, I think our concern should be any AI that is able to find enough security holes to attain that kind of freedom. Given the current state of cybersecurity, that seems like a big concern.

---

Footnotes
---------

1. Assuming you accept other points made in the [most important century](https://www.cold-takes.com/most-important-century/) series, e.g. that AI that can do most of what humans do to [advance science and technology](https://www.cold-takes.com/transformative-ai-timelines-part-1-of-4-what-kind-of-ai/) could be developed this century. [↩](#fnref1)- See [Superintelligence](https://smile.amazon.com/Superintelligence-Dangers-Strategies-Nick-Bostrom-ebook/dp/B00LOOCGB2/) chapter 6. [↩](#fnref2)- See the "Nanotechnology blue box," in particular. [↩](#fnref3)- * The report estimates the amount of computing power it would take to *train* (create) a transformative AI system, and the amount of computing power it would take to *run* one. This is a [bounding exercise](https://www.cold-takes.com/biological-anchors-is-about-bounding-not-pinpointing-ai-timelines/) and isn't supposed to be literally predicting that transformative AI will arrive in the form of a single AI system trained in a single massive run, but here I am interpreting the report that way for concreteness and simplicity.

	* As explained in the next footnote, I use the report's figures for transformative AI arriving on the soon side (around 2036). Using its central estimates instead would strengthen my point, but we'd then be talking about a longer time from now; I find it helpful to imagine how things could go in a world where AI comes relatively soon. [↩](#fnref4)- I assume that transformative AI ends up costing about 10^14 FLOP/s to run (this is about 1/10 the Bio Anchors central estimate, and well within its error bars) and about 10^30 FLOP to train (this is about 10x the Bio Anchors central estimate for how much will be available in 2036, and corresponds to about the 30th-percentile estimate for how much will be needed based on the "short horizon" anchor). That implies that the 10^30 FLOP needed to *train* a transformative model could *run* 10^16 seconds' worth of transformative AI models, or about 300 million years' worth. This figure would be higher if we use Bio Anchors's central assumptions, rather than assumptions consistent with transformative AI being developed on the soon side. [↩](#fnref5)- They might also run fewer copies of scaled-up models or more copies of scaled-down ones, but the idea is that the total productivity of all the copies should be at *least* as high as that of several hundred million copies of a human-ish model. [↩](#fnref6)- [Intel](https://en.wikipedia.org/wiki/Intel), [Google](https://en.wikipedia.org/wiki/Google) [↩](#fnref7)- Working-age population: about [65%](https://data.worldbank.org/indicator/SP.POP.1564.TO.ZS) \* [7.9 billion](https://ourworldindata.org/world-population-growth) =~ 5 billion. [↩](#fnref8)- Humans could rent hardware using money they made from running AIs, or - if AI systems were operating on their own - they could potentially rent hardware themselves via human allies or just via impersonating a customer (you generally don't need to physically show up in order to e.g. rent server time from Amazon Web Services). [↩](#fnref9)- (I had a speculative, illustrative possibility here but decided it wasn't in good enough shape even for a footnote. I might add it later.) [↩](#fnref10)
- I don't go into detail about how AIs might coordinate with each other, but it seems like there are many options, such as by opening their own email accounts and emailing each other.  [↩](#fnref11)- Alien invasions seem unlikely if only because we have no evidence of one in millions of years. [↩](#fnref12)- Here's a recent [comment exchange](https://forum.effectivealtruism.org/posts/ajBYeiggAzu6Cgb3o/biological-anchors-is-about-bounding-not-pinpointing-ai?commentId=59wqZk8wcKdEWQG3J) I was in on this topic. [↩](#fnref13)- E.g., individual AI systems may occasionally get caught trying to steal, lie or exploit security vulnerabilities, due to various unusual conditions including bugs and errors. [↩](#fnref14)- E.g., see this [list of high-stakes security breaches](https://docs.google.com/document/d/1_smEDPWDVIaLuZ14Cm7KLHcWx4LkJ0DCTk8bcHjYy_Y/edit) and a [list of quotes about cybersecurity](https://docs.google.com/document/d/1VtV_eX-vU3bC41Il-x0OnT3DGqlL6zawWqtkisqr5Cg/edit), both courtesy of Luke Muehlhauser. For some additional not-exactly-rigorous evidence that at least shows that "cybersecurity is in really bad shape" is seen as relatively uncontroversial by at least one cartoonist, see: <https://xkcd.com/2030/>  [↩](#fnref15)- Purchases and contracts could be carried out by human allies, or just by AI systems themselves with humans willing to make deals with them (e.g., an AI system could digitally sign an agreement and wire funds from a bank account, or via cryptocurrency). [↩](#fnref16)- See [above](#fn15) note about my general assumption that today's cybersecurity has a lot of holes in it. [↩](#fnref17)