Iterated Distillation and Amplification

This is a guest post summarizing Paul Christiano’s proposed scheme for training machine learning systems that can be robustly aligned to complex and fuzzy values, which I call Iterated Distillation and Amplification (IDA) here. IDA is [notably similar](/alphago-zero-and-capability-amplification-ede767bb8446) to [AlphaGoZero](https://www.nature.com/articles/nature24270) and [expert iteration](https://arxiv.org/abs/1705.08439).

The hope is that if we use IDA to train each learned component of an AI then the overall AI will remain aligned with the user’s interests while achieving state of the art performance at runtime — provided that any non-learned components such as search or logic are also built to preserve alignment and maintain runtime performance. This document gives a high-level outline of IDA.

Motivation: The alignment/capabilities tradeoff
===============================================

Assume that we want to train a learner **A** to perform some complex fuzzy task, e.g. “Be a good personal assistant.” Assume that **A** is *capable* of learning to perform the task at a superhuman level — that is, if we could perfectly specify a “personal assistant” objective function and trained **A** to maximize it, then **A** would become afar better personal assistant than any human.

There is a spectrum of possibilities for how we might train **A** to do this task. On one end, there are techniques which allow the learner to discover powerful, novel policies that improve upon human capabilities:

* Broad reinforcement learning: As **A** takes actions in the world, we give it a relatively sparse reward signal based on how satisfied or dissatisfied we are with the eventual consequences. We then allow **A** to optimize for the expected sum of its future rewards
* Broad inverse reinforcement learning: **A** attempts to infer our deep long-term values from our actions, perhaps using a sophisticated model of human psychology and irrationality to select which of many possible extrapolations is correct.

However, it is difficult to specify a broad objective that captures everything we care about, so in practice **A** will be optimizing for some proxy that is not completely aligned with our interests. Even if this proxy objective is “almost” right, its optimumcould be disastrous according to our true values.

On the other end, there are techniques that try to narrowly emulate human judgments:

* Imitation learning: We could train **A** to exactly mimic how an expertwould do the task, e.g. by training it to fool a discriminative model trying to tell apart **A**’s actionsfrom the human expert’s actions.
* Narrow inverse reinforcement learning: We could train **A** to infer our near-term instrumental values from our actions, with the presumption that our actions are roughly optimal according to those values.
* Narrow reinforcement learning: As **A** takes actions in the world, we give it a dense reward signal based on how reasonable we judge its choices are (perhaps we directly reward state-action pairs themselves rather than outcomes in the world, as in [TAMER](http://www.cs.utexas.edu/~bradknox/papers/icdl08-knox.pdf)). **A** optimizes for the expected sum of its future rewards.

Using these techniques, the risk of misalignment is reduced significantly (though not eliminated) by restricting agents to the range of known human behavior — but this introduces severe limitations on capability. This tradeoff between allowing for novel capabilities and reducing misalignment risk applies across different learning schemes (with imitation learning generally being narrowest and lowest risk) as well as within a single scheme.

The motivating problem that IDA attempts to solve: if we are only able to align agents that narrowly replicate human behavior, how can we build an AGI that is both aligned and ultimately much more capable than the best humans?

Core concept: Analogy to AlphaGoZero
====================================

The core idea of Paul’s scheme is similar to [AlphaGoZero](https://deepmind.com/blog/alphago-zero-learning-scratch/) (AGZ): We use a learned model many times as a subroutine in a more powerful decision-making process, and then re-train the model to imitate those better decisions.

AGZ’s policy network **p** is the learned model. At each iteration, AGZ selects moves by an expensive Monte Carlo Tree Search (MCTS) which uses policy **p** as its prior; **p** is then trained to directly predict the distribution of moves that MCTS ultimately settles on. In the next iteration, MCTS is run using the new more accurate **p**, and **p** is trained to predict the eventual outcome of *that* process, and so on. After enough iterations, a fixed point is reached — **p** is unable to learn how running MCTS will change its current probabilities.

MCTS is an *amplification* of **p** — it uses **p** as a subroutine in a larger process that ultimately makes better moves than **p** alone could. In turn, **p** is a *distillation* of MCTS: it learns to directly guess the results of running MCTS, achieving comparable performance while short-cutting the expensive computation. The idea of IDA is to use the basic iterated distillation and amplification procedure in a much more general domain.

The IDA Scheme
==============

IDA involves repeatedly improving a learned model through an amplification and distillation process over multiple iterations.

Amplification is interactive and human-directed in IDA
------------------------------------------------------

In AGZ, the amplification procedure is Monte Carlo Tree Search — it’s a simple and well-understood algorithm, and there’s a clear mechanism for how it improves on the policy network’s original choices (it traverses the game tree more deeply). But in IDA, amplification is not necessarily a fixed algorithm that can be written down once and repeatedly applied; it’s an interactive process directed by human decisions.

In most domains, humans are capable of improving their native capabilities by delegating to assistants (e.g. because CEOs can delegate tasks to a large team, they can produce orders of magnitude more output per day than they could on their own). This means if our learning procedure can create an adequate *helper* for the human, the human can use the AI to amplify their ability — this human/AI system may be capable of doing things that the human couldn’t manage on their own.

Below I consider the example of using IDA to build a superhuman personal assistant. Let **A**[t] to refer to the state of the learned model after the end of iteration *t*; the initial agent **A**[0] is trained by a human overseer **H**.

Example: Building a superhuman personal assistant
-------------------------------------------------

**H** trains **A**[0] using a technique from the narrow end of the spectrum, such as imitation learning. *Here we are imagining a much more powerful version of “imitation learning” than current systems are actually capable of* — we assume that **A**[0] can acquire nearly human-level capabilities through this process. That is, the trained **A**[0] model executes all the tasks of a personal assistant as **H** would (including comprehending English instructions, writing emails, putting together a meeting schedule, etc).

Even though **A**[0] cannot discover any novel capabilities, it has two key advantages over **H**: it can run much faster, and many copies or versions of it can be run at once. We hope to leverage these advantages to construct a larger system — involving **H** and many copies of **A**[0] — that will substantially improve on **H**’s capabilities while preserving alignment with **H**’s values.

**H** can use calls to **A**[0] (along with other tools such as external memory) to become a better personal assistant. For example, **H** could assign one copy of **A**[0] to figuring out the best time to schedule the client’s recurring team meetings, another copy to figure out what to order the client for lunch, another copy to balance the client’s personal budget, etc. **H** now has the ability to get very quick solutions to sub-problems that are roughly as good as the ones **H** would have come up with on their own over a longer time period, andcan combine these results to make much better decisions than an unaided human.

Let Amplify(**H**, **A**[0]) refer to the larger system of **H** + many copies of **A**[0] + aids. Compared to **A**[0] alone, the Amplify(**H**, **A**[0]) system has much higher time and resource costs but its eventual decisions are much better. Moreover, because in each of its individual decisions each copy of **A**[0] continues to act just as a human personal assistant would act, we can hope that Amplify(**H**, **A**[0]) preserves alignment.

In the next iteration of training, the Amplify(**H**, **A**[0]) system takes over the role of **H** as the overseer. **A**[1] is trained with narrow and safe techniques to quickly reproduce the results of Amplify(**H**, **A**[0]). Because we assumed Amplify(**H**, **A**[0]) was aligned, we can hope that **A**[1] is also aligned if it is trained using sufficiently narrow techniques which introduce no new behaviors. **A**[1] is then used in Amplify(**H**, **A**[1]), which serves as an overseer to train **A**[2], and so on.

Pseudocode
----------

```
**def** IDA(**H**):
 **A** <- random initialization
 repeat:
 **A** <-Distill(Amplify(**H**, **A**))**def** Distill(overseer):
 """
 *Returns an AI trained using narrow, robust techniques to perform
 a task that the overseer already understands how to perform.*
 """**def** Amplify(human, AI): """
 Interactive process in which human uses many calls to AI to
 improve on human's native performance at relevant task(s).
 """
```
What properties must hold for IDA to work?
==========================================

The IDA scheme is a template with “slots” for Amplify and Distill procedures that have not been fully specified yet — in fact, they rely on capabilities we don’t yet have. Because IDA itself is not fully specified, it’s not clear what minimal set of properties are necessary for it to succeed.

Achieving alignment and high capability
---------------------------------------

That said, here are some general properties which seem necessary — though likely not sufficient — for IDA agents to achieve robust alignment and high capability:

1. *The* Distill *procedure robustly preserves alignment:* Given an aligned agent **H** we can use narrow safe learning techniques to train a much faster agent **A** which behaves as **H** would have behaved, without introducing any misaligned optimization or losing important aspects of what **H** values.
2. *The* Amplify *procedure robustly preserves alignment:* Given an aligned agent **A**, it is possible to specify an amplification scheme which calls **A** multiple times as a subroutine in a way that reliably avoids introducing misaligned optimization.
3. *At least some human experts are able to iteratively apply amplification to achieve arbitrarily high capabilities at the relevant task:* a) there is some threshold of general capability such that if someoneis above this threshold, they can eventually solve any problem that an arbitrarily intelligent system could solve, provided they can delegate tasks to similarly-intelligent assistants and are given arbitrary amounts of memory and time; b) at least some human experts are above this threshold of generality — given enough time and resources, they can figure out how to use AI assistants and tools to improve their capabilities arbitrarily far.

The non-profit [Ought](https://ought.org/projects) is working on gathering more evidence about assumptions 2 and 3.

Achieving competitive performance and efficiency
------------------------------------------------

Paul aims for IDA agents to be competitive with traditional RL agents in time and resource costs at *runtime* — this is a reasonable expectation because an IDA agent is ultimately just another learned model whose weights were tuned with an unusual training procedure.

Resource and time cost during *training* is a more open question; I haven’t explored the assumptions that would have to hold for the IDA training process to be practically feasible or resource-competitive with other AI projects.


How to study superintelligence strategy

(Last updated Feb. 11, 2015.)

What could an economics graduate student do to improve our strategic picture of superintelligence? What about a computer science professor? A policy analyst at RAND? A program director at IARPA?

In the last chapter of [*Superintelligence*](http://smile.amazon.com/Superintelligence-Dangers-Strategies-Nick-Bostrom/dp/0199678111/), Nick Bostrom writes:

> We find ourselves in a thicket of strategic complexity and surrounded by a dense mist of uncertainty. Though many considerations have been discerned, their details and interrelationships remain unclear and iffy — and there might be other factors we have not thought of yet. How should we act in this predicament?
>
>
> … Against a backdrop of perplexity and uncertainty, [strategic] analysis stands out as being of particularly high expected value. Illumination of our strategic situation would help us target subsequent interventions more effectively. Strategic analysis is especially needful when we are radically uncertain not just about some detail of some peripheral matter but about the cardinal qualities of the central things. For many key parameters, we are radically uncertain even about their *sign…*
>
>
> The hunt for crucial considerations… will often require crisscrossing the boundaries between different academic disciplines and other fields of knowledge.
>
>

Bostrom does not, however, provide a list of specific research projects that could illuminate our strategic situation and thereby “help us target subsequent interventions more effectively.”

Below is **my personal list of studies which could illuminate our strategic situation with regard to superintelligence**. I’m hosting it on my personal site rather than [MIRI’s blog](http://intelligence.org/blog/) to make it clear that this is *not* “MIRI’s official list of project ideas.” Other researchers at MIRI would, I’m sure, put together a different list.

I should also note that in addition to “strategic” work there is also [direct technical work on AGI safety](http://intelligence.org/research/) to be done — in fact, that’s what MIRI *focuses* on, for reasons partially enumerated [here](http://intelligence.org/2014/06/11/mid-2014-strategic-plan/).

**I’ll keep adding to this list as time passes, but I’ll preserve project numbering** so the ideas can be referred to easily and stably, e.g. “CS project 14” or “Psych project 4.” If listed projects are completed or no longer seem valuable, I’ll “cross them off” rather than deleting them from the list and thereby changing the project numbering.

**Each project description below is merely a *seed idea*** for a project; I assume published studies will differ substantially from my descriptions, depending on the judgments and affordances and creativity of each investigator.

If carried out, these studies could be published as papers, reports, dissertations, or books. Some of them could be very large in scope, others could be quite small, and most of them could be tweaked in various ways to be made more or less ambitious.

Most of the project ideas below are of broad interest, but also have implications for superintelligence strategy — implications which could be spelled out in the study itself, or not, depending on the limitations of the publication venue.

I’ve described each seed idea in a single paragraph, but each project can be described in substantially more detail if someone who would credibly carry out the study asks for more detail. Here are some examples of elaborated project idea descriptions (2-5 pages), including potential research methods, comparison studies, publishing venues, and expert consultants:

* [Survey AI experts on past progress](https://docs.google.com/document/d/1Wm5AWq52afXsrmCTemJZDmMWtNeN_3dLRvJOyQDkWMs/edit?usp=sharing)
* [Map the computing landscape](https://docs.google.com/document/d/171EPom_IKT0RQt4TSJqBb_VdlY0lE_RIaCzKVeOZbKE/edit?usp=sharing)
* [Concrete AI paths to influence](https://docs.google.com/document/d/1LnP_717GjOcKofIdgugQwcCx9be7ICYe-YfWmS2fFls/edit?usp=sharing)
* [How IQ predicts metacognition and philosophical success](https://docs.google.com/document/d/1vs_l2NHLnsfWyPyqpTDe2YBJosj93xSMgszP8yxFS40/edit?usp=sharing)

I couldn’t find a “natural” way to organize all these project ideas, so I chose what seemed like the least-terrible option and organized them by *home field of the project’s most plausible publication venues.* Many of these projects are highly interdisciplinary, but I’ve tried to guess at which publication venues were the most plausible candidates (*if* the study results were written up in a *paper*), and which field of inquiry those venues were perceived as “belonging to.” Still, this often involved fairly arbitrary guesswork, so e.g. if you’re a computer scientist then please also check the list of projects under the ‘economics’ heading, in case you find a project there that strikes your fancy and can indeed be published in a venue on the border of computer science and economics.

Finally: for brevity’s sake, I make substantial use of field-specific jargon. The list below might make more sense after you’ve read *Superintelligence*.

Okay, on to my list of quick-and-dirty research project seed ideas.

### Computer science

1. Another survey of AI scientists’ estimates on AGI timelines, takeoff speed, and likely social outcomes, with more respondents and a higher response rate than the best current survey, which is probably [Müller & Bostrom (2014)](http://www.sophia.de/pdf/2014_PT-<abbr>AI</abbr>_polls.pdf).
2. Survey AI subfield experts on rates of progress within their subfields. See project guide [here](https://docs.google.com/document/d/1-eqYP1LumqZohBTGrujyPwj9q9WUx2c2leawzbaXrV0/edit?usp=sharing).
3. How large is the field of AI currently? How many quality-adjusted researcher years, funding, and available computing resources per year? How big was the AI field in 1960, 1970, 1980, 1990, 2000, 2010? Given current trends, how large will the field be in 2020, 2030, 2040? Initial steps taken [here](http://intelligence.org/2014/01/28/how-big-is-ai/).
4. How well does an AI system’s transparency to human inspection scale, using different kinds of architectures and methods? See [here](http://intelligence.org/2013/08/25/transparency-in-safety-critical-systems/).
5. Can computational complexity theory place any interesting bounds on AI progress or AI takeoff speeds? ([Dewey](http://www.danieldewey.net/) has looked into this some.)
6. Summarize the current capabilities and limitations of methods for gaining “high assurance” in autonomous and semi-autonomous systems, e.g. in hybrid systems control, formal verification, program synthesis, simplex architectures. Explain which extant methods seem most likely to tractably scale well (for systems that are more complex, more autonomous, more general than those we have now), and what work is needed to extend the most promising methods to handle those challenges.
7. Continue [Grace (2013)](http://intelligence.org/files/AlgorithmicProgress.pdf) in measuring rates of algorithmic improvement. Filter not for ease of data collection but for other properties of algorithms, for example economic significance.
8. Construct a first-step “map of mind design space.” Are there principal components? Where do human minds sit in that space relative to apes, dolphins, current AIs, future AIs, etc.? See also [Yampolskiy (2014)](http://arxiv.org/pdf/1410.0369.pdf).
9. What are the nearest neighbors of narrow-AI “takeoff” that have actually occurred? Can they teach us anything about the plausibility of various AGI takeoff scenarios?
10. Produce an initial feasibility analysis of Christiano’s proposal for addressing the value-loading problem (*Superintelligence*, ch. 12).
11. How could one monitor and track AGI development nationally or globally?
12. Could one construct a [cryptographic box](http://lesswrong.com/lw/3cz/cryptographic_boxes_for_unfriendly_ai/) for an untrusted autonomous system?
13. Produce improved measures of (substrate-independent) general intelligence. Build on the ideas of Legg, Yudkowsky, Goertzel, Hernandez-Orallo & Dowe, etc.
14. Investigate steep temporal discounting as an incentives control method for an untrusted AGI.
15. …

### Psychology, Neuroscience, and Biology

1. How strongly does IQ predict rationality, metacognition, and philosophical sophistication, especially in the far right tail of the IQ distribution? Relevant to the interaction of [intelligence amplification and FAI chances](http://lesswrong.com/lw/iqi/intelligence_amplification_and_friendly_ai/). See the project guide [here](https://docs.google.com/document/d/18io8lEK-JVl2rWSRjwcRJMHjmJz_ik2cqVCODJFXU7c/edit?usp=sharing).
2. Is the first functional WBE likely to be (1) an emulation of low-level functionality that doesn’t require much understanding of human cognitive neuroscience at the computational level, as described in [Sandberg & Bostrom (2008)](http://www.fhi.ox.ac.uk/brain-emulation-roadmap-report.pdf), or is it more likely to be (2) an emulation that makes heavy use of advanced human cognitive neuroscience, as [described](http://intelligence.org/2014/09/09/hayworth/) by Ken Hayworth, or is it likely to be (3) something else?
3. Can we get WBE without producing neuromorphic AGI slightly earlier or shortly afterward? See section 3.2 for [Eckersley & Sandberg (2013)](http://www.degruyter.com/view/j/jagi.2013.4.issue-3/jagi-2013-0011/jagi-2013-0011.xml?format=INT).
4. List some feasible but non-realized cognitive talents for humans, and explore what could be achieved if they were given to some humans. (See *Superintelligence*, ch. 3.)
5. What can we learn about AI takeoff dynamics by studying primate brain evolution? See [Yudkowsky (2013)](http://intelligence.org/files/IEM.pdf).
6. How powerful is evolution? In what ways does it have its hands tied that human programmers aimed at general intelligence don’t? How much more efficient can we expect human researchers to be at finding general intelligence algorithms, compared to evolution? (See *Superintelligence*, ch. 2.)
7. Investigate the feasibility of emulation modulation solutions, based on currently known cognitive neuroscience.
8. Can a person’s willingness to cooperate with future generations be increased? Conduct follow-ups to e.g. [Hauser et al. (2014)](https://lukemuehlhauser.com/wp-content/uploads/Hauser-et-al-Cooperating-with-the-future.pdf).
9. …

### Economics

1. Can endogenous growth theory or unified growth theory give us any insight into AI takeoff dynamics? See [Yudkowsky (2013)](http://intelligence.org/files/IEM.pdf).
2. …

### History, forecasting, and general social science

1. Do another [GJP](http://www.goodjudgmentproject.com/)/[SciCast](https://scicast.org/)-style forecasting tournament, but with 5-year and 10-year time horizons for predictions.
2. Did most early AI scientists really think AI was right around the corner, or was it just a few people? The earliest survey available ([Michie 1973](http://commonsenseatheism.com/wp-content/uploads/2013/05/Michie-Machines-and-the-theory-of-intelligence.pdf)) suggests it may have been just a few people. For those that thought AI was right around the corner, how much did they think about the safety and ethical challenges? If they thought and talked about it substantially, why was there so little published on the subject? If they really didn’t think much about it, what does that imply about how seriously AI scientists will treat the safety and ethical challenges of AI in the future? Some relevant sources [here](http://lesswrong.com/r/discussion/lw/bd6/ai_risk_opportunity_a_timeline_of_early_ideas_and/).
3. [TG-style](http://oai.dtic.mil/oai/oai?verb=getRecord&metadataPrefix=html&identifier=ADA568107) studies of predictions from (1) *The Futurist* and *World Future Review*, (2) *Technological Forecasting and Social Change*, (3) *Foresight* and *International Journal of Forecasting*, (4) *Journal of Forecasting*, (5) publications of the Hudson Institute, (6) publications of the Institute for the Future, (7) publications of the Club of Rome, (8) *Journal of Future Studies*, (9) Ray Kurzweil (more thorough than section 5.4 [here](http://www.tandfonline.com/doi/abs/10.1080/0952813X.2014.895105?journalCode=teta20)), (10) Alvin Toffler, (11) John Naisbitt, (12) the *State of the World* reports by the Worldwatch Institute, and/or (13) other sources. What kinds of *long-term* forecasts are most accurate, by whom, and under what conditions?
4. Conduct a broad survey of past and current civilizational competence. In what ways, and under what conditions, do human civilizations show competence vs. incompetence? Which kinds of problems do they handle well or poorly? Similar in scope and ambition to, say, Perrow’s *[Normal Accidents](http://www.amazon.com/Normal-Accidents-Living-High-Risk-Technologies/dp/0691004129/)* and Sagan’s *[The Limits of Safety](http://www.amazon.com/The-Limits-Safety-Scott-Sagan/dp/0691021015/).* The aim is to get some insight into the likelihood of our civilization handling various aspects of the superintelligence challenge well or poorly. Some initial steps were taken [here](http://intelligence.org/2013/09/12/how-well-will-policy-makers-handle-agi-initial-findings/) and [here](http://intelligence.org/2014/10/16/agi-outcomes-civilizational-competence/).
5. Conduct a [Delphi](http://en.wikipedia.org/wiki/Delphi_method) study of likely AGI impacts. Participants could be AI scientists, researchers who work on high-assurance software systems, and AGI theorists.
6. Is macro-structural acceleration net good or net bad for FAI chances? See “Rates of change and cognitive enhancement” in chapter 14 of *Superintelligence*, and also Yudkowsky’s “[Do Earths with slower economic growth have a better chance at FAI?](http://lesswrong.com/lw/hoz/do_earths_with_slower_economic_growth_have_a/)”
7. Build an improved AGI forecasting model *ala* [The Uncertain Future](http://theuncertainfuture.com/). Decompose the AGI forecasting problem further, and update the program based on the latest analysis — perhaps, build it on the organization of ideas in *Superintelligence*.
8. How scalable is innovative project secrecy? Examine past cases: Manhattan project, Bletchly park, Bitcoin, Anonymous, Stuxnet, Skunk Works, Phantom Works, Google X.
9. What is the world’s distribution of computation, and what are the trends? Initial steps taken [here](http://intelligence.org/2014/02/28/the-worlds-distribution-of-computation-initial-findings/). See the project guide [here](https://docs.google.com/document/d/19K37J6VzN7aigZC4IwydEWDAYMSVBTWFcrxN6YFMxig/edit?usp=sharing).
10. Which aspects of information technology hardware and software have exhibited exponential price-performance trends in recent decades? Some notes and sources available [here](http://intelligence.org/2014/05/12/exponential-and-non-exponential/).
11. How networked will the world be in 2020, 2030, 2040?
12. How extensive and capable will robots be in 2020, 2030, 2040?
13. Scenario analysis: What are some concrete AI paths to influence over world affairs? See project guide [here](https://docs.google.com/document/d/1D7ifyRORAPD8qk5OwVlF29iSSPprxXDG6x9p7aZFlTM/edit?usp=sharing).
14. Is [Bostrom (2009)](http://www.nickbostrom.com/papers/future.pdf)’s “technological completion conjecture” true? If not, what are some predictable kinds of exceptions?
15. Produce an initial feasibility analysis of Bostrom’s “Hail Mary” approach to the value-loading problem (*Superintelligence*, ch. 12).
16. Analyze our “epistemic deference” situation: which problems do humans need to solve *before* we produce superintelligence, and which problems can be left to a properly designed superintelligence? (See *Superintelligence*, ch. 13.)
17. What is the overall current level of “state risk” from existential threats? (See *Superintelligence*, ch. 14.)
18. What are the major existential-threat “step risks” ahead of us, besides those from superintelligence? (See *Superintelligence*, ch. 14.)
19. What are some additional “technology couplings,” in addition to those named in *Superintelligence*, ch. 14?
20. What are some plausible “second-guessing arguments” with regard to superintelligence? (See *Superintelligence*, ch. 14.)
21. In practice, to what degree do human values and preferences converge upon learning new facts? To what degree has this happened in history? (Nobody values the will of Zeus anymore, presumably because we all learned the truth of Zeus’ non-existence. But perhaps such examples don’t tell us much.) See also philosophical analyses of the issue, e.g. Sobel ([1999](http://commonsenseatheism.com/wp-content/uploads/2013/10/Sobel-Do-the-desires-of-rational-agents-converge.pdf)).
22. Do we gain any insight by modeling an intelligence explosion not with two parameters (as in *Superintelligence*, ch. 4) but with four parameters — recalcitrance, algorithms, information, and computational resources? ([Dewey](http://www.danieldewey.net/) has done some thinking on this.)
23. List and examine some types of problems better solved by a speed superintelligence than by a collective superintelligence, and vice versa. Also, what are the returns on “more brains applied to the problem” (collective intelligence) for various problems? (See *Superintelligence*, ch. 3.)
24. What are the optimization power gains from mere content? What have people figured out without original theoretical advances or new experiments, but just by reading lots of known facts and putting together the pieces in a way that nobody had before?
25. What will be some major milestone for various kinds of people “taking AI seriously” in various ways? How did public perception respond to previous AI milestones? How will the public react to self-driving taxis? Etc. See the debates on [this thread](http://lesswrong.com/lw/hp5/after_critical_event_w_happens_they_still_wont/).
26. Provide more examples of decisive advantages: the ones in *Superintelligence*, ch. 5 are all pre-internet. Examine other strategically significant technology races. When do actors not maximize EV given a decisive strategic advantage of some kind?
27. Examine international collaboration on major innovative technology. How often does it happen? What blocks it from happening more? What are the necessary conditions? Examples: Concord jet, LHC, international space station, etc.
28. Signpost the future. *Superintelligence* explores many different ways the future might play out with regard to superintelligence, but cannot help being somewhat agnostic about which *particular* path the future will take. Come up with clear diagnostic signals that policy makers can use to gauge whether things are developing toward or away from one set of scenarios or another. If X does or does not happen by 2030, what does that suggest about the path we’re on? If Y ends up taking value A or B, what does that imply?
29. Which kinds of technological innovations produce public panic or outrage, under which conditions?
30. Which kinds of multipolar scenarios would predictably resolve into a singleton, and how quickly? See *Superintelligence*, ch. 11.
31. What happens when governments ban or restrict certain kinds of technological development? What happens when a certain kind of technological development is banned or restricted in one country but not in other countries where technological development sees heavy investment?
32. What kinds of innovative technology projects do governments monitor, shut down, or nationalize? How likely are major governments to monitor, shut down, or nationalize serious AGI projects?
33. Explore uploading FAI researchers as a potential solution. (See Salamon & Shulman, “[Whole Brain Emulation, as a platform for creating safe AGI](http://intelligence.org/2014/01/31/two-miri-talks-from-agi-11/).”)
34. What is the construct validity of non-anthropomorphic intelligence measures? In other words, are there convergently instrumental prediction+planning algorithms? E.g. can one tend to get agents that are good at predicting economies but not astronomical events? Or do self-modifying agents in a competitive environment tend to converge toward a specific stable attractor in general intelligence space?
35. Sure, “any level of intelligence could in principle be combined with more or less any final goal,” but what kinds of general intelligences are *plausible*? Should we expect some correlation between level of intelligence and final goals in de novo AI? How true is this in humans, and in WBEs?
36. How quickly would different kinds of agents become optimizery? How strong is the ‘optimizer’ stable attractor? Are there other stable attractors? Are tool-ish or Oracle-ish things stable attractors?
37. What does the bargaining-with-a-future-superintelligence calculus look like for guaranteeing Earth, or our galaxy, or some other slice of the observable universe for humans rather than for the AGI?
38. Do approximately all final goals make an optimizer want to control a spatial region of linearly increasing radius?
39. If a kludge AI stumbles its way into strong self-modification and becomes a maximizer, would its goal function end up being as “alien” as a paperclip maximizer?
40. Are multipolar scenarios safer at all? One intuition for thinking so might be that “inaction is safer because it leaves us with status quo”. This intuition seems wrong but may have a little something to it — e.g. maybe you can use multipolar scenarios to formalize inaction (maybe there’s a Schelling point for multiple AIs that looks more like inaction than the fixed point for a singleton which is just paperclip everything). Secondly, if you just want a sliver of the universe, maybe multipolar outcomes are safer because maybe there’s at least one superintelligence who will give us a sliver.
41. How likely is it that AGI will be a surprise to most policy-makers and industry leaders? How much advance warning are they likely to have? Some notes on this [here](http://lesswrong.com/lw/ke6/will_agi_surprise_the_world/).
42. Copied from the [AI Impacts list](http://www.aiimpacts.org/possible-investigations): “Look at the work of ancient or enlightenment mathematicians and control for possible selection effects in [this analysis](http://www.aiimpacts.org/resolutions-of-mathematical-conjectures) of historical mathematical conjectures.” This is relevant to questions of AGI development, AGI surprise, and AI takeoff speed.
43. Copied from the [AI Impacts list](http://www.aiimpacts.org/possible-investigations): “Obtain a clearer picture of the extent to which historical developments in neuroscience have played a meaningful role in historical progress in AI.”
44. How much do the goals of powerful AI agents determine outcomes in a multipolar scenario? By analogy, it’s not clear that the goals of animal agents determine ecological or animal population outcomes as much as other dynamics in the ecological system do.
45. Robin Hanson is writing a book which explores a multipolar WBE scenario, starting with the assumptions that WBEs can’t be qualitatively modified from their human sources much, and that there’s a competitive market for WBEs. One could do a similar analysis on the possible consequences of widely available AGI software, starting with the assumption that AGI software is like all other software we know in terms of reliability, design, development time, synergies between different modules, etc.
46. Enumerate the risks unique to a multipolar scenario, more thoroughly than Bostrom does in *Superintelligence*.
47. Run several principal agent problem analyses, but vary the assumptions as per different AI/WBE scenarios. E.g. if people with capital build agents, then how much of the future is controlled by the goals of those with capital compared to the goals of the created agents?
48. In a multipolar outcome, which things last as a result of initial conditions? City locations, standards, etc. can last a while. But what else?
49. When nuclear weapons arrived, experts first treated the strategic situation as the same as before but with bigger bombs. But some people thought this was a different situation, and they managed to convince others to treat this differently and to build new strategic analysis tools for this new strategic situation. There was no precedent for this at the time. How did they do that, and what can we learn from them about developing new strategic analysis tools for AI scenarios?
50. …

### Philosophy

1. What are the optimal solutions to normative uncertainty under various conditions? See [this interview](http://intelligence.org/2014/04/08/will-macaskill/) with Will MacAskill.
2. Do we need to solve the paradoxes of population ethics ([Arrhenius 2011](http://people.su.se/~guarr/Texter/The%20Impossibility%20of%20a%20Satisfactory%20Population%20Ethics%20in%20Descriptive%20and%20Normative%20Approaches%20to%20Human%20Behavior%202011.pdf)) before we have superintelligence? If so, what’s the best solution?
3. Address various problems relating to [infinite ethics](http://www.nickbostrom.com/ethics/infinite.html).
4. Copied from [Beckstead’s list](http://www.nickbeckstead.com/advice/ea-research-topics): “What do currently known approaches to decision-making under moral uncertainty imply about the case for the overwhelming importance of shaping the far future?”  Start by reading my interviews with [Beckstead](http://intelligence.org/2013/07/17/beckstead-interview/) and [MacAskill](http://intelligence.org/2014/04/08/will-macaskill/).
5. …

### Other

1. How much of humanity’s cosmic endowment can we plausibly make productive use of given AGI? One way to explore this question is via various follow-ups to [Armstrong & Sandberg (2013)](http://commonsenseatheism.com/wp-content/uploads/2013/05/Armstrong-Sandberg-Eternity-in-six-hours-intergalactic-spreading-of-intelligent-life-and-sharpening-the-Fermi-paradox.pdf). Sandberg lists several potential follow-up studies in [this interview](http://intelligence.org/2014/03/02/anders-sandberg/), for example (1) get more precise measurements of the distribution of large particles in interstellar and intergalactic space, and (2) analyze how well different long-term storable energy sources scale. See [Beckstead (2014)](http://www.effective-altruism.com/will-we-eventually-be-able-to-colonize-other-stars-notes-from-a-preliminary-review/).
2. Clarify what it would take for there to be perfectly loyal parts to an AGI, coordinating over millions of miles or light-years, etc. Distributed computing challenges in a vast space.
3. …

Acknowledgements
----------------

My thanks to Katja Grace and Amanda House for their help in preparing the elaborated project guides, and to the many people who contributed research project ideas to this list, including Nick Beckstead, Nick Bostrom, Paul Christiano, Daniel Dewey, Benja Fallenstein, Robin Hanson, Katja Grace, Louie Helm, Anna Salamon, Anders Sandberg, Carl Shulman, Qiaochu Yuan, Eliezer Yudkowsky, and probably several people whose contributions I’m forgetting. Additional project suggestions are welcome. Also see [List of Multipolar Research Projects](http://aiimpacts.org/multipolar-research-projects/) at AI Impacts, with which my own list has some overlap.