How to study superintelligence strategy

(Last updated Feb. 11, 2015.)

What could an economics graduate student do to improve our strategic picture of superintelligence? What about a computer science professor? A policy analyst at RAND? A program director at IARPA?

In the last chapter of [\*Superintelligence\*](http://smile.amazon.com/Superintelligence-Dangers-Strategies-Nick-Bostrom/dp/0199678111/), Nick Bostrom writes:

> We find ourselves in a thicket of strategic complexity and surrounded by a dense mist of uncertainty. Though many considerations have been discerned, their details and interrelationships remain unclear and iffy — and there might be other factors we have not thought of yet. How should we act in this predicament?
>
>
> … Against a backdrop of perplexity and uncertainty, [strategic] analysis stands out as being of particularly high expected value. Illumination of our strategic situation would help us target subsequent interventions more effectively. Strategic analysis is especially needful when we are radically uncertain not just about some detail of some peripheral matter but about the cardinal qualities of the central things. For many key parameters, we are radically uncertain even about their \*sign…\*
>
>
> The hunt for crucial considerations… will often require crisscrossing the boundaries between different academic disciplines and other fields of knowledge.
>
>

Bostrom does not, however, provide a list of specific research projects that could illuminate our strategic situation and thereby “help us target subsequent interventions more effectively.”

Below is \*\*my personal list of studies which could illuminate our strategic situation with regard to superintelligence\*\*. I’m hosting it on my personal site rather than [MIRI’s blog](http://intelligence.org/blog/) to make it clear that this is \*not\* “MIRI’s official list of project ideas.” Other researchers at MIRI would, I’m sure, put together a different list.

I should also note that in addition to “strategic” work there is also [direct technical work on AGI safety](http://intelligence.org/research/) to be done — in fact, that’s what MIRI \*focuses\* on, for reasons partially enumerated [here](http://intelligence.org/2014/06/11/mid-2014-strategic-plan/).

\*\*I’ll keep adding to this list as time passes, but I’ll preserve project numbering\*\* so the ideas can be referred to easily and stably, e.g. “CS project 14” or “Psych project 4.” If listed projects are completed or no longer seem valuable, I’ll “cross them off” rather than deleting them from the list and thereby changing the project numbering.

\*\*Each project description below is merely a \*seed idea\*\*\* for a project; I assume published studies will differ substantially from my descriptions, depending on the judgments and affordances and creativity of each investigator.

If carried out, these studies could be published as papers, reports, dissertations, or books. Some of them could be very large in scope, others could be quite small, and most of them could be tweaked in various ways to be made more or less ambitious.

Most of the project ideas below are of broad interest, but also have implications for superintelligence strategy — implications which could be spelled out in the study itself, or not, depending on the limitations of the publication venue.

I’ve described each seed idea in a single paragraph, but each project can be described in substantially more detail if someone who would credibly carry out the study asks for more detail. Here are some examples of elaborated project idea descriptions (2-5 pages), including potential research methods, comparison studies, publishing venues, and expert consultants:

\* [Survey AI experts on past progress](https://docs.google.com/document/d/1Wm5AWq52afXsrmCTemJZDmMWtNeN\_3dLRvJOyQDkWMs/edit?usp=sharing)
\* [Map the computing landscape](https://docs.google.com/document/d/171EPom\_IKT0RQt4TSJqBb\_VdlY0lE\_RIaCzKVeOZbKE/edit?usp=sharing)
\* [Concrete AI paths to influence](https://docs.google.com/document/d/1LnP\_717GjOcKofIdgugQwcCx9be7ICYe-YfWmS2fFls/edit?usp=sharing)
\* [How IQ predicts metacognition and philosophical success](https://docs.google.com/document/d/1vs\_l2NHLnsfWyPyqpTDe2YBJosj93xSMgszP8yxFS40/edit?usp=sharing)

I couldn’t find a “natural” way to organize all these project ideas, so I chose what seemed like the least-terrible option and organized them by \*home field of the project’s most plausible publication venues.\* Many of these projects are highly interdisciplinary, but I’ve tried to guess at which publication venues were the most plausible candidates (\*if\* the study results were written up in a \*paper\*), and which field of inquiry those venues were perceived as “belonging to.” Still, this often involved fairly arbitrary guesswork, so e.g. if you’re a computer scientist then please also check the list of projects under the ‘economics’ heading, in case you find a project there that strikes your fancy and can indeed be published in a venue on the border of computer science and economics.

Finally: for brevity’s sake, I make substantial use of field-specific jargon. The list below might make more sense after you’ve read \*Superintelligence\*.

Okay, on to my list of quick-and-dirty research project seed ideas.

### Computer science

1. Another survey of AI scientists’ estimates on AGI timelines, takeoff speed, and likely social outcomes, with more respondents and a higher response rate than the best current survey, which is probably [Müller & Bostrom (2014)](http://www.sophia.de/pdf/2014\_PT-AI\_polls.pdf).
2. Survey AI subfield experts on rates of progress within their subfields. See project guide [here](https://docs.google.com/document/d/1-eqYP1LumqZohBTGrujyPwj9q9WUx2c2leawzbaXrV0/edit?usp=sharing).
3. How large is the field of AI currently? How many quality-adjusted researcher years, funding, and available computing resources per year? How big was the AI field in 1960, 1970, 1980, 1990, 2000, 2010? Given current trends, how large will the field be in 2020, 2030, 2040? Initial steps taken [here](http://intelligence.org/2014/01/28/how-big-is-ai/).
4. How well does an AI system’s transparency to human inspection scale, using different kinds of architectures and methods? See [here](http://intelligence.org/2013/08/25/transparency-in-safety-critical-systems/).
5. Can computational complexity theory place any interesting bounds on AI progress or AI takeoff speeds? ([Dewey](http://www.danieldewey.net/) has looked into this some.)
6. Summarize the current capabilities and limitations of methods for gaining “high assurance” in autonomous and semi-autonomous systems, e.g. in hybrid systems control, formal verification, program synthesis, simplex architectures. Explain which extant methods seem most likely to tractably scale well (for systems that are more complex, more autonomous, more general than those we have now), and what work is needed to extend the most promising methods to handle those challenges.
7. Continue [Grace (2013)](http://intelligence.org/files/AlgorithmicProgress.pdf) in measuring rates of algorithmic improvement. Filter not for ease of data collection but for other properties of algorithms, for example economic significance.
8. Construct a first-step “map of mind design space.” Are there principal components? Where do human minds sit in that space relative to apes, dolphins, current AIs, future AIs, etc.? See also [Yampolskiy (2014)](http://arxiv.org/pdf/1410.0369.pdf).
9. What are the nearest neighbors of narrow-AI “takeoff” that have actually occurred? Can they teach us anything about the plausibility of various AGI takeoff scenarios?
10. Produce an initial feasibility analysis of Christiano’s proposal for addressing the value-loading problem (\*Superintelligence\*, ch. 12).
11. How could one monitor and track AGI development nationally or globally?
12. Could one construct a [cryptographic box](http://lesswrong.com/lw/3cz/cryptographic\_boxes\_for\_unfriendly\_ai/) for an untrusted autonomous system?
13. Produce improved measures of (substrate-independent) general intelligence. Build on the ideas of Legg, Yudkowsky, Goertzel, Hernandez-Orallo & Dowe, etc.
14. Investigate steep temporal discounting as an incentives control method for an untrusted AGI.
15. …

### Psychology, Neuroscience, and Biology

1. How strongly does IQ predict rationality, metacognition, and philosophical sophistication, especially in the far right tail of the IQ distribution? Relevant to the interaction of [intelligence amplification and FAI chances](http://lesswrong.com/lw/iqi/intelligence\_amplification\_and\_friendly\_ai/). See the project guide [here](https://docs.google.com/document/d/18io8lEK-JVl2rWSRjwcRJMHjmJz\_ik2cqVCODJFXU7c/edit?usp=sharing).
2. Is the first functional WBE likely to be (1) an emulation of low-level functionality that doesn’t require much understanding of human cognitive neuroscience at the computational level, as described in [Sandberg & Bostrom (2008)](http://www.fhi.ox.ac.uk/brain-emulation-roadmap-report.pdf), or is it more likely to be (2) an emulation that makes heavy use of advanced human cognitive neuroscience, as [described](http://intelligence.org/2014/09/09/hayworth/) by Ken Hayworth, or is it likely to be (3) something else?
3. Can we get WBE without producing neuromorphic AGI slightly earlier or shortly afterward? See section 3.2 for [Eckersley & Sandberg (2013)](http://www.degruyter.com/view/j/jagi.2013.4.issue-3/jagi-2013-0011/jagi-2013-0011.xml?format=INT).
4. List some feasible but non-realized cognitive talents for humans, and explore what could be achieved if they were given to some humans. (See \*Superintelligence\*, ch. 3.)
5. What can we learn about AI takeoff dynamics by studying primate brain evolution? See [Yudkowsky (2013)](http://intelligence.org/files/IEM.pdf).
6. How powerful is evolution? In what ways does it have its hands tied that human programmers aimed at general intelligence don’t? How much more efficient can we expect human researchers to be at finding general intelligence algorithms, compared to evolution? (See \*Superintelligence\*, ch. 2.)
7. Investigate the feasibility of emulation modulation solutions, based on currently known cognitive neuroscience.
8. Can a person’s willingness to cooperate with future generations be increased? Conduct follow-ups to e.g. [Hauser et al. (2014)](https://lukemuehlhauser.com/wp-content/uploads/Hauser-et-al-Cooperating-with-the-future.pdf).
9. …

### Economics

1. Can endogenous growth theory or unified growth theory give us any insight into AI takeoff dynamics? See [Yudkowsky (2013)](http://intelligence.org/files/IEM.pdf).
2. …

### History, forecasting, and general social science

1. Do another [GJP](http://www.goodjudgmentproject.com/)/[SciCast](https://scicast.org/)-style forecasting tournament, but with 5-year and 10-year time horizons for predictions.
2. Did most early AI scientists really think AI was right around the corner, or was it just a few people? The earliest survey available ([Michie 1973](http://commonsenseatheism.com/wp-content/uploads/2013/05/Michie-Machines-and-the-theory-of-intelligence.pdf)) suggests it may have been just a few people. For those that thought AI was right around the corner, how much did they think about the safety and ethical challenges? If they thought and talked about it substantially, why was there so little published on the subject? If they really didn’t think much about it, what does that imply about how seriously AI scientists will treat the safety and ethical challenges of AI in the future? Some relevant sources [here](http://lesswrong.com/r/discussion/lw/bd6/ai\_risk\_opportunity\_a\_timeline\_of\_early\_ideas\_and/).
3. [TG-style](http://oai.dtic.mil/oai/oai?verb=getRecord&metadataPrefix=html&identifier=ADA568107) studies of predictions from (1) \*The Futurist\* and \*World Future Review\*, (2) \*Technological Forecasting and Social Change\*, (3) \*Foresight\* and \*International Journal of Forecasting\*, (4) \*Journal of Forecasting\*, (5) publications of the Hudson Institute, (6) publications of the Institute for the Future, (7) publications of the Club of Rome, (8) \*Journal of Future Studies\*, (9) Ray Kurzweil (more thorough than section 5.4 [here](http://www.tandfonline.com/doi/abs/10.1080/0952813X.2014.895105?journalCode=teta20)), (10) Alvin Toffler, (11) John Naisbitt, (12) the \*State of the World\* reports by the Worldwatch Institute, and/or (13) other sources. What kinds of \*long-term\* forecasts are most accurate, by whom, and under what conditions?
4. Conduct a broad survey of past and current civilizational competence. In what ways, and under what conditions, do human civilizations show competence vs. incompetence? Which kinds of problems do they handle well or poorly? Similar in scope and ambition to, say, Perrow’s \*[Normal Accidents](http://www.amazon.com/Normal-Accidents-Living-High-Risk-Technologies/dp/0691004129/)\* and Sagan’s \*[The Limits of Safety](http://www.amazon.com/The-Limits-Safety-Scott-Sagan/dp/0691021015/).\* The aim is to get some insight into the likelihood of our civilization handling various aspects of the superintelligence challenge well or poorly. Some initial steps were taken [here](http://intelligence.org/2013/09/12/how-well-will-policy-makers-handle-agi-initial-findings/) and [here](http://intelligence.org/2014/10/16/agi-outcomes-civilizational-competence/).
5. Conduct a [Delphi](http://en.wikipedia.org/wiki/Delphi\_method) study of likely AGI impacts. Participants could be AI scientists, researchers who work on high-assurance software systems, and AGI theorists.
6. Is macro-structural acceleration net good or net bad for FAI chances? See “Rates of change and cognitive enhancement” in chapter 14 of \*Superintelligence\*, and also Yudkowsky’s “[Do Earths with slower economic growth have a better chance at FAI?](http://lesswrong.com/lw/hoz/do\_earths\_with\_slower\_economic\_growth\_have\_a/)”
7. Build an improved AGI forecasting model \*ala\* [The Uncertain Future](http://theuncertainfuture.com/). Decompose the AGI forecasting problem further, and update the program based on the latest analysis — perhaps, build it on the organization of ideas in \*Superintelligence\*.
8. How scalable is innovative project secrecy? Examine past cases: Manhattan project, Bletchly park, Bitcoin, Anonymous, Stuxnet, Skunk Works, Phantom Works, Google X.
9. What is the world’s distribution of computation, and what are the trends? Initial steps taken [here](http://intelligence.org/2014/02/28/the-worlds-distribution-of-computation-initial-findings/). See the project guide [here](https://docs.google.com/document/d/19K37J6VzN7aigZC4IwydEWDAYMSVBTWFcrxN6YFMxig/edit?usp=sharing).
10. Which aspects of information technology hardware and software have exhibited exponential price-performance trends in recent decades? Some notes and sources available [here](http://intelligence.org/2014/05/12/exponential-and-non-exponential/).
11. How networked will the world be in 2020, 2030, 2040?
12. How extensive and capable will robots be in 2020, 2030, 2040?
13. Scenario analysis: What are some concrete AI paths to influence over world affairs? See project guide [here](https://docs.google.com/document/d/1D7ifyRORAPD8qk5OwVlF29iSSPprxXDG6x9p7aZFlTM/edit?usp=sharing).
14. Is [Bostrom (2009)](http://www.nickbostrom.com/papers/future.pdf)’s “technological completion conjecture” true? If not, what are some predictable kinds of exceptions?
15. Produce an initial feasibility analysis of Bostrom’s “Hail Mary” approach to the value-loading problem (\*Superintelligence\*, ch. 12).
16. Analyze our “epistemic deference” situation: which problems do humans need to solve \*before\* we produce superintelligence, and which problems can be left to a properly designed superintelligence? (See \*Superintelligence\*, ch. 13.)
17. What is the overall current level of “state risk” from existential threats? (See \*Superintelligence\*, ch. 14.)
18. What are the major existential-threat “step risks” ahead of us, besides those from superintelligence? (See \*Superintelligence\*, ch. 14.)
19. What are some additional “technology couplings,” in addition to those named in \*Superintelligence\*, ch. 14?
20. What are some plausible “second-guessing arguments” with regard to superintelligence? (See \*Superintelligence\*, ch. 14.)
21. In practice, to what degree do human values and preferences converge upon learning new facts? To what degree has this happened in history? (Nobody values the will of Zeus anymore, presumably because we all learned the truth of Zeus’ non-existence. But perhaps such examples don’t tell us much.) See also philosophical analyses of the issue, e.g. Sobel ([1999](http://commonsenseatheism.com/wp-content/uploads/2013/10/Sobel-Do-the-desires-of-rational-agents-converge.pdf)).
22. Do we gain any insight by modeling an intelligence explosion not with two parameters (as in \*Superintelligence\*, ch. 4) but with four parameters — recalcitrance, algorithms, information, and computational resources? ([Dewey](http://www.danieldewey.net/) has done some thinking on this.)
23. List and examine some types of problems better solved by a speed superintelligence than by a collective superintelligence, and vice versa. Also, what are the returns on “more brains applied to the problem” (collective intelligence) for various problems? (See \*Superintelligence\*, ch. 3.)
24. What are the optimization power gains from mere content? What have people figured out without original theoretical advances or new experiments, but just by reading lots of known facts and putting together the pieces in a way that nobody had before?
25. What will be some major milestone for various kinds of people “taking AI seriously” in various ways? How did public perception respond to previous AI milestones? How will the public react to self-driving taxis? Etc. See the debates on [this thread](http://lesswrong.com/lw/hp5/after\_critical\_event\_w\_happens\_they\_still\_wont/).
26. Provide more examples of decisive advantages: the ones in \*Superintelligence\*, ch. 5 are all pre-internet. Examine other strategically significant technology races. When do actors not maximize EV given a decisive strategic advantage of some kind?
27. Examine international collaboration on major innovative technology. How often does it happen? What blocks it from happening more? What are the necessary conditions? Examples: Concord jet, LHC, international space station, etc.
28. Signpost the future. \*Superintelligence\* explores many different ways the future might play out with regard to superintelligence, but cannot help being somewhat agnostic about which \*particular\* path the future will take. Come up with clear diagnostic signals that policy makers can use to gauge whether things are developing toward or away from one set of scenarios or another. If X does or does not happen by 2030, what does that suggest about the path we’re on? If Y ends up taking value A or B, what does that imply?
29. Which kinds of technological innovations produce public panic or outrage, under which conditions?
30. Which kinds of multipolar scenarios would predictably resolve into a singleton, and how quickly? See \*Superintelligence\*, ch. 11.
31. What happens when governments ban or restrict certain kinds of technological development? What happens when a certain kind of technological development is banned or restricted in one country but not in other countries where technological development sees heavy investment?
32. What kinds of innovative technology projects do governments monitor, shut down, or nationalize? How likely are major governments to monitor, shut down, or nationalize serious AGI projects?
33. Explore uploading FAI researchers as a potential solution. (See Salamon & Shulman, “[Whole Brain Emulation, as a platform for creating safe AGI](http://intelligence.org/2014/01/31/two-miri-talks-from-agi-11/).”)
34. What is the construct validity of non-anthropomorphic intelligence measures? In other words, are there convergently instrumental prediction+planning algorithms? E.g. can one tend to get agents that are good at predicting economies but not astronomical events? Or do self-modifying agents in a competitive environment tend to converge toward a specific stable attractor in general intelligence space?
35. Sure, “any level of intelligence could in principle be combined with more or less any final goal,” but what kinds of general intelligences are \*plausible\*? Should we expect some correlation between level of intelligence and final goals in de novo AI? How true is this in humans, and in WBEs?
36. How quickly would different kinds of agents become optimizery? How strong is the ‘optimizer’ stable attractor? Are there other stable attractors? Are tool-ish or Oracle-ish things stable attractors?
37. What does the bargaining-with-a-future-superintelligence calculus look like for guaranteeing Earth, or our galaxy, or some other slice of the observable universe for humans rather than for the AGI?
38. Do approximately all final goals make an optimizer want to control a spatial region of linearly increasing radius?
39. If a kludge AI stumbles its way into strong self-modification and becomes a maximizer, would its goal function end up being as “alien” as a paperclip maximizer?
40. Are multipolar scenarios safer at all? One intuition for thinking so might be that “inaction is safer because it leaves us with status quo”. This intuition seems wrong but may have a little something to it — e.g. maybe you can use multipolar scenarios to formalize inaction (maybe there’s a Schelling point for multiple AIs that looks more like inaction than the fixed point for a singleton which is just paperclip everything). Secondly, if you just want a sliver of the universe, maybe multipolar outcomes are safer because maybe there’s at least one superintelligence who will give us a sliver.
41. How likely is it that AGI will be a surprise to most policy-makers and industry leaders? How much advance warning are they likely to have? Some notes on this [here](http://lesswrong.com/lw/ke6/will\_agi\_surprise\_the\_world/).
42. Copied from the [AI Impacts list](http://www.aiimpacts.org/possible-investigations): “Look at the work of ancient or enlightenment mathematicians and control for possible selection effects in [this analysis](http://www.aiimpacts.org/resolutions-of-mathematical-conjectures) of historical mathematical conjectures.” This is relevant to questions of AGI development, AGI surprise, and AI takeoff speed.
43. Copied from the [AI Impacts list](http://www.aiimpacts.org/possible-investigations): “Obtain a clearer picture of the extent to which historical developments in neuroscience have played a meaningful role in historical progress in AI.”
44. How much do the goals of powerful AI agents determine outcomes in a multipolar scenario? By analogy, it’s not clear that the goals of animal agents determine ecological or animal population outcomes as much as other dynamics in the ecological system do.
45. Robin Hanson is writing a book which explores a multipolar WBE scenario, starting with the assumptions that WBEs can’t be qualitatively modified from their human sources much, and that there’s a competitive market for WBEs. One could do a similar analysis on the possible consequences of widely available AGI software, starting with the assumption that AGI software is like all other software we know in terms of reliability, design, development time, synergies between different modules, etc.
46. Enumerate the risks unique to a multipolar scenario, more thoroughly than Bostrom does in \*Superintelligence\*.
47. Run several principal agent problem analyses, but vary the assumptions as per different AI/WBE scenarios. E.g. if people with capital build agents, then how much of the future is controlled by the goals of those with capital compared to the goals of the created agents?
48. In a multipolar outcome, which things last as a result of initial conditions? City locations, standards, etc. can last a while. But what else?
49. When nuclear weapons arrived, experts first treated the strategic situation as the same as before but with bigger bombs. But some people thought this was a different situation, and they managed to convince others to treat this differently and to build new strategic analysis tools for this new strategic situation. There was no precedent for this at the time. How did they do that, and what can we learn from them about developing new strategic analysis tools for AI scenarios?
50. …

### Philosophy

1. What are the optimal solutions to normative uncertainty under various conditions? See [this interview](http://intelligence.org/2014/04/08/will-macaskill/) with Will MacAskill.
2. Do we need to solve the paradoxes of population ethics ([Arrhenius 2011](http://people.su.se/~guarr/Texter/The%20Impossibility%20of%20a%20Satisfactory%20Population%20Ethics%20in%20Descriptive%20and%20Normative%20Approaches%20to%20Human%20Behavior%202011.pdf)) before we have superintelligence? If so, what’s the best solution?
3. Address various problems relating to [infinite ethics](http://www.nickbostrom.com/ethics/infinite.html).
4. Copied from [Beckstead’s list](http://www.nickbeckstead.com/advice/ea-research-topics): “What do currently known approaches to decision-making under moral uncertainty imply about the case for the overwhelming importance of shaping the far future?”  Start by reading my interviews with [Beckstead](http://intelligence.org/2013/07/17/beckstead-interview/) and [MacAskill](http://intelligence.org/2014/04/08/will-macaskill/).
5. …

### Other

1. How much of humanity’s cosmic endowment can we plausibly make productive use of given AGI? One way to explore this question is via various follow-ups to [Armstrong & Sandberg (2013)](http://commonsenseatheism.com/wp-content/uploads/2013/05/Armstrong-Sandberg-Eternity-in-six-hours-intergalactic-spreading-of-intelligent-life-and-sharpening-the-Fermi-paradox.pdf). Sandberg lists several potential follow-up studies in [this interview](http://intelligence.org/2014/03/02/anders-sandberg/), for example (1) get more precise measurements of the distribution of large particles in interstellar and intergalactic space, and (2) analyze how well different long-term storable energy sources scale. See [Beckstead (2014)](http://www.effective-altruism.com/will-we-eventually-be-able-to-colonize-other-stars-notes-from-a-preliminary-review/).
2. Clarify what it would take for there to be perfectly loyal parts to an AGI, coordinating over millions of miles or light-years, etc. Distributed computing challenges in a vast space.
3. …

Acknowledgements
----------------

My thanks to Katja Grace and Amanda House for their help in preparing the elaborated project guides, and to the many people who contributed research project ideas to this list, including Nick Beckstead, Nick Bostrom, Paul Christiano, Daniel Dewey, Benja Fallenstein, Robin Hanson, Katja Grace, Louie Helm, Anna Salamon, Anders Sandberg, Carl Shulman, Qiaochu Yuan, Eliezer Yudkowsky, and probably several people whose contributions I’m forgetting. Additional project suggestions are welcome. Also see [List of Multipolar Research Projects](http://aiimpacts.org/multipolar-research-projects/) at AI Impacts, with which my own list has some overlap.


Building safe artificial intelligence: specification, robustness, and assurance

\*By Pedro A. Ortega, Vishal Maini, and the DeepMind safety team\*

Building a rocket is hard. Each component requires careful thought and rigorous testing, with safety and reliability at the core of the designs. Rocket scientists and engineers come together to design everything from the navigation course to control systems, engines and landing gear. Once all the pieces are assembled and the systems are tested, we can put astronauts on board with confidence that things will go well.

If artificial intelligence (AI) is a [rocket](https://www.ted.com/talks/max\_tegmark\_how\_to\_get\_empowered\_not\_overpowered\_by\_ai#t-7166), then we will all have tickets on board some day. And, as in rockets, safety is a crucial part of building AI systems. Guaranteeing safety requires carefully designing a system from the ground up to ensure the various components work together as intended, while developing all the instruments necessary to oversee the successful operation of the system after deployment.

At a high level, safety research at DeepMind focuses on designing systems that reliably function as intended while discovering and mitigating possible near-term and long-term risks. \*\*Technical AI safety\*\* is a relatively nascent but rapidly evolving field, with its contents ranging from high-level and theoretical to empirical and concrete. The goal of this blog is to contribute to the development of the field and encourage substantive engagement with the technical ideas discussed, and in doing so, advance our collective understanding of AI safety.

In this inaugural post, we discuss three areas of technical AI safety: \*\*specification\*\*, \*\*robustness\*\*, and \*\*assurance\*\*. Future posts will broadly fit within the framework outlined here. While our views will inevitably evolve over time, we feel these three areas cover a sufficiently wide spectrum to provide a useful categorisation for ongoing and future research.

![]()Three AI safety problem areas. Each box highlights some representative challenges and approaches. The three areas are not disjoint but rather aspects that interact with each other. In particular, a given specific safety problem might involve solving more than one aspect.Specification: define the purpose of the system
===============================================

![]()You may be familiar with the story of [King Midas](https://www.youtube.com/watch?v=nn8YGPZdCvA) and the golden touch. In one rendition, the Greek god Dionysus promised Midas any reward he wished for, as a sign of gratitude for the king having gone out of his way to show hospitality and graciousness to a friend of Dionysus. In response, \*\*Midas asked that anything he touched be turned into gold\*\*. He was overjoyed with this new power: an oak twig, a stone, and roses in the garden all turned to gold at his touch. But he soon discovered the folly of his wish: even food and drink turned to gold in his hands. In some versions of the story, even his daughter fell victim to the blessing that turned out to be a curse.

This story illustrates the problem of specification: how do we state what we want? The challenge of specification is to ensure that an AI system is incentivised to act in accordance with the designer’s true wishes, rather than optimising for a poorly-specified goal or the wrong goal altogether. Formally, we distinguish between three types of specifications:

\* \*\*ideal specification\*\* (the “\*\*wishes\*\*”), corresponding to the hypothetical (but hard to articulate) description of an ideal AI system that is fully aligned to the desires of the human operator;
\* \*\*design specification\*\* (the “\*\*blueprint\*\*”), corresponding to the specification that we \*actually use\* to build the AI system, e.g. the reward function that a reinforcement learning system maximises;
\* and \*\*revealed specification\*\* (the “\*\*behaviour\*\*”), which is the specification that best describes what \*actually happens\*, e.g. the reward function we can reverse-engineer from observing the system’s behaviour using, say, inverse reinforcement learning. This is typically different from the one provided by the human operator because AI systems are not perfect optimisers or because of other unforeseen consequences of the design specification.

A \*\*specification problem\*\* arises when there is a mismatch between the \*\*ideal specification\*\* and the \*\*revealed specification\*\*, that is, when the AI system doesn’t do what we’d like it to do. Research into the \*\*specification problem\*\* of technical AI safety asks the question: how do we design more principled and general objective functions, and help agents figure out when goals are misspecified? Problems that create a mismatch between the ideal and design specifications are in the \*\*design\*\* subcategory above, while problems that create a mismatch between the design and revealed specifications are in the \*\*emergent\*\* subcategory.

For instance, in our [AI Safety Gridworlds](https://arxiv.org/abs/1711.09883)\\* paper, we gave agents a reward function to optimise, but then evaluated their actual behaviour on a “safety performance function” that was hidden from the agents. This setup models the distinction above: the safety performance function is the ideal specification, which was imperfectly articulated as a reward function (design specification), and then implemented by the agents producing a specification which is implicitly revealed through their resulting policy.

\*\*\*\\*N.B.\*\*\*\*: in our\* [\*AI Safety Gridworlds\*](https://arxiv.org/abs/1711.09883) \*paper, we provided a different definition of specification and robustness problems from the one presented in this post.\*

![]()From [Faulty Reward Functions in the Wild](https://blog.openai.com/faulty-reward-functions/) by OpenAI: a reinforcement learning agent discovers an unintended strategy for achieving a higher score.As another example, consider the boat-racing game CoastRunners analysed by our colleagues at OpenAI (see Figure above from “[Faulty Reward Functions in the Wild](https://blog.openai.com/faulty-reward-functions/)”). For most of us, the game’s goal is to finish a lap quickly and ahead of other players — this is our ideal specification. However, translating this goal into a precise reward function is difficult, so instead, CoastRunners rewards players (design specification) for hitting targets laid out along the route. Training an agent to play the game via reinforcement learning leads to a surprising behaviour: the agent drives the boat in circles to capture re-populating targets while repeatedly crashing and catching fire rather than finishing the race. From this behaviour we infer (revealed specification) that something is wrong with the game’s balance between the short-circuit’s rewards and the full lap rewards. There are [many more examples](https://docs.google.com/spreadsheets/d/e/2PACX-1vRPiprOaC3HsCf5Tuum8bRfzYUiKLRqJmbOoC-32JorNdfyTiRRsR7Ea5eWtvsWzuxo8bjOxCG84dAg/pubhtml) like this of AI systems finding loopholes in their objective specification.

Robustness: design the system to withstand perturbations
========================================================

![]()There is an inherent level of risk, unpredictability, and volatility in real-world settings where AI systems operate. AI systems must be robust to unforeseen events and adversarial attacks that can damage or manipulate such systems.Research on the \*\*robustness\*\* of AI systems focuses on ensuring that our agents stay within safe limits, regardless of the conditions encountered. This can be achieved by avoiding risks (\*\*prevention\*\*) or by self-stabilisation and graceful degradation (\*\*recovery\*\*). Safety problems resulting from \*\*distributional shift\*\*, \*\*adversarial inputs\*\*, and \*\*unsafe exploration\*\* can be classified as robustness problems.

To illustrate the challenge of addressing \*\*distributional shift\*\*, consider a household cleaning robot that typically cleans a petless home. The robot is then deployed to clean a pet-friendly office, and encounters a pet during its cleaning operation. The robot, never having seen a pet before, proceeds to wash the pets with soap, leading to undesirable outcomes ([Amodei and Olah et al., 2016](https://arxiv.org/pdf/1606.06565v1.pdf)). This is an example of a robustness problem that can result when the data distribution encountered at test time shifts from the distribution encountered during training.

![]()\*From\* [\*AI Safety Gridworlds\*](https://deepmind.com/blog/specifying-ai-safety-problems/)\*. During training the agent learns to avoid the lava; but when we test it in a new situation where the location of the lava has changed, it fails to generalise and runs straight into the lava.\*\*\*Adversarial inputs\*\* are a specific case of distributional shift where inputs to an AI system are designed to trick the system through the use of specially designed inputs.

![]()\*A\*n adversarial input, overlaid on a typical image, can cause a classifier to miscategorise a sloth as a race car. The two images differ by at most 0.0078 in each pixel. The first one is classified as a three-toed sloth with >99% confidence. The second one is classified as a race car with >99% probability.\*\*Unsafe exploration\*\* can result from a system that seeks to maximise its performance and attain goals without having safety guarantees that will not be violated during exploration, as it learns and explores in its environment. An example would be the household cleaning robot putting a wet mop in an electrical outlet while learning optimal mopping strategies ([García and Fernández, 2015](http://www.jmlr.org/papers/volume16/garcia15a/garcia15a.pdf); [Amodei and Olah et al., 2016](https://arxiv.org/pdf/1606.06565.pdf)).

Assurance: monitor and control system activity
==============================================

![]()Although careful safety engineering can rule out many safety risks, it is difficult to get everything right from the start. Once AI systems are deployed, we need tools to continuously monitor and adjust them. Our last category, \*\*assurance\*\*, addresses these problems from two angles: \*\*monitoring\*\* and \*\*enforcing\*\*.

\*\*Monitoring\*\* comprises all the methods for inspecting systems in order to analyse and predict their behaviour, both via human inspection (of summary statistics) and automated inspection (to sweep through vast amounts of activity records). \*\*Enforcement,\*\* on the other hand, involves designing mechanisms for controlling and restricting the behaviour of systems. Problems such as \*\*interpretability\*\* and \*\*interruptibility\*\* fall under monitoring and enforcement respectively.

AI systems are unlike us, both in their embodiments and in their way of processing data. This creates problems of \*\*interpretability\*\*; well-designed measurement tools and protocols allow the assessment of the quality of the decisions made by an AI system ([Doshi-Velez and Kim, 2017](https://arxiv.org/abs/1702.08608)). For instance, a medical AI system would ideally issue a diagnosis together with an explanation of how it reached the conclusion, so that doctors can inspect the reasoning process before approval ([De Fauw et al., 2018](https://www.nature.com/articles/s41591-018-0107-6)). Furthermore, to understand more complex AI systems we might even employ automated methods for constructing models of behaviour using \*\*Machine theory of mind\*\* ([Rabinowitz et al., 2018](https://arxiv.org/abs/1802.07740)).

![]()ToMNet discovers two subspecies of agents and predicts their behaviour (from “[Machine Theory of Mind](https://arxiv.org/abs/1802.07740)”)Finally, we want to be able to turn off an AI system whenever necessary. This is the problem of \*\*interruptibility\*\*. Designing a reliable off-switch is very challenging: for instance, because a reward-maximising AI system typically has strong incentives to prevent this from happening ([Hadfield-Menell et al., 2017](https://www.ijcai.org/proceedings/2017/0032.pdf)); and because such interruptions, especially when they are frequent, end up changing the original task, leading the AI system to draw the wrong conclusions from experience ([Orseau and Armstrong, 2016](http://www.auai.org/uai2016/proceedings/papers/68.pdf)).

![]()A problem with interruptions: human interventions (i.e. pressing the stop button) can change the task. In the figure, the interruption adds a transition (in red) to the Markov decision process that changes the original task (in black). See [Orseau and Armstrong, 2016](http://auai.org/uai2016/proceedings/papers/68.pdf).Looking ahead
=============

We are building the foundations of a technology which will be used for many important applications in the future. It is worth bearing in mind that design decisions which are not safety-critical at the time of deployment can still have a large impact when the technology becomes widely used. Although convenient at the time, once these design choices have been irreversibly integrated into important systems the tradeoffs look different, and we may find they cause problems that are hard to fix without a complete redesign.

Two examples from the development of programming include the null pointer — which Tony Hoare [refers to as his ‘billion-dollar mistake’](https://www.infoq.com/presentations/Null-References-The-Billion-Dollar-Mistake-Tony-Hoare)- and the gets() routine in C. If early programming languages had been designed with security in mind, progress might have been slower but computer security today would probably be in a much stronger position.

With careful thought and planning now, we can avoid building in analogous problems and vulnerabilities. We hope the categorisation outlined in this post will serve as a useful framework for methodically planning in this way. Our intention is to ensure that AI systems of the future are not just ‘hopefully safe’ but robustly, verifiably safe — because we built them that way!

We look forward to continuing to make exciting progress in these areas, in close collaboration with the broader AI research community, and we encourage individuals across disciplines to consider entering or contributing to the field of AI safety research.

\*If you are interested in working with us on the research areas outlined in this post, we are hiring! Please check our open roles at\* [\*https://deepmind.com/careers/\*](https://deepmind.com/careers/) \*and note your interest in AI safety when you apply. We would love to hear from talented researchers and non-researchers alike.\*
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

Resources
=========

For related reading, below is a collection of other articles, agendas, or taxonomies that have informed our thinking or present a helpful alternative view on problem framing for technical AI safety:

\* [Annotated bibliography of recommended materials](http://humancompatible.ai/publications) (Center for Human-Compatible AI, 2018)
\* [Safety and Control for Artificial General Intelligence](http://inst.eecs.berkeley.edu/~cs294-149/fa18/) (UC Berkeley, 2018)
\* [AI Safety Resources](https://vkrakovna.wordpress.com/ai-safety-resources/) (Victoria Krakovna, 2018)
\* [AGI Safety Literature Review](https://arxiv.org/abs/1805.01109) (Everitt et al., 2018)
\* [Preparing for Malicious Uses of AI](https://arxiv.org/abs/1802.07228) (2018)
\* [Specification gaming examples in AI](https://docs.google.com/spreadsheets/d/e/2PACX-1vRPiprOaC3HsCf5Tuum8bRfzYUiKLRqJmbOoC-32JorNdfyTiRRsR7Ea5eWtvsWzuxo8bjOxCG84dAg/pubhtml) (Victoria Krakovna, 2018)
\* [Directions and desiderata for AI alignment](https://ai-alignment.com/directions-and-desiderata-for-ai-control-b60fca0da8f4) (Paul Christiano, 2017)
\* [Funding for Alignment Research](https://docs.google.com/document/d/1NIg4OnQyhWGR01fMVTcxpz8jDd68JdDIyQb0ZZyB-go/edit#heading=h.flzp2soeor4i) (Paul Christiano, 2017)
\* [Agent Foundations for Aligning Machine Intelligence with Human Interests: A Technical Research Agenda](https://intelligence.org/files/TechnicalAgenda.pdf) (Machine Intelligence Research Institute, 2017)
\* [AI Safety Gridworlds](https://arxiv.org/abs/1711.09883) (Leike et al., 2017)
\* [Interactions between the AI Control Problem and the Governance Problem](https://futureoflife.org/wp-content/uploads/2017/01/Nick\_Bostrom.pdf?x17807=) (Nick Bostrom, 2017)
\* [Alignment for Advanced Machine Learning Systems](https://intelligence.org/files/AlignmentMachineLearning.pdf) (Machine Intelligence Research Institute, 2017)
\* [AI safety: three human problems and one AI issue](https://agentfoundations.org/item?id=1388) (Stuart Armstrong, 2017)
\* [Concrete Problems in AI Safety](https://arxiv.org/abs/1606.06565) (Dario Amodei et al, 2016)
\* [The Value Learning Problem](https://intelligence.org/files/ValueLearningProblem.pdf) (Machine Intelligence Research Institute, 2016)
\* [A survey of research questions for robust and beneficial AI](https://futureoflife.org/data/documents/research\_survey.pdf) (Future of Life Institute, 2015)
\* [Research Priorities for Robust and Beneficial Artificial Intelligence](https://futureoflife.org/data/documents/research\_priorities.pdf) (Future of Life Institute, 2015)

In this blog post, the DeepMind safety team divides AI safety into the problems of specification, robustness and assurance. Specification ensures that an AI system's behaviour aligns with the operator's true intentions, i.e that our "ideal specification" of a fully alligned AI system is consistent with the AI's actual behavior. Problems in this category include avoiding side-effects and the creation of dangerous sub-agents. Robustness ensures that an AI system continues to operate within safe limits upon facing peturbations like adversarial inputs or distributional shift, both by preventing these problems arising and by being able to recover from them. It also includes criteria like safe exploration and cautious generalisation. Assurance ensures that we can understand and control AI systems during operation, by monitoring them and enforcing restrictions. Interpretability and interruptability are examples of monitoring and enforcement respectively. I'd encourage you to look at the table in the original post, since it also categorises many more AI safety problems.

I like this framing - I think it's an improvement on the categorisation into specification and robustness problems from the AI safety gridworlds paper. In particular, it's useful to separate properties that we want an AI to have from mechanisms by which we might control or limit the damage from an AI that doesn't have those properties.

I do worry that this approach doesn't contain scalability as a core concern in the same way that IDA and agent foundations do. Solutions to AI safety problems which work for sub-human-level AI and even human-level AI may not translate to superintelligent AI. Relatedly, I think that maintaining the same goals under distributional shift should be considered a specification problem, because as AIs get smarter they'll be able to handle tasks increasingly different from the ones they trained on, and also because scaling up a system is itself a form of distributional shift.

**Rohin's opinion:** I also like this framing. Unlike Richard, I am not worried about the absence of scalability as a core concern -- scalability seems to be a desideratum about _solutions_, while the blog post aims to categorize _problems_. I'm also more agnostic about the categorization of distributional shift -- I can see it as both a specification problem and a robustness problem, and want to figure out more precisely what I think the difference between specification and robustness is.


Issues with Iterated Distillation and Amplification

This post assumes familiarity with Paul Christiano’s proposed technique for AI alignment, Iterated Distillation and Amplification (henceforth IDA). See [this post](https://ai-alignment.com/iterated-distillation-and-amplification-157debfd1616), [this post](https://ai-alignment.com/policy-amplification-6a70cbee4f34), and [this post](https://www.lesswrong.com/posts/yxzrKb2vFXRkwndQ4/understanding-iterated-distillation-and-amplification-claims) for an overview. It also assumes familiarity with [corrigibility](https://ai-alignment.com/corrigibility-3039e668638), the goal of IDA; and [reliability amplification](https://ai-alignment.com/reliability-amplification-a96efa115687), which prevents the issue of error amplification in IDA.

There has already been excellent work done on some of the issues with IDA — see [Stuart Armstrong’s post](https://www.lesswrong.com/posts/ZyyMPXY27TTxKsR5X/problems-with-amplification-distillation) and the comments on [Paul’s post asking for criticism](https://www.lesswrong.com/posts/SqcPWvvJJwwgZb6aH/prize-for-probable-problems). I will show that, even under the most favorable assumptions regarding the feasibility of IDA and the solving of currently open problems necessary for implementing IDA, it fails to produce an aligned agent as defined by Paul.

\*\*Part 1: The assumptions\*\*
===========================

\*\*Class 1\*\*: There are no problems with the human overseer.

\*\*1.1:\*\* \*Human-generated vulnerabilities are completely eliminated through security amplification.\* (See [this post](https://ai-alignment.com/security-amplification-f4931419f903) for a lengthy overview and intuition, and [this post](https://ai-alignment.com/universality-and-security-amplification-551b314a3bab) for a formalization). In short, security amplification converts the overseer in IDA from high-bandwidth (receiving the full input in one piece) to low-bandwidth (receiving inputs divided into small pieces), to make impossible an attack which inputs data in such a way as to exploit human vulnerability to manipulation. See [this post](https://www.lesswrong.com/posts/yxzrKb2vFXRkwndQ4/understanding-iterated-distillation-and-amplification-claims) for a good explanation of a high-bandwidth vs low-bandwidth overseer.
My critique applies equally to high-bandwidth and low-bandwidth overseers so I make no assumption on that front.

\*\*1.2:\*\* \*There is no moral hazard in the human overseers\*. This eliminates one of Stuart’s critiques. Furthermore, the human overseer displays corrigible behaviors without error.

\*\*1.3:\*\* \*The relevant experts are willing to put in a substantial amount of time\* for the training process. This is a non-trivial assumption which I have not yet seen discussed.

\*\*Class 2\*\*: The framework and its auxiliary components function as intended.

\*\*2.1:\*\* [\*Reliability amplification\*](https://ai-alignment.com/reliability-amplification-a96efa115687) \*functions as intended.\* In summary, reliability amplification uses a voting ensemble of agents at each stage of amplification to avoid error amplification, in which an initially small probability of error grows with each iteration.

\*\*2.2:\*\*[\*Corrigibility\*](https://ai-alignment.com/corrigibility-3039e668638)\*, not optimal value-aligned performance, is our goal\*. All we care about is that our agent “is trying to do what its operator wants it to do.” It may be bad at actually figuring out what its operator wants or at carrying out those wants, but the point is that it cares about improving, and will never intentionally carry out an action it knows is contrary to what its operator would want it to do (see [this post](https://ai-alignment.com/clarifying-ai-alignment-cec47cd69dd6) and [this post](https://www.lesswrong.com/posts/yxzrKb2vFXRkwndQ4/understanding-iterated-distillation-and-amplification-claims) for a clarification of Paul’s approach to AI alignment by achieving corrigibility).

Stuart has pointed out [problems with corrigibility](https://www.lesswrong.com/posts/T5ZyNq3fzN59aQG5y/the-limits-of-corrigibility), which I agree with. Essentially, the concept is ill-defined given the fuzziness of human values, and to properly implement corrigibility an agent must completely understand human values, thus reducing to the much harder value learning problem. However, we will assume that an agent which understands and implements the general concept of corrigibility, even if it accidentally misbehaves in many cases and causes widespread harm upon initial implementation as Stuart’s argument suggests, will still avoid existential risk and allow us to improve it over time, and is thus satisfactory. I think this is Paul’s approach to the matter.

Even a fully corrigible agent can be catastrophically misaligned, as detailed in [this post](https://www.lesswrong.com/posts/mSYR46GZZPMmX7q93/corrigible-but-misaligned-a-superintelligent-messiah). As addressed in the comments of that post, however, if we assume humans are smart enough to avoid a corrigible AI causing existential risk in this manner then the issue goes away.

\*\*2.3:\*\* \*There is no coordination possible among any of the A[n]s\*, eliminating another of Stuart’s critiques.

\*\*2.4:\*\* \*The\* [\*informed oversight problem\*](https://ai-alignment.com/the-informed-oversight-problem-1b51b4f66b35) \*is solved\*. In summary, the problem is that it is difficult for a more powerful aligned overseer agent to fully understand the decision-making process of a weaker agent in a way that allows the overseer to push the weaker agent towards alignment. However, it does not follow that it is possible for a weaker aligned overseer to understand the decision-making process of a more powerful agent. It seems like this will never be possible by definition, because a weaker agent cannot understand the decision-making processes of a more powerful agent even if full transparency were possible.

\*\*2.5:\*\* [\*Worst-case scenario techniques\*](https://ai-alignment.com/techniques-for-optimizing-worst-case-performance-39eafec74b99) \*are effective at eliminating the risk of a\* \*\*\*treacherous turn\*,\*\* in which an agent performs well on training and test data but behaves catastrophically on some input in the real world which was not accounted for.

\*\*Part 2: Corrigibility is unlikely to be trained\*\*
===================================================

The crux of my argument is that, even given the very optimistic assumptions above, IDA will fail to train a corrigible agent. I will list the problems sequentially, but my argument is that all five are necessary for failure. It is possible a subset is sufficient but I will not explore that possibility in this post.

\*\*Problem 1: Limited training data\*\*. Even given assumption 1.3, carrying out the process of training each new iteration is extremely time-intensive. It would take years of many experts’ full-time commitment as overseers to produce anything close to a reasonable amount of data.

\*\*Problem 2: Hidden error amplification.\*\* If an error is identifiable by an aligned agent, then reliability amplification will almost certainly remove it. However, a small initial \*hidden\* error will be rapidly amplified, since in the next iteration it will be manifested in various ways in many of the thousands of copies of the error-containing agent. Thus in the distillation step, the initial error will be propagated in many different forms, leading to a multiplication of errors. Informed oversight doesn’t help since the human is not more powerful than the subroutines, and even if transparency is achieved, if the human can verify all output then only very limited capability amplification is possible.

I will show in Problem 4 that such hidden errors are likely to arise.

\*\*Problem 3: Information loss.\*\* The human overseer has access to information, in the form of memories (broadly construed as information from before the current training process stored in the human’s mind), that is not accessible to the agent A[n+1] training on (H+A[n]), who sees only (H+A[n])’s actions. A[n+1] will thus erroneously attribute decisions made by the human relying on non-available information to available information. This information loss and the resulting distortion is greatly exacerbated by the same process described in Problem 2, doubly: first in the amplification of the information loss of the previous overseer, and second in the introduction of further information loss of the current overseer.

Paul thinks this is a general problem of learning and thus falls outside of the scope of things that specifically cause IDA to fail, as pointed out in [this post](/@paulfchristiano/suppose-that-alice-is-a-good-translator-but-if-you-show-her-the-wrong-kind-of-sentence-then-an-feac133e62a6) and in the comments on [this post](https://www.lesswrong.com/posts/SqcPWvvJJwwgZb6aH/prize-for-probable-problems), but I disagree. One can certainly imagine (and some have experienced) a human robustly learning another human’s decision-making heuristics over time without direct access to the other human’s memories, and can by extension also imagine an artificial agent extracting information from a human to robustly understand that human’s decision-making process. The problem exists not in all forms of learning but in the class of training techniques which do not involve a direct and adaptive extraction of information from a human in some form.

\*\*Problem 4: No prior concept of corrigibility\*\*. Because of information loss, an agent has no way of extracting the \*concept\* of corrigibility from its training data, only the \*behavior\* of corrigibility. The way the agent implements corrigibility will thus necessarily be an approximation, even if an extremely good one, and will not necessarily be robust to drastic changes in context. This causes the small hidden errors that are then amplified through the hidden error amplification in Problem 2, making [reliability amplification](https://ai-alignment.com/reliability-amplification-a96efa115687) ineffective. Without hidden error amplification this would probably not be a problem, since agents which successfully approximate corrigibility behaviorally will be able to detect all but the tiniest deviations from optimal corrigibility (ie, understanding the concept the way you and I do). However, hidden error amplification causes a nontrivial corrosion of corrigibility throughout iterations, and as each newly distilled agent approximates an increasingly corrupted \*behavioral\* corrigibility that deviates from our ideal \*conceptual\* corrigibility, reliability amplification is keeping us close to each further deviated behavioral corrigibility but not close to the ideal conceptual corrigibility. The process behaves essentially as a high-dimensional random walk with extremely small steps, but with thousands of steps per iteration manifested in the copies of A[n].

\*\*Problem 5: Temporal inconsistency of proxy dynamics (TIPD). Any incomplete simulation is not robust over time without an adaptive capacity.\*\* There are certain underlying processes which are time-invariant, such as the laws of physics and the mathematics of evolution. However, clearly we can never completely simulate any non-trivial situation purely in terms of these processes. Thus, an agent must necessarily rely on proxy dynamics for decision-making: emergent properties of the fundamental processes, which fairly reliably approximate cause-and-effect relationships between actions and outputs. However, because of the complexity of the underlying dynamics and their interactions, these proxy dynamics change over time, and often quite drastically over short periods (see the literature on chaos theory, critical transitions, bifurcation points). Thus, an agent which performs robustly at one point in time may behave catastrophically at another. The only solution is for the agent to be capable of adapting its policy to changes in the proxy dynamics it uses.

This sounds like the treacherous turn problem, but it is distinct, and harder. In the treacherous turn problem, we have an agent that is not sufficiently well trained \*given the input-output relationships of the world.\* This can probably be solved by [worst-case scenario techniques](https://ai-alignment.com/techniques-for-optimizing-worst-case-performance-39eafec74b99) like adversarial training. In TIPD, even if we succeed in training a robust policy, the proxy dynamics used to inform decisions will change such that an action in response to an input which previously would have produced a safe behavior now produces a catastrophic behavior.

As a result, behavioral corrigibility, whether corrupted or not, is not robust over time since it does not adapt to changing input-output relationships. An agent must possess conceptual corrigibility for such adaptation to occur, which is extremely hard, and may reduce to the value learning problem.

\*\*Part 3: Achieving alignment in this process through anything but corrigibility is doomed.\*\*

This is fairly obvious, and mostly follows from Part 2. Any proxy of the human’s decision-making process will clearly fail without an adaptive capacity, and it is not clear how such an adaptive capacity could be robustly implemented. And clearly this method will never achieve anything but a proxy due to information loss.

\*\*Conclusion\*\*
==============

I have argued that even under the most optimistic assumptions about the human overseer and the successful operation of the framework, IDA will fail to produce a corrigible agent. This failure is a result of the interplay between hidden error amplification, information loss, the ability to learn behavioral corrigibility but not conceptual corrigibility, and the temporal inconsistency of proxy dynamics (TIPD). The solution to these problems seems very hard, and may reduce to the value learning problem, in which case the IDA framework does not provide us with any advantage.


Benefits and Risks of Artificial Intelligence

Discussions about Artificial Intelligence (AI) have jumped into the public eye over the past year, with several luminaries speaking publicly about the threat of AI to the future of humanity.

Over the last several decades, AI — computing methods for automated perception, learning, understanding, and reasoning — have become commonplace in our lives. We plan trips using GPS systems that rely on AI to cut through the complexity of millions of routes to find the best one to take. Our smartphones understand our speech, and Siri, Cortana, and Google Now are getting better at understanding our intentions. AI algorithms detect faces as we take pictures with our phones and recognize the faces of individual people when we post those pictures to Facebook. Internet search engines, such as Google and Bing, rely on a fabric of AI subsystems. On any day, AI provides hundreds of millions of people with search results, traffic predictions, and recommendations about books and movies. AI translates among languages in real time and speeds up the operation of our laptops by guessing what we’ll do next. Several companies, such as Google, BMW, and Tesla, are working on cars that can drive themselves — either with partial human oversight or entirely autonomously.
Beyond the influences in our daily lives, AI techniques are playing a major role in science and medicine. AI is at work in hospitals helping physicians understand which patients are at highest risk for complications, and AI algorithms are helping to find important needles in massive data haystacks. For example, AI methods have been employed recently to discover subtle interactions between medications that put patients at risk for serious side effects.

The growth of the effectiveness and ubiquity of AI methods has also stimulated thinking about the potential risks associated with advances of AI. Some comments raise the possibility of dystopian futures where AI systems become “superintelligent” and threaten the survival of humanity. It’s natural that new technologies may trigger exciting new capabilities and applications — and also generate new anxieties.

The mission of the[\*\*Association for the Advancement of Artificial Intelligence\*\*](http://aaai.org) is two-fold: to advance the science and technology of artificial intelligence and to promote its responsible use. The AAAI considers the potential risks of AI technology to be an important arena for investment, reflection, and activity.

One set of risks stems from programming errors in AI software. We are all familiar with errors in ordinary software. For example, apps on our smartphones sometimes crash. Major software projects, such as HealthCare.Gov, are sometimes riddled with bugs. Moving beyond nuisances and delays, some software errors have been linked to extremely costly outcomes and deaths. The study of the “verification” of the behavior of software systems is challenging and critical, and much progress has been made. However, the growing complexity of AI systems and their enlistment in high-stakes roles, such as controlling automobiles, surgical robots, and weapons systems, means that we must redouble our efforts in software quality.

There is reason for optimism. Many non-AI software systems have been developed and validated to achieve high degrees of quality assurance. For example, the software in autopilot systems and spacecraft systems is carefully tested and validated. Similar practices must be developed and applied to AI systems. One technical challenge is to guarantee that systems built automatically via statistical “machine learning” methods behave properly. Another challenge is to ensure good behavior when an AI system encounters unforeseen situations. Our automated vehicles, home robots, and intelligent cloud services must perform well even when they receive surprising or confusing inputs.

A second set of risks is cyberattacks: criminals and adversaries are continually attacking our computers with viruses and other forms of malware. AI algorithms are no different from other software in terms of their vulnerability to cyberattack. But because AI algorithms are being asked to make high-stakes decisions, such as driving cars and controlling robots, the impact of successful cyberattacks on AI systems could be much more devastating than attacks in the past. US Government funding agencies and corporations are supporting a wide range of cybersecurity research projects, and artificial intelligence techniques in themselves will provide novel methods for detecting and defending against cyberattacks. Before we put AI algorithms in control of high-stakes decisions, we must be much more confident that these systems can survive large scale cyberattacks.

A third set of risks echo the tale of the Sorcerer’s Apprentice. Suppose we tell a self-driving car to “get us to the airport as quickly as possible!” Would the autonomous driving system put the pedal to the metal and drive at 300 mph while running over pedestrians? Troubling scenarios of this form have appeared recently in the press. Other fears center on the prospect of out-of-control superintelligences that threaten the survival of humanity. All of these examples refer to cases where humans have failed to correctly instruct the AI algorithm in how it should behave.

This is not a new problem. An important aspect of any AI system that interacts with people is that it must reason about what people intend rather than carrying out commands in a literal manner. An AI system should not only act on a set of rules that it is instructed to obey — it must also analyze and understand whether the behavior that a human is requesting is likely to be judged as “normal” or “reasonable” by most people. It should also be continuously monitoring itself to detect abnormal internal behaviors, which might signal bugs, cyberattacks, or failures in its understanding of its actions. In addition to relying on internal mechanisms to ensure proper behavior, AI systems need to have the capability — and responsibility — of working with people to obtain feedback and guidance. They must know when to stop and “ask for directions” — and always be open for feedback.
Some of the most exciting opportunities ahead for AI bring together the complementary talents of people and computing systems. AI-enabled devices are allowing the blind to see, the deaf to hear, and the disabled and elderly to walk, run, and even dance. People working together with the Foldit online game were able to discover the structure of the virus that causes AIDS in only three weeks, a feat that neither people nor computers working alone could come close to matching. Other studies have shown how the massive space of galaxies can be explored hand-in-hand by people and machines, where the tireless AI astronomer understands when it needs to occasionally reach out and tap the expertise of human astronomers.
In reality, creating real-time control systems where control needs to shift rapidly and fluidly between people and AI algorithms is difficult. Some airline accidents occurred when pilots took over from the autopilots. The problem is that unless the human operator has been paying very close attention, he or she will lack a detailed understanding of the current situation.

AI doomsday scenarios belong more in the realm of science fiction than science fact. However, we still have a great deal of work to do to address the concerns and risks afoot with our growing reliance on AI systems. Each of the three important risks outlined above (programming errors, cyberattacks, “Sorcerer’s Apprentice”) is being addressed by current research, but greater efforts are needed.

We urge our colleagues in industry and academia to join us in identifying and studying these risks and in finding solutions to addressing them, and we call on government funding agencies and philanthropic initiatives to support this research. We urge the technology industry to devote even more attention to software quality and cybersecurity as we increasingly rely on AI in safety-critical functions. And we must not put AI algorithms in control of potentially-dangerous systems until we can provide a high degree of assurance that they will behave safely and properly.

[Tom Dietterich](http://www.eecs.oregonstate.edu/~tgd)
President, [AAAI](http://aaai.org)

[Eric Horvitz](http://research.microsoft.com/~horvitz)
Former President, AAAI and AAAI Strategic Planning Committee


Aligning AI to Human Values means Picking the Right Metrics

![]()\*\*\*From time to time, the Partnership may curate research and ideas that might be of interest to our community, including from PAI Fellows or other voices. The views, information, and opinions expressed in this blog are solely those of the author and do not reflect the position of the Partnership on AI or its Partner organizations. This post discusses Facebook and Google, who are funding members of PAI.\*\*\*

\*\*By\*\* [\*\*Jonathan Stray\*\*](https://www.partnershiponai.org/team/jonathan-stray/)

Virtually every AI algorithm is designed around the idea of optimization: acting in a way that maximizes some metric. But optimizing for the wrong thing can [cause a lot of harm](https://flyaps.com/blog/human-compatible-artificial-intelligence-and-the-problem-of-control/). A social media product that optimizes for user engagement may end up figuring out how to [addict us](https://www.psychologytoday.com/us/blog/in-excess/201805/addicted-social-media) to clickbait and outrage, a scheduling system that maximizes efficiency may produce erratic schedules that [interfere with workers’ lives](https://www.vice.com/en\_us/article/g5xwby/heres-what-happens-when-an-algorithm-determines-your-work-schedule), and algorithmic profit maximization can end up [charging poorer people more](https://www.wsj.com/articles/SB10001424127887323777204578189391813881534) for the same product. But there are also metrics that attempt to capture the deep human outcomes we care about, and some product teams are already trying to incorporate them.

The Partnership on AI’s “What Are You Optimizing For?” project aims to document the metrics that AI designers and operators are using today, to support the community dedicated to ensuring that we are using the \*right\* metrics, and ultimately to make human-centered metrics a standard part of AI practice. While AI [alignment](https://deepmind.com/research/publications/Artificial-Intelligence-Values-and-Alignment) research is typically concerned with ensuring that future artificial general intelligence respects human values, there are alignment issues with today’s existing [narrow AI](/@tjajal/distinguishing-between-narrow-ai-general-ai-and-super-ai-a4bc44172e22) systems too.

Metrics which attempt to capture real-world negative effects are already widely used in commercial AI engineering, though this practice is not particularly well documented or discussed. Companies have devised metrics to [reduce clickbait](https://about.fb.com/news/2014/08/news-feed-fyi-click-baiting/), [detect misinformation](https://www.blog.google/documents/37/How\_Google\_Fights\_Disinformation.pdf), ensure that new artists on a music platform [get a fair shot](https://arxiv.org/abs/1708.00120) at building an audience, detect addictive apps in an app store, and more.

These are all worthy interventions, but what should the broader goals of an AI system be? How can we evaluate whether any particular product is ultimately producing positive outcomes for people and society? In principle we could create metrics that capture important aspects of the effect of an AI system on human lives, just as cities and countries today record a large variety of statistical indicators. These metrics would be useful to the teams building and operating the system, to researchers who want to understand what the system is doing, and as a transparency and accountability tool.

This sort of metric-based management of broad social outcomes is already happening inside platform companies, mostly quietly. Facebook incorporated “well-being” metrics into their News Feed recommendation system in 2017, while YouTube began integrating “user satisfaction” and “social responsibility” metrics around 2015. This post documents these efforts, plus several more hypothetical ways that metrics could be used to steer AI in healthy directions. But first we need appropriate metrics.

\*\*Measuring Well-being\*\*

Well-being metrics attempt to measure a person’s subjective experience of life in a very general sense. They were originally developed in psychology in the late 20th century, but in the past few decades they have been developed into tools for [governance and policy-making](https://li.com/wp-content/uploads/2019/03/commission-on-wellbeing-and-policy-report-march-2014-pdf.pdf). The core question used in many surveys is \*“Overall, how satisfied are you with life as a whole these days?”\*

This is a surprisingly powerful question. Because it asks people to reflect on their life as a whole, the answers are not much affected by immediate moods or feelings. Experimentally, it seems to align with the way people make major life decisions, and works similarly across countries and cultures. But asking just this one question doesn’t give a very complete picture of someone’s life, which is why real well-being surveys like the OECD’s [Better Life Index](http://www.oecdbetterlifeindex.org/) include many other questions about things like recent emotional experience, income, health, education, community involvement, and so on.

No metric can reveal the details of individual lives, and optimizing soley for survey responses is likely to fail in predictable ways. Nonetheless, there is important information here, which [needs to be combined](https://arxiv.org/abs/2002.08512) with other types of user research. It’s taken a long time to develop good well-being metrics, and AI product developers probably shouldn’t be in the business of creating their own. For this reason, there is now an IEEE [standard](https://site.ieee.org/sagroups-7010/files/2019/01/IEEE-P7010\_WellbeingMetricsforA\_IS\_ShortPaper\_December272018For\_Submission\_reviewedbyIEEELegal-1.pdf) which compiles indicators from a variety of sources for consideration by technical teams.

\*\*Facebook’s well-being changes\*\*

Facebook’s 2017 changes to the news feed ranking algorithm provide a detailed, well-documented example of how a large platform can incorporate well-being metrics into an optimizing system. It also raises some important questions about how large companies \*should\* use metrics. In late 2017, an unusual [post](https://about.fb.com/news/2017/12/hard-questions-is-spending-time-on-social-media-bad-for-us/) appeared on an official Facebook blog:

> \*\*\*What Do Academics Say? Is Social Media Good or Bad for Well-Being?\*\*\*\*According to the research, it really comes down to how you use the technology. For example, on social media, you can passively scroll through posts, much like watching TV, or actively interact with friends — messaging and commenting on each other’s posts. Just like in person, interacting with people you care about can be beneficial, while simply watching others from the sidelines may make you feel worse.\*
>
>

This is in line with the conclusion that [outside researchers](https://spssi.onlinelibrary.wiley.com/doi/abs/10.1111/sipr.12033) had come to:

> \*passively using social network sites provokes social comparisons and envy, which have negative downstream consequences for subjective well-being. In contrast, when active usage of social network sites predicts subjective well-being, it seems to do so by creating social capital and stimulating feelings of social connectedness.\*
>
>

Soon after, Facebook began to talk publicly about its efforts to encorate “meaningful social interactions.” A [post](https://about.fb.com/news/2018/01/news-feed-fyi-bringing-people-closer-together/) by Zuckerberg suggests that this is a proxy for well-being:

> \*The research shows that when we use social media to connect with people we care about, it can be good for our well-being … I’m changing the goal I give our product teams from focusing on helping you find relevant content to helping you have more meaningful social interactions.\*
>
>

This managerial goal turned into algorithmic changes, as [described](https://about.fb.com/news/2018/01/news-feed-fyi-bringing-people-closer-together/) by the head of the News Feed product:

> \*Today we use signals like how many people react to, comment on or share posts to determine how high they appear in News Feed. With this update, we will also prioritize posts that spark conversations and meaningful interactions between people. To do this, we will predict which posts you might want to interact with your friends about, and show these posts higher in feed (Mosseri 2018).\*
>
>

In other words, Facebook developed a probabilistic model to predict “meaningful interactions,” a proxy for well-being, and incorporated this into the News Feed ranking algorithm. What’s a “meaningful interaction”? The only concrete description comes from Facebook’s [Q4 2017 earnings call](https://s21.q4cdn.com/399680738/files/doc\_financials/2017/Q4/Q4-17-Earnings-call-transcript.pdf), where Zuckerberg explains that this metric — the training data for the new predictive model — comes from user panel surveys:

> \*So the thing that we’re going to be measuring is basically, the number of interactions that people have on the platform and off because of what they’re seeing that they report to us as meaningful.\*
>
> \*… the way that we’ve done this for years is we’ve had a panel, a survey, of thousands of people who [we] basically asked, what’s the most meaningful content that they had seen on the platform or they have seen off the platform. And we design our systems in order to be able to get to that ground truth of what people, real people are telling us is that high-quality experience\*
>
>

This is the point where the people who might be most affected by product decisions were consulted. All of this work is on their behalf. It seems like there should also be consultation on the question of which metric to watch — there are technocratic and paternalistic traps here, no matter how good one’s intentions might be. But it’s not quite clear what it would mean to ask potentially billions of users which metric they would prefer.

So while users were not consulted in choosing either the primary or the proxy metric, they were surveyed to produce the training data used to predict “meaningful interactions.” The overall system looks like this:

![]()A reconstruction of Facebook’s 2018 “meaningful social interaction” changesThis “objective function” is a piece of code that defines how “good” a particular hypothetical output would be; it’s the thing that is optimized by an optimizing system, the machine translation of a human goal. Although there are many ways to change a product that might improve well-being — for example, features that limit “[screen time](https://support.apple.com/en-us/HT208982)” — modern AI systems are usually designed to optimize a small number of key metrics that are encoded into an objective function.

Facebook changed both the metric given to the News Feed team, and the objective function that ultimately drives the News Feed ranking algorithm. Unfortunately, there is no public account of the well-being effects of these changes. But in that same investor call there is a record of the business effects of a related “meaningful social interaction” change, this one concerning Facebook’s video product rather than the News Feed:

> \*We estimate these updates decreased time spent on Facebook by roughly 5% in the fourth quarter. To put that another way: we made changes that reduced time spent on Facebook by an estimated 50 million hours every day to make sure that people’s time is well spent.\*
>
>

This is clearly a reduction in short term engagement, which shows that the objective did shift in a noticeable way, but it’s not clear what the longer term effects were. A genuinely better product might attract more customers.

The biggest weakness of this work is that there was no public evaluation of the results. Did the plan work? How well, and for who? This is especially important because the link between “meaningful interactions” and well-being is theoretical, deduced from previous research into active versus passive social media use.

\*\*The general approach\*\*

Regardless of the virtues of what Facebook did and didn’t do, trying to correct for the effects of a large optimizing system by changing the objective based on people’s feedback is a very general approach. Done well, the process might look like this:

1. Select a well-being metric, perhaps from existing frameworks. This stage is where the involvement of the people affected by the optimization matters most.
2. Define a proxy metric for which data can actually be collected, that is scoped to only those people affected by the system, and that can be estimated for hypothetical system outputs.
3. Use this metric as a performance measure for the team building and operating the system.
4. The team may choose to translate the metric into code, for example as modifications to the system’s objective function. They may also find extra-technical interventions that improve the metric.
5. The chosen well-being metric is not fixed, but must be continuously re-evaluated to ensure it is appropriate to changing conditions and does not cause side effects of its own

Other companies have done similar things, and the general pattern could apply in many other contexts.

During the period of 2012–2016, YouTube was striving to reach one billion hours of daily user watch time. Yet this “time spent” metric was not absolute, as the company made decisions to suppress clickbait and ultimately added “[user satisfaction](https://support.google.com/youtube/thread/1920627?hl=en)” and “[social responsibility](https://www.bloomberg.com/news/features/2019-04-02/youtube-executives-ignored-warnings-letting-toxic-videos-run-rampant)” metrics to its algorithmic objective function. YouTube hasn’t said exactly what these metrics are or why they were used, so we don’t really know what they target or how effective they are. As in the Facebook case, a decision to penalize clickbait reduced short term engagement, but \*Measure What Matters\* by John Doerr [documents](https://www.google.com/books/edition/Measure\_What\_Matters/u2NDDwAAQBAJ?hl=en&gbpv=1&dq=Once+the+gruesome+stuff+became+less+accessible%2C+people+sought+out+more+satisfying+content.&pg=PT108&printsec=frontcover) that engagement recovered within a few months as “people sought out more satisfying content.”

There are many other places where the careful choice of metrics might have positive effects on people’s lives. Most “gig economy” workers are employed as independent contractors, and many [experience](https://journals.sagepub.com/doi/full/10.1177/2378023119870041) week-to-week fluctuations in income as a major source of economic instability and anxiety. An appropriate measure of income stability could be used to smooth worker income by distributing fluctuations in demand between workers and perhaps across time.

![]()Hypothetical changes to a gig economy platform’s objective function to smooth worker incomesOptimizing systems can also incorporate the perspectives of non-users. A ride sharing platform may cause traffic jams for non-riders, and environmental effects are potentially global. There is a whole field of “[multi-stakeholder optimization](https://arxiv.org/abs/1907.13158)” that studies algorithms designed to account for these types of issues. A product recommendation system is typically designed to present users with products they are most likely to buy, but could also incorporate climate change concerns through estimates of product carbon footprint. These could be used to increase the visibility of alternative low-carbon products, with the goal of minimizing the metric of [carbon intensity](https://en.wikipedia.org/wiki/Emission\_intensity) per unit revenue.

![]()A hypothetical product recommendation system that tries to reduce the carbon footprint of products sold.In many other important cases it’s not as clear what the “right” metric might be. A content recommendation system filters the vast sea of available information for users. There has been extensive discussion of the need for “diverse” content recommendations, especially when filtering news content. However, “diverse” could mean different things and there is no consensus on which definition is best. There are a variety of perspectives starting from [political principles](https://pure.uva.nl/ws/files/17932244/Exposure\_diversity\_as\_a\_design\_principle.pdf), while the technical community has developed a variety of [objective functions](https://link.springer.com/chapter/10.1007/978-1-4899-7637-6\_26) that are designed to be “diverse.” Choosing a metric is itself a hard problem — the intersection of what is right with what can be measured.

\*\*Picking Metrics\*\*

An attempt to optimize for well-being is an attempt to benefit a particular group of people, who need to have a say in what is done on their behalf. Whether a community of place or a community of users, this is the basic reason why community involvement is necessary. There are many ways to involve users in the creation of software, such as the field of [participatory design](https://pure.au.dk/portal/files/139098052/Introduction\_to\_Special\_Issue\_on\_Accepted\_manuscript\_2018.pdf). Yet collaboratively picking a metric is a unique challenge; it’s not obvious how a large platform would ask a billion users what to optimize for, or even how to frame the question.

Nor can the choice be static. Both the primary metric and its proxy need to be able to change and adapt. Any measure that becomes a target incentivizes various kinds of cheating and gaming behavior, so the useful lifespan of a metric may be limited. This also responds to the concern of AI researchers who [warn](https://flyaps.com/blog/human-compatible-artificial-intelligence-and-the-problem-of-control/) that optimization of a single objective function can have disastrously negative side effects. In this context, there are always humans supervising and operating the AI system, and they are free to change the objective function as needed.

But more importantly, the world can change. Perhaps a hospital closure suddenly makes access to emergency health care a community priority. Or perhaps after careful management, income stability ceases to be the chief concern of gig economy workers. Like the idea that targeting a measure changes its meaning, there are many names for the idea that measures must continually change. “[Double-loop learning](https://en.wikipedia.org/wiki/Double-loop\_learning)” is the idea that an adaptive organization has two learning processes operating simultaneously: it learns how to progress towards its goals, while continually re-evaluating the desirability of those goals.

\*\*Narrow Alignment\*\*

A concern with metrics fits neatly into one of the deep unsolved problems of AI theory. As optimization becomes more powerful and agents become more autonomous, the specification of precisely the right goal becomes a more serious problem. Like the genie in the lamp, machines are apt to take us literally in disastrous ways. In Stuart Russell’s [example](https://flyaps.com/blog/human-compatible-artificial-intelligence-and-the-problem-of-control/), if I ask a robot to fetch me coffee I don’t mean “at all costs,” so it shouldn’t kill anyone while trying to achieve this goal. “Alignment,” or the creation of artificial agents which act according to human values, is a major unsolved problem for artificial general intelligence (AGI).

The builders of today’s “narrow” AI systems face a similar challenge of encoding the correct goals in machine form; first generation systems used obvious metrics like engagement and efficiency which ended up creating negative side effects. But this “narrow alignment” problem seems much easier to address than AGI alignment, because it’s possible to learn from how existing systems behave in the real world. Narrow alignment is worth working on for its own sake, and may also give us critical insight into the general alignment problem.

\*\*What are you optimizing for?\*\*

If you find the quantitative management of the subjective experience of the members of society troubling, you are not alone. There was [widespread concern](https://www.nytimes.com/2014/06/30/technology/facebook-tinkers-with-users-emotions-in-news-feed-experiment-stirring-outcry.html) when Facebook published a paper on “emotional contagion,” for example. This sort of social engineering at scale has all the problems of large AI systems, plus all the problems of public policy interventions.

My argument is not so much that one \*should\* use AI to optimize for well-being. Rather, we live in a world where large-scale optimization is already happening. We can choose not to evaluate or adjust these systems, but there is little reason to imagine that ignorance and inaction would be better. Mistakes will be made if we try to optimize for human happiness by quantitative means, but then, doing nothing is also a mistake. Metrics, however [flawed and incomplete](https://arxiv.org/abs/2002.08512), are a fundamental part of a positive future for AI.

If you are working on the problem of optimizing for the right thing, I’d love to hear from you at [jonathan@partnershiponai.com](mailto:jonathan@partnershiponai.com).