March 2017 Newsletter

|  |
| --- |
|
**Research updates**
* New at IAFF: [Some Problems with Making Induction Benign](https://agentfoundations.org/item?id=1263); [Entangled Equilibria and the Twin Prisoners’ Dilemma](https://agentfoundations.org/item?id=1279); [Generalizing Foundations of Decision Theory](https://agentfoundations.org/item?id=1302)
* New at AI Impacts: [Changes in Funding in the AI Safety Field](http://aiimpacts.org/changes-in-funding-in-the-ai-safety-field/); [Funding of AI Research](http://aiimpacts.org/funding-of-ai-research/)
* MIRI Research Fellow Andrew Critch has started a two-year stint at UC Berkeley’s [Center for Human-Compatible AI](http://humancompatible.ai), helping launch the research program there.
* “[Using Machine Learning to Address AI Risk](https://intelligence.org/2017/02/28/using-machine-learning/)”: Jessica Taylor explains our AAMLS agenda (in video and blog versions) by walking through six potential problems with highly performing ML systems.

**General updates**
* [Why AI Safety?](https://intelligence.org/why-ai-safety/): A quick summary (originally posted during our [fundraiser](http://effective-altruism.com/ea/12n/miri_update_and_fundraising_case/)) of the case for working on AI risk, including notes on distinctive features of our approach and our goals for the field.
* Nate Soares attended “[Envisioning and Addressing Adverse AI Outcomes](https://www.bloomberg.com/news/articles/2017-03-02/ai-scientists-gather-to-plot-doomsday-scenarios-and-solutions),” an event pitting red-team attackers against defenders in a variety of AI risk scenarios.

* We also attended an AI safety strategy retreat run by the Center for Applied Rationality.

**News and links**
* Ray Arnold provides a useful list of [ways the average person help with AI safety](http://effective-altruism.com/ea/17s/what_should_the_average_ea_do_about_ai_alignment/).
* New from OpenAI: [attacking machine learning with adversarial examples](https://openai.com/blog/adversarial-example-research/).
* OpenAI researcher Paul Christiano [explains his view of human intelligence](https://sideways-view.com/2017/02/19/the-monkey-and-the-machine-a-dual-process-theory/):

I think of my brain as a machine driven by a powerful reinforcement learning agent. The RL agent chooses what thoughts to think, which memories to store and retrieve, where to direct my attention, and how to move my muscles.
The “I” who speaks and deliberates is *implemented by* the RL agent, but is distinct and has different beliefs and desires. My thoughts are outputs and inputs to the RL agent, they are not what the RL agent “feels like from the inside.”
* Christiano describes three [directions and desiderata](https://medium.com/ai-control/directions-and-desiderata-for-ai-control-b60fca0da8f4#.3j8dflohv) for AI control: reliability and robustness, reward learning, and deliberation and amplification.
* Sarah Constantin argues that existing techniques [won’t scale up to artificial general intelligence](https://srconstantin.wordpress.com/2017/02/21/strong-ai-isnt-here-yet/) absent major conceptual breakthroughs.
* The Future of Humanity Institute and the Centre for the Study of Existential Risk ran a “[Bad Actors and AI](https://www.fhi.ox.ac.uk/bad-actors-and-artificial-intelligence-workshop/)” workshop.
* FHI is [seeking interns](https://www.fhi.ox.ac.uk/fhi-is-accepting-applications-for-internships-in-the-area-of-ai-safety-and-reinforcement-learning-2/) in reinforcement learning and AI safety.
* Michael Milford [argues against](http://theconversation.com/merging-our-brains-with-machines-wont-stop-the-rise-of-the-robots-73275) brain-computer interfaces as an AI risk strategy.
* Open Philanthropy Project head Holden Karnofsky explains why he sees [fewer benefits to public discourse](http://effective-altruism.com/ea/17o/some_thoughts_on_public_discourse/) than he used to.

 |

The post [March 2017 Newsletter](https://intelligence.org/2017/03/15/march-2017-newsletter/) appeared first on [Machine Intelligence Research Institute](https://intelligence.org).


At the [EA Global](https://www.eaglobal.org/) 2016 conference, I gave a talk on “**Using Machine Learning to Address AI Risk**”:

> It is plausible that future artificial general intelligence systems will share many qualities in common with present-day machine learning systems. If so, how could we ensure that these systems robustly act as intended? We discuss the technical agenda for a new project at MIRI focused on this question.
>
>

A recording of my talk is now up online:

The talk serves as a quick survey (for a general audience) of the kinds of technical problems we’re working on under the “[Alignment for Advanced ML Systems](https://intelligence.org/2016/07/27/alignment-machine-learning/)” research agenda. Included below is a version of the talk in blog post form.[1](https://intelligence.org/2017/02/28/using-machine-learning/#footnote_0_15536 "I also gave a version of this talk at the MIRI/FHI Colloquium on Robust and Beneficial AI.")

Talk outline:

>
> 1. [Goal of this research agenda](https://intelligence.org/2017/02/28/using-machine-learning/#1)
>
>
> 2. [Six potential problems with highly capable AI systems](https://intelligence.org/2017/02/28/using-machine-learning/#2)
>
>
> 2.1. [Actions are hard to evaluate](https://intelligence.org/2017/02/28/using-machine-learning/#problem-1)
>
> 2.2. [Ambiguous test examples](https://intelligence.org/2017/02/28/using-machine-learning/#problem-2)
>
> 2.3. [Difficulty imitating human behavior](https://intelligence.org/2017/02/28/using-machine-learning/#problem-3)
>
> 2.4. [Difficulty specifying goals about the real world](https://intelligence.org/2017/02/28/using-machine-learning/#problem-4)
>
> 2.5. [Negative side-effects](https://intelligence.org/2017/02/28/using-machine-learning/#problem-5)
>
> 2.6. [Edge cases that still satisfy the goal](https://intelligence.org/2017/02/28/using-machine-learning/#problem-6)
>
>
> 3. [Technical details on one problem: inductive ambiguity identification](https://intelligence.org/2017/02/28/using-machine-learning/#3)
>
>
> 3.1. [KWIK learning](https://intelligence.org/2017/02/28/using-machine-learning/#kwik-learning)
>
> 3.2. [A Bayesian view of the problem](https://intelligence.org/2017/02/28/using-machine-learning/#a-bayesian-view-of-the-problem)
>
>
> 4. [Other agendas](https://intelligence.org/feed/?paged=25#4)
>
>
>

---

---

### Goal of this research agenda

This talk is about a new research agenda aimed at using machine learning to make AI systems safe even at very high capability levels. I’ll begin by summarizing the goal of the research agenda, and then go into more depth on six problem classes we’re focusing on.

The goal statement for this technical agenda is that we want to know how to train a smarter-than-human AI system to perform one or more large-scale, useful *[tasks](https://arbital.com/p/task_goal/)* in the world.

Some assumptions this research agenda makes:

1. Future AI systems are likely to look like more powerful versions of present-day ML systems in many ways. We may get better deep learning algorithms, for example, but we’re likely to still be relying heavily on something like deep learning.[2](https://intelligence.org/2017/02/28/using-machine-learning/#footnote_1_15536 "Alternatively, you may think that AGI won’t look like modern ML in most respects, but that the ML aspects are easier to productively study today and are unlikely to be made completely irrelevant by future developments.")
2. Artificial general intelligence (AGI) is likely to be developed relatively soon (say, in the next couple of decades).[3](https://intelligence.org/2017/02/28/using-machine-learning/#footnote_2_15536 "Alternatively, you may think timelines are long, but that we should focus on scenarios with shorter timelines because they’re more urgent.")
3. Building [task-directed AGI](https://arbital.com/p/genie/) is a good idea, and we can make progress today studying how to do so.

I’m not confident that all three of these assumptions are true, but I think they’re plausible enough to deserve about as much attention from the AI community as the likeliest alternative scenarios.

A task-directed AI system is a system that pursues a semi-concrete objective in the world, like “build a million houses” or “cure cancer.” For those who have read *Superintelligence*, task-directed AI is similar to the idea of genie AI. Although these tasks are kind of fuzzy — there’s probably a lot of work you’d need to do to clarify what it really means to build a million houses, or what counts as a good house — they’re at least somewhat concrete.

An example of an AGI system that *isn’t* task-directed would be one with a goal like “learn human values and do things humans would consider good upon sufficient reflection.” This is too abstract to count as a “task” in the sense we mean; it doesn’t directly cash out in things in the world.

The hope is that even though task-directed AI pursues a less ambitious objective then “learn human values and do what we’d want it to do,” it’s still [sufficient](https://arbital.com/p/minimality_principle/) to prevent global catastrophic risks. Once the immediate risks are averted, we can then work on building more ambitious AI systems under reduced time pressure.

Task-directed AI uses some (moderate) amount of human assistance to clarify the goal and to evaluate and implement its plans. A goal like “cure cancer” is vague enough that humans will have to do some work to clarify what they mean by it, though most of the intellectual labor should be coming from the AI system rather than from humans.

Ideally, task-directed AI also shouldn’t require [significantly more computational resources](https://medium.com/ai-control/directions-and-desiderata-for-ai-control-b60fca0da8f4#.ytgacb3cm) than competing systems. You shouldn’t get an exponential slowdown from building a safe system vs. a generic system.

In order to think about this overall goal, we need some kind of model for these future systems. The general approach that I’ll take is to look at current systems and imagine that they’re more powerful. A lot of the time you can look at tasks that people do in ML and you can see that the performance improves over time. We’ll model more advanced AI systems by just supposing that systems will continue to achieve higher scores in ML tasks. We can then ask what kinds of failure modes are likely to arise as systems improve, and what we can work on today to make those failures less likely or less costly.

### Six potential problems with highly capable AI systems

##### Problem 1: Actions are hard to evaluate

[![Slide 13](https://intelligence.org/wp-content/uploads/2017/02/presentation-13.png)](https://intelligence.org/wp-content/uploads/2017/02/presentation-13.png)Suppose an AI system composes a story, and a human gives the system a reward based on how good the story is.[4](https://intelligence.org/2017/02/28/using-machine-learning/#footnote_3_15536 "Although I’ll use the example of stories here, in real life it could be a system generating plans for curing cancers, and humans evaluating how good the plans are.")

This is similar to some RL tasks: the agent wants to do something that will cause it to receive a high reward in the future. The formalism of RL would say that the objective of this RL agent is to write a story that the human is expected to give a high score to.

For this objective to actually help us receive very high-quality stories, however, we also need to know that the human understands the RL agent’s actions well enough to [correctly administer rewards](https://medium.com/ai-control/the-reward-engineering-problem-30285c779450#.oy88ehql4). This assumption seems less likely to hold for systems that are optimizing the objective much more powerfully than any present-day system. For example:

* A system much smarter than a human may be able to manipulate or coerce the human into giving a bad story a high score.
* Even if the system is less intelligent than that, it might resort to plagiarism. Plagiarism can be easier to generate than to detect, since detection often requires scouring a larger pool of source texts.
* A subhuman system might also have an advantage in inserting steganography into the story; it might take polynomial time to embed a secret message, and exponential time to detect such a message. Finding a way to discourage agents from taking covert actions like these would make it easier to monitor those actions’ effects and keep operators in the loop.

Do we have a general way of preventing this? Can we train an RL system to not only output an action (e.g., a story), but also a report that might help an overseer better evaluate the system’s performance? Following OpenAI researcher Paul Christiano, we call this the problem of [informed oversight](https://medium.com/ai-control/the-informed-oversight-problem-1b51b4f66b35#.eisw1fly9).[5](https://intelligence.org/2017/02/28/using-machine-learning/#footnote_4_15536 "See the Q&A section of the talk for questions like “Won’t the report be subject to the same concerns as the original story?”")

##### Problem 2: Ambiguous test examples

[![Slide 19](https://intelligence.org/wp-content/uploads/2017/02/presentation-19.png)](https://intelligence.org/wp-content/uploads/2017/02/presentation-19.png)Another problem: Consider a classifier trained to distinguish images of cats from images not containing cats, or trained to detect cancer. You may have lots of life experience that tells you “wild cats are cats.” If the training set only contains images of house cats and dogs, however, then it may not be possible to infer this fact during training.

An AI system that was superhumanly good at classifying images from a particular data set might not construct the same generalizations as a human, making it unreliable in new environments.

[![Slide 21](https://intelligence.org/wp-content/uploads/2017/02/presentation-21.png)](https://intelligence.org/wp-content/uploads/2017/02/presentation-21.png)In safety-critical settings, ideally we would like the classifier to say, “This is ambiguous,” to alert us that the image’s label is underdetermined by the labels of training set images. We could then leverage the classifier’s proficiency at classification to intervene in contexts where the system is relatively likely to misclassify things, and could also supply training data that’s tailored to the dimensions along which the original data was uninformative. Formalizing this goal is the problem of [inductive ambiguity detection](https://arbital.com/p/inductive_ambiguity/).

##### Problem 3: Difficulty imitating human behavior

[![Slide 25](https://intelligence.org/wp-content/uploads/2017/02/presentation-25.png)](https://intelligence.org/wp-content/uploads/2017/02/presentation-25.png)One way we might want to deal with the problems of smarter-than-human AI is to just have it imitate a human. By having the system observe a human attempting to perform some task and having it learn to behave similarly, we might be able to leverage the AI system’s cognitive advantages while having it mostly stick to fairly normal strategies.

As an example, we can consider an AI system tasked with producing the kind of picture a human would draw. How could you actually train a system on this objective?

[![Slide 27](https://intelligence.org/wp-content/uploads/2017/02/presentation-27.png)](https://intelligence.org/wp-content/uploads/2017/02/presentation-27.png&quot)In one approach, generative adversarial networks, you have a human and two AI agents: an imitator, and a distinguisher.[6](https://intelligence.org/2017/02/28/using-machine-learning/#footnote_5_15536 "Ian J. Goodfellow et al. “Generative Adversarial Nets”. In: Advances in Neural Information Processing 27. Ed. by Z. Ghahramani et al. Curran Associates, Inc., 2014, pp. 2672-2680. URL: https://papers.nips.cc/paper/5423-generative-adversarial-nets.pdf.") The distinguisher tries to guess whether a given picture came from the human or from the imitator, and the imitator tries to produce the kinds of image that the distinguisher classifies as coming from the human.

This is a pretty cool setup, but it raises some theoretical issues. We could ask the question, “Does the distinguisher have to be smarter than the imitator? And if so, how much smarter?” If the distinguisher is less smart than the imitator, then the imitator could covertly do very non-human-like things to the image, as long as they’re too subtle for the distinguisher to notice. An open avenue of research here is to analyze schemes like this and ask under what conditions we can get nice robustness guarantees.

For more on this problem, see Paul Christiano’s “[Act-Based Agents](https://medium.com/ai-control/act-based-agents-8ec926c79e9c#.x1cluonuo)” and “[Mimicry and Meeting Halfway](https://medium.com/ai-control/mimicry-maximization-and-meeting-halfway-c149dd23fc17#.r0yv5vgz0)”.

##### Problem 4: Difficulty specifying goals about the real world

What if we want to train our AI system to perform a task like making a sandwich? One option might be to hook the system up to a robot in a room full of sandwich ingredients, have it perform an action sequence, and then have a human observer rate the robot’s performance based on how close it came to making a sandwich. That rating determines the robot’s reward.

We previously noted that sufficiently capable RL agents might pick actions that are hard to evaluate. Here we face the additional problem that useful tasks will often require taking physical action in the world. If the system is capable enough, then this setup gives it an incentive to take away the reward button and press it itself. This is what the formalism of RL would tell you is the best action, if we imagine AI systems that continue to be trained in the RL framework far past current capability levels.

A natural question, then, is whether we can train AI systems that just keep getting better at producing a sandwich as they improve in capabilities, without ever reaching a tipping point where they have an incentive to do something else. Can we avoid relying on proxies for the task we care about, and just train the system to value completing the task in its own right? This is the [generalizable environmental goals](https://arbital.com/p/environmental_goals/) problem.

##### Problem 5: Negative side-effects

Suppose we succeeded in making a system that wants to put a sandwich in the room. In choosing between plans, it will favor whichever plan has the higher probability of resulting in a sandwich. Perhaps the policy of just walking over and making a sandwich has a 99.9% chance of success; but there’s always a chance that a human could step in and shut off the robot. A policy that drives down the probability of interventions like that might push up the probability of the room ending up containing a sandwich to 99.9999%. In this way, sufficiently advanced ML systems can end up with incentives to interfere with their developers and operators even when there’s no risk of reward hacking.

This is the problem of designing task-directed systems that can become superhumanly good at achieving their task, without causing negative side-effects in the process.

One response to this problem is to try to [quantify how much total impact](https://arbital.com/p/low_impact/) different policies have on the world. We can then add a penalty term for actions that have a high impact, causing the system to favor low-impact strategies.

Another approach is to ask how we might design an AI system to be satisfied with a merely 99.9% chance of success — just have the system stop trying to think up superior policies once it finds one meeting that threshold. This is the problem of formalizing [mild optimization](https://arbital.com/p/soft_optimizer/).

Or one can consider advanced AI systems from the perspective of [convergent instrumental strategies](https://arbital.com/p/convergent_strategies/). No matter what the system is trying to do, it can probably benefit by having more computational resources, by having the programmers like it more, by having more money. A sandwich-making system might want money so it can buy more ingredients, whereas a story-writing system might want money so it can buy books to learn from. Many different goals imply similar instrumental strategies, a number of which are likely to introduce conflicts due to resource limitations.

One, approach, then, would be to study these instrumental strategies directly and try to find a way to design a system [that doesn’t exhibit them](https://arbital.com/p/avert_instrumental_pressure/). If we can identify common features of these strategies, and especially of the adversarial strategies, then we could try to proactively avert the incentives to pursue those strategies. This seems difficult, and is very underspecified, but there’s some initial research pointed in this direction.

##### Problem 6: Edge cases that still satisfy the goal

[![Slide 40](https://intelligence.org/wp-content/uploads/2017/02/presentation-40.png)](https://intelligence.org/wp-content/uploads/2017/02/presentation-40.png)Another problem that’s likely to become more serious as ML systems become more advanced is [edge cases](https://arbital.com/p/edge_instantiation/).

Consider our ordinary concept of a sandwich. There are lots of things that technically count as sandwiches, but are unlikely to have the same practical uses a sandwich normally has for us. You could have an extremely small or extremely large sandwich, or a toxic sandwich.

For an example of this behavior in present-day systems, we can consider this image that an image classifier correctly classified as a panda (with 57% confidence). Goodfellow, Shlens, and Szegedy found that they could add a tiny vector to this image that causes the classifier to misclassify it as a gibbon with 99% confidence.[7](https://intelligence.org/2017/02/28/using-machine-learning/#footnote_6_15536 "Ian J. Goodfellow, Jonathon Shlens, and Christian Szegedy. “Explaining and Harnessing Adversarial Examples”. In: (2014). arXiv: 1412.6572 [stat.ML].")

Such edge cases are likely to become more common and more hazardous as ML systems begin to search wider solution spaces than humans are likely (or even able) to consider. This is then another case where systems might become increasingly good at maximizing their score on a conventional metric, while becoming *less* reliable for achieving realistic goals we care about.

[Conservative concepts](https://arbital.com/p/conservative_concept/) are an initial idea for trying to address this problem, by biasing systems to avoid assigning positive classifications to examples that are near the edges of the search space. The system might then make the mistake of thinking that some perfectly good sandwiches are inadmissible, but it would not make the more risky mistake of classifying toxic or otherwise bizarre sandwiches as admissible.

### Technical details on one problem: inductive ambiguity identification

I’ve outlined eight research directions for addressing six problems that seem likely to start arising (or to become more serious) as ML systems become better at optimizing their objectives — objectives that may not exactly match programmers’ intentions. The research directions were:

* [Informed oversight](https://arbital.com/p/informed_oversight/), for making it easier to interpret and assess ML systems’ actions.
* [Inductive ambiguity identification](https://arbital.com/p/inductive_ambiguity/), for designing classifiers that stop and check in with overseers in circumstances where their training data was insufficiently informative.
* Robust [human imitation](https://medium.com/ai-control/mimicry-maximization-and-meeting-halfway-c149dd23fc17#.1s9kwuq86), for recapitulating the safety-conducive features of humans in ML systems.
* [Generalizable environmental goals](https://arbital.com/p/environmental_goals/), for preventing RL agents’ instrumental incentives to seize control of their reward signal.
* [Impact measures](https://arbital.com/p/low_impact/), [mild optimization](https://arbital.com/p/soft_optimizer/), and [averting instrumental incentives](https://arbital.com/p/avert_instrumental_pressure/), for preventing negative side-effects of superhumanly effective optimization in a general-purpose way.
* [Conservative concepts](https://arbital.com/p/conservative_concept/), for steering clear of edge cases.

These problems are discussed in more detail in “[Alignment for Advanced ML Systems](https://intelligence.org/2016/07/27/alignment-machine-learning/).” I’ll go into more technical depth on an example problem to give a better sense of what working on these problems looks like in practice.

##### KWIK learning

[![Slide 44](https://intelligence.org/wp-content/uploads/2017/02/presentation-44.png)](https://intelligence.org/wp-content/uploads/2017/02/presentation-44.png)Let’s consider the inductive ambiguity identification problem, applied to a classifier for 2D points. In this case, we have 4 positive examples and 4 negative examples.

When a new point comes in, the classifier could try to label it by drawing a whole bunch of models that are consistent with the previous data. Here, I draw just 4 of them. The question mark falls on opposite sides of these different models, suggesting that all of these models are plausible given the data.

[![Slide 45](https://intelligence.org/wp-content/uploads/2017/02/presentation-45.png)](https://intelligence.org/wp-content/uploads/2017/02/presentation-45.png)We can suppose that the system infers from this that the training data is ambiguous with respect to the new point’s classification, and asks the human to label it. The human might then label it with a plus, and the system draws new conclusions about which models are plausible.

This approach is called “Knows What It Knows” learning, or KWIK learning. We start with some input space *X* ≔ ℝ*n* and assume that there exists some true mapping from inputs to probabilities. E.g., for each image the cat classifier encounters we assume that there is a true answer in the set *Y* ≔ [0,1] to the question, “What is the probability that this image is a cat?” This probability corresponds to the probability that a human will label that image “1” as opposed to “0,” which we can represent as a weighted coin flip. The model maps the inputs to answers, which in this case are probabilities.[8](https://intelligence.org/2017/02/28/using-machine-learning/#footnote_7_15536 "The KWIK learning framework is much more general than this; I’m just giving one example.")

The KWIK learner is going to play a game. At the beginning of the game, some true model *h*\* gets picked out. The true model is assumed to be in the hypothesis set *H*. On each iteration *i* some new example *xi* ∈ ℝ*n* comes in. It has some true answer *yi* = *h*\*(*xi*), but the learner is unsure about the true answer. The learner has two choices:

* Output an answer *ŷi* ∈ [0,1].
	+ If |*ŷi* – *yi*| > ε, the learner then loses the game.
* Output ⊥ to indicate that the example is ambiguous.
	+ The learner then gets to observe the true label *zi* = FlipCoin(*yi*) from the observation set *Z* ≔ {0,1}.

The goal is to not lose, and to not output ⊥ too many times. The upshot is that it’s actually possible to win this game with a high probability if the hypothesis class *H* is a small finite set or a low-dimensional linear class. This is pretty cool. It turns out that there are certain forms of uncertainty where we can just resolve the ambiguity.

The way this works is that on each new input, we consider multiple models *h* that have done well in the past, and we consider something “ambiguous” if the models disagree on *h*(*xi*) by more than ε. Then we just refine the set of models over time.

The way that a KWIK learner represents this notion of inductive ambiguity is: ambiguity is about not knowing which model is correct. There’s some set of models, many are plausible, and you’re not sure which one is the right model.

There are some problems with this. One of the main problems is KWIK learning’s realizability assumption — the assumption that the true model *h*\* is actually in the hypothesis set *H*. Realistically, the actual universe won’t be in your hypothesis class, since your hypotheses need to fit in your head. Another problem is that this method only works for these very simple model classes.

##### A Bayesian view of the problem

That’s some existing work on inductive ambiguity identification. What’s some work we’ve been doing at MIRI related to this?

Lately, I’ve been trying to approach this problem from a Bayesian perspective. On this view, we have some kind of prior *Q* over mappings *X* → {0,1} from the input space to the label. The assumption we’ll make is that our prior is wrong in some way and there’s some unknown “true” prior *P* over these mappings. The goal is that even though the system only has access to *Q*, it should perform the classification task almost as well (in expectation over *P*) as if it already knew *P*.

It seems like this task is hard. If the real world is sampled from *P*, and *P* is different from your prior *Q*, there aren’t that many guarantees. To make this tractable, we can add a grain of truth assumption:

$$\forall f : Q(f) \geq \frac{1}{k} P(f) $$

This says that if *P* assigns a high probability to something, then so does *Q*. Can we get good performance in various classification tasks under this kind of assumption?

We haven’t completed this research avenue, but initial results suggest that it’s possible to do pretty well on this task while avoiding catastrophic behaviors in at least in some cases (e.g., online supervised learning). That’s somewhat promising, and this is definitely an area for future research.

How this ties in to inductive ambiguity identification: If you’re uncertain about what’s true, then there are various ways of describing what that uncertainty is about. You can try taking your beliefs and partitioning them into various possibilities. That’s in some sense an ambiguity, because you don’t know which possibility is correct. We can think of the grain of truth assumption as saying that there’s some way of splitting up your probability distribution into components such that one of the components is right. The system should do well even though it doesn’t initially know which component is right.

(For more recent work on this problem, see Paul Christiano’s “[Red Teams](https://medium.com/ai-control/red-teams-b5b6de33dc76#.y6lwslork)” and “[Learning with Catastrophes](https://medium.com/ai-control/learning-with-catastrophes-59387b55cc30#.ii2tpjntu)” and research forum results from me and Ryan Carey: “[Bias-Detecting Online Learners](https://agentfoundations.org/item?id=1025)” and “[Adversarial Bandit Learning with Catastrophes](https://agentfoundations.org/item?id=1086).”)

### Other research agendas

Let’s return to a broad view and consider other research agendas focused on long-run AI safety. The first such agenda was outlined in MIRI’s 2014 [agent foundations](https://intelligence.org/files/TechnicalAgenda.pdf) report.[9](https://intelligence.org/2017/02/28/using-machine-learning/#footnote_8_15536 "Nate Soares and Benja Fallenstein. Agent Foundations for Aligning Machine Intelligence with Human Interests: A Technical Research Agenda. Tech. rep. 2014-8. Forthcoming 2017 in “The Technological Singularity: Managing the Journey” Jim Miller, Roman Yampolskiy, Stuart J. Armstrong, and Vic Callaghan, Eds. Berkeley, CA. Machine Intelligence Research Institute. 2014.")

The agent foundations agenda is about developing a better theoretical understanding of reasoning and decision-making. An example of a relevant gap in our current theories is ideal reasoning about mathematical statements (including statements about computer programs), in contexts where you don’t have the time or compute to do a full proof. This is the basic problem we’re responding to in “[Logical Induction](https://intelligence.org/2016/09/12/new-paper-logical-induction/).” In this talk I’ve focused on problems for advanced AI systems that broadly resemble present-day ML; in contrast, the agent foundations problems are agnostic about the details of the system. They apply to ML systems, but also to other possible frameworks for good general-purpose reasoning.

Then there’s the “[Concrete Problems in AI Safety](https://openai.com/blog/concrete-ai-safety-problems/)” agenda.[10](https://intelligence.org/2017/02/28/using-machine-learning/#footnote_9_15536 "Dario Amodei, Chris Olah, Jacob Steinhardt, Paul Christiano, John Schulman, and Dan Mané. “Concrete Problems in AI Safety”. In: (2016). arXiv: 1606.06565 [cs.AI].") Here the idea is to study AI safety problems with a more empirical focus, specifically looking for problems that we can study using current ML methods, and perhaps can even demonstrate in current systems or in systems that might be developed in the near future.

As an example, consider the question, “How do you make an RL agent that behaves safely while it’s still exploring its environment and learning about how the environment works?” It’s a question that comes up in current systems all the time, and is relatively easy to study today, but is likely to apply to more capable systems as well.

These different agendas represent different points of view on how one might make AI systems more reliable in a way that scales with capabilities progress, and our hope is that by encouraging work on a variety of different problems from a variety of different perspectives, we’re less likely to completely miss a key consideration. At the same time, we can achieve more confidence that we’re on the right track when relatively independent approaches all arrive at similar conclusions.

I’m leading the team at MIRI that will be focusing on the “Alignment for Advanced ML Systems” agenda going forward. It seems like there’s a lot of room for more eyes on these problems, and we’re hoping to hire a number of new researchers and kick off a number of collaborations to tackle these problems. If you’re interested in these problems and have a solid background in mathematics or computer science, I definitely recommend [getting in touch](mailto:jessica@intelligence.org) or [reading more about these problems](https://intelligence.org/2016/07/27/alignment-machine-learning/).

---

1. I also gave [a version of this talk](https://www.youtube.com/watch?v=_sGTqI5qdD4) at the MIRI/FHI Colloquium on Robust and Beneficial AI.
2. Alternatively, you may think that AGI won’t look like modern ML in most respects, but that the ML aspects are easier to productively study today and are unlikely to be made completely irrelevant by future developments.
3. Alternatively, you may think timelines are long, but that we should focus on scenarios with shorter timelines because they’re more urgent.
4. Although I’ll use the example of stories here, in real life it could be a system generating plans for curing cancers, and humans evaluating how good the plans are.
5. See the [Q&A section](http://www.youtube.com/watch?v=TSe3p1zIvVI&t=35m40s) of the talk for questions like “Won’t the report be subject to the same concerns as the original story?”
6. Ian J. Goodfellow et al. “Generative Adversarial Nets”. In: *Advances in Neural Information Processing 27*. Ed. by Z. Ghahramani et al. Curran Associates, Inc., 2014, pp. 2672-2680. URL: <https://papers.nips.cc/paper/5423-generative-adversarial-nets.pdf>.
7. Ian J. Goodfellow, Jonathon Shlens, and Christian Szegedy. “Explaining and Harnessing Adversarial Examples”. In: (2014). [arXiv: 1412.6572 [stat.ML]](https://arxiv.org/abs/1412.6572).
8. The KWIK learning framework is much more general than this; I’m just giving one example.
9. Nate Soares and Benja Fallenstein. [Agent Foundations for Aligning Machine Intelligence with Human Interests: A Technical Research Agenda](https://intelligence.org/files/TechnicalAgenda.pdf). Tech. rep. 2014-8. Forthcoming 2017 in “The Technological Singularity: Managing the Journey” Jim Miller, Roman Yampolskiy, Stuart J. Armstrong, and Vic Callaghan, Eds. Berkeley, CA. Machine Intelligence Research Institute. 2014.
10. Dario Amodei, Chris Olah, Jacob Steinhardt, Paul Christiano, John Schulman, and Dan Mané. “Concrete Problems in AI Safety”. In: (2016). [arXiv: 1606.06565 [cs.AI]](https://arxiv.org/abs/1606.06565).

The post [Using machine learning to address AI risk](https://intelligence.org/2017/02/28/using-machine-learning/) appeared first on [Machine Intelligence Research Institute](https://intelligence.org).


February 2017 Newsletter

|  |
| --- |
|
Following up on a post outlining some of the reasons MIRI researchers and OpenAI researcher Paul Christiano are [pursuing different research directions](https://agentfoundations.org/item?id=1129), Jessica Taylor has written up the key [motivations for MIRI’s highly reliable agent design research](https://agentfoundations.org/item?id=1220).

**Research updates**
* A new paper: “[Toward Negotiable Reinforcement Learning: Shifting Priorities in Pareto Optimal Sequential Decision-Making](https://intelligence.org/2017/01/25/negotiable-rll/)“
* New at IAFF: [Pursuing Convergent Instrumental Subgoals on the User’s Behalf Doesn’t Always Require Good Priors](https://agentfoundations.org/item?id=1149); [Open Problem: Thin Logical Priors](https://agentfoundations.org/item?id=1206)
* MIRI has a new [research advisor](https://intelligence.org/team/#advisors): Google DeepMind researcher Jan Leike.
* MIRI and the Center for Human-Compatible AI are [looking for research interns](https://intelligence.org/2017/02/11/chcai-miri/) for this summer. Apply by March 1!

**General updates**
* We attended the Future of Life Institute’s [Beneficial AI conference](https://futureoflife.org/bai-2017/) at Asilomar. See Scott Alexander’s [recap](http://slatestarcodex.com/2017/02/06/notes-from-the-asilomar-conference-on-beneficial-ai/). MIRI executive director Nate Soares was on a technical safety panel discussion with representatives from DeepMind, OpenAI, and academia ([video](https://www.youtube.com/watch?v=UMq4BcRf-bY)), also featuring a back-and-forth with Yann LeCun, the head of Facebook’s AI research group (at [22:00](http://www.youtube.com/watch?v=UMq4BcRf-bY&t=22m0s)).
* MIRI staff and a number of top AI researchers are signatories on FLI’s new [Asilomar AI Principles](https://futureoflife.org/ai-principles/), which include cautions regarding arms races, value misalignment, recursive self-improvement, and superintelligent AI.
* The Center for Applied Rationality [recounts](http://rationality.org/studies/2016-case-studies) MIRI researcher origin stories and other cases where their workshops have been a big assist to our work, alongside examples of CFAR’s impact on other groups.
* The Open Philanthropy Project has awarded a $32,000 [grant](http://www.openphilanthropy.org/focus/global-catastrophic-risks/potential-risks-advanced-artificial-intelligence/ai-impacts-general-support) to AI Impacts.
* Andrew Critch spoke at Princeton’s [ENVISION](http://envision-conference.com/) conference ([video](https://www.youtube.com/watch?v=qeGQ3FhTmKo)).
* Matthew Graves has joined MIRI as a staff writer. See his first piece for our blog, a [reply](https://intelligence.org/2017/01/13/response-to-ceglowski-on-superintelligence/) to “Superintelligence: The Idea That Eats Smart People.”
* The audio version of [*Rationality: From AI to Zombies*](https://intelligence.org/rationality-ai-zombies/) is temporarily unavailable due to the shutdown of Castify. However, fans are already putting together [a new free recording](http://from-ai-to-zombies.eu/files.html) of the full collection.

**News and links**
* An Asilomar panel on superintelligence ([video](https://www.youtube.com/watch?v=h0962biiZa4)) gathers Elon Musk (OpenAI), Demis Hassabis (DeepMind), Ray Kurzweil (Google), Stuart Russell and Bart Selman (CHCAI), Nick Bostrom (FHI), Jaan Tallinn (CSER), Sam Harris, and David Chalmers.
* Also from Asilomar: Russell on corrigibility ([video](https://www.youtube.com/watch?v=pARXQnX6QS8)), Bostrom on openness in AI ([video](https://www.youtube.com/watch?v=_H-uxRq2w-c)), and LeCun on the path to general AI ([video](https://www.youtube.com/watch?v=bub58oYJTm0)).
* From *MIT Technology Review*‘s “[AI Software Learns to Make AI Software](https://www.technologyreview.com/s/603381/ai-software-learns-to-make-ai-software/?set=603387)”:

Companies must currently pay a premium for machine-learning experts, who are in short supply. Jeff Dean, who leads the Google Brain research group, mused last week that some of the work of such workers could be supplanted by software. He described what he termed “automated machine learning” as one of the most promising research avenues his team was exploring.

* AlphaGo [quietly defeats the world’s top Go professionals](https://qz.com/877721/the-ai-master-bested-the-worlds-top-go-players-and-then-revealed-itself-as-googles-alphago-in-disguise/) in a crushing 60-win streak. AI also bests the top human players [in no-limit poker](https://www.theguardian.com/technology/2017/jan/30/libratus-poker-artificial-intelligence-professional-human-players-competition).
* More signs that artificial general intelligence is becoming a trendier goal in the field: FAIR proposes [an AGI progress metric](https://arxiv.org/abs/1701.08954).
* Representatives from Apple and OpenAI join the [Partnership on AI](http://www.wired.co.uk/article/ai-partnership-facebook-google-deepmind), and MIT and Harvard announce a new [Ethics and Governance of AI Fund](http://news.mit.edu/2017/mit-media-lab-to-participate-in-ai-ethics-and-governance-initiative-0110).
* The World Economic Forum’s 2017 [Global Risks Report](http://www3.weforum.org/docs/GRR17_Report_web.pdf) includes [a discussion of AI safety](http://reports.weforum.org/global-risks-2017/part-3-emerging-technologies/3-2-assessing-the-risk-of-artificial-intelligence/): “given the possibility of an AGI working out how to improve itself into a superintelligence, it may be prudent – or even morally obligatory – to consider potentially feasible scenarios, and how serious or even existential threats may be avoided.”
* On the other hand, the JASON advisory group [reports to the US Department of Defense](https://motherboard.vice.com/en_us/article/elite-scientists-have-told-the-pentagon-that-ai-wont-threaten-humanity) that “the claimed ‘existential threats’ posed by AI seem at best uninformed,” adding, “In the midst of an AI revolution, there are no present signs of any corresponding revolution in AGI.”
* Data scientist Sarah Constantin argues that ML algorithms are exhibiting [linear or sublinear performance returns](https://srconstantin.wordpress.com/2017/01/28/performance-trends-in-ai/) to linear improvements in processing power, and that deep learning represents a break from trend in image and speech recognition, but not in strategy games or language processing.
* New safety papers discuss [human-in-the-loop reinforcement learning](https://arxiv.org/abs/1701.04079) and [ontology identification](https://dspace.ut.ee/bitstream/handle/10062/54240/Rao_Parnpuu_MA_2016.pdf), and Jacob Steinhardt writes on [latent variables and counterfactual reasoning](https://jsteinhardt.wordpress.com/2017/01/10/latent-variables-and-model-mis-specification/) in AI alignment.

 |

The post [February 2017 Newsletter](https://intelligence.org/2017/02/16/february-2017-newsletter/) appeared first on [Machine Intelligence Research Institute](https://intelligence.org).


We’re looking for talented, driven, and ambitious technical researchers for a summer research internship with the [Center for Human-Compatible AI](http://humancompatible.ai/) (CHCAI) and the Machine Intelligence Research Institute (MIRI).

##### About the research:

CHCAI is a research center based at UC Berkeley with PIs including Stuart Russell, Pieter Abbeel and Anca Dragan. CHCAI describes its goal as “to develop the conceptual and technical wherewithal to reorient the general thrust of AI research towards provably beneficial systems”.

MIRI is an independent research nonprofit located near the UC Berkeley campus with a mission of helping ensure that smarter-than-human AI has a positive impact on the world.

CHCAI’s research focus includes work on inverse reinforcement learning and human-robot cooperation ([link](http://humancompatible.ai/publications)), while MIRI’s focus areas include [task AI](https://arbital.com/p/task_goal/) and computational reflection ([link](https://intelligence.org/research)). Both groups are also interested in theories of (bounded) rationality that may help us develop a deeper understanding of general-purpose AI agents.

##### To apply:

1. Fill in the form here: <https://goo.gl/forms/bDe6xbbKwj1tgDbo1>

2. Send an email to [beth.m.barnes@gmail.com](mailto:beth.m.barnes@gmail.com) with the subject line “AI safety internship application”, attaching your CV, a piece of technical writing on which you were the primary author, and your research proposal.

The research proposal should be one to two pages in length. It should outline a problem you think you can make progress on over the summer, and some approaches to tackling it that you consider promising. We recommend reading over [CHCAI’s annotated bibliography](http://humancompatible.ai/bibliography) and the [concrete problems agenda](https://openai.com/blog/concrete-ai-safety-problems/) as good sources for open problems in AI safety, if you haven’t previously done so.

You should target your proposal at a specific research agenda or a specific adviser’s interests. Advisers’ interests include:

• **Andrew Critch** (CHCAI, MIRI): anything listed in [CHCAI’s open technical problems](http://humancompatible.ai/bibliography#open_technical_problems%3a); [negotiable reinforcement learning](https://arxiv.org/abs/1701.01302); game theory for agents with transparent source code (e.g., “[Program Equilibrium](https://arxiv.org/abs/1701.01302)” and “[Parametric Bounded Löb’s Theorem and Robust Cooperation of Bounded Agents](https://arxiv.org/abs/1602.04184)“).

• **Daniel Filan** (CHCAI): the contents of “Foundational Problems,” “Corrigibility,” “Preference Inference,” and “Reward Engineering” in CHCAI’s [open technical problems list](http://humancompatible.ai/bibliography#open_technical_problems%3a).

• **Dylan Hadfield-Menell** (CHCAI): application of game-theoretic analysis to models of AI safety problems (specifically by people who come from a theoretical economics background); formulating and analyzing AI safety problems as [CIRL games](https://arxiv.org/pdf/1606.03137.pdf); the relationships between AI safety and principal-agent models / theories of incomplete contracting; reliability engineering in machine learning; questions about fairness.

• **Jessica Taylor**, **Scott Garrabrant**, and **Patrick LaVictoire** (MIRI): open problems described in MIRI’s [agent foundations](https://intelligence.org/files/TechnicalAgenda.pdf) and [alignment for advanced ML systems](https://intelligence.org/2016/07/27/alignment-machine-learning/) research agendas.

This application does not bind you to work on your submitted proposal. Its purpose is to demonstrate your ability to make concrete suggestions for how to make progress on a given research problem.

##### Who we’re looking for:

This is a new and somewhat experimental program. You’ll need to be self-directed, and you’ll need to have enough knowledge to get started tackling the problems. The supervisors can give you guidance on research, but they aren’t going to be teaching you the material. However, if you’re deeply motivated by research, this should be a fantastic experience.

Successful applicants will demonstrate examples of technical writing, motivation and aptitude for research, and produce a concrete research proposal. We expect most successful applicants will either:

• have or be pursuing a PhD closely related to AI safety;

• have or be pursuing a PhD in an unrelated field, but currently pivoting to AI safety, with evidence of sufficient knowledge and motivation for AI safety research; or

• be an exceptional undergraduate or masters-level student with concrete evidence of research ability (e.g., publications or projects) in an area closely related to AI safety.

##### Logistics:

Program dates are flexible, and may vary from individual to individual. However, our assumption is that most people will come for twelve weeks, starting in early June.

The program will take place in the San Francisco Bay Area. Basic living expenses will be covered. We can’t guarantee that housing will be all arranged for you, but we can provide assistance in finding housing if needed.

Interns who are not US citizens will most likely need to apply for J-1 intern visas. Once you have been accepted to the program, we can help you with the required documentation.

##### Deadlines:

The deadline for applications is the March 1. Applicants should hear back about decisions by March 20.

The post [CHCAI/MIRI research internship in AI safety](https://intelligence.org/2017/02/11/chcai-miri/) appeared first on [Machine Intelligence Research Institute](https://intelligence.org).


[![Toward Negotiable Reinforcement Learning](https://intelligence.org/files/paretooptimal.png)](https://arxiv.org/abs/1701.01302)MIRI Research Fellow Andrew Critch has developed a new result in the theory of conflict resolution, described in “[**Toward negotiable reinforcement learning: Shifting priorities in Pareto optimal sequential decision-making**](https://arxiv.org/abs/1701.01302).”

Abstract:

> Existing multi-objective reinforcement learning (MORL) algorithms do not account for objectives that arise from players with differing beliefs. Concretely, consider two players with different beliefs and utility functions who may cooperate to build a machine that takes actions on their behalf. A representation is needed for how much the machine’s policy will prioritize each player’s interests over time.
>
>
> Assuming the players have reached common knowledge of their situation, this paper derives a recursion that any Pareto optimal policy must satisfy. Two qualitative observations can be made from the recursion: the machine must (1) use each player’s own beliefs in evaluating how well an action will serve that player’s utility function, and (2) shift the relative priority it assigns to each player’s expected utilities over time, by a factor proportional to how well that player’s beliefs predict the machine’s inputs. Observation (2) represents a substantial divergence from naïve linear utility aggregation (as in Harsanyi’s utilitarian theorem, and existing MORL algorithms), which is shown here to be inadequate for Pareto optimal sequential decision-making on behalf of players with different beliefs.
>
>

If AI alignment is as difficult as it looks, then there are already strong reasons for different groups of developers to collaborate and to steer clear of race dynamics: the difference between a superintelligence aligned with one group’s values and a superintelligence aligned with another group’s values pales compared to the difference between any aligned superintelligence and a misaligned one. As Seth Baum of the Global Catastrophic Risk Institute notes [in a recent paper](https://papers.ssrn.com/sol3/papers.cfm?abstract_id=2816323):

> Unfortunately, existing messages about beneficial AI are not always framed well. One potentially counterproductive frame is the framing of strong AI as a powerful winner-takes-all technology. This frame is implicit (and sometimes explicit) in discussions of how different AI groups might race to be the first to build strong AI. The problem with this frame is that it makes a supposedly dangerous technology seem desirable. If strong AI is a winner-takes-all technologies race, then AI groups will want to join the race and rush to be the first to win. This is exactly the opposite of what the discussions of strong AI races generally advocate—they postulate (quite reasonably) that the rush to win the race could compel AI groups to skimp on safety measures, thereby increasing the probability of dangerous outcomes.
>
>
> Instead of framing strong AI as a winner-takes-all race, those who are concerned about this technology should frame it as a dangerous and reckless pursuit that would quite likely kill the people who make it. AI groups may have some desire for the power that might accrue to whoever builds strong AI, but they presumably also desire to not be killed in the process.
>
>

Researchers’ discussion of mechanisms to disincentivize arms races should therefore not be read as implying that self-defeating arms races are rational. Empirically, however, developers have a wide range of beliefs about the difficulty of alignment. Mechanisms for formally resolving policy disagreements may help create more evident incentives for cooperation and collaboration; hence there may be some value in developing formal mechanisms that advanced AI systems can use to generate policies that each party prefers over simple compromises between all parties’ goals (and beliefs), and that each prefers over racing.

Critch’s recursion relation provides a framework in which players may negotiate for the priorities of a jointly owned AI system, producing a policy that is more attractive than the naïve linear utility aggregation approaches already known in the literature. The mathematical simplicity of the result suggests that there may be other low-hanging fruit in this space that would add to and further illustrate the value of collaboration. Critch identifies six areas for future work (presented in more detail in the paper):

1. *[Best-alternative-to-negotiated-agreement](https://en.wikipedia.org/wiki/Best_alternative_to_a_negotiated_agreement) dominance.* Critch’s result considers negotiations between agents with differing beliefs, but does not account for the possibility that parties may have different BATNAs.
2. *Targeting specific expectation pairs.* A method for modifying the players’ utility functions to make this possible would be useful for specifying various fairness or robustness criteria, including BATNA dominance.
3. *Information trade.* Critch’s algorithm gives a large advantage to any contributor that is better able to predict the AI system’s inputs from its outputs. In realistic setting where players lack common knowledge of each other’s priors and observations, it would therefore make sense for agents to be able to trade away some degree of control over the system for information; but it is not clear how one should carry out such trades in practice.
4. *Learning priors and utility functions.* Realistic smarter-than-human AI systems will need to learn their utility function over time, e.g., through [cooperative inverse reinforcement learning](http://arxiv.org/abs/1606.03137). A realistic negotiation procedure will need to account for the fact that the developers’ goals are imperfectly known and the AI system’s goals are a “work in progress.”
5. *Incentive compatibility.* The methods used to learn players’ beliefs and utility functions additionally need to incentivize honest representations of one’s beliefs and goals, or they will need to be robust to attempts to game the system.
6. *Naturalized decision theory.* The setting used in this result assumes a separation between the inner workings of the machine (and the players) and external reality, as opposed to modeling it as part of its environment. More realistic formal frameworks would allow us to better model the players’ representations of each other, opening up new [negotiation possibilities](https://intelligence.org/2016/03/31/new-paper-on-bounded-lob/).[1](https://intelligence.org/2017/01/25/negotiable-rll/#footnote_0_15443 "Thanks to Matthew Graves, Andrew Critch, and Jessica Taylor for helping draft this post.")

---

#### Sign up to get updates on new MIRI technical results

*Get notified every time a new technical paper is published.*

*
*

×

---

1. Thanks to Matthew Graves, Andrew Critch, and Jessica Taylor for helping draft this post.

The post [New paper: “Toward negotiable reinforcement learning”](https://intelligence.org/2017/01/25/negotiable-rll/) appeared first on [Machine Intelligence Research Institute](https://intelligence.org).