Low-stakes alignment

Right now I’m working on finding a good objective to optimize with ML, rather than trying to make sure our models are robustly optimizing that objective. (This is roughly “[outer alignment](https://www.alignmentforum.org/s/r9tYkB2a8Fp4DN8yB/p/pL56xPoniLvtMDQ4J).”)

That’s pretty vague, and it’s not obvious whether “find a good objective” is a meaningful goal rather than being inherently confused or sweeping key distinctions under the rug.

So I like to focus on a more precise special case of alignment: solve alignment when decisions are “low stakes.” I think this case effectively isolates the problem of “find a good objective” from the problem of ensuring robustness and is precise enough to focus on productively.

In this post I’ll describe what I mean by the low-stakes setting, why I think it isolates this subproblem, why I \*want\* to isolate this subproblem, and why I think that it’s valuable to work on crisp subproblems.

1. What is the low-stakes setting?
----------------------------------

A situation is low-stakes if we care very little about any small number of decisions. That is, we only care about the average behavior of the system over long periods of time (much longer than the amount of time it takes us to collect additional data and retrain the system).

For example, this requires that all of the AI systems in the world can’t corrupt the training process quickly or seize control of resources from humans. If they try, we can keep collecting data and fine-tuning them, and this will cause their behavior to change before anything irreversibly bad happens.

For a more formal definition see section 6.

2. Why do low stakes require only outer alignment?
--------------------------------------------------

If the stakes are low, we can train our model on the decisions that actually arise in practice rather than needing to anticipate tricky decisions in advance. Moreover, because the payoff from an individual action is always small, we can focus on average-case performance and achieve reasonable sample complexities without any additional tricks.

The main substantive claim is that we don’t need to worry about the “distributional shift” between past decisions and future decisions. When the distribution of inputs change, the system may behave poorly for a while, but if we keep retraining on the new data then it will eventually adapt. If individual decisions are low stakes, then the total cost of all of this adaptation is small. I give this argument in more detail in section 7.

Formally this resembles an online regret bound ([this textbook](https://arxiv.org/pdf/1909.05207.pdf) gives a nice introduction to online learning). SGD satisfies such a bound in the case of convex losses. For messy model classes like neural networks we usually can’t prove much interesting about SGD (either for online \*or\* offline learning), but for a variety of reasons I think it’s reasonable to expect a similar online bound. I discuss this in more detail in section 8.

This isn’t to say that we can totally ignore optimization difficulties, or the online nature of the problem. But it appears that the \*main\* difficulty is constructing a good enough objective and arguing that it is sufficiently easy to optimize.

3. Why focus on this subproblem first?
--------------------------------------

I think it’s really great to focus on a good subproblem if you can find one.

If you solve your subproblem, then you’ve made progress. If you get stuck, well then you were probably going to get stuck anyway and at least you’re stuck on something easier. When working on a big problem like alignment, I feel like it’s easy to bounce off of \*every\* solution because it doesn’t handle the whole problem immediately, and splitting into subproblems is a key way to get over that failure.

I think that the low-stakes setting is a particularly good and clean subproblem: it’s definitely not harder than the original, there are clear ways in which it’s much easier, and solving it would represent real progress.

Why do I focus on this problem first, rather than starting with the other side (something more like robustness / inner alignment)?

\* I think that finding a “good” objective is likely to be similar to finding a “good” specification for adversarial training or verification, and understanding the structure of our specification will change how we approach robustness.
\* I think that defining a good objective likely [requires](/informed-oversight-18fcb5d3d1e1) something like “knowing what the model knows.” If this is successful, it’s likely to be an important ingredient for robustness as well (especially for treacherous behavior, where in some sense the model “knows” about the problem).
\* Put differently, I feel like we can approximately split the full alignment problem into two parts: low stakes and [handling catastrophes](/learning-with-catastrophes-59387b55cc30). We know how to define the low-stakes part but don’t know quite how to formulate catastrophes, so it’s more natural to start with low stakes.

4. Is the low-stakes setting actually scary?
--------------------------------------------

Many AI safety problems involve AI systems behaving badly in an abrupt and irreversible way. Does the low-stakes assumption dismiss the central concerns?

I think that you could have very risky situations even if every individual decision is low-stakes. For example, suppose that our world was full of safeguards that made it very slow and hard to change anything big (our infrastructure, our government, individual decision-making…). Perhaps any big change will need to take place in tiny pieces spread over thousands of days (though smaller changes occur more easily).

In such a world AI can still cause huge amounts of trouble \*if humans can’t understand what it is doing\*. Rather than “taking over” in a single unanticipated shock, the situation can deteriorate in a thousand tiny pieces each of which humans cannot understand.

\*\*5. Why focus on “low stakes” rather than “outer alignment”?\*\*
---------------------------------------------------------------

Theoretical work in general, and [my research methodology in particular](http://v), is a lot easier when we can cleanly evaluate proposed solutions rather than being fuzzy about the boundaries of our problem. Alignment is always frustratingly fuzzy, but we don’t have to make it any \*worse\*.

I think that if I tried to work on “outer alignment” defined more fuzzily, I might have ended up sweeping some critical issues under the rug or pursuing dead ends. For example:

\* If I’d tried to assume that our models generalize “correctly” as long as the objective is good, I might not have been thinking clearly about how models generalize from questions we can evaluate to [those we can’t](/inaccessible-information-c749c6a88ce). But I think that’s actually a core issue that can’t be cleanly separated from the rest of outer alignment.
\* If I’d tried to assume that models are “trying” to get a low value for the training loss, I might have ended up relying on our ability to incentivize the model to make very long-term predictions. But I think that approach is basically a dead end.

Overall I think that inner alignment and outer alignment are useful intuitive handles but don’t carve the problem space cleanly enough to be good research problems.

6. More formal definition of low-stakes
---------------------------------------

Say that our AI receives a sequence of inputs \*x\*[1], \*x\*[2], … \*x\*[T], and produces a sequence of outputs \*y\*[1], \*y\*[2], … \*y\*[T].

Let U[\*t\*](\*y\*) be our expected utility if we intervene to have the AI output \*y\*[\*t\*] = \*y\* and then be aligned in all future timesteps. Let \*ρ\* be a bound on the maximum possible utility difference U[\*t\*](\*y\*) − U[\*t\*](\*y\*′).

(Assume that our AI sometimes behaves randomly for exploration, so that we can define these as conditional expectations given the state of the world before the system chooses \*y\*[\*t\*].)

Let \*k\* be a constant (the “latency”) such that we can afford to train the AI on all data up to time \*t\* before we need to deploy it to make a decision at time \*t\*+\*k\*.

Then the “low-stakes” goal is to achieve a total utility within O(\*ρk\*√T) of an aligned model. I call this “low-stakes” because the bound is only meaningful when \*ρk\* — the total damage that can be done by ML systems before retraining — is small relative to the total value at stake. Because this gap grows sublinearly with T, note that it is \*eventually\* small if ML is deployed for long enough relative to the stakes and the time required for models to learn.

7. More formal argument that outer alignment is sufficient
----------------------------------------------------------

Suppose that we could compute the utilities U[\*t\*](\*y\*) exactly, i.e. that we had a perfect objective. I claim that we could then satisfy the low-stakes goal by performing online RL, i.e. performing SGD with a loss function that is an unbiased estimator for the expectation of U[\*t\*](\*y\*) for an action \*y\* sampled from the model.

I’ll focus on the case \*k\*=1 for simplicity, but I think \*k\*>1 is basically the same.

For each timestep \*t\*, let M[\*t\*] be our model at time \*t\*, let \*y\*[\*t\*] be the random output we sample, let M\\*[\*t\*] be the aligned model that is competitive with M, and let \*y\*\\*[\*t\*] be the output of M\\*.

Then:

\* U[0](\*y\*\\*[0]) is the utility we’d obtain by taking aligned decisions at every step. U[T](y[T]) is the actual utility we receive. So our “regret” is U[T](\*y\*[T]) − U[0](\*y\*\\*[0])
\* U[\*t\*+1](\*y\*\\*[\*t\*+1]) = U[\*t\*](\*y\*[\*t\*]), since U is defined assuming the system is aligned at all future times.
\* Thus the regret is the sum of U[\*t\*](\*y\*\\*[\*t\*]) − U[\*t\*](\*y\*[\*t\*]) across all time steps \*t\*.
\* But this is identical to the difference in performance between the sequence of models M[\*t\*] and M\\*[\*t\*].

If our loss function was convex and M\\* was fixed, then SGD would have a regret bound of O(\*ρk\*√T), as desired. If we are optimizing over the space of neural networks then our loss function is obviously not convex and so we can’t easily prove a bound of this form, but I’ll argue in section 8 that we can aim for a similar guarantee as long as M\\* is “easy” to find with SGD.

Of course we can’t hope to actually compute the real utility differences U[\*t\*] (since e.g. which decision is optimal may depend on hard empirical facts where we don’t get any feedback until years later). So we’ll need to set our sights a bit lower (e.g. to say that in subjective expectation we do as well as the aligned model, rather than being able to say that for every possible sequence we actually do as well). I discuss similar issues in section 3 of [Towards formalizing universality](/towards-formalizing-universality-409ab893a456), and I don’t think they change the basic picture.

8. Why expect SGD to work online even for neural networks?
----------------------------------------------------------

SGD enjoys a √T online regret bound only for convex losses. Convexity implies that the iterates of SGD are optimal for a regularized loss function, which is needed to get a bound.

Other than that hiccough I think the regret bound basically goes through. But why am I not concerned about the suboptimality of SGD?

\* To the extent that SGD can’t find the optimum, it hurts the performance of both the aligned model and the unaligned model. In some sense what we really want is a regret bound compared to the “best learnable model,” where the argument for a regret bound is heuristic but seems valid.
\* Moreover, a satisfactory alignment scheme already needs to explain why finding the aligned model M\\* is not much harder than finding the unaligned model M. And that’s basically what we need in order to argue that SGD has a regret bound relative to M\\*.

Overall I’m prepared to revise my view if the empirical evidence suggests that online bounds are problematic, but so far all the experiments I’m aware of are broadly consistent with the theoretical picture.

The other slight subtlety in section 7 is that we care about the regret between the actual model M[\*t\*] and the aligned version M\\*[\*t\*], whereas we conventionally define regret relative to some fixed target model. I’m not concerned about this either: (i) regret bounds that compare the actual model M to the best transformed model can be obtained by similar methods, (ii) for our purposes we’d also be fine just bounding our regret compared to the single maximally-competent aligned model at the end of training (although I actually expect that to be somewhat harder).

Overall I don’t think this is completely straightforward, but I think it looks good enough that the main difficulty is probably finding a good objective. I’d personally want to start thinking about these more technical issues only after we’ve solved the thornier conceptual issues.

We often split AI alignment into two parts: outer alignment, or ``finding a good reward function'', and inner alignment, or ``robustly optimizing that reward function''. However, these are not very precise terms, and they don't form clean subproblems. In particular, for outer alignment, how good does the reward function have to be? Does it need to incentivize good behavior in all possible situations? How do you handle the no free lunch theorem? Perhaps you only need to handle the inputs in the training set? But then what specifies the behavior of the agent on new inputs?

This post proposes an operationalization of outer alignment that admits a clean subproblem: _low stakes alignment_. Specifically, we are given as an assumption that we don't care much about any small number of decisions that the AI makes -- only a large number of decisions, in aggregate, can have a large impact on the world. This prevents things like quickly seizing control of resources before we have a chance to react. We do not expect this assumption to be true in practice: the point here is to solve an easy subproblem in the hopes that the solution is useful for solving the hard version of the problem.

The main power of this assumption is that we no longer have to worry about distributional shift. We can simply keep collecting new data online and training the model on the new data. Any decisions it makes in the interim period could be bad, but by the low-stakes assumption, they won't be catastrophic. Thus, the primary challenge is in obtaining a good reward function that incentivizes the right behavior after the model is trained. We might also worry about whether gradient descent will successfully find a model that optimizes the reward even on the training distribution -- after all, gradient descent has no guarantees for non-convex problems -- but it seems like, to the extent that gradient descent doesn't do this, it will probably affect aligned and unaligned models equally.

Note that this subproblem is still non-trivial, and existential catastrophes still seem possible if we fail to solve it. For example, one way that the low-stakes assumption could be made true was if we had a lot of bureaucracy and safeguards that the AI system had to go through before making any big changes to the world. It still seems possible for the AI system to cause lots of trouble if none of the bureaucracy or safeguards can understand what the AI system is doing.

I like the low-stakes assumption as a way of saying "let's ignore distributional shift for now". Probably the most salient alternative is something along the lines of "assume that the AI system is trying to optimize the true reward function". The main way that low-stakes alignment is cleaner is that it uses an assumption on the _environment_ (an input to the problem) rather than an assumption on the _AI system_ (an output of the problem). This seems to be a lot nicer because it is harder to "unfairly" exploit a not-too-strong assumption on an input rather than on an output. See [this comment thread](https://www.alignmentforum.org/posts/TPan9sQFuPP6jgEJo/low-stakes-alignment?commentId=askebCP36Ce96ZiJa) for more discussion.


Mundane solutions to exotic problems

I’m looking for alignment techniques that are [indefinitely scalable](/directions-and-desiderata-for-ai-control-b60fca0da8f4) and that [work in any situation we can dream up](/my-research-methodology-b94f2751cb2c). That means I spend time thinking about “exotic” problems — like AI systems reasoning about their own training process or about humanity’s far future.

Yet I’m very optimistic about finding practical alignment strategies that are relatively simple and usable today. I expect the results of my research to look mundane and even boring.

These two stances may appear to be in tension. If I’m worrying about all kinds of exotic considerations, how can the end result be something mundane?

I’m not too concerned. These failures seem exotic because they involve exotic \*capabilities.\* I’m looking for a mundane algorithm that trains the model to use whatever capabilities it has for good purposes. So it will lead to exotic countermeasures if and when the model has exotic capabilities, i.e. just in time to avoid exotic failures (by which time they’ll no longer feel exotic).

I want to illustrate this phenomenon with a few examples.

Example 1: Gradient hacking
---------------------------

[\*Gradient hacking\*](https://www.lesswrong.com/posts/uXH4r6MmKPedk8rMA/gradient-hacking)is a possible failure mode where a powerful misaligned model behaves in a way that is apparently benign but biases the gradient of the loss to point in an unfavorable direction, eventually trapping optimization at a bad local minimum. This feels exotic today because it requires the model to have an incredible amount of introspective access and control, as well as strategic awareness over long time horizons. (“[Forward Pass](http://karpathy.github.io/2021/03/27/forward-pass/)” by Andrej Karpathy explores some similar ideas.)

Normally I imagine using SGD to optimize a “behavioral” objective: to update the parameters θ you use them to make a randomized decision \*y\* on an input \*x\*, then you calculate a reward \*r\*(\*x\*, \*y\*), then you compute an unbiased estimate for the gradient of the expectation \*r\*(\*x\*, \*y\*) with respect to θ.

Gradient hacking is troubling because it suggests that if we optimize a purely behavioral objective we could get a bad outcome even if the reward function \*r\*(\*x\*, \*y\*) perfectly captures the quality of the output \*y\*.

Rather than fixing this problem in a specific way, I’d like to rely on competitiveness. That is, assume that (throughout training) we have a question-answering policy Q → A which reflects \*everything\* that our learned model “knows” about the world.

Then it seems like the best way to compute the system’s loss should be by by asking some question like “What should the loss be for the current parameters?”

Our usual behavioral loss then is a great \*heuristic\* for computing the loss (which we can justify on principled Bayesian grounds if we want). In mundane situations it’s not clear we have anything further to say about what the loss should be, so “What should the loss be?” is almost the same as hard-coding the heuristic.

But when gradient hacking occurs, we are no longer in a mundane situation. By hypothesis, our learned model is able to reason introspectively about its own parameters and the structure of the loss landscape! It is thinking about how to change its behavior in order to affect the loss, change the optimization trajectory, and ultimately disempower humans.

That means that a competitive question-answerer \*also\* is able to answer questions about its own parameters and the structure of the loss landscape. It is able to talk about how assigning a loss could mess up the optimization trajectory and ultimately disempower humans. If we incorporate all that knowledge into our assessment of “What should the loss be for the current parameters?” then it will assign a low loss to some of the actions that the gradient-hacker tried to sabotage. That is, the malicious AI can outsmart the behavioral heuristic, but it can’t outsmart an overseer \*who knows everything that it knows\*.

It’s not clear if this works (and my strong guess is that we’ll ultimately end up with an algorithm that looks quite different from this). But overall I think that if gradient hacking actually came up, it’s more likely than not that the core difficulty in avoiding it would be the same version of epistemic competitiveness that we need in more mundane situations. The only real change to our algorithm we need to make is applying a competitive overseer to the right question.

You might worry that the knowledge a model has about its own parameters is qualitatively different than the knowledge it has about the world. But I don’t think so — they are both produced by transfer from the task of “get a low training loss,” combined with a bunch of computation. It’s possible that after we have an epistemically competitive solution we’ll see that it doesn’t apply to a model’s introspective knowledge. If that happens then we may indeed need some more exotic solution that talks about introspection \*per se\*, but I personally doubt it.

Example 2: far future forecasts and inaccessible influence
----------------------------------------------------------

Suppose that there’s [some domain that our AI can understand but we cannot](/inaccessible-information-c749c6a88ce), for vividness you can imagine humans without microscopes and an AI that has a working model of the microscopic world. For simplicity, imagine that the world factors nicely into an accessible part and an inaccessible part, and that there are influences in both directions but to us the the (inaccessible) → (accessible) influences just look like noise. (If you don’t know about the mechanics of biology, then events like “getting cancer” can just look like noise.)

One of my concerns is that an AI that understands the inaccessible part may be able to cause trouble in the very long term. Even if we’re able to prevent the AI from doing anything overtly bad in the accessible part for the next few years (like killing humans), it may still be that humans aren’t really in control of the situation because the real power is in the inaccessible part of the world, and that power will eventually assert itself.

So if we can only observe the accessible part of the world, then we would need to look \*very\* far ahead to avoid problem. Basically we’d need to look far enough ahead that the future humans have done science and learned about the previously-inaccessible parts of the world. This kind of lookahead feels particularly exotic because it involves extreme generalization. Training typically involves feedback loops over seconds, but now we are talking about a model that makes good predictions about events that literally occur \*generations\* from now.

That kind of extreme lookahead sounds exotic, but in some cases my current alignment plans require it.

As in the last example, I think this situation is only exotic because our model had exotic capabilities. We’re in this situation because we are trying to handle a malicious AI that makes plans that have no observable consequences for many generations, building up power in the inaccessible part of the world that it predicts will eventually interfere with humans. That AI is already apparently able to generalize well to extremely long time horizons — if it wasn’t, we’d have no problem at all.

So once again, to avoid this problem we really just need epistemically competitive question answering, i.e. we need to learn a Q → A function that generalizes as well to novel questions (e.g. about the future) as the underlying model’s knowledge. And that’s a pretty mundane problem that’s core to the idea of epistemic competitiveness that we need to solve for other reasons.

If the malicious model is able to generalize from its experiences to make plans that unfold over generations, and the Q → A model generalizes as well as the malicious model, then our Q → A system can also answer questions about what happens over generations. That only sounds exotic because we assumed some exotic capabilities. Nothing about the solution or training needs to deal with ultra-long horizons specifically.

Conclusion
----------

In my work I don’t shy away from exotic problems (I often find them useful as extreme cases to illustrate some principle). At the same time, I’m aiming for mundane solutions and optimistic about finding them.

I think those positions are consistent because my intermediate goal is to ensure that the oversight process is able to leverage all of the capabilities developed by the model — so if the model develops exotic capabilities which pose exotic challenges, then we get an exotic oversight process automatically.

The author’s goal is to find “mundane” or simple algorithms that solve even “exotic” problems in AI alignment. Why should we expect this is possible? If an AI system is using powerful, exotic capabilities to evade detection, shouldn’t we need powerful, exotic algorithms to fight that? The key idea here is that we can instead have a mundane algorithm that leverages the exotic capabilities of the AI system to produce an exotic oversight process. For example, we could imagine that a mundane algorithm could be used to create a question-answerer that knows everything the model knows. We could then address <@gradient hacking@>(@Gradient hacking@) by asking the question “what should the loss be?” In this case, our model has an exotic capability: very strong introspective access to its own reasoning and the training process that modifies it. (This is what is needed to successfully hack gradients). As a result, our question answerer should be able to leverage this capability to assign high loss (low reward) to cases where our AI system tries to hack gradients, even if our normal hardcoded loss would not do so.


Techniques for optimizing worst-case performance

If powerful ML systems fail catastrophically, they may be able to quickly cause irreversible damage. To be safe, it’s not enough to have an average-case performance guarantee on the training distribution — we need to ensure that even if our systems fail on new distributions or with small probability, they will never fail \*too\* badly.

The difficulty of optimizing worst-case performance is one of the most likely reasons that I think [prosaic AI alignment](/prosaic-ai-control-b959644d79c2) might turn out to be impossible (if combined with an [unlucky empirical situation](https://arbital.com/p/daemons/)).

In this post I want to explain my view of the problem and enumerate some possible angles of attack. My goal is to communicate why I have hope that worst-case guarantees are achievable.

None of these are novel proposals. The intention of this post is to explain my view, not to make a new contribution. I don’t currently work in any of these areas, and so this post should be understood as an outsider looking in, rather than coming from the trenches.

Malign vs. benign failures and corrigibility
--------------------------------------------

I want to distinguish two kinds of failures:

\* “Benign” failures, where our system encounters a novel situation, doesn’t know how to handle it, and so performs poorly. The resulting behavior may simply be erratic, or may serve an external attacker. Their effect is similar to physical or cybersecurity vulnerabilities — they create an opportunity for destructive conflict but don’t systematically disfavor human values. They may pose an existential risk when combined with high-stakes situations, in the same way that human incompetence may pose an existential risk. Although these failures are important, I don’t think it is necessary or possible to eliminate them in the worst case.
\* “Malign” failures, where our system continues to behave competently but applies its intelligence in the service of an unintended goal. These failures systematically favor whatever goals AI systems tend to pursue in failure scenarios, at the expense of human values. They constitute an existential risk independent of any other destructive technology or dangerous situation. Fortunately, they seem both less likely and potentially possible to avoid even in the worst case.

I’m most interested in malign failures, and the narrower focus is important to my optimism.

The distinction between malign and benign failures is not always crisp. For example, suppose we try to predict a human’s preferences, then search over all strategies to find the one that best satisfies the predicted preferences. Guessing the preferences even a little bit wrong would create an adversarial optimizer incentivized to apply its intelligence to a purpose at odds with our real preferences. If we take this approach, incompetence does systematically disfavor human values.

By aiming for corrigible rather than optimal behavior (see [here](https://arbital.com/p/hard\_corrigibility/) or [here](/corrigibility-3039e668638)) I’m optimistic that it is possible to create a sharper distinction between benign and malign failures, which can be leveraged by the techniques below. But for now, this hope is highly speculative.

Amplification
-------------

I believe that these techniques are much more likely to work if we have access to an overseer who is significantly smarter than the model that we are trying to train. I hope that [amplification](/alphago-zero-and-capability-amplification-ede767bb8446) makes this possible.

It seems realistic for a strong overseer to recognize an (input, output) pair as a malign failure mode (though it may require a solution to [informed oversight](/the-informed-oversight-problem-1b51b4f66b35)). So now we have a concrete goal: find a model that never gives an output the overseer would diagnose as catastrophically bad.

Historically researchers in the AI safety community have been extremely pessimistic about reliability. I think part of that pessimism is because they have been imagining working with models much smarter than the overseer.

Techniques
==========

I’ll describe three categories of techniques:

\* Adversarial training
\* Verification
\* Transparency

Previous versions of this list included implicit ensembles, e.g. Bayesian neural nets. I still think there might be useful techniques in that space, but I’ve become pessimistic enough that it no longer makes the list.

Adversarial training
--------------------

(\*Examples:\* [\*Explaining and Harnessing Adversarial Examples\*](https://arxiv.org/abs/1412.6572)\*,\* [\*Adversarial Machine Learning at Scale\*](https://arxiv.org/abs/1611.01236)\*.)\*

If we want to know whether our model ever fails catastrophically, it’s natural to train an adversary to search for catastrophe-inducing inputs. If the adversary can’t find one, then we hope that none exist. In order to produce a catastrophe-free model, we can integrate the results of testing into the training process.

The [basic problem](http://www.cleverhans.io/security/privacy/ml/2017/06/14/verification.html) with this strategy is that no matter how hard you test, it’s always possible (realistically, it’s likely) that there are possible failure cases your adversary didn’t notice. For example, suppose that it takes 2ᴺ seconds to find the Nth failure. Because nature will be many times more complex than our training procedure, it’s inevitable that there are many types of failures that nature can find but we can’t.

We can attack this problem by trying to make the adversary’s job easier. Some possibilities:

\* Use interpretability techniques (see below) to help the adversary identify possible failures.
\* Use interpretability to expand what counts as catastrophic — if the adversary can get the model to think in a way that looks like it \*could\* lead to catastrophic behavior under different situations, that could count as a success for the adversary.
\* Combine with verification (see below) to help guide the adversary’s search and expand the space of allowed attacks.
\* Allow the adversary to directly control some of the internal computation of our model rather than merely controlling inputs. This requires a way to constrain the actions of the adversary — we want them to be able to produce any failure that could arise naturally, but if we give them too much power then they can trivially cause a catastrophic failure.
\* Change the system’s inputs so they are easier for the adversary to fake: see the “distorting observations” and “abstracting observations” sections of [this post](/red-teams-b5b6de33dc76).
\* Make the adversary more powerful, without changing their task, by applying amplification.

Adversarial training faces other difficulties as well. For example, it might take a long time for adversarial training to converge, or training might never converge due to non-convexity. I think these are significantly less likely to be deal-breakers, but they are certainly important research problems if we actually want to apply adversarial training.

Verification
------------

\*(Examples:\* [\*Certified Defenses against Adversarial Examples\*](https://arxiv.org/abs/1801.09344)\*,\* [\*Reluplex: An Efficient SMT Solver for Verifying Deep Neural Networks\*](https://arxiv.org/pdf/1702.01135.pdf))

In verification the goal is to find a certificate that there is no input on which a system behaves badly.

Verification has (at least) two problems:

\* \*\*Complexity\*\*. It’s really, really hard to verify properties of a computationally interesting model.
\* \*\*Specification\*\*. It’s unclear what we should be trying to verify.

Handling complexity is definitely challenging. But if we are free to train the model in order to facilitate verification, and if we are only interested in certifying some “easy” property that the model satisfies with slack, then it’s not clearly doomed.

Prima facie, specification looks more like an unfixable deal-breaker. In the rest of this section I’ll give three possible approaches for finding specifications. I think none of these is satisfactory on their own, but they leave me optimistic that verification can be useful despite specification difficulty.

My current view is that specification and complexity are both potentially surmountable difficulties, but that it’s likely their resolutions will result in partial solutions that need to be combined with other approaches.

\*\*Use a slow model of overseer as a specification\*\*. If our overseer is a physical process, to actually get a specification we’d need to learn a model of the overseer. So we’ve just shifted the problem to the model of the overseer.

Fortunately, this move does potentially made the problem one step better. Because we don’t need to use the overseer model at runtime, we can afford to use a much slower model. So verification could distill a very slow reliable model into a fast reliable model.

\*\*Amplification\*\*. If our overseer is produced by amplifying some process H, and we have a trusted model of H, then we can obtain a trusted model of the overseer. So if we were able to form a trusted model of the first step of amplification, then we could iteratively use verification to construct trusted models at each subsequent step.

We are still left with the problem at the first step of amplification. But at this step we might be able to restrict attention to a small range of questions for which worst-case guarantees are more feasible; we may be able to train weaker models for which worst-case guarantees are more feasible; or we might be able to eliminate the human altogether and begin amplification from some explicit simple core of reasoning. Any of these seems plausible.

\*\*“Verification” without specification.\*\* I find it helpful to think of verification from the primal/dual perspective, which also makes the connection to adversarial training clearer.

In adversarial training, the adversary wants to find an input on which the system behaves poorly. The dual to the adversary is an explicit list of all inputs, showing that each one of them performs well. The dual is exponentially large, and (relatedly) the adversary’s problem is exponentially hard.

We can view verification in a given proof system as making the adversary’s job easier, by expanding the set of allowable “attacks.” If we do this carefully, we can make the dual certificate much smaller and make the adversary’s task much more tractable.

For example, if our proofs (dual solutions) are in first order logic, the adversary’s task is finding any model of the axioms in which there exists an attack. (This is only different in the infinite case.) If our proofs are sum of squares proofs, the adversary’s task is finding a [pseudodistribution](https://www.dsteurer.org/talk/cargese.pdf) over attacks. For certain linear programming proofs, the adversary’s goal is finding an attack on a particular extrapolation of the model. And so on.

On this perspective, we don’t necessarily need an actual specification. Instead, we can take some external catastrophe-checker, relax it to a catastrophe checker over the adversary’s relaxed action space, and then try to implement an adversary against that catastrophe checker. This would change the meaning of the dual, which would now have a subtle statistical interpretation rather than being a traditional certificate.

Transparency
------------

(\*Examples:\* [\*Feature Visualization\*](https://distill.pub/2017/feature-visualization/)\*,\* [\*Deep Inside Convolutional Networks\*](https://arxiv.org/pdf/1312.6034.pdf)\*)\*

While training a model, we are free to look inside and “see what it’s thinking.” If we could actually understand those thoughts, we might be able to more easily predict under what conditions it would fail malignantly.

\*\*A (weird) example\*\*: suppose that human evolution was part of an alien project to train a mathematician, and some humans are extracted from our simulation and trained to solve alien math problems. If aliens are only able to test the humans’ behavior they might conclude that they always faithfully solve whatever problem they are posed — especially after a few generations of selection removed all the humans who didn’t do so. But if the aliens were able to look inside the humans’ heads they might discover that this behavior is extremely contingent — perhaps humans are only solving problems because they believe they will die otherwise, and the humans would behave very differently as soon as they saw an opportunity to escape.

That example is exotic and somewhat implausible, but I think that the same basic dynamic is likely to apply in much messier practical situations. If a model would fail malignantly under some conditions, I think there is some sense in which the model must be testing those conditions during training.

\*\*Malign vs. benign failures:\*\* I think the prospects for using transparency to detect malign failures in models trained by gradient descent are \*much\* better than the prospects for detecting benign failures, or for detecting failures in arbitrary models.

The key point is that a malign failure requires leveraging the intelligence of the model to do something actively bad. If our model is trained by gradient descent, its behavior can only be intelligent when it is exercised on the training distribution — if part of the model never (or very rarely) does anything on the training distribution, then that part of the model can’t be intelligent. So in some sense a malign failure mode needs to use a code path that gets run on the training distribution, just under different conditions that cause it to behave badly.

That means that we don’t necessarily need to analyze our model in the abstract, we can look at its behavior on actual training data. This looks much more likely to be tractable. (Though this entire discussion is extremely subtle, and without being more precise it’s very easy for us to go wrong.)

\*\*Amplification\*\*: Transparency will be hard if we ultimately want to train models much smarter than we are, but deeply understanding what a model is thinking probably requires being much smarter than it. I think that problem can potentially be overcome by using amplification to stay ahead of the models we are training. This means that current work on transparency, which considers models doing tasks that are within the abilities of the humans who are trying to understand them, could potentially remain relevant even as AI improves significantly.

\*\*What do you do with transparency?\*\* Merely understanding that a model might behave catastrophically could be useful, but it would be much nicer to actually fix the problem. Adversarial training gives a natural mechanism: once we understand a failure we can synthesize appropriate data and then train on that data.

This approach puts significantly more stress on our transparency techniques. Even if were initially able to use transparency to see how our model might fail, after we perform many generations of selection we might weed out exactly the comprehensible failures and leave the incomprehensible ones. You would only want to apply this technique if you had a great deal of faith in your methods; if you were feeling at all shaky about your ability to achieve worst-case guarantees, and transparency techniques let you see one potential catastrophic failure, it would be better to consider that a near-miss and seriously rework your project rather than plowing on.

Conclusion
==========

Making ML systems work in the worst case is hard, even if we are only concerned with malign failures and have access to an overseer who can identify them. If we can’t solve this problem, I think it seriously calls into question the feasibility of aligned ML.

Fortunately there are at least a few plausible angles of attack on this problem. All of these approaches feel very difficult, but I don’t think we’ve run into convincing deal-breakers. I also think these approaches are complementary, which makes it feel even more plausible that they (or their descendants) will eventually be successful. I think that exploring these angles of attack, and identifying new approaches, should be a priority for researchers interested in alignment.


The Steering Problem

Most AI research focuses on reproducing human abilities: to learn, infer, and reason; to perceive, plan, and predict. There is a complementary problem which (understandably) receives much less attention: if you \*had\* these abilities, what would you do with them?

\*\*The steering problem:\*\* Using black-box access to human-level cognitive abilities, can we write a program that is as useful as a well-motivated human with those abilities?

This document explains what the steering problem is and why I think it’s worth spending time on.

1. Introduction
===============

A capable, well-motivated human can be extremely useful: they can work without oversight, produce results that need not be double-checked, and work towards goals that aren’t precisely defined. These capabilities are critical in domains where decisions cannot be easily supervised, whether because they are too fast, too complex, or too numerous.

In some sense “be as useful as possible” is just another task at which a machine might reach human-level performance. But it is different from the concrete capabilities normally considered in AI research.

We can say clearly what it means to “predict well,” “plan well,” or “reason well.” If we ignored computational limits, machines could achieve any of these goals today. And before the existing vision of AI is realized, we must \*necessarily\* achieve each of these goals.

For now, “be as useful as possible” is in a different category. We can’t say exactly what it means. We could not do it no matter how fast our computers could compute. And even if we resolved the most salient challenges in AI, we could remain in the dark about this one.

Consider a capable AI tasked with running an academic conference. How should it use its capabilities to make decisions?

\* We could try to specify exactly what makes a conference good or bad. But our requirements are complex and varied, and so specifying them exactly seems time-consuming or impossible.
\* We could build an AI that imitates successful conference organizers. But this approach can never do any better than the humans we are imitating. Realistically, it won’t even match human performance unless we somehow communicate what characteristics are important and why.
\* We could ask an AI to maximize our satisfaction with the conference. But we’ll get what we measure. An extensive evaluation would greatly increase the cost of the conference, while a superficial evaluation would leave us with a conference optimized for superficial metrics. Everyday experience with humans shows how hard delegation can be, and how much easier it is to assign a task to someone who actually cares about the outcome.

Of course there is already pressure to write \*useful\* programs in addition to smart programs, and some AI research studies how to efficiently and robustly communicate desired behaviors. For now, available solutions apply only in limited domains or to weak agents. The steering problem is to close this gap.

Motivation
----------

A system which “merely” predicted well would be extraordinarily useful. Why does it matter whether we know how to make a system which is “as useful as possible”?

Our machines will probably do \*some\* things very effectively. We know what it means to “act well” in the service of a given goal. For example, using human cognitive abilities as a black box, we could probably design autonomous corporations which very effectively maximized growth. If the black box was cheaper than the real thing, such autonomous corporations could displace their conventional competitors.

If machines can do everything equally well, then this would be great news. If not, society’s direction may be profoundly influenced by what can and cannot be done easily. For example, if we can only maximize what we can precisely define, we may inadvertently end up with a world filled with machines trying their hardest to build bigger factories and better widgets, uninterested in anything we consider intrinsically valuable.

All technologies are more useful for some tasks than others, but machine intelligence might be particularly problematic because it can entrench itself. For example, a rational profit-maximizing corporation might distribute itself throughout the world, pay people to help protect it, make well-crafted moral appeals for equal treatment, or campaign to change policy. Although such corporations could bring large benefits in the short term, in the long run they may be difficult or impossible to uproot, even once they serve no one’s interests.

\*\*Why now?\*\*
------------

Reproducing human abilities gets a lot of deserved attention. Figuring out exactly what you’d do once you succeed feels like planning the celebration before the victory: it might be interesting, but why can’t it wait?

1. \*\*Maybe it’s hard\*\*. Probably the steering problem is much easier than the AI problem, but it might turn out to be surprisingly difficult. If it \*is\* difficult, then learning that earlier will help us think more clearly about AI, and give us a head start on addressing the steering problem.
2. \*\*It may help us understand AI.\*\* The difficulty of saying exactly what you want is a basic challenge, and the steering problem is a natural perspective on this challenge. A little bit of research on natural theoretical problems is often worthwhile, even when the direct applications are limited or unclear. In section 4 we discuss possible approaches to the steering problem, many of which are new perspectives on important problems.
3. \*\*It should be developed alongside AI.\*\* The steering problem is a long-term goal in the same way that understanding human-level prediction is a long-term goal. Just as we do theoretical research on prediction before that research is commercially relevant, it may be sensible to do theoretical research on steering before it is commercially relevant. Ideally, our ability to build useful systems will grow in parallel with our ability to build capable systems.
4. \*\*Nine women can’t make a baby in one month.\*\* We could try to save resources by postponing work on the steering problem until it seems important. At this point it will be easier to work on the steering problem, and if the steering problem turns out to be unimportant then we can avoid thinking about it altogether. But at large scales it becomes hard to speed up progress by increasing the number of researchers. Fewer people working for longer may ultimately be more efficient even if earlier researchers are at a disadvantage. In general, scaling up fields rapidly is difficult.
5. \*\*AI progress may be surprising\*\*. We probably won’t reproduce human abilities in the next few decades, and we probably won’t do it without ample advance notice. That said, AI is too young, and our understanding too shaky, to make confident predictions. A mere 15 years is 20% of the history of modern computing. If important human-level capabilities are developed surprisingly early or rapidly, then it would be worthwhile to better understand the implications in advance.
6. \*\*The field is sparse\*\*. Because the steering problem and similar questions have received so little attention, individual researchers are likely to make rapid headway. There are perhaps three to four orders of magnitude between basic research on AI and research directly relevant to the steering problem, lowering the bar for arguments 1–5.

In section 3 we discuss some other reasons not to work on the steering problem: Is work done now likely to be relevant? Is there any concrete work to do now? Should we wait until we can do experiments? Are there adequate incentives to resolve this problem already?

2. Defining the problem precisely
=================================

Recall our problem statement:

\*\*The steering problem:\*\* Using black-box access to human-level cognitive abilities, can we write a program that is as useful as a well-motivated human with those abilities?

We’ll adopt a particular human, Hugh, as our “well-motivated human:” we’ll assume that we have black-box access to Hugh-level cognitive abilities, and we’ll try to write a program which is as useful as Hugh.

Abilities
---------

In reality, AI research yields complicated sets of related abilities, with rich internal structure and no simple performance guarantees. But in order to do concrete work in advance, we will model abilities as black boxes with well-defined contracts.

We’re particularly interested in tasks which are “AI complete” in the sense that human-level performance on that task could be used as a black box to achieve human-level performance on a very wide range of tasks. For now, we’ll further focus on domains where performance can be unambiguously defined.

Some examples:

\* \*\*Boolean question-answering\*\*. A question-answerer is given a statement and outputs a probability. A question-answerer is Hugh-level if it never makes judgments predictably worse than Hugh’s. We can consider question-answerers in a variety of languages, ranging from natural language (“Will a third party win the US presidency in 2016?”) to precise algorithmic specifications (“Will this program output 1?”).
\* \*\*Online learning\*\*. A function learner is given a sequence of labelled examples (x, y) and predicts the label of a new data point, x’. A function learner is Hugh-level if, after training on any sequence of data (xᵢ, yᵢ), the learner’s guess for the label of the next point is—on average—at least as good as Hugh’s.
\* \*\*Embodied reinforcement learning\*\*. A reinforcement learner interacts with an environment and receives periodic rewards, with the goal of maximizing the discounted sum of its rewards. A reinforcement learner is Hugh-level if, following any sequence of observations, it achieves an \*expected\* performance as good as Hugh’s in the subsequent rounds. The expectation is taken using our subjective distribution over the physical situation of an agent who has made those observations.

When talking about Hugh’s predictions, judgments, or decisions, we imagine that Hugh has access to a reasonably powerful computer, which he can use to process or display data. For example, if Hugh is given the binary data from a camera, he can render it on a screen in order to make predictions about it.

We can also consider a particularly degenerate ability:

\* \*\*Unlimited computation\*\*. A box that can run any algorithm in a single time step is—in some sense—Hugh level at every precisely stated task.

Although unlimited computation seems exceptionally powerful, it’s not immediately clear how to solve the steering problem even using such an extreme ability.

Measuring usefulness
--------------------

What does it mean for a program to be “as useful” as Hugh?

We’ll start by defining “as useful for X as Hugh,” and then we will informally say that a program is “as useful” as Hugh if it’s as useful for the tasks we care most about.

Consider \*\*H,\*\* a black box which simulates Hugh or perhaps consults a version of Hugh who is working remotely. We’ll suppose that running \*\*H\*\* takes the same amount of time as consulting our Hugh-level black boxes. A project to accomplish X could potentially use as many copies of \*\*H\*\* as it can afford to run.

A program \*\*P\*\* is as useful than Hugh for X if, for every project using \*\*H\*\* to accomplish X, we can efficiently transform it into a new project which uses \*\*P\*\* to accomplish X. The new project shouldn’t be much more expensive—-it shouldn’t take much longer, use much more computation or many additional resources, involve much more human labor, or have significant additional side-effects.

Well-motivated
--------------

What it does it mean for Hugh to be well-motivated?

The easiest approach is universal quantification: for \*any\* human Hugh, if we run our program using Hugh-level black boxes, it should be as useful as Hugh.

Alternatively, we can leverage our intuitive sense of what it means for someone to be well-motivated to do X, and define “well-motivated” to mean “motivated to help the user’s project succeed.”

Scaling up
----------

If we are given better black boxes, we should make a better program. This is captured by the requirement that our program should be as useful as Hugh,no matter how capable Hugh is (as long as the black boxes are equally capable).

Ideally, our solutions should scale far past human-level abilities. This is not a theoretical concern—in many domains computers already have significantly superhuman abilities. This requirement is harder to make precise, because we can no longer talk about the “human benchmark.” But in general, we would like to build systems which are (1) working towards their owner’s interests, and (2) nearly as effective as the best goal-directed systems that can be built using the available abilities. The ideal solution to the steering problem will have these characteristics in general, even when the black-box abilities are radically superhuman.

Scaling down
------------

“Human-level abilities” could refer to many different things, including:

1. Human-level performance on high-level tasks.
2. The level of functionality embodied in the human brain. Human-level perception, intuition, motor control, subsymbolic reasoning, and so on.

In general, as we shift from 1 towards 2 the steering problem becomes more difficult. It may be difficult to produce simple or predictable high-level functions using low-level abilities.

For example, humans pursue a complicated set of goals that would be very difficult to determine by looking at the human brain (and some of which are quite distant from the evolutionary pressures that produced us). When given a task that doesn’t serve these goals, a human may simply decide to pursue their own agenda. If we build human-like abilities out of human-like low-level functions, we may find ourselves with similarly unpredictable high-level functions.

It is harder to formalize or understand low-level abilities than high-level functions. One approach is to consider very short time periods. For example, we could consider black boxes which learn functions as well as a human who spends only 500 milliseconds per example. Unfortunately, at this level it is harder to encapsulate human abilities in a small number of simple functions, and we must pay more attention to the way in which these abilities can be connected.

If the steering problem were satisfactorily resolved, “scaling down” to these lower-level abilities would be a natural but challenging next step.

\*\*3. Objections\*\*
=================

\*\*The simple, abstract capabilities we can think of now are much harder to use productively than the rich and messy AI capabilities we will actually develop.\*\*

For now we can’t clearly state \*anything\* a machine could do that would make the steering problem easy (short of exactly reproducing human behavior). Filling in this gap would be an appropriate response to the steering problem.

Perhaps we don’t yet know exactly what we want machines to do, but figuring it out is inextricably bound up with getting them to do it. If so, it might be easier to say what we want once we know how to do it. But by the same token, it might be easier to figure out how to do it once we can better say what we want.

In either case, it seems likely that the steering problem fills in its own niche: either it is a distinct problem that won’t be solved automatically en route to AI; or else it is a different perspective on the same underlying difficulties, and can be productively explored in parallel with other AI research.

Because the steering problem is non-trivial for simple, precisely stated abilities, it may well be non-trivial for the abilities we actually obtain. Certainly we can imagine developing a human-level predictor without learning too much about how to build useful systems. So it seems unreasonable to be confident that the steering problem will turn out to be a non-problem.

\*\*The simple, abstract abilities we can think of now are much \*easier\* to work with than the human-level abilities we will actually develop, or at least much different. Building a robust system is easier when all of the pieces have clean, reliable functions; in practice things won’t be so pretty.\*\*

It would be a larger leap to continue “…and the ideas required to work with simple, reliable components will have no relevance to their more realistic counterparts.” \*Whatever\* abilities we end up with, many solutions to the steering problem will turn out to be inapplicable, and they will all be incomplete. But we can still find useful general techniques by developing ideas that are helpful for many versions of the steering problem; and we can identify important technical challenges by understanding what makes each version easy or hard.

We can gradually scale up the difficulty of the steering problem by demanding more robust solutions, making weaker guarantees on our black boxes, or working with less manageable abilities. Our choices can be informed by ongoing progress in AI, focusing on those capabilities we think are most realistic and the forms of robustness we consider most likely to be necessary.

\*\*Why is autonomy necessary?\*\*

One apparent solution to the steering problem is to retain human decision-making, with AI systems acting as assistants and tools to help humans accomplish their goals.

This is an appropriate solution while AI systems remain relatively limited. It has serious problems when scaling:

\* If large numbers of machines make large numbers of decisions, with human wages orders of magnitude larger than the operating costs of machines, then the cost of human oversight becomes prohibitive. Imagine a million humans overseeing a billion or trillion human-level machines.
\* If machines make very rapid decisions, human oversight can introduce unacceptable latency. Imagine human engineers overseeing the handling of individual Google searches.
\* If machines work on complex problems, human overseers may not be able to understand their reasoning process. Imagine a physics undergraduate overseeing a team of world-class physicists.

All of these problems become particularly severe when we consider \*thinking about thinking\*. That is, machines must make numerous, rapid decisions about how to process information, what to investigate or compute, how to organize their resources, and so on. If we want to use machine intelligence to make those decisions better, that will have to be done without substantial human oversight.

It may be possible to maintain human involvement in all important automation, but doing so will eventually become a serious bottleneck. Tasks that can be performed without human oversight will become increasingly efficient, and without explicit coordination (and a willingness to make short-term sacrifices) it seems likely that more autonomous operations will outcompete their less autonomous counterparts.

\*\*Is there any concrete work to do on the steering problem?\*\*

In the next section I’ll describe a handful of existing research directions that bear on the steering problem. I think the steering problem suggests an interesting and unusual perspective on each of these domains; I don’t know whether it will prove to be a fruitful perspective, but if it fails it won’t be because of a lack of first steps.

I have done some work motivated explicitly by the steering problem: [a formalization](http://www.google.com/url?q=http%3A%2F%2Fordinaryideas.wordpress.com%2F2012%2F04%2F21%2Findirect-normativity-write-up%2F&sa=D&sntz=1&usg=AFQjCNGtdsc2V3j\_mhZbr8uvywxV0R2eJA) of “judgment upon reflection,” which can be expressed entirely algorithmically based on (experimentally controlled) observations of human behavior, [an alternative](https://medium.com/@paulfchristiano/model-free-decisions-6e6609f5d99e) to goal-directed behavior which may enjoy similar productivity benefits while being more robust, and [some](http://ordinaryideas.wordpress.com/2014/07/18/adversarial-collaboration/) [simple](https://medium.com/@paulfchristiano/delegating-to-a-mixed-crowd-dda2b8e22cd8) [protocols](https://medium.com/@paulfchristiano/of-arguments-and-wagers-ee16a0e84cf7) for delegating to untrusted agents.

\*\*4. Approaches, ingredients, and related work\*\*
================================================

Rational agency
---------------

One natural approach to the steering problem is to build goal-directed agents who want to be useful or who share their creators’ goals.

There are two main difficulties:

\* Specifying goals in an appropriate language. What does it mean to “be useful”? How can we define what we want?
\* Building agents that reliably pursue goals specified in that language.

Deploying a goal-directed agent is somewhat worrying: an agent with an almost-but-not-quite-correct goal will be working at cross-purposes to its creator, and will be motivated (for example) to avoid revealing that its goal is not quite correct. These concerns motivate a third line of research:

\* Designing goals or goal-directed agents which “fail gracefully,” i.e. which don’t behave adversarially or resist correction, even if their goals are not perfectly aligned with their creators’.

Several lines of existing research bear on each of these questions.

\*\*Specifying goals\*\*

Rather than directly specifying what outcomes are good, it seems more promising to specify how to learn what outcomes are good. This is a topic of existing research, although the focus is typically on pragmatic considerations rather than on the more general theoretical problem.

\* Inverse reinforcement learning (for example see [Russell](http://www.cs.berkeley.edu/~russell/papers/colt98-uncertainty.pdf), [Ng and Russell](http://www-cs.stanford.edu/people/ang/papers/icml00-irl.pdf), or [Ziebart et al](http://www.aaai.org/Papers/AAAI/2008/AAAI08-227.pdf).) and goal inference (for example see [Baker, Tenenbaum, and Saxe](http://web.mit.edu/clbaker/www/papers/cogsci2007.pdf) or [Verma and Rao](http://homes.cs.washington.edu/~rao/nips05imit.pdf)) attempt to infer underlying preferences by observing behavior, despite a complex relationship between actions and outcomes. To apply to the steering problem, the techniques would need to be generalized to learners who are much better informed and more capable than the human models they are learning from, and who have much noisier information about the human models and their environment. This requires understanding the limitations and errors of the human models, and generalizing the human’s goals robustly so that they remain acceptable even when they are pursued in an unfamiliar way.
\* Preference learning attempts to infer underlying preferences from observed decisions, despite noisy information and potentially irrational behavior (for a small sample, see [Fürnkranz and Hüllermeier](http://www.springer.com/computer/ai/book/978-3-642-14124-9), [Fürnkranz and Hüllermeier](http://pdf.aminer.org/000/172/223/pairwise\_preference\_learning\_and\_ranking.pdf), or [Gervasio et al](http://pdf.aminer.org/000/172/223/pairwise\_preference\_learning\_and\_ranking.pdf).). Existing work considers small domains with explicitly represented preferences, and there seem to be serious challenges when scaling to preferences over complete states-of-affairs. As a result, preference learning seems less directly applicable than inverse reinforcement learning or goal inference.
\* Some more speculative and philosophical research (for example, see [my post](http://ordinaryideas.wordpress.com/2012/04/21/indirect-normativity-write-up/) on the subject, or [this](https://intelligence.org/files/CEV.pdf) more discursive article by Yudkowsky) has explored how to formalize our preferences in a general and precise way. The focus is on determining what processes of deliberation correctly capture our informed judgment and how we might formalize those processes. The primary challenge, on this perspective, is defining our preferences about outcomes that are difficult for us to describe or reason about.

We could also investigate goals of the form “maximize user satisfaction,” but it seems hard to find a satisfactory definition along these lines.

\*\*Pursuing goals\*\*

Even with desirable goals in hand, it may be challenging to design systems that reliably pursue those goals. There are questions about how goal-directed behavior relates to reasoning and to the behavior of subsystems (does the system pursue the goals it appears to?), about the theoretical basis for optimal rational behavior (does it pursue them well? ), and about how an agent should behave in light of uncertainty about what outcomes are desirable.

\* Some existing work in reinforcement learning (for a summary, see [Sutton and Barto](http://webdocs.cs.ualberta.ca/~sutton/book/the-book.html)) and probabilistic inference (for example, see [Attas](http://research.goldenmetallic.com/aistats03.pdf)) shows how goal-directed behavior can be implemented using other faculties.
\* Some work in philosophy studies the formal basis for rational agency, from decision theory to epistemology. This work clarifies what it means to rationally pursue a particular goal. A lack of understanding may result in systems that appear to pursue one goal but actually pursue another, or that generalize to novel environments in undesirable ways.
\* Some work on AI safety (see [Dewey](http://www.danieldewey.net/learning-what-to-value.pdf) or [Bostrom](http://books.google.com/books?hl=en&lr=&id=C-\_8AwAAQBAJ&oi=fnd&pg=PP1&dq=superintelligence+nick+bostrom&ots=UES5TvoQLp&sig=k55cFr\_T1nLzunEALbvgI50KoX8#v=onepage&q=superintelligence%20nick%20bostrom&f=false), ch. 12) explores frameworks for pursuing uncertain goals, in the interest of understanding how and to what extent “learning what is good” is different from “learning what is true.”
\* Some work in multi-objective optimization considers settings with a large space of possible objectives, and seeks policies which are appropriate in light of uncertainty about which objective we really care about.

\*\*Failing gracefully\*\*

Even a “near miss” when defining a goal-directed agent might have undesirable consequences. In addition to minimizing the probability of failure, it would be nice to minimize the costs of a near miss—-and to allow us to use the kind of “trial and error” approach that is more typical of software development.

\* Researchers interested in AI safety have introduced and discussed the notion of “corrigible” agents, who cooperate with “corrective” interventions by their programmers—even if we cannot implement goals consistent with that cooperation.
\* Some researchers have worked to make AI reasoning understandable (For example, see [Bullinaria](https://www.cs.bham.ac.uk/~jxb/PUBS/AIRTNN.pdf) or [Craven](http://research.cs.wisc.edu/machine-learning/shavlik-group/craven.thesis.pdf)). Understandability can reduce the scope for malignant failure modes, since engineers might be able to directly monitor the motivation for decisions (and in particular to distinguish between honest and deceptive behavior).

Delegation
----------

Every day humans work productively on projects that they don’t intrinsically care about, motivated by a desire for money, recognition, or satisfaction. We could imagine an AI doing the same thing. For example, a reinforcement learner might do useful work in order to earn a reward, without having any intrinsic concern for the work being done.

Unfortunately, such delegation runs into some problems. The problems appear even when delegating to humans, but they get considerably worse as machines become more powerful and more numerous:

\* Naively, the agent will only do good work when the principal can verify the quality of that work. The cost of this oversight can be non-trivial.
\* Unless the oversight is extremely thorough or the problem particularly straightforward, there will be some gap between what the principal wants and what the principal evaluates. The reward-driven agent will maximize whatever the principal evaluates.
\* If agents are granted much autonomy, they may find other ways to get rewards. This depends in detail on how the “reward” is implemented, and what the agent cares about.

There are many tools available to address these problems: breaking a system up into pieces with differing values and limited autonomy, performing randomized audits, automating audits and auditing auditors, relying on agents with short time horizons or extreme risk aversion, and so on. So far there are no compelling proposals that put the pieces together.

A full solution is likely to rely on agents with different values, combined with an appropriate system of checks and balances. But if the agents coordinate to pursue their collective values, they could fatally undermine such a system. We can try to minimize the risk by making the agents’ interactions essentially zero sum, or employing other agents to oversee interactions and report signs of collusion. But the possibility of collusion remains a serious obstacle.

Even if delegation cannot fully resolve the steering problem, a weak solution might be useful as part of a bootstrapping protocol (see the section on bootstrapping below).

This problem is similar in spirit to mechanism design, but the details (and apparently the required tools) are quite different. Nevertheless, some ideas from mechanism design or the economics of delegation may turn out to be applicable. Conversely, some approaches to the steering problem might be of interest to economists in these areas.

Shared language and concepts
----------------------------

When delegating to a helpful human, we would say what we want done in natural language, relying on a rich network of shared concepts that can be used to specify goals or desired actions. Writing programs with the same capability would greatly simplify or perhaps solve the steering problem.

In some sense, human-level language understanding is already encapsulated in human-level cognitive abilities. For example, if we were pursuing the delegation approach in the last section, we could describe tasks in natural language. The agents would infer what we expect them to do and under what conditions we will give them rewards, and they would behave appropriately in light of that knowledge. But this “language understanding” only appears in the agent’s goal-directed behavior.

To address the steering problem, we would like something stronger. We would like to build agents that share human concepts, such that we can write code that operates in terms of those concepts: specifying a goal in terms of higher-level concepts, or executing instructions defined in terms of these concepts. These tasks don’t seem to be possible using only goal-directed language understanding.

Understanding concept learning and the relationship to language is a fundamental problem in cognitive science and AI. Work in these areas thus bears directly on the steering problem.

For now, we cannot say formally what it would mean to have a program that reliably acquired “the same” concepts as humans, so that instructions expressed in those concepts would have the intended meaning. Even given unlimited computation, it’s not clear how we would solve the steering problem using concept learning. This is not at all to say it is not possible, merely that it has not yet been done.

There may be a tight connection between the theoretical question—-what would it mean to learn human concepts, and how could you do it with any amount of computation—-and the pragmatic computational issues. If there is a connection, then the theoretical question might be easier once the pragmatic issues are better understood. But conversely, the pragmatic question might also be easier once the theoretical issues are better understood.

Non-consequentialist intelligence
---------------------------------

If describing our real goals is too demanding, and describing a crude approximation is hazardous, then we might try to build systems without explicitly defined goals. This makes the safety problem much easier, but probably makes it harder to build systems which are sufficiently powerful at all.

\*\*Non-agents\*\*

One idea is to focus on systems with some narrower function: answering questions, proposing plans, executing a narrow task, or so on\*.\* On their own these systems might not be terribly useful, but the hope is that as tools they can be nearly as useful as a goal-directed assistant. Moreover, because these systems don’t need to be aligned with human goals, they may be easier to construct.

For example, research in decision support and multi-objective optimization aims to find good solutions to an optimization problem (or a planning problem) by interacting with a decision-maker rather than giving a scalar representation of their preferences (for example, see [Fonseca and Fleming](http://pdf.aminer.org/000/310/607/genetic\_algorithms\_for\_multiobjective\_optimization\_formulationdiscussion\_and\_generalization.pdf) or [Deb](http://link.springer.com/chapter/10.1007/978-1-4614-6940-7\_15)).

These systems can certainly be useful (and so may be appropriate as part of a bootstrapping strategy or as a component in a more complex solution; see the next sections). But most of them inherently require significant human oversight, and do not seem suitable as solutions to the full steering problem: for projects involving large numbers of agents with a small number of human overseers, the involvement of human oversight in all substantive decisions is probably an unacceptable overhead.

This issue applies at every level simultaneously, and seems particularly serious when we consider systems thinking about how to think. For example, in order to produce a plan, a simulated human or team would first plan how to plan. They would seek out relevant information, talk to people with necessary expertise, focus their attention on the highest-priority questions, allocate available computational resources, and so on. If human oversight is necessary for every step of this process, the resulting system is not even as useful as a black box that outputs good plans.

Of course, even if humans rely on goal-directed behavior to accomplish these tasks, this doesn’t imply that machines must as well. But that would require a concrete alternative approach that still captures the benefits of goal-directed reasoning without substantial oversight.

\*\*Non-consequentialist agents\*\*

Instead we might build agents with no explicitly defined goals which can nevertheless accomplish the same tasks as a helpful human.

Perhaps most straightforward is an agent that asks “What would a helpful human do?” and then does that. If we have a particular helpful human available as a template, we could build a predictive model of that human template’s decisions, and use this model to guide our agent’s decisions. With a good enough model, the result would be precisely as useful as the human template.

This proposal has a number of potential problems. First, it does not scale to the available abilities—-it is never more useful than a simulation of the human template. So it is not a general solution to the steering problem, unless we have access to arbitrarily capable human templates. Second, if our predictions are imperfect, the behavior may be significantly worse than the helpful human template. Third, making accurate predictions about a human is itself a superhuman skill, and so asking Alice to do what she thinks Bob would do can result in behavior worse than Alice would produce on her own, no matter how smart Bob is.

We can address some of these problems by instead asking “What choice would the human overseer most approve of?” and then taking the most-approved-of option. And a bootstrapping procedure can address some limitations of using a human template. I explore these ideas [here](https://medium.com/@paulfchristiano/model-free-decisions-6e6609f5d99e).

Research on inverse reinforcement learning or goal inference could also be used to learn imitative behavior which is sensitive to human goals, rather than to build systems that infer a human’s long-term goals.

There may be many other alternatives to goal-directed behavior. Any proposal would probably be combined with some ideas under “rational agency” and “shared language” above.

Bootstrapping
-------------

We might try to deal with two cases separately:

\* We are dealing with machines \*much\* smarter, faster, or more numerous than humans.
\* We aren’t.

In the first case, we could try to delegate the steering problem to the much more capable machines. In the second case, we might be able to make do with a weak solution to the steering problem which wouldn’t scale to more challenging cases.

It’s hard to know how this strategy would work out, and to a greater extent than the other approaches we would expect to play it by ear. But by thinking about some of the difficulties in advance, we can get a better sense for how hard the steering problem will actually be, and whether a strong solution is necessary.

Roughly, there are two pieces to a bootstrapping protocol:

1. We need to solve a weak version of the steering problem. We don’t have to make our machine as useful as a human, but we do need to get \*some\* useful work, beyond what we could have done ourselves. Ideally we’ll come as close as possible to a useful human.
2. We need to use the available machines to solve a stronger version of the steering problem. This is repeated until we’ve solved the full problem.

Both of these steps would rely on the kinds of techniques we have discussed throughout this section. The difference is that the humans, as well as the machine intelligences, only need to solve the steering problem for machines somewhat smarter or faster than themselves.

One implication is that weak solutions to the steering problem might be amplified into strong solutions, and so are worth considering even if they can’t be scaled up to strong solutions directly. This includes many of the approaches listed under “Delegation” and “Goal-less intelligence.”