Two guarantees

When I imagine proving something about an AI, or making an inductive argument about [amplification](/iterated-distillation-and-amplification-157debfd1616), I think about two different guarantees:

1. The AI behaves well \*on average\* over some particular distribution of inputs. (The performance guarantee.)
2. The AI never does anything “actively malicious,” on \*any\* input.
(The control guarantee.)

Definitions
-----------

We can define “behaves well on average” by looking at the average impact of decisions on our utility function. [This post](/thoughts-on-reward-engineering-82b193ec03f6) addresses some subtleties with that definition, including inconsistent preferences, different scales of impact, and long-term consequences. Actually evaluating this quantity, so that it can be used in training, requires [amplification](/iterated-distillation-and-amplification-157debfd1616) and [informed oversight](/the-informed-oversight-problem-1b51b4f66b35).

My intuition about “actively malicious” is best captured by the (informal) idea of [incorrigible](/corrigibility-3039e668638) optimization: our AI should never be actively trying to undermine our understanding or control of the situation. This property seems “easy,” in the sense that it can be satisfied without any domain knowledge or expertise, which makes me optimistic that it is possible to [satisfy it for every input](/techniques-for-optimizing-worst-case-performance-39eafec74b99).

Rationale
=========

Rationale for the performance guarantee
---------------------------------------

The performance guarantee is what does the work; it’s the reason you’d want an AI instead of a pet rock.

If we construct the reward so it reflects the difference in utility between different actions, then the average case is actually a pretty good thing to optimize.

In reality the data distribution is constantly changing over time. So we want an [online guarantee](https://www.cs.huji.ac.il/~shais/papers/OLsurvey.pdf), maximizing the average case performance over a varying sequence, rather than a batch guarantee. Normally deep learning is designed for the batch setting under the iid assumption, and data distributions change slowly enough that this is acceptable. Overall, I don’t think this distinction is a huge deal.

Note that the “average case” guarantee still reflects the overseer’s desire for [corrigibility](/corrigibility-3039e668638). This is formally reflected when (a) the overseer evaluates the expected utility impact of a decision by marginalizing over all of the mistakes that the AI might make in the future, (b) the overseer evaluates individual decisions by assigning a large penalty for incorrigibly doing the wrong thing, but only a small penalty for behaving corrigibly but slightly inefficiently.

Rationale for control guarantee
-------------------------------

Unfortunately, it is only possible to get an average performance guarantee when the rewards are small relative to the total amount of data. If a few bad decisions can destroy all of the value from a lifetime of good decisions, then traditional ML can’t do well on average.

If an AI is \*trying\* to cause a catastrophe, it can do so very quickly even in apparently benign situations (by breaking the mechanisms that humans use to remain in control of the situation). The goal of the control guarantee is to prevent this kind of catastrophe.

Similarly, if an AI is trying to be difficult, it could intentionally undermine whatever testing regime we use (by failing selectively on inputs that don’t look like tests). The control guarantee should also rule out this kind of intentional sabotage.

Hopefully the control guarantee cuts off the worst catastrophes, making the performance guarantee meaningful.

Other catastrophes
------------------

Even if our AI satisfies the control guarantee, it can make other kinds of catastrophic mistakes (just as an [aligned human](/clarifying-ai-alignment-cec47cd69dd6) could make a catastrophic mistake). For example, the system might be in charge of running a nuclear defense system, and a single failure could lead to a nuclear war. Or an attacker might fool the AI into executing an instruction by impersonating the operator, which in turn open the system to further attack.

Neither the control nor performance guarantee directly address these problems. Instead, anyone who deploys an AI needs to be aware of the system’s limits, to test the system to see where it might fail, to design mechanisms with redundancy, to protect the system for attackers, and to avoid incorrectly assuming perfect performance. (The same measures we would take if delegating to a human who sometimes made mistakes.)

Corrigibility
-------------

The performance and control guarantees interact to create a system that is corrigible: the performance guarantee ensures the system is typically trying to give the human more effective understanding and control over the situation. The control guarantee ensures the system isn’t undermining those measures, for example by constructing a fake narrative for the human or illusory control, by leaving backdoors that can be exploited, etc.

The performance guarantee leaves open the possibility that the system will sometimes fail to inform or empower the human effectively. But as long as those are failures aren’t optimized to be unrecoverable, it seems the human can work around them by recognizing the shortcoming and having the AI optimize for the robustness of human control and understanding.

I think there is a meaningful analogy between this picture and this [post about monitoring and whistleblowing](https://sideways-view.com/2018/02/01/honest-organizations/). and I have a vague intuition that there is some important underlying dynamic that could be better understood.

Amplification
=============

Amplification and performance
-----------------------------

[Amplification](/policy-amplification-6a70cbee4f34) solves a task by (adaptively) breaking it into several subtasks, solving the subtasks, and combining the results.

If we are making an inductive argument about amplification, then the performance guarantee implies the expected \*average\* performance on subtasks is good, and requires that the expected performance on the overall task is good.

If some of the subtasks make “honest mistakes,” then I think this is a reasonable hope.

In order to make it work we definitely need to have an unequal weighting over subtasks, which corresponds to an unequal sampling of subtasks during training. For example, if we are trying to figure out if any of {A, B, C} are positive, we proceed by independently estimating A, B, C, and we conclude that B is positive, then the estimation of B was a more important subtask than the estimation of A or C.

Even with arbitrary weights on subtasks, preserving average performance might seem too optimistic. In the previous example, if we conclude that all of A, B, C are negative, then any one of the three errors could lead to an erroneous overall conclusion. So if our original error rate was 5%, then our new error rate will be 15%.

In cases like this, it’s wrong of think of “error rate” as the measure of performance that should be preserved inductively. Some questions are harder than others, and should be expected to have higher error rate — if any rational agent tries to figure out whether any one of N quantities is positive, without any clever trick for figuring it out, then they’ll have a harder time than if they were just trying to estimate whether a single quantity is positive. This makes it harder to talk precisely about the guarantees of amplification, but leaves me feeling relatively optimistic about eventually finding a working analysis.

Amplification and control
-------------------------

I find the above intuition plausible when errors are “honest mistakes.” If instead one of the subtasks is optimized to cause trouble, then I think it is very hard (and likely impossible) to correct the problem. So in order to have any hope of maintaining performance, we need to guarantee that none of the subtask solutions are actively malicious.

But in order to guarantee that a subtask isn’t actively malicious, we also need to assume that none of \*its\* subtasks are actively malicious. If we want to carry out the induction to a reasonable depth, this basically requires a worst-case guarantee. Otherwise even a single failure, in one leaf of an exponentially large tree, could cause trouble.

Unfortunately, a human probably does not satisfy the control guarantee, since they may behave very strangely on some small subset of possible inputs or with small probability. I originally described [security amplification](/universality-and-security-amplification-551b314a3bab) and [reliability amplification](/reliability-amplification-a96efa115687) to gradually eliminate the human control failures (as well as new failures that crop up, e.g. from memetic selection inside the amplification process).

Interestingly, some ways of achieving the control guarantee in the distillation step could also be applied even if the overseer \*doesn’t\* satisfy the control guarantee, and so could be used as a replacement for security/reliability amplification. This isn’t as crazy as it sounds, since the ability to look at the entire code of a learned model gives us a substantial advantage over dealing with an opaque overseer. It gives a further reason to prioritize [techniques for worst-case performance](/techniques-for-optimizing-worst-case-performance-39eafec74b99) (and particularly interpretability).

Conclusion
==========

We can’t guarantee an AI is aligned if we have only an average-case guarantee, \*or\* onlyaworst-case guarantee. So achieving both seems like the “minimum viable product” for alignment research.

My original intuition (in mid-2016) was that having two separate guarantees must be an ugly hack, and that the real goal should encapsulate both. That’s no longer so clear to me: I think these two properties interact surprisingly nicely, such that they may actually suffice to get good behavior even though it looks like a weird combination. At the same time, I think attempts to capture both are much less promising than I’d initially believed.

I still think we need more clarity about what we should be trying to prove. I think that having two separate guarantees, one for the worst case and one for the average case, is the current best guess and is the most promising starting point for further research. In the short term, my focus will be on understanding ways in which this structure is inadequate and on independently refining each of these two subgoals.

The "minimum viable product" of AI alignment research would be an AI that satisfies two guarantees -- first, that it achieves good average-case behavior (the performance guarantee), and second, that it achieves reasonable worst-case behavior (the control guarantee). There's then some discussion of how we might establish these guarantees inductively about [amplification](https://ai-alignment.com/policy-amplification-6a70cbee4f34).

I like this framing of what guarantees we want to achieve. Hopefully we can apply this to other AI systems as well.

Techniques for optimizing worst-case performance


Universality and consequentialism within HCH

The scalability of iterated amplification is closely tied to whether very large teams of humans are “[ascription universal](/towards-formalizing-universality-409ab893a456).” If they are, we [could](/informed-oversight-18fcb5d3d1e1) [hope](/universality-and-model-based-rl-b08701394ddd) that their approval would be an adequate training signal for powerful aligned behavior, and that we can approximate this signal with iterated amplification.

There are many reasons that a team of humans could fail to be universal. For example:

1. They could be unable to understand some kinds of effective reasoning, because they can’t represent relevant concepts, understand particular inferences, or whatever else.
2. They could make errors that derail an otherwise-universal reasoning process.
3. They could use processes that work “most of the time,” but fail catastrophically for a small number of inputs.
4. Errors of type #2 or #3 might produce extreme outputs that tend to propagate through the reasoning process and compound into larger failures (even if they occur with small probability or for a small fraction of inputs).
5. In a large enough computation, errors of type #2 or #3 could eventually give rise to intelligent patterns of behavior that systematically identify and exploit mechanisms by which they could propagate themselves or cause cascading failures. For intuition, note that memetic selection amongst a group of 10¹⁰⁰ humans could be much stronger than the genetic selection that gave rise to life on Earth.

(This is not an exhaustive, exclusive, or even properly parallel list.)

My current view is that #2 and #3 can probably be corrected by traditional methods, even when compounded by #4: computing carefully or redundantly, sanity-checking outputs, structuring computations to be more robust to isolated failures, and so on\*.\* These problems seem be easier to solve for smarter teams(though also more problematic if left unchecked), and I suspect there is some competence threshold above which they become manageable.

A priori, #2 or #3 compounded by #5 could be harder to cope with:

\* An intelligent optimization process could systematically subvert whatever error-correction or robustness mechanisms we try to design.
\* For “random” failures, it’s easy to imagine that a sufficiently diverse and extensive set of checks can drive failure rates exponentially low, but malicious optimization is a natural source of highly correlated failures.
\* Problem #5 potentially becomes \*worse\* as we move to larger and larger teams, because such teams may give rise to larger and larger adversaries. This means that our defenses are engaged in a race, instead of fighting against a fixed target which we might expect them to eventually overcome. (Absent an adversary, we still have a “race” in that we need to drive failure probabilities lower and lower for larger teams, but if we get exponential or even linear scaling then this seems feasible.)
\* Intuitively, I would feel a bit surprised if random errors and propagating patterns managed to totally overwhelm the optimization being applied by a very large team of humans. But I would not feel so surprised if a large team of humans gave rise to other misaligned optimization power, and if that misaligned optimization eventually became the “main affair.” This is a little bit like the difference between concerns that could extinguish all life on Earth (which seems quite hard to me) and those that could replace life on Earth with more effective competitors (which feels very possible).

This post
---------

I’m going to argue that problem #5, the appearance of consequentialism within HCH, might be manageable. That is, if HCH would be ascription universal if not forthe possibility of malign consequentialism within HCH, then we can probably make it universal even in light of that possibility.

The basic idea is to apply a “sanity check” to each of HCH’s computations. If a computation is optimized for negative consequences, then by ascription universality HCH can identify those consequences, and any output affected by such a malicious computation can be censored. In practice I expect this to be contiguous with the use of [reflection](https://ought.org/projects/factored-cognition/networks-of-workspaces#stage-3-reflection) to sanity-check large computations (and similar mechanisms may be the best way to protect against more prosaic cascading failures).

The arguments in this post are not tight, and indeed it’s not clear they could be totally tight without clarifying the assumption “HCH could be universal if not forthe possibility of malicious consequentialism.” But I find these arguments suggestive enough that I think the current focus should be on problems #1 and #3 — whether it is possible for HCH to robustly understand complex reasoning even in principle — without worrying about the possible emergence of consequentialism. If this is successful, I think there is a good chance that we could address issues #2, #4, and #5.

Filtering
=========

Given a “transcript” τ of HCH(Q, N)’s behavior, i.e. a record of all of the questions, subquestions, and answers, we can ask two questions:

\* Info(\*a\*, \*a\*′, τ) : “Which of the two arguments \*a\* and \*a\*′ is more useful for determining whether the transcript τ contains problematic subcomputations?”
\* Trouble(\*a\*, τ) : “in light of argument \*a\*, does transcript τ contain problematic subcomputations?”

Now we’ll jointly define filtered-HCH(Q, N) and Filter(τ, N) as follows.

Define Filter(τ, N) as:

\* Find the \*a\*\\*[maximizing](/optimizing-with-comparisons-c02b8c0d7877) filtered-HCH(Info(·, ·, τ), N).
(Interpret ⟂ as a tie.)
\* Run filtered-HCH(Trouble(\*a\*\\*, τ), N) 3 times, and return the majority vote.
(Interpret a response of ⟂ as a judgment that τ is problematic.)

Define filtered-HCH(Q, N) as:

\* First call HCH(Q, N) to get an answer A with transcript τ. But every time you would call HCH to answer a subquestion, call filtered-HCH instead.
\* Call Filter(τ, N-1) to test whether this answer is problematic.
\* If τ is problematic, return a special symbol ⟂, otherwise return A unmodified. (If you return ⟂, you can eliminate this subtree from the transcript.)

Within HCH, if a question is answered with ⟂, the question-asker needs to respond appropriately. They could try re-asking the question in a different way; if that is unsuccessful they might assume that the question itself represents an attack on HCH and so needs to avoided. What to do next depends on the context — HCH might decide that a particular hypothesis is “malicious” and therefore best dropped from consideration; or that a particular idea is hazardous to think about directly and so it needs to engage in indirect reasoning to try to mitigate the harms; or it might just pass the failure up the tree with some explanation of where the problem appeared.

Note that the pattern of which questions get censored could itself cause trouble if it were maliciously optimized. However, achieving that would require a malicious computation to sometimes decide \*not\* to get the surrounding computation censored, which shouldn’t be possible. That is, if a computation thinks to itself “should I do something malicious in order to cause the computation to be censored?” and concludes “no” in order to cause trouble, then that should already be grounds for censoring.

Informal analysis
=================

Assumption
----------

I’ll assume that using HCH to find a maximally useful argument \*a\*\\* about Q, and then using HCH to answer Q given \*a\*\\*, is [ascription universal](/towards-formalizing-universality-409ab893a456) — or rather would be universal if not for the possibility of malicious internal consequentialists.

Moreover, I’ll assume that it’s universal with some small blow-up, i.e. that if we give HCH a budget \*k\* N\*,\* then it epistemically dominates any computation that could be simulated by HCH with budget N.

Note that this is a very strong assumption. This verifier receives the argument \*a\*\\* that they’d evaluate most highly, and so this will fail if there is \*any\* argument that would cause them to give a high score and behave badly. I’m trying to modify HCH to be robust to the possibility of internal consequentialists, but there are all sorts of other ways that a carefully chosen input could compromise a system that is “mostly” universal, which I’m assuming are already resolved.

Hope
----

Why might we expect this filtering procedure to fix problems caused by malicious consequentialists?

Then the rough hope is:

\* If there is a consequentialist whose behavior is optimized to break HCH, then it “knows” something about why that behavior will cause trouble.
\* So by ascription universality, filtered-HCH(Trouble(\*a\*\\*, τ)) also knows why the behavior within τ will cause trouble.

This is only plausible once the original human H always behaves in the “intended” way and never performs malicious optimization, which I’d hope to achieve by [only showing them inputs from some small set](/universality-and-security-amplification-551b314a3bab). So any malicious consequentialist is implemented “on top of” these humans rather than within.

(Easy mode) Handling stochastic failures
----------------------------------------

Suppose that HCH(·, N) has internal malice with probability ε ≪ 1 on some inputs. This is a special case of the full problem (where errors occur with small probability) and the solution won’t be reused to solve the more general problem, so an impatient reader should skip ahead. I’m including it because I feel quite uncertain about the general solution, and I think this simpler solution already shows some of the upside of filtering and has a reasonable chance of being useful.

If we only want to cope with stochastic failures, the easiest approach is to use a slightly modified definition of Filter:

\* Use HCH(·, \*k\*N) to evaluate Trouble(\*a\*\\*, τ) and Info(\*a\*, \*a\*′, τ), rather than filtered-HCH(·, N).

We hope that HCH(·, \*k\*N) has a probability at most \*k\*εof encountering malicious internal computations. This seems intuitively plausible, since it is \*k\* times larger than HCH(·, N).

Internal malicious computations can affect Filter(τ, \*k\*N) in two ways:

\* HCH(Trouble(\*a\*, τ), \*k\*N) may behave badly with probability \*k\*ε. But the only possible “bad” behavior is to give the wrong answer, either a false positive or false negative. If we take a best-of-3 vote, this probability is reduced to \*k²\*ε², so we have significantly reduced the failure probability as long as ε ≪ 1/\*k\*² (even if we consider a false positive, leading a subcomputation to be censored, to be as bad as an arbitrary failure).
\* HCH(Info(\*a\*, \*a\*′, τ), \*k\*N) may behave badly, which affects which argument \*a\*\\* will be optimal. However, we still get a distribution over computations which is “almost optimal” in the sense that it is preferred to every other argument with probability at least 1/2 − \*k\*ε (while the optimal argument would be preferred with probability at least 1/2). It seems reasonable to assume this small difference doesn’t affect the universality of HCH (and we can drive this probability exponentially close to 1/2 by ensembling).

This seems like a significant improvement over my previous proposal for [reliability amplification](/reliability-amplification-a96efa115687) because Filter(τ, N) returns yes or no, even if Q itself has complex answers. This lets us use a simple majority vote to improve reliability. In particular, we no longer require that no agent in the ensemble be too much smarter than the others — we have shifted a vague assumption about relative capabilities to a slightly more precise assumption about universality.

This mechanism only fixes errors that occur with sufficiently small probability. But we can apply it throughout a large tree, reducing the error probability frequently enough that it never gets large.

(Hard mode) Handling “bad” questions
------------------------------------

The problem is much harder when there are “bad” questions Q for which HCH has a significant probability of encountering malicious consequentialism and failing. If τ is the transcript from a question where HCH failed, Trouble(\*a\*\\*, τ) may well be another question on which HCH predictably fails (e.g. because the same vulnerability is triggered by the attempt to diagnose the original failure). This could in turn prevent Filter() from censoring the original failure.

In this case we are going to argue inductively based on budgets: we can assume that filtered-HCH(·, N-1) probably has no malicious consequentialism by induction, and hence is universal.

Because we want to use an induction, it’s important that we answer Trouble(\*a\*\\*, τ) and Info(\*a\*, \*a\*′, τ) using a \*smaller\* budget than was used in τ. But our assumption about universality requires the reasoner to be \*k\* times \*bigger\* than the computation it wants to compete with.

I hope that if there is malicious consequentialism within HCH(Q, N), then it needs to be “small” enough that we can rely on filtered-HCH(Trouble(\*a\*\\*, τ), N-1) to compete with it. Intuitively:

\* None of the unfiltered subquestions in HCH(Q, N) involve any malicious consequentialism, so we hope each subquestion either “behaves as intended” or produces the special symbol ⟂.
\* If all subquestions of HCH(Q, N) are behaving as intended, and if the human at the root of HCH(Q, N) is behaving as intended (which we attempted to guarantee by limiting their input space), then any malicious consequentialism is very tightly constrained.
\* Intuitively, it’s restricted to two options:
(a) be computed implicitly with “side effects” of the intended behavior, repurpose valid parts of the computation for unintended effects, \*etc.\*
(b) make use of the intended behavior of questions
\* When using option (a), it feels like the malicious optimization will be significantly less powerful than HCH with the same budget. For example, when I’m answering a question, it seems like most of my useful computational work is going into actually answering the question; if someone wanted to try to do computational work based on details of how I formatted my answer, I think they’d get less done. So we can hope that this part of the malicious consequentialism could be simulated by HCH with total budget much less than N.
\* When using option (b), the malicious consequentialism can be very efficient — exactly as efficient as HCH itself, since it’s just using that computation in the intended way. But filtered-HCH(Trouble(\*a\*\\*, τ)) may not need to understand these parts of that computation in order to do its job. Instead it can look at the transcript τ and take the relevant answers at face value, just like the malicious computation is doing. (This relies on the inductive invariant, and even with that isn’t obvious, but it looks pretty plausible to me.)

Note that this strategy really leans on the assumption that HCH works well for \*every\* input. In addition to the fact that \*a\*\\* is optimized, Filter calls HCH with a large and potentially malicious transcript τ as its input. The distinction between “untrusted input that you consult when appropriate” and “subcomputation that has a privileged position from which it can cause harm.”

In all the settings discussed so far, we hope to use <@HCH@>(@Humans Consulting HCH@). What if HCH itself contains malicious optimization? Specifically, recall that HCH involves an arbitrarily large tree of humans breaking questions down into subquestions. Memetic selection on 10^100 humans should be _much_ stronger than the selection done by evolution to produce us, and so could be quite dangerous. We would like to use HCH to check whether its own computations are malicious.

So, we'll define a pair of mutually recursive functions, Filter and Filtered-HCH. Filter takes as input a transcript from HCH, and using filtered-HCH it finds the best argument suggesting that the transcript is problematic. It then uses filtered-HCH to ask whether, in light of this argument, the transcript should be treated as problematic. This check is repeated thrice for robustness. (Note that the original HCH treats the transcript as trusted computations, while the checker treats it as an untrusted input.) Filtered-HCH has the same type signature as HCH (i.e. it produces answers to questions). Just like HCH, it computes an answer to the question, with the help of recursive calls to filtered-HCH to get answers to subquestions. Instead of returning the answer directly, it calls Filter to check that the computation done was problematic, and returns the special symbol ⟂ instead of the answer if so.

The basic hope is that if there's some malicious subcomputation implemented "on top of" the base agents in HCH, and it tries to break HCH, then it "knows" why that attempt could work, and so if filtered-HCH is universal with respect to computations that can be instantiated by (smaller versions of) HCH, then filtered-HCH will also "know" that the computation could break HCH, and will censor it as a result. Consider the threat model from <@Reliability amplification@>, where the chance of any particular HCH computation with budget n becoming malicious is ε, independent of the input or any past HCH calls. Then, intuitively, filtered-HCH (which needs a higher budget, say kn) has a kε chance of becoming malicious, but since the check is repeated three times and a majority vote is taken, the chance of a malicious computation getting through is k^2ε^2, which will be much smaller than the original ε if k^2 ε << 1. (This is significantly better than the scheme in <@Reliability amplification@>, because the question "is there a problem?" is a binary yes/no question, and so majority vote works a lot better.) The harder case is when specific inputs cause very high probabilities of failure (which I believe is the threat model from <@Security amplification@>). I particularly don't understand this section, but I believe the hope is that since the malicious computation is implemented "on top of" the (aligned) humans, only a small "part" of the computation can be malicious, and this can be detected by a _small_ instance of Filtered-HCH on which the malicious computation cannot be implemented.

I don't understand this post that well, so I'll refrain from opining on it. I think I would benefit from a more concrete example of how malicious optimization could be implemented "on top of" HCH, even when the humans are restricted to a small set of inputs on which they are "secure".


Universality and model-based RL

[Ascription universality](/towards-formalizing-universality-409ab893a456) seems sufficient for [informed oversight](/informed-oversight-18fcb5d3d1e1), and I’ve argued it could be used to [prevent cascading failures in HCH](/universality-and-consequentialism-within-hch-c0bee00365bd). In this post I’ll argue that ascription universality also appears to address two problems with aligning model-based RL:

1. It allows you to perform induction without being hijacked by malicious predictors (this is the “[benign induction](https://agentfoundations.org/item?id=1263)” problem, which is extremely speculative)
2. It seems necessary, though maybe not sufficient, for extracting “all the relevant facts a model knows.” (Which is at least necessary, and I think probably sufficient, for defining a reward function that yields aligned behavior when optimized.)

These arguments aren’t rigorous enough that I find them convincing, but they are enough to convince me that better understanding universality is a top priority. At this point I wouldn’t be surprised if universality is enough to resolve most of the classical philosophical difficulties for AI safety. If so, first we’ll need to better understand in what sense it could be achievable.

Setup
-----

I’m interested in the following hopefully-aligned version of model-based RL:

\* Use iterated amplification to learn a (distribution over) models of the world.
\* Use iterated amplification to learn a utility function over sequences of states and actions.
\* Plan in that model+utility function (e.g. using MCTS)
(You might want to combine learning a value function and terminal values, but for now I’ll simplify.)

This is similar to the traditional model-based RL approach, except that we learn a model and utility function with iterated amplification, instead of learning a model to predict future observations and defining a utility function over observations by hand.

What is a model?
----------------

The simplest kind of model to consider formally is a \*predictor\*, which we can view as a probability distribution over sequences of observations. We can evaluate predictors on past observations, and use them to predict future observations.

This is not enough for making the decisions, because the utility of an outcome is not a (simple) function of our future observations. We care about lots of features of the world that are hard to directly observe, and if we tried to define our preferences in terms of our observations alone we’d need an additional step where we map our observations to a prediction about what’s “really happening” out there in the world. For example, given a bunch of observations we might ask “given those observations, is Alice actually happy?” At some point we need to get a a distribution over models that we can use to answer questions about \*the stuff we care about\*, and not just our observations.

(Closely related: [Formalizing Two Problems of Realistic World Models](https://intelligence.org/files/RealisticWorldModels.pdf), though the basic philosophical issue is a classic.)

As a first step we can ask a model to answer arbitrary questions about the world rather than only reproducing observations. We can evaluate a model by looking at questions whose answers we already know. A problem with this is that in order to behave well a model only needs to correctly answer \*the kinds of questions that we might know the answer to\*, and the simplest model may well behave strangely on other questions.

A second step is to view a model M as an object that HCH can reason about, and use HCH to answer questions like “If model M describes reality, then what would be true?” For example, a model might posit some kinds of physical objects, and HCH could reason from those objects both to explain our observations and to infer the existence of other objects we care about. This seems to more closely track the reasoning humans perform about possible models, and (a) gives us more traction on a good prior over models, (b) gives us traction to pose other questions about the model (beyond what its predictions are).

I’ll use this notion of “model” throughout the post.

Aside: defining goals
---------------------

If we are able to learn a “correct” and “understandable” model, I’m optimistic about being able to find a utility function that can induce aligned behavior. In particular, I think that function can look something like:

\* Identify a thing that is “me” in the resulting state
\* Conservatively evaluate how many resources “I” effectively control
(working together with AI systems, \*etc.\*)
\* Conservatively evaluate how “I” changed over the history — do I endorse the way in which my beliefs and values shifted?

We can’t hope to have completely robust answers to any of these questions. But we can at least hope to have a utility function that is safe to maximize within our model, by being aware of the history leading to high utility and penalizing the act if that history leads to high utility for bad reasons.

For example, if we can see that we control resources only because other agents disagree with our assessment of which resources are important, then we can revise our estimates appropriately — we don’t have to fix a single way of valuing resources which is robust to arbitrary adversarial pressure.

(Realizing this kind of hope is related to the ideas expressed by Jessica Taylor [here](https://agentfoundations.org/item?id=1149) and [here](https://agentfoundations.org/item?id=1224).)

As another example, our preferences over changes to ourselves will not be completely robust. But maximization over the utility function will only exert pressure for such changes if they somehow lead to \*higher\* reward. And in order to do that, the changed behavior needs to actually manifest over the history, and we can use that manifestation to detect a corruption that allows the agent to receive higher reward even if we wouldn’t recognize most possible corruptions.

“Knowing what the model knows” seems to be a critical ingredient for constructing such a utility function. It’s not clear whether or not it’s sufficient, but I won’t discuss this issue more in this post.

Problem 1: Benign induction
===========================

If we take the “simplest” model that explains our observations, it could potentially be [very](https://ordinaryideas.wordpress.com/2016/11/30/what-does-the-universal-prior-actually-look-like/) [strange](https://agentfoundations.org/item?id=1263). In particular, it could produce correct observations because it contains an intelligent predictor rather than because it captures “real” physics. This could cause it to generalize in unpredictable and dangerous ways (depending on the motivations of that agent).

Two variants of this problem have been discussed:

1. The predictor may literally \*contain\* intelligent agents who are trying to predict our observations. These agents may be motivated by the desire to survive the training process, but then may generalize maliciously once off the training distribution.
2. The predictor could \*reason\* \*about\* intelligent agents who change their behavior in order to explain our behavior, because the existence of such agents is the “simplest” explanation for our observations. A notion of simplicity that produces this behavior is in some sense not the “right” one, but I believe it would probably apply to the simple formalisms we currently understand (such as minimum description length).

(A single situation may have some features of both #1 and #2, this is not a clean dichotomy.)

If we have a universal reasoning procedure, I think we can likely avoid these problems.

Option 1: filter malign models
------------------------------

We use the following procedure:

\* Compute log likelihoods for models as normal, by asking HCH(“how likely are facts F if model M is true?”)
\* Additionally, compute HCH(“is model M likely to lead to catastrophically bad predictions on inputs we’ll encounter in the future?”) and judge a model “unsafe” is the answer is “yes.”
\* Apply Bayes as usual but restricted to the safe models.

Why might this work?

If a model M is malign, then it is making good predictions \*because\* of the existence of some adversarial agent. Then:

\* If that agent is actually contained in the model (situation #1), then universality means we can learn everything that agent knows. But the agent has formed the plan of giving good answers in order to increase the model’s log likelihood and so have more influence later, and if we know this then we will realize the model will generalize poorly and filter it out.
\* If the agent is being reasoned about by the model (situation #2) then things are even easier, since the model knows about the existence of the agent and its motivation. Then applying universality to the model itself will let us know the model would generalize poorly.

Going beyond the simplest safety test, we could also try to throw out any “abnormal” models, e.g. where the stars don’t actually exist or our local universe is actually a fiction. This could leave value on the table if these weird hypotheses are important, but it could also reduce the risk of making a serious error based on strange hypotheses. Universality is relevant because it potentially allows us to avoid adversarial pressure on this definition of “normal” — if a model looks normal because an adversary is trying to make it look normal, then we can recognize that and throw it out as unsafe.

Option 2: outcompete malign models
----------------------------------

Suppose that I have a prior μ over models, and I believe that the posterior puts significant probability on “malicious” models that make good predictions only because they contain agents who are motivated to make good predictions. (Or because they reason about such agents.)

This is a strange epistemic situation to be in. I must believe that on average these malicious agents have a “better” prior μ than I do — that’s what they need in order to assign a higher probability to the data than the rest of μ. Jessica Taylor writes about this situation [here](https://www.lesswrong.com/posts/5bd75cc58225bf067037535e/are-daemons-a-problem-for-ideal-agents), and has similar intuitions about it.

Intuitively, something like universality should allow you to avoid this state. That is, the malicious models must perform some reasoning that we believe allows them to assign better prior probabilities; by ascription universality HCH must be able to capture this reasoning and use it to inform \*its\* prior. As long as malicious agents have a reasonably low prior they will therefore end up with a reasonably low posterior.

To implement this strategy, we could use the same procedure as in the last section, but instead of using a simplicity prior over models M, we would use HCH to compute those prior probabilities by asking HCH(“a priori how likely is model M to describe the observations we are going to provide to it?”) or something along those lines.

I don’t know whether this works, and I’m not sure it will really be possible to say much precisely without having a cleaner picture of the sense in which HCH is or isn’t universal. If it does work, this seems like a much cleaner solution than filtering. The fact that there are two plausible options on the table makes me more optimistic that universality will address benign induction one way or another.

Problem 2: extracting facts from a model
========================================

Consider a model M that specifies low-level physical principles and some basic “bridging” facts about how everyday objects are implemented in low-level physics. HCH can answer questions about the world by performing a detailed physical calculation, and then using bridging facts to relate the resulting state to things we care about.

(Modeling physics itself is a somewhat exotic example, but I think the same phenomenon would occur in more prosaic cases, wherever there are effective but opaque models.)

Unfortunately, M can be a good model even if the bridging facts are not exhaustive. They might allow HCH to answer some kinds of questions — e.g. to predict the observations that we’re using to test models — but not to answer other important questions.

For example, it may be the case that M predicts that Alice will report high satisfaction with the AI’s behavior, by performing some physics calculation and then using bridging facts that relate physical states to Alice’s reports. At the same time, the physics calculation might actually simulate Alice being bribed to report high satisfaction, but it might not be possible to infer that from the available bridging facts.

But the physics computation contains a complete simulation of Alice being bribed, and so we can ascribe knowledge of the bribery to it. Thus if HCH is ascription universal, we can hope it recovers all of these facts.

It’s not obvious whether universality is actually sufficient for this purpose, and it will depend on exactly what form of universality ends up being achievable. It might be that a “reasonable” ascription strategy would recognize that the model knows facts about atoms, but not facts about Alice.

At a minimum universality is \*necessary\* for this purpose — if M can believe facts that HCH can’t identify, then those facts could turn out to be relevant to the utility evaluation. It’s just that there might or might not be a further step not captured by universality.

My current guess is that the most natural form of universality will give us everything we want, and that thinking about “ontology identification” is mostly useful for finding hard cases for universality. Regardless of whether that’s true, I think the next step should be a clearer picture of prospects for universality, after which we can revisit this question.

So far, we've been talking about the model-free setting, where the overseer provides the incentives. What about model-based RL? Here, we might want to learn separate distributions over models and utility functions using iterated amplification or HCH, and then plan using any off-the-shelf algorithm, such as MCTS. The first new problem that arises is that our distribution over models may be malicious (see [benign induction](https://agentfoundations.org/item?id=1263)). Using universality, we can solve this problem in two different ways. First, we could use HCH to filter out malicious models. By universality, since a malicious model knows what its goals are, so does HCH, and thus HCH will detect it. Second, we could ask HCH to give us a good prior over models. By universality, this prior over models must be better than the prior used by the malicious models, and so the malicious models' predictions will be worse than that of HCH as a whole, and so the malicious models will not be able to affect the computation much.

The second new problem in the model-based setting is that it isn't clear how our utility function can extract all the relevant information from the model. For example, if the model is a detailed simulation of physics, how do you find the "humans" and evaluate how "happy" they are? Universality is at least necessary for this: if there are facts the model "knows", those facts may be relevant to the utility calculation, and so the utility function also needs to know them. We might hope that universality is sufficient: in the previous case, we might think that the physics simulation "knows" whether humans are happy. But this depends on details of what "know" means that haven't yet been fleshed out, and it depends on what "kind" of universality we can achieve in practice.


Universality and security amplification

If many copies of Paul [collaborate to solve a problem](/policy-amplification-6a70cbee4f34), I expect their behavior to be [corrigible](/corrigibility-3039e668638): each copy will make an honest effort to help the group accomplish its goal, to help the group notice and correct mistakes, and so on.

Sometimes by chance one copy may behave incorrigibly. For example, one in every million days I might be struck by an irresistible urge to watch the world burn. [Reliability amplification](/reliability-amplification-a96efa115687) is the problem of combining several probably\*-\*corrigible agents into a more-corrigible ensemble (which only wants to destroy the world on one in every \*billion\* days). I think reliability amplification is probably possible, because a large ensemble of sufficiently-corrigible agents is very unlikely to have more than one incorrigible member.

Unfortunately, some situations might cause me to consistently behave incorrigibly. Let’s call such a situation an “attack.” For example, a clever argument might convince a copy of Paul that he should sabotage the collaboration. If we expose ten copies of Paul to the same argument it won’t fix the problem, we’ll just have ten copies all of whom are behaving incorrigibly.

[Security amplification](/security-amplification-f4931419f903) is the problem of combining many corrigible agents, into a group which is harder to attack than any individual. I hope to iterate security amplification and end up with a secure system.

This post explains why I think security amplification is likely to possible.

Setting
-------

I’ll be thinking about dialog systems that communicate over a text channel. A “query” is anything that you can ask interactively over a text channel.

In this context, corrigible behavior means behaving cooperatively to the person on the other end of the conversation — being honest, volunteering helpful information, following their instructions, \*etc.\*

An “attack” is a sequence of messages which cause an agent to stop behaving cooperatively.

I. Breaking it down
===================

The basic hope
--------------

A large group can answer a big query by breaking it down into less complex pieces. Let’s suppose we had some formal measure of complexity, and that we can answer a query of complexity \*k\* by breaking it into pieces each of which has complexity <\*k,\* until we reach some set of “foundational” queries of complexity \*k\*⁰ that are too simple to break down any further.

This suggests a route to security amplification:

\* Let \*k\*ᴬ be the complexity of the \*simplest possible attack\* on an individual.
\* We hope that \*k\*ᴬ > \*k\*⁰. That is, we hope that the foundational questions are simple enough that all of them are safe.
\* Then if we give any input of complexity \*k\*ᴬ to the group, it will be broken down into pieces of complexity <\*k\*ᴬ (which are necessarily safe).
\* Therefore the complexity of the simplest attack \*on the group\* is higher than the complexity of the simplest attack \*on the individual\*.
\* If we iterate this process, building a group out of groups, then the simplest attack will get more and more complex, until eventually there are no attacks at any feasible level of complexity.

In order to realize this hope, we need:

1. to define a procedure for breaking queries down,
2. to define an appropriate notion of complexity,
3. to argue that all queries above some foundational complexity \*k\*⁰ are broken down into strictly simpler pieces,
4. to argue that all queries below complexity \*k\*⁰ are safe,
5. to argue that the group behaves corrigibly as long as all of the individuals behave corrigibly.

1. Meta-execution
-----------------

[Meta-execution](/meta-execution-27ba9b34d377) can potentially answer a query Q without any individual needing to consider anything as complex as Q itself.

I’ve sketched out part of an example [here](https://medium.com/@paulfchristiano/an-example-of-meta-execution-e1ddc99291c5), to give some sense of what this actually looks like.

2. Defining complexity
----------------------

I’ll start by imagining what would happen if you asked me to pick a query at random.

By “at random” I’m imagining something like the benign universal distribution relative to Paul: I think of all of the possible not-actively-evil strategies I could use for picking queries at random, and then I pick one of those \*strategies\* at random.

For example, my meta-strategy might be: with 5% probability I give a uniformly random string. With 2% probability I think of events in my life and ask a question relevant to one of them. With 15% probability I try splitting up the space of possible questions into a hierarchy, and flip a coin at each step to decide which part of the hierarchy to explore. With 80% probability I do something else.

One of these strategies is particularly important: I could generate an intermediate random query Q′, and then let Q be a random query that arises in the process of answering Q′. For concreteness, let’s say that I use this strategy half of the time.

Let µ(Q) be the probability that this process outputs query Q. We can define the complexity of a query Q as \*k\*(Q) = −log µ(Q).

We can also use µ to measure the complexity of a distribution η over queries, by letting \*k\*(Q) be the maximum value of log (η(Q)/µ(Q)) for any query Q.

Informally, this is the log of “How many samples would you have to draw from µ, in order to get one sample from η?”, i.e how many bits of selection pressure you need to get a sample from η.

(It may be more useful to consider the ε-smooth complexity: what is the \*smallest\* complexity of any distribution η′ that is within ε total variation distance of η? Informally: how many samples would you have to draw from µ, in order to get a sample ε-close to η?)

\*\*Note\*\*: There’s no way that this is the right definition to use and I expect we’ll be able to do better. I think an important ingredient is having a decomposition where the (subquestion → question) graph is a small set expander. If anyone wants to take up this analysis, I think it’s an important question, I expect to come back to it at some point.

3. Can we break a query into simpler parts?
-------------------------------------------

Suppose that I start with a query Q, and split off a subquery Q′. When can we hope that \*k\*(Q′) < \*k\*(Q)?

Suppose there are many possible queries Q for which Q′ appears as a subquery. For example, the subquery “when did [\*a\*] happen?” might appear in the course of answering “was [\*a\*] a cause of the Great Depression?”, or it might occur in the course of answering “was [\*a\*] a consequence of the Spanish flu?”

Each of these corresponds to a way that Q′ could be sampled by µ — since one of the strategies in µ is to generate a random query and then generate a random subquery.

So µ(Q′) > Σµ(Qi) / 2N, where {Qi} is the set of superqueries that give rise to subquery Q′, N is the total number of subqueries for each superquery, and 50% is how often µ decides to sample a random subquery.

As long as each possible superquery has at most 1/(2N) of the total probability under µ — i.e., as long as you can’t guess the superquery with reasonable probability from the subquery — then we have reduced the complexity. If N is 16, this requires that each subquery must throw out about 5 bits of information about the original query.

In fact the situation is better than this:

\* In reality our queries are sampled from some distribution η rather than chosen deterministically. As we increase the entropy of η the set of preimages {Qi} effectively becomes larger. (The analysis also gets a little bit more complicated.)
\* µ mixes in strategies other than (pick a random query, then pick a random subquery). If Q′ is qualitatively simpler than Q, then we may have µ(Q′) > µ(Q) without even considering the possibility of Q′ being sampled as a subquery of Q.

4. Are simple queries secure?
-----------------------------

What is the complexity of the simplest attack on an unaided human? Equivalently: how many queries do we expect to sample, before we find an attack?

This will vary from human to human, so for concreteness I’ll consider a thoughtful futurist or AI researcher who has been briefed about all of the issues we can think of.

If we wanted to actually test for the existence of attacks, we could conduct a giant brute force search for “simple” attacks.

If we imagine generating a query every 30 seconds for 40 hours a week for a year, we generate about 200,000 queries a year. Hiring ten people gets us up to 2M queries a year. 30 seconds may be low, but note that the generation process can produce many samples per invocation, and we only need to test the candidate attacks.

If we put in a little bit of selection pressure I suspect we would find attacks at least 1–3 orders of magnitude more often than by sampling randomly.

So it seems easy to test explicitly whether 100 million random queries contain an attack. This could be done by dedicated [red teams](/red-teams-b5b6de33dc76), and mixed freely with the process used to collect training data. I suspect that thoughtful analysis could get us even further.

Intuitively, I would be surprised if such a search turned up an attack against a careful overseer. If it did turn up attacks, I suspect it would be possible to strengthen the overseer against them.

Overall, I feel pretty safe relying on 10⁸ queries not containing an attack, pretty unsafe with numbers like 10³⁰, and am not sure where in between attacks would crop up.

5. Is a collaboration of corrigible agents corrigible?
------------------------------------------------------

Even if every individual behaves corrigibly, the group need not behave corrigibly. This is a bit of a tangent from the real thrust of this post, which is about removing the security failures of the human. But it’s a tangent worth discussing briefly, since I think it’s an important open question.

There are two very salient sources of incorrigibility (though without a crisper picture of optimization and corrigibility this list isn’t exhaustive):

\* The group is applying selection pressure to patterns and behaviors propagating through the group. Those patterns may themselves become powerful incorrigible consequentialists.
\* The group is implementingincorrigible optimization.
For example, suppose meta-execution asks the subquestion “What does the user want?”, gets a representation of their values, and then asks the subquestion “What behavior is best according to those values?” I’ve then generated incorrigible behavior by accident, after taking innocuous steps.

I think the first problem is likely resolvable by applying security and reliability amplification, which operate on groups as well as individuals. I think the biggest thing that’s missing here are tools for analysis (though maybe once we have those tools we’ll notice more problems). Currently this feels pretty complicated and confusing to me.

The second problem clearly requires the human operator to do something other than just “being sensible.” That is, in order for this scheme to possibly work, the human needs to understand something about what kind of optimization causes incorrigibility and the human would have to avoid actions that introduce problematic optimization. It’s conceivable that this can be done by an existing human if they are paranoid, but realistically the only reason I expect this to go well is because I expect us to make progress on the theoretical issue of understanding incorrigibility. I think this is an important theoretical problem.

II. Universality
================

So far I’ve avoided a key question: why think we can break tasks down at all?

Indeed, some human capabilities \*can’t\* be broken down into pieces. Suppose that a human has seen a bunch of examples (\*x\*, \*f\*(\*x\*)) — such as English phrases and their French translations — and has learned a predictive model of \*f\*. The human may not be able to access their model except by running it, in which case they have no hope of breaking down the task of computing \*f\* — in this case translating a sentence. (Wei Dai proposed the translation example.)

The proposal
------------

I propose to go on breaking tasks down anyway. This means that we will lose certain abilities as we apply amplification.

For example, given the task “Translate the sentence [\*x\*] from French to English” we will answer it without having any translator look at the entire sentence \*x\*. This means that the quality of translation will fall.

After enough steps of amplification, we may eventually arrive at an agent that doesn’t know French at all, and is stuck with recommendations like “Consult an English-to-French dictionary.”

Effectively, this proposal replaces our original human overseer with an impoverished overseer, who is only able to respond to the billion most common queries.

Is this OK?
-----------

The first key question is whether this impoverished overseer remains [universal](/of-humans-and-universality-thresholds-24b473e0c898).

That is, if we put together enough copies of this impoverished overseer (by iteratively apply [meta-execution](/meta-execution-27ba9b34d377)) would we be able to obtain arbitrarily smart groups? Or would we get stuck?

Here we need to be careful about “arbitrarily smart.” There are clearly problems the group will never be able to solve — due to lacking knowledge/expertise— including problems that individual humans \*can\* solve.

This is potentially OK, as long as we learn a good policy for leveraging the information in the environment (including human expertise). This can then be [distilled](/approval-maximizing-representations-56ee6a6a1fe6) into a state maintained by the agent, which can be as expressive as whatever state the agent might have learned. Leveraging external facts requires making a tradeoff between the benefits and risks, so we haven’t eliminated the problem, but we’ve potentially isolated it from the problem of training our agent.

Comparison to agent foundations
-------------------------------

If the impoverished overseer is universal, then the set of questions of complexity \*k\* < \*k\*⁰ form a simple “core” for reasoning: by creating a giant lookup table of human responses to these questions and simply using that lookup table enough times, we can produce arbitrarily sophisticated behavior.

If humans are universal at all then of course such a core exists (just take all questions that a human can articulate in their lifetime). But finding a small core seems like it requires a better understanding of intelligence.

I think that coming up with such a core is a very natural and important problem for researchers interested in philosophical issues in AI. My view is that if MIRI-style research adds value, it is most likely to be by finding an explicit core for reasoning rather than finding an explicit recipe for AGI. This core would then be combined with iterated amplification to yield a competitive AI. However, I don’t think that such a core is likely to encode an answer to questions like “what is the right decision theory to use?” — instead I expect it to look more like a solution to metaphilosphy, automating the process whereby humans answer questions of that form.

Conditioned on amplification working well, I think there is about a 50% chance that it uses an explicit core that we understand and a 50% chance that it uses a messy core learned from humans.

In addition to making it much easier to analyze amplification, having an explicit core of reasoning would also potentially make verification much easier, as discussed [here](/techniques-for-optimizing-worst-case-performance-39eafec74b99). Overall, I think that this kind of perspective might capture most of what is useful about the MIRI view while still being able to leverage the benefits of modern ML.

When you are learning from a source that you can't fully trust (such as humans), one major issue is that the errors that the source makes may be correlated, so you can't just correct for them by averaging over many samples. Paul's overall approach to AI alignment requires the overseer to be corrigible, but there may be attacks that cause the overseer to behave incorrigibly. This post explains how we might solve this, even though the errors can be correlated. Specifically, we assume that the overseer is corrigible for queries that are sufficiently low complexity (for some definition of complexity, he gives an example definition). Then, by automatically breaking down an input task into low complexity subqueries (eg. using meta-execution), we can make sure the overseer always remains corrigible. The post has a lot more detail on each of these points, of course.

You do need to be familiar with IDA. I found it useful to learn about the particular assumption made (corrigibility for low complexity queries). In addition, this post makes the proof of safety substantially more concrete.


“Unsupervised” translation as an (intent) alignment problem

Suppose that we want to translate between English and an alien language (Klingon). We have plenty of Klingon text, and separately we have plenty of English text, but it’s not matched up and there are no bilingual speakers.

We train GPT on a mix of English and Klingon text and find that it becomes fluent in both. In some sense this model “knows” quite a lot about both Klingon and English, and so it should be able to read a sentence in one language, understand it, and then express the same idea in the other language. But it’s not clear how we could train a translation model.

Of course some concepts won’t have translations, and the model will often be uncertain about the translation of a term. But we can still ask for a model to explain the meaning of a Klingon expression as best as it can to an English-speaking user. For example, it could say “This is an idiomatic expression that’s often used to express great uncertainty” or “This is a small animal that is familiar to most Klingon speakers, I think it’s kind of like a frog but am not really sure” rather than translating a sentence directly.

How can we construct an objective that incentivizes the model to “try its best” at this translation task?

Translation-specific approaches
===============================

There are many published heuristics for unsupervised translation (e.g. [Lample et al](https://arxiv.org/pdf/1711.00043.pdf)). I don’t think those techniques should completely satisfy us:

\* Existing methods can’t lead to a model that appropriately describes its uncertainty or talks the user through a hard-to-translate expression. (At least as far as I’m aware.)
\* We have no real reason to think existing methods fully utilize the model’s understanding, or to expect those methods to scale well. (In practice, I think they are impressive but still lag behind the quality of our models’ understanding.)
\* These heuristics are specific to translation, whereas we’d like to find general methods that can scale up to harder problems.

Existing alignment techniques
=============================

If we try to apply RL from human feedback to translation, we immediately run into a problem: how am I supposed to judge which of two English explanations of a Klingon sentence is better, given that I don’t know Klingon?

Debate doesn’t easily address this difficulty either — if one model claims that “qapla” means “great success” and the other claims it means “minor success,” I can’t easily decompose that disagreement into simpler sub-questions that debaters disagree about. Debaters could cite phrases in the database where “qapla” is used, but they’d need to average weak evidence over many phrases. Making things worse, to interpret each usage they’d need to agree about the meaning of the rest of the phrase — -which isn’t necessarily any simpler than the original disagreement about “qapla.” Even if this process was possible, it’s not at all clear that GPT would be able to do it — -being able to translate between Spanish and English doesn’t mean I have an encyclopedic knowledge of all the documents from which I built up my intuitive sense of a particular word’s meaning (which I’d need in order to win such a debate).

Right now I don’t think we have any scalable strategies to this kind of problem; I think it’s a core open question for alignment.

Unsupervised translation seems like a good problem to think about for alignment
===============================================================================

I think the key feature of this situation is that our model has acquired a bunch of intuitions about the domain which are only justified empirically — the model “knows” about the meaning of phrases only insofar as it has a very complex hypothesis that was supported by the data.

This situation is going to become increasingly common as we train more powerful models, and will immediately be a real problem if we are applying human feedback to fine-tune GPT; while GPT is subhuman in many ways, it’s already acquired plenty of knowledge that any given human contractor would lack.

Most of GPT’s knowledge is something that came from \*some\* human, but ultimately we will be training models that generate new knowledge (e.g.by searching over plans in realistic environments, or by writing code on their own and learning about what works), and \*no\* human will have that knowledge. So we can’t hope to get around this problem by simply hiring more knowledgeable contractors.

This can leave us in a situation where it’s extremely difficult for humans to oversee AI decisions. If a model says “My intuition is that this business plan will make a lot of money” the user will need to decide whether or not to trust it. If they don’t, then they may find themselves at an increasing economic disadvantage. If they do, then they may have lost the ability to effectively oversee AI systems except by evaluating the consequences of their actions. That leads directly into the classical challenges of AI safety, namely that AI systems evaluated exclusively on the basis of measured outcomes have a tendency to push the world in undesirable directions (since we can’t measure what we care about) and to corrupt our measurements.

My vague hope
=============

I’m hoping we can address this using the kind of approach discussed in [learning the prior](/learning-the-prior-48f61b445c04). That might look like:

\* In parallel with training GPT, train a helper model that explains the meaning of phrases (it can also provide other intuitions or background facts that are useful for predicting the next word).
\* As we train on Klingon text, we sample phrases and then ask a human “which word will come next?” The human uses the helper model to understand what is being discussed and make a prediction.
\* We optimize the helper model to make the human’s next-word predictions good (in parallel with generative pre-training).
\* Finally, a human uses the same helper model to evaluate a proposed Klingon → English translation, and we use this to train a translator by RL.

That short description sweeps a lot of complexity under the rug. Most importantly, the success of the scheme relies on the correctness of the prior over helper models (or else the helper could just be another copy of GPT-Klingon), and we don’t have a credible strategy for representing and manipulating our prior over complex programs.

Overall, I’d say that this is more at the level of “vague hope” rather than “concrete proposal.” I think it’s an open question whether anything in this space will work.

I think that this is the kind of problem which makes e.g. MIRI researchers justifiably skeptical that scalable ML alignment is possible at all, and it’s the main focus of my current conceptual work on AI alignment. I’m glad that this kind of theoretical crux also looks like it will soon be relevant to ML practice, since I think it will make it much easier to close the gap between people who work on ML and people who work on alignment.


Safer ML paradigms team: the story – AI Safety Research Program

Safer ML paradigms team: the story
==================================

\*This is a summary of the project the team focussing on combining Inductive Logic Programming with Deep Learning undertook.\*

Exploration phase
-----------------

During the first retreat we formed a team based on a vague shared interest to re-interpret existing AI technologies. Our interests included symbolic Reinforcement Learning (RL), emergent communication between RL agents, translating open philosophy problems to RL experiments, factored cognition, and combining Inductive Logic Programming (ILP) with deep learning.

We ranked all these project ideas based on “expected quality of the output if the project is successful”, “tractability”, “novelty”, “how much we’d learn”, “non-tediousness”, “how well it ties in with existing work” and “safety impact”. Combining ILP with deep learning came out on top.

In the time between the pre-retreat and the Louti retreat we read up on ILP (as two out of the four team members had never heard of the method) and generated ideas about how to buff it up with deep learning. Before the start of the second retreat we had collected about 7 rough ideas and lost our only member with prior ILP experience to deadly Oxford deadlines.

We considered extending  the existing literature that combines higher order logic with ILP to obtain better performance, either by improving their open-source implementation or by proving theoretical results. The latter would entail making a stronger argument for their design choices, by building intuition for why they led to empirical improvements, formulating hypotheses, and proving theorems. Although we were reasonably confident that we could produce some results, this project seemed like it wouldn’t have any impact whatsoever on the direction that AI or ILP research is taking.

A project related to factored cognition was writing an algorithm that takes in a question, uses a neural network or deep RL to produce subquestions, answers the subquestions using ILP, and then combines the subquestions to produce an answer. We could have either approached this with a focus on “how to learn how to factorize problems” or “how can we improve ILP performance by decomposing tasks”. Ought has worked on factored cognition for a while, so it seems unlikely that we would have very interesting insights on the first perspective that they have not yet realized. We considered reproducing OpenAI’s iterated amplification results on experiments in five algorithmic domains. We decided against this, because we thought it would be a lot of work in proportion to what we would end up contributing. On the whole, we decided not to work on factored cognition because we expected the project would end up being brittle in the face of bottlenecks: if one part of the pipeline is not working then the entire algorithm is useless.

We had some very underspecified ideas on combining ILP with ML. Among which, “predict the distance between a theory and a completed theory (in number of predicates)” and “use ILP or some other proof checking algorithm to check a hypothesis and use a generative algorithm to generate hypotheses”. In the end we went with the idea that seemed most straightforward, which is developing a toy version an algorithm that predicts which pieces of the background in an ILP problem are relevant. This was a natural choice since it has a small base case (bag of words input to ordinary MLP), many possible extensions, and since the exact “background forgetting” algorithm is NP-hard. ([Cropper 2019](https://arxiv.org/abs/1911.06643))

ILP implementation project
--------------------------

ILP takes as input background information (rule-like facts about the world), \*B\*, a set of positive examples, \*E+\*, and a set of negative examples, \*E-\*, and outputs a hypothesis, \*H\*, that is consistent with all the positive examples and inconsistent with the negative examples while using the background to minimise hypothesis size. ILP takes longer as the background is bigger, so reducing the background size reduces the computation time. Our goal is pruning the background set without losing relevant predicates that would change the output hypothesis.

Approaches include:

\* Within some domain, such as computational chemistry, there is a broad and fixed background. For each combination of positive and negative examples (problem statement) only a subset of the background is needed to find the correct hypothesis. If we can predict which subset of the background is needed for which problem, then we can reduce the time it takes to solve those problems. Such an algorithm would probably  be based on recognizing similarities between background predicates and examples. For each element in the background it could predict whether that predicate is relevant for the problem at hand.
\* Alternatively, we could develop a general purpose “background size reduction” algorithm that works on any background. Such a distillation process would have to be based on intrinsic similarities between elements of the background. The algorithm could for example recognize which elements are duplicates. However, if we want to deal with the relationships between predicates in the background, then we can not use a predictor that only predicts relevance of individual elements. For example, if we have five duplicates of one predicate, then we may predict that each individual predicate should be thrown away, but we would like to throw away exactly four. Hence, in this setting we probably want to make predictions about the redundancy of subsets of the background (rather than the relevance of individual predicates).

![](https://lh3.googleusercontent.com/gaWUDixdpBAc3XU\_ua0\_vlz3-Vr8Gi55IKWDq8bm5Fo9-l4SUcOHVFXSR\_KM3kn3pN8hYy0ve61NXRfsXIcP5jl9oz\_EFMuKXrsZylNBOe3r937T-Sp3iRNsF9yNUJwYFdnUPoiY)
Relevance predictor

\* Input to the relevance predictor. Train on:
 + One specific background within a specific domain.
 + Backgrounds of a specific size.
 + Backgrounds of variable sizes smaller than n and just have some empty input vector placeholders (in random slots).
 + Backgrounds of variable sizes.
\* Output of relevance predictor. Given a background B:

1. Return a relevance prediction for each element (predicting if after removing this element the resulting ILP hypothesis will or will not stay the same).
2. For a background B return a subset B’ (which is a smallest set for which the ILP hypothesis H\\_B and H\\_B’ are the same).

To go from predictions about individual predicates to predictions about subsets of the background, we could iteratively apply that system.

Very ambitious ideas (that we did not focus on) are:

\* Instead of just taking out semantic duplicates, we could build a system that generates summarizing predicates.
\* We could also be more ambitious and hope to clean up noise by for example eliminating contradictory predicates. This is more difficult than reducing redundancy, because in this case we are trying to change the background such that the output hypothesis could change (or could be found in the first place), but there may be many changes that look valid at face value. Presumably, statistics could predict which predicates are more likely to be noise if a predicate contradicts many other predicates, while the background set would be consistent without the bad predicate. However, when there are two predicates that are inconsistent, but that are both consistent with the rest of the background set, then it seems almost necessary to have domain knowledge to distinguish noise from true predicates.

The prototype developed over the course of the second retreat consists of the following modules:

\* A \*\*formal grammar generator\*\*. For a fixed set of symbols, which constitute the background, we generate sets of positive (resp. negative) examples by using (breaking) the rules of the grammar on a subset of symbols. For each set of examples, the background knowledge that we can dispose of is the set of predicates corresponding to the symbols not used in the grammar. We intend to use this very simple task as a first proof-of-concept, but the rest of the pipeline is agnostic about the ILP problem to solve, and we plan to implement other tasks in the future.
\* A Prolog \*\*parser\*\* that reads the predicates in the background and examples files and returns their parse trees. This way we obtain a graph representation of the input that encodes the relations between terms of the theory.
\* A \*\*predicate embedder\*\* module that maps parse trees to vectors. This is implemented as a graph net. Graph nets are models that receive an annotated graph as input and return the same graph as output with updated annotations. The annotations are simply labels on the graph nodes and/or edges and they can have any type. Update functions are usually implemented by neural networks and they respect certain locality rules e.g. a node label is updated using only the current values of the labels on neighbouring nodes and/or incoming edges. In our case, the labels are vectors attached to the nodes. They are initialized to one-hot vectors and updated following the rules of [FormulaNet](https://github.com/princeton-vl/FormulaNet).
\* A \*\*relevance prediction\*\* module that receives the vector representation of the triple \*(B,E+,E-)\* and outputs the probability that each predicate in the background is relevant for the ILP task, given the examples. This module is implemented as an MLP with dropout. The embedder-predictor model is trained end-to-end using cross-entropy loss.

Implementation choices:

\* We could just have inputted a string into our network, but we thought that would be very hard to learn from. We tried two approaches to translating predicates into vectors that can be used as input for a predictor network.
 + One approach is to count the number of occurrences of words/symbols in the predicate (bag of words).
 + The other is to first translate the predicate to an abstract syntax tree, then use that tree as an input to a graph network, which learns to output a useful vector, which is input to an MLP. This pipeline was trained end-to-end with a training signal from (at first) known ground truth, and later a full ILP system as oracle.
\* Initially we planned to produce ground truth to train our predictor using an ILP module, but we realized we could just adjust our example generator such that it automatically generates the solutions (which was possible because of the simple grammar domain that we used). For a predictor that can be applied to a wider range of problems, such an ILP module is still needed.

We have two outputs planned: a case for and against ILP for safety, and the writeup and repo from our relevance predictor. We are seeking feedback from the ILP community, and have sufficient ideas for a number of future projects of similar size.