The strategy-stealing assumption

Suppose that 1% of the world’s resources are controlled by unaligned AI, and 99% of the world’s resources are controlled by humans. We might hope that at least 99% of the universe’s resources end up being used for stuff-humans-like (in expectation).

Jessica Taylor argued for this conclusion in [Strategies for Coalitions in Unit-Sum Games](https://www.lesswrong.com/posts/5bd75cc58225bf0670375325/strategies-for-coalitions-in-unit-sum-games): if the humans divide into 99 groups each of which acquires influence as effectively as the unaligned AI, then by symmetry each group should end up with as much influence as the AI, i.e. in aggregate they should end up with 99% of the influence.

This argument rests on what I’ll call the \*strategy-stealing assumption\*: for any strategy an unaligned AI could use to influence the long-run future, there is an analogous strategy that a similarly-sized group of humans can use in order to capture a similar amount of flexible influence over the future. By “flexible” I mean that humans can decide later what to do with that influence — which is important since humans don’t yet know what we want in the long run.

Why might the strategy-stealing assumption be true?
===================================================

Today there are a bunch of humans, with different preferences and different kinds of influence. Crudely speaking, the long-term outcome seems to be determined by some combination of {which preferences have how much influence?} and {what is the space of realizable outcomes?}.

I expect this to become more true over time — I expect groups of agents with diverse preferences to eventually approach efficient outcomes, since otherwise there are changes that everyagent would prefer (though this is not obvious, especially in light of bargaining failures). Then the question is just about \*which\* of these efficient outcomes we pick.

I think that our actions don’t effect the space of realizable outcomes, because long-term realizability is mostly determined by facts about distant stars that we can’t yet influence. The obvious exception is that if we colonize space faster, we will have access more resources. But [quantitatively this doesn’t seem like a big consideration](https://rationalaltruist.com/2013/04/30/astronomical-waste/), because astronomical events occur over millions of millennia while our decisions only change colonization timelines by decades.

So I think our decisions mostly affect long-term outcomes by changing the relative weights of different possible preferences (or by causing extinction).

Today, one of the main ways that preferences have weight is because agents with those preferences control resources and other forms of influence. Strategy-stealing seems most possible for this kind of plan — an aligned AI can exactly copy the strategy of an unaligned AI, except the money goes into the aligned AI’s bank account instead. The same seems true for most kinds of resource gathering.

There are lots of strategies that give influence to other people instead of helping me. For example, I might preferentially collaborate with people who share my values. But I can still steal these strategies, as long as my values are just as common as the values of the person I’m trying to steal from. So a majority can steal strategies from a minority, but not the other way around.

There can be plenty of strategies that don’t involve acquiring resources or flexible influence. For example, we could have a parliament with obscure rules in which I can make maneuvers that advantage one set of values or another in a way that can’t be stolen. Strategy-stealing may only be possible at the level of groups — you need to retain the option of setting up a different parliamentary system that doesn’t favor particular values. Even then, it’s unclear whether strategy-stealing is possible.

There isn’t a clean argument for strategy-stealing, but I think it seems plausible enough that it’s meaningful and productive to think of it as a plausible default, and to look at ways it can fail. (If you found enough ways it could fail, you might eventually stop thinking of it as a default.)

Eleven ways the strategy-stealing assumption could fail
=======================================================

In this section I’ll describe some of the failures that seem most important to me, with a focus on the ones that would interfere with the argument in the introduction.

1. AI alignment
---------------

If we can build smart AIs, but not aligned AIs, then humans can’t necessarily use AI to capture flexible influence. I think this is theπ most important way in which strategy-stealing is likely to fail. I’m not going to spend much time talking about it here because I’ve spent so much time elsewhere.

For example, if smart AIs inevitably want to fill the universe with paperclips, then “build a really smart AI” is a good strategy for someone who wants to fill the universe with paperclips, but it can’t be easily stolen by someone who wants anything else.

2. Value drift over generations
-------------------------------

The values of 21st century humans are determined by some complicated mix of human nature and the modern environment. If I’m a 16th century noble who has really specific preferences about the future, it’s not really clear how I can act on those values. But if I’m a 16th century noble who thinks that future generations will inevitably be wiser and should get what they want, then I’m in luck, all I need to do is wait and make sure our civilization doesn’t do anything rash. And if I have some kind of crude intermediate preferences, then I might be able to push our culture in appropriate directions or encourage people with similar genetic dispositions to have more kids.

This is the most obvious and important way that strategy-stealing has failed historically. It’s not something I personally worry about too much though.

The big reason I don’t worry is some combination of common-sense morality and decision-theory: our values are the product of many generations each giving way to the next one, and so I’m pretty inclined to “pay it forward.” Put a different way, I think it’s relatively clear I should empathize with the next generation since I might well have been in their place (whereas [I find it much less clear under what conditions I should empathize with AI](/sympathizing-with-ai-e11a4bf5ef6e)). Or from yet another perspective, the same intuition that I’m “more right” than previous generations makes me very open to the possibility that future generations are more right still. This question gets very complex, but my first-pass take is that I’m maybe an order of magnitude less worried than about other kinds of value drift.

The small reason I don’t worry is that I think this dynamic is probably going to be less important in the future (unless we actively want it to be important — which seems quite possible). I believe there is a good chance that within 60 years most decisions will be made by machines, and so the handover from one generation to the next will be optional.

That all said, I am somewhat worried about more “out of distribution” changes to the values of future generations, in scenarios where AI development is slower than I expect. For example, I think it’s possible that genetic engineering of humans will substantially change what we want, and that I should be less excited about that kind of drift. Or I can imagine the interaction between technology and culture causing similarly alien changes. These questions are even harder to think about than the basic question of “how much should I empathize with future generations?” which already seemed quite thorny, and I don’t really know what I’d conclude if I spent a long time thinking. But at any rate, these things are not at the top of my priority queue.

3. Other alignment problems
---------------------------

AIs and future generations aren’t the only optimizers around. For example, we can also build institutions that further their own agendas. We can then face a problem analogous to AI alignment — if it’s easier to build effective institutions with some kinds of values than others, then those values could be at a structural advantage. For example, we might inevitably end up with a society that optimizes generalizations of short-term metrics, if big groups of humans are much more effective when doing this. (I say “generalizations of short-term metrics” because an exclusive focus on short-term metrics is the kind of problem that can fix itself over the very long run.)

I think that institutions are currently considerably weaker than humans (in the sense that’s relevant to strategy-stealing) and this will probably remain true over the medium term. For example:

\* A company with 10,000 people might be much smarter than any individual humans, but mostly that’s because of its alliance with its employees and shareholders — most of its influence is just used to accumulate more wages and dividends. Companies do things that seem antisocial not because they have come unmoored from any human’s values, but because plenty of influential humans want them to do that in order to make more money. (You could try to point the “market” as an organization with its own preferences, but it’s even worse at defending itself than bureaucracies — it’s up to humans who benefit from the market to defend it.)
\* Bureaucracies can seem unmoored from any individual human desire. But their actual ability to defend themselves and acquire resources seems much weaker than other optimizers like humans or corporations.

Overall I’m less concerned about this than AI alignment, but I do think it is a real problem. I’m somewhat optimistic that the same general principles will be relevant both to aligning institutions and AIs. If AI alignment wasn’t an issue, I’d be more concerned by problems like institutional alignment.

4. Human fragility
------------------

If AI systems are aligned with humans, they may want to keep humans alive. Not only do humans prefer being alive, humans may need to survive if they want to have the time and space to figure out what they really want and to tell their AI what to do. (I say “may” because at some point you might imagine e.g. putting some humans in cold storage, to be revived later.)

This could introduce an asymmetry: an AI that just cares about paperclips can get a leg up on humans by threatening to release an engineered plague, or trashing natural ecosystems that humans rely on. (Of course, this asymmetry may also go the other way — values implemented in machines are reliant on a bunch of complex infrastructure which may be more or less of a liability than humanity’s reliance on ecosystems.)

Stepping back, I think the fundamental long-term problem here is that “do what this human wants” is only a simple description of human values if you actually have the human in hand, and so an agent with these values does have a big extra liability.

I do think that the extreme option of “storing” humans to revive them later is workable, though most people would be very unhappy with a world where that becomes necessary. (To be clear, I think it almost certainly won’t.) We’ll return to this under “short-term terminal preferences” below.

5. Persuasion as fragility
--------------------------

If an aligned AI defines its values with reference to “whatever Paul wants,” then someone doesn’t need to kill Paul to mess with the AI, they just need to change what Paul wants. If it’s very easy to manipulate humans, but we want to keep talking with each other and interacting with the world despite the risk, then this extra attack surface could become a huge liability.

This is easier to defend against — just stop talking with people except in extremely controlled environments where you can minimize the risk of manipulation — but again humans may not be willing to pay that cost.

The main reason this might be worse than point 4 is that humans may be relatively happy to physically isolate themselves from anything scary, but it would be much more costly for us to cut off from contact with other humans.

6. Asymmetric persuasion
------------------------

Even if humans are the only optimizers around, it might be easier to persuade humans of some things than others. For example, you could imagine a world where it’s easier to convince humans to endorse a simple ideology like “maximize the complexity of the universe” than to convince humans to pursue some more complex and subtle values.

This means that people with easily-persuadable values can use persuasion as a strategy, and people with other values cannot copy it.

I think this is ultimately more important than fragility, because it is relevant before we have powerful AI systems. It has many similarities to “value drift over generations,” and I have some mixed feelings here as well — there are some kinds of argument and deliberation that I certainly do endorse, and to the extent that my current views are the product of significant amounts of non-endorsed deliberation I am more inclined to be empathetic to future people who are influenced by increasingly-sophisticated arguments.

But as I described in section 2, I think these connections can get weaker as technological progress moves us further out of distribution, and if you told me that e.g. it was possible to perform a brute force search and find an argument that could convince someone to maximize the complexity of the future, I wouldn’t conclude that it’s probably fine if they decided to do that.

(Credit to Wei Dai for emphasizing this failure mode.)

7. Value-sensitive bargaining
-----------------------------

If a bunch of powerful agents collectively decide what to do with the universe, I think it probably won’t look like “they all control their own slice of the universe and make independent decisions about what to do.” There will likely be opportunities for trade, they may have meddling preferences (where I care what you do with your part of the universe), there may be a possibility of destructive conflict, or it may look completely different in an unanticipated way.

In many of these settings the outcome is influenced by a complicated bargaining game, and it’s unclear whether the majority can steal a minority’s strategy. For example, suppose that there are two values X and Y in the world, with 99% X-agents and 1% Y-agents. The Y-agents may be able to threaten to destroy the world unless there is an even split, and the X-agents have no way to copy such a strategy. (This could also occur over the short term.)

I don’t have a strong view about the severity of this problem. I could imagine it being a big deal.

8. Recklessness
---------------

Some preferences might not care about whether the world is destroyed, and therefore have access to productive but risky strategies that more cautious agents cannot copy. The same could happen with other kinds of risks, like commitments that are game-theoretically useful but risk sacrificing some part of the universe or creating long-term negative outcomes.

I tend to think about this problem in the context of particular technologies that pose an extinction risk, but it’s worth keeping in mind that it can be compounded by the existence of more reckless agents.

Overall I think this isn’t a big deal, because it seems much easier to cause extinction by trying to kill everyone than as an accident. There are fewer people who are in fact trying to kill everyone, but I think not enough fewer to tip the balance. (This is a contingent fact about technology though; it could change in the future and I could easily be wrong even today.)

9. Short-term unity and coordination
------------------------------------

Some actors may have long-term values that are easier to talk about, represent formally, or reason about. Relative to humans, AIs may be especially likely to have such values. These actors could have an easier time coordinating, e.g. by pursuing some explicit compromise between their values (rather than being forced to find a governance mechanism for some resources produced by a joint venture).

This could leave us in a place where e.g. an unaligned AI controls 1% resources, but the majority of resources are controlled by humans who want to acquire flexible resources. Then the unaligned AIs can form a coalition which achieves very high efficiencies, while the humans cannot form 99 other coalitions to compete.

This could theoretically be a problem without AI, e.g. a large group of human with shared explicit values might be able to coordinate better and so leave normal humans at a disadvantage, though I think this is relatively unlikely as a major force in the world.

The seriousness of this problem is bounded by both the efficiency gains for a large coalition, and the quality of governance mechanisms for different actors who want to acquire flexible resources. I think we have OK solutions for coordination between people who want flexible influence, such that I don’t think this will be a big problem:

\* The humans can participate in lotteries to concentrate influence. Or you can gather resources to be used for a lottery in the future, while still allowing time for people to become wiser and then make bargains about what to do with the universe before they know who wins.
\* You can divide up the resources produced by a coalition equitably (and then negotiate about what to do with them).
\* You can modify other mechanisms by allowing votes that could e.g. overrule certain uses of resources. You could have more complex governance mechanisms, can delegate different kinds of authority to different systems, can rely on trusted parties, etc.
\* Many of these procedures work much better amongst groups of humans who expect to have relatively similar preferences or have a reasonable level of trust for other participants to do something basically cooperative and friendly (rather than e.g. demanding concessions so that they don’t do something terrible with their share of the universe or if they win the eventual lottery).

(Credit to Wei Dai for describing and emphasizing this failure mode.)

10. Weird stuff with simulations
--------------------------------

I think civilizations like ours mostly have an impact via the common-sense channel where we ultimately colonize space. But there may be many civilizations like ours in simulations of various kinds, and influencing the results of those simulations could also be an important part of what we do. In that case, I don’t have any particular reason to think strategy-stealing breaks dow but I think stuff could be very weird and I have only a weak sense of how this influences optimal strategies.

Overall I don’t think much about this since it doesn’t seem likely to be a large part of our influence and it doesn’t break strategy-stealing in an obvious way. But I think it’s worth having in mind.

11. Other preferences
---------------------

People care about lots of stuff other than their influence over the long-term future. If 1% of the world is unaligned AI and 99% of the world is humans, but the AI spends all of its resources on influencing the future while the humans only spend one tenth, it wouldn’t be too surprising if the AI ended up with 10% of the influence rather than 1%. This can matter in lots of ways other than literal spending and saving: someone who only cared about the future might make different tradeoffs, might be willing to defend themselves at the cost of short-term value (see sections 4 and 5 above), might pursue more ruthless strategies for expansion, and so on\*.\*

I think the simplest approximation is to restrict attention to the part of our preferences that is about the long-term (I discussed this a bit in [Why might the future be good?](https://rationalaltruist.com/2013/02/27/why-will-they-be-happy/)). To the extent that someone cares about the long-term less than the average actor, they will represent a smaller fraction of this “long-term preferences” mixture. This may give unaligned AI systems a one-time advantage for influencing the long-term future (if they care more about it) but doesn’t change the basic dynamics of strategy-stealing. Even this advantage might be clawed back by a majority (e.g. by taxing savers).

There are a few places where this picture seems a little bit less crisp:

\* Rather than being able to spend resources on either the short or long-term, sometimes you might have preferences about \*how\* you acquire resources in the short-term; an agent without such scruples could potentially pull ahead. If these preferences are strong, it probably violates strategy-stealing unless the majority can agree to crush anyone unscrupulous.
\* For humans in particular, it may be hard to separate out “humans as repository of values” from “humans as an object of preferences,” and this may make it harder for us to defend ourselves (as discussed in sections 4 and 5).

I mostly think these complexities won’t be a big deal quantitatively, because I think our short-term preferences will mostly be compatible with defense and resource acquisition. But I’m not confident about that.

Conclusion
==========

I think strategy-stealing isn’t really true; but I think it’s a good enough approximation that we can basically act as if it’s true, and then think about the risk posed by possible failures of strategy-stealing.

I think this is especially important for thinking about AI alignment, because it lets us formalize the lowered goalposts I discussed [here](/a-possible-stance-for-ai-control-research-fe9cf717fc1b): we just want to ensure that AI is compatible with strategy-stealing. These lowered goalposts are an important part of why I think we can solve alignment.

In practice I think that a large coalition of humans isn’t reduced to strategy-stealing — a majority can simply stop a minority from doing something bad, rather than by copying it. The possible failures in this post could potentially be addressed by either a technical solution or some kind of coordination.


Towards formalizing universality

The scalability of [iterated amplification](https://arxiv.org/pdf/1810.08575.pdf) or [debate](https://arxiv.org/abs/1805.00899) seems to depend on whether large enough teams of humans can carry out arbitrarily complicated reasoning. Are these schemes “universal,” or are there kinds of reasoning that work but which humans fundamentally can’t understand?

This post defines the concept of “ascription universality,” which tries to capture the property that a question-answering system \*\*A\*\* is better-informed than any particular simpler computation \*\*C\*\*.

[These](/informed-oversight-18fcb5d3d1e1) [parallel](/universality-and-consequentialism-within-hch-c0bee00365bd) [posts](/universality-and-model-based-rl-b08701394ddd) explain why I believe that the alignment of iterated amplification largely depends on whether HCH is ascription universal. Ultimately I think that the “right” definition will be closely tied to the use we want to make of it, and so we should be refining this definition in parallel with exploring its applications.

I’m using the awkward term “ascription universality” partly to explicitly flag that this is a preliminary definition, and partly to reserve linguistic space for the better definitions that I’m optimistic will follow.

(Thanks to Geoffrey Irving for discussions about many of the ideas in this post.)

I. Definition
=============

We will try to define what it means for a question-answering system \*\*A\*\* to be “ascription universal.”

1. Ascribing beliefs to A
-------------------------

Fix a language (e.g. English with arbitrarily big compound [terms](/approval-directed-algorithm-learning-bf1f8fad42cd)) in which we can represent questions and answers.

To ascribe beliefs to \*\*A\*\*, we ask it. If \*\*A\*\*(“are there infinitely many twin primes?”) = “probably, though it’s hard to be sure” then we ascribe that belief about twin primes to \*\*A\*\*.

This is not a general way of ascribing “belief.” This procedure wouldn’t capture the beliefs of a native Spanish speaker, or for someone who wasn’t answering questions honestly. But it can give us a sufficient condition, and is particularly useful for someone who wants to use \*\*A\*\* as part of an alignment scheme.

Even in this “straightforward” procedure there is a lot of subtlety. In some cases there are questions that we can’t articulate in our language, but which (when combined with \*\*A\*\*’s other beliefs) have consequences that we can articulate. In this case, we can infer something about \*\*A\*\*’s beliefs from its answers to the questions that we can articulate.

2. Ascribing beliefs to arbitrary computations
----------------------------------------------

We are interested in whether \*\*A\*\* “can understand everything that could be understood by someone.” To clarify this, we need to be more precise about what we mean by “could be understood by someone.”

This will be the most informal step in this post. (Not that any of it is very formal!)

We can imagine various ways of ascribingbeliefs to an arbitrary computation \*\*C\*\*. For example:

\* We can give \*\*C\*\* questions in a particular encoding and assume its answers reflect its beliefs. We can either use those answers directly to infer \*\*C\*\*’s beliefs (as in the last section), or we can ask what set of beliefs about latent facts would explain \*\*C\*\*’s answers.
\* We can view \*\*C\*\* as optimizingsomething andask what set of beliefs rationalize that optimization. For example, we can give \*\*C\*\* a chess board as input, see what move it produces, assume it is trying to win, and infer what it must believe. We might conclude that \*\*C\*\* believes a particular line of play will be won by black, or that \*\*C\*\* believes general heuristics like “a pawn is worth 3 tempi,” or so on.
\* We can reason about how \*\*C\*\*’s behavior depends on facts about the world, and ask what state of the world is determined by its current behavior. For example, we can observe that \*\*C\*\*(113327) = 1 but that \*\*C\*\*(113327) “would have been” 0 if 113327 had been composite, concluding that \*\*C\*\*(11327) “knows” that 113327 is prime. We can extend to probabilistic beliefs, e.g. if \*\*C\*\*(113327) “probably” would have been 0 if 113327 had been composite, then we might that \*\*C\*\* knows that 113327 is “probably prime.” This certainly isn’t a precise definition, since it involves considering logical counterfactuals, and I’m not clear whether it can be made precise. (See also ideas along the lines of [“knowledge is freedom”](https://www.lesswrong.com/posts/b3Bt9Cz4hEtR26ANX/knowledge-is-freedom).)
\* If a computation behaves differently under different conditions, then we could use restrict attention to a particular condition. For example, if a question-answering system appears to be bilingual but answers questions differently in Spanish and English, we could ascribe two different sets of beliefs. Similarly, we could ascribe beliefs to any subcomputation. For example, if a part of \*\*C\*\* can be understood as optimizing the way data is laid out in memory, then we can ascribe beliefs to that computation about the way that data will be used.

Note that these aren’t intended to be efficient procedures that we could actually apply to a given computation \*\*C\*\*. They are hypothetical procedures that we will use to define what it means for \*\*A\*\* to be universal.

I’m not going to try to ascribe a single set of beliefs to a given computation; instead, I’ll consider all of the reasonable ascription procedures. For example, I think different procedures would ascribe different beliefs to a particular human, and don’t want to claim there is a unique answer to what a human “really” believes. A universal reasoner needs to have more reasonable beliefs than the beliefs ascribed to that a human using any particular method.

An ascription-universal reasoner needs to compete with any beliefs that can be ascribed to \*\*C\*\*, so I want to be generous with this definition. For example, given a chess-playing algorithm, we might rationalize it as trying to win a game and infer its beliefs about the rules of chess. Or we might rationalize it as trying to look like a human and infer its beliefs about what a human would do. Or something different altogether. Most of these will be kind of crazy ascriptions, but I want to compete with them anyway (competing with crazier beliefs will turn out to just be easier).

It’s not totally clear what counts as a “reasonable” ascription procedure, and that’s the biggest source of informality. Intuitively, the key property is that the ascription itself isn’t doing the “hard work.” In practice I’m using an informal extensional definition, guided by examples like those in the bulleted list.

3. Comparing beliefs
--------------------

What does it mean to say that one agent is “better-informed” than another?

It’s natural to try to express this in terms of empirical information about the world, but we are particularly interested in the different inferencesthat agents are able to draw from the same data. Another natural approach is to compare their “knowledge,” but I have no idea how to define knowledge or justified belief. So I’m reduced to working directly with sets of beliefs.

Consider two sets of beliefs, described by the subjective expectations 𝔼¹ and 𝔼². What does it mean to say that 𝔼¹ is better-informed than 𝔼²?

This framing makes it tempting to try something simple: “for every quantity, 𝔼¹’s belief about that quantity is more accurate.” But this is property is totally unachievable. Even if 𝔼¹ is obtained by conditioning 𝔼² on a true fact, it will almost certainly happen to update in the “wrong” direction for some claims.

We will instead use a subjective definition, i.e. we’ll define this concept from a particular epistemic position represented by another subjective expectation 𝔼.

Then we say that 𝔼¹ \*\*dominates\*\* 𝔼² (w.r.t. 𝔼) if, for every bounded quantity X and for every “nice” property Φ:

\* 𝔼[X|Φ(𝔼¹, 𝔼²)] = 𝔼[𝔼¹[X]|Φ(𝔼¹, 𝔼²)]

(By “nice” I mean something like: simple to define and open in the product topology, viewing 𝔼¹ and 𝔼² as infinite tables of numbers.)

Intuitively, this means that 𝔼 always “trusts” 𝔼¹, even if given arbitrary information about 𝔼¹ and 𝔼². For example, if 𝔼 was told that 𝔼¹[X] ≈ \*x\* and
𝔼²[X] ≈ \*y\*, then it would expect X to be around \*x\* (rather than \*y\*)\*.\* Allowing arbitrary predicates Φ allows us to make stronger inferences, effectively that 𝔼 thinks that 𝔼¹ captures \*everything\* useful about 𝔼².

I’m not sure if this is exactly the right property, and it becomes particularly tricky if the quantity X is itself related to the behavior of 𝔼¹ or 𝔼² (continuity in the product topology is the minimum plausible condition to avoid a self-referential paradox). But I think it’s at least roughly what we want and it may be exactly what we want.

Note that dominance is \*subjective\*, i.e. it depends on the epistemic vantage point 𝔼 used for the outer expectation. This property is a little bit stronger than what we originally asked for, since it also requires 𝔼 to trust 𝔼¹, but this turns out to be implied anyway by our definition of universality so it’s not a big defect.

Note that dominance is a property of the \*descriptions\* of 𝔼¹ and 𝔼². There could be two different computations that in fact compute the same set of expectations, such that 𝔼 trusts one of them but not the other. Perhaps one computation hard-codes a particular result, while the other does a bunch of work to estimate it. Even if the hard-coded result happened to be correct, such that the two computations had the same outputs, 𝔼 might trust the hard work but not the wild guess.

4. Complexity and parameterization
----------------------------------

There are computations with arbitrarily sophisticated beliefs, so no fixed \*\*A\*\* can hope to dominate everything. To remedy this, rather than comparing to a fixed question-answerer \*\*A\*\*, we’ll compare to a parameterized family \*\*A\*\*[\*\*C\*\*].

I’ll consider two different kinds of potentially-universal reasoners \*\*A\*\*:

\* In the “idealized” case, \*\*A\*\*[\*\*C\*\*] depends only on the complexity of \*\*C\*\*.
For example, we might hope that an \*n\*-round debate dominates any beliefs that could be ascribed to a fast computation with (\*n\*-1) rounds of [alternation](https://en.wikipedia.org/wiki/Alternating\_Turing\_machine). In particular, this \*\*A\*\*[\*\*C\*\*] is the same for any two computations \*\*C\*\* of the same complexity.
\* In the “practical” case, \*\*A\*\*[\*\*C\*\*] depends on the complexity of \*\*C\*\* but also uses the computation \*\*C\*\* as a hint. For example, if \*\*C\*\* is the training process for a neural net, then we might take \*\*A\*\*[\*\*C\*\*] to be a debate in which the debaters are able to share weights and activations with the neural net throughout the entire training process.

I’m generally interested in the case where \*\*A\*\*[\*\*C\*\*] is only slightly more powerful than \*\*C\*\* itself. This mirrors the setting where a universal Turing machine is able to run any other Turing machine with only a modest slowdown.

Putting it all together
-----------------------

We say that a set of beliefs 𝔼ᴬ \*epistemically dominates\* a computation \*\*C\*\* (w.r.t. some beliefs 𝔼 and language L) if the beliefs ascribed to A by the “straightforward” procedure, using L, dominate (w.r.t. 𝔼) the beliefs ascribed to \*\*C\*\* by any reasonable ascription procedure.

We say that a family of question-answering systems \*\*A\*\*[\*\*·\*\*] are \*ascription universal\* (w.r.t. 𝔼 and L) if \*\*A\*\*[\*\*C\*\*] epistemically dominates \*\*C\*\* for every computation \*\*C\*\*.

II. Discussion
==============

Why is (subjective) dominance sufficient?
-----------------------------------------

This universality condition requires that \*we\* believe that \*\*A\*\*[\*\*C\*\*] is better-informed than \*\*C\*\*. Naively we might have wanted it to \*actually\* be the case that \*\*A\*\*[\*\*C\*\*] is better-informed than \*\*C\*\*; the stronger condition is clearly unachievable, but why should we be satisfied with the weaker condition?

In [applications](/informed-oversight-18fcb5d3d1e1) [of this](/universality-and-consequentialism-within-hch-c0bee00365bd) [property](/universality-and-model-based-rl-b08701394ddd), the subjective condition is what we need in order for \*us\* to believe that \*\*A\*\*[\*\*C\*\*] will cope with the challenges posed by \*\*C\*\*. For example, suppose that \*\*C\*\* formulates a plan to “trick” \*\*A\*\*[\*\*C\*\*]. Then the subjective universality condition implies that we don’t expect \*\*C\*\* to succeed.

This isn’t as good as actually knowing that \*\*C\*\* won’t succeed. But I think it should be good enough for us — the reason we are thinking about AI safety is because we are concerned that something bad will happen. If we find a technique that defuses this argument, then we’ve addressed the motivating problem. It may still be the case that bad things happen (and we should still search for additional reasons that bad things might happen), but we don’t particularly expect them to.

Of course if you select over a large number of computations, then you may find one that will succeed in tricking \*\*A\*\*. But if we are concerned about that, then we can instead apply ascription universality to the entire process including the selection.

Why trust opaque computation?
-----------------------------

If \*\*C\*\* uses some clever heuristics that I don’t understand, then \*\*C\*\*’s “beliefs” might be excellent, but I might not \*expect\* them to be excellent. In this sense understanding may seem almost vacuous. If there is some heuristic that I trust, wouldn’t \*\*A\*\* just use it?

To see why the definition is demanding, consider the special case where \*\*C\*\* performs an extensive search to find a computation that works well empirically. For example, \*\*C\*\* might be the following computation:

\* Start with a training set of (image, label) pairs.
\* Search over simple programs to find one that makes good predictions.
\* Run that simple program on a new image to predict its label.

In this case, we can ascribe beliefs to \*\*C\*\* about the contents of the new image. And because those beliefs are coming from a simple program that works empirically, I expect them to be accurate (in some respects).

For example, a simple classifier \*\*C\*\* may “believe” that the new image contains a particular curve that typically appears in images labeled “dog;” or a really sophisticated classifier may perform complex deductions about the contents of the scene, starting from premises that were empirically validated on the training set.

So it’s not OK for \*\*A\*\* to simply ignore whatever heuristics \*\*C\*\* is using — if those heuristics have the kind of empirical support that makes us think they actually work, then A needs to be able to understand everything that those heuristics imply about the domain.

Why be so general?
------------------

I’ve formulated universality as competing with arbitrary computations \*\*C\*\*. It seems totally possible that the form of \*\*C\*\* discussed in the last section — searching for a program that works well in practice and then using it in a new situation — is so central that the definition of universality should focus entirely on it.

One reason to use the broader definition is because sometimes this “selection” process can be embedded in a non-trivial way in a larger computation. For example, if I have a sufficiently large group of humans, I might expect memetic selection to occur and produce systems that could be said to have “beliefs,” and I’d like universal systems to dominate those beliefs as well.

The other reason to use this very general definition is because I don’t see an easy way to simplify the definition by using the additional structural assumption about \*\*C\*\*. I do think it’s likely there’s a nicer statement out there that someone else can find.

Universal from whose perspective?
---------------------------------

Unfortunately, achieving universality depends a lot on the epistemic perspective 𝔼 from which it is being evaluated. For example, if 𝔼 knows any facts, than a universal agent must know all of those facts as well. Thus “a debate judged by Paul” may be universal from Paul’s perspective, but “a debate arbitrated by Alice” cannot be universal from my perspective unless I believe that Alice knows everything I know.

This isn’t necessarily a big problem. It will limit us to conclusions like: Google engineers believe that the AI they’ve built serves the user’s interests reasonably well. The user might not agree with that assessment, if they have different beliefs from Google engineers. This is what you’d expect in any case where Google engineers build a product, however good their intentions.

(Of course Google engineers’ notion of “serving the user’s interests” can involve deferring to the user’s beliefs in cases where they disagree with Google engineers, just as they could defer to the user’s beliefs with other products. That gives us reason to be less concerned about such divergences, but eventually these evaluations do need to bottom out somewhere.)

This property becomes more problematic when we ask questions like: is there a way to [seriously limit the inputs and outputs to a human while preserving universality of HCH](/universality-and-security-amplification-551b314a3bab)? This causes trouble because even if limiting the human intuitively preserves universality, it will effectively eliminate some of the human’s knowledge and know-how that can [only be accessed on large inputs](https://medium.com/@weidai/to-put-it-another-way-a-human-translator-has-learned-a-lot-of-valuable-information-much-of-it-48457f95b9bf), and hence violate universality.

So when investigating schemes based on this kind of impoverished human, we would need to evaluate universality from some impoverished epistemic perspective. We’d like to say that the impoverished perspective is still “good enough” for us to feel safe, despite not being good enough to capture literally everything we know. But now we risk begging the question: how do we evaluate whether the impoverished perspective is good enough? I think this is probably OK, but it’s definitely subtle.

I think that defining universality w.r.t. 𝔼 is an artifact of this definition strategy, and I’m optimistic that a better definition wouldn’t have this dependence, probably by directly attacking the notion of “justified” belief (which would likely also be useful for actually establishing universality, and may even be necessary). But that’s a hard problem. Philosophers have thought about very similar problems extensively without making the kind of progress that seems adequate for our purposes, and I don’t see an immediate angle of attack.

III. Which A might be universal?
================================

Two regimes
-----------

I’m interested in universality in two distinct regimes:

\* Universality of idealized procedures defined in terms of perfect optimization, such as [debate](https://arxiv.org/abs/1805.00899) under optimal play or [max-HCH](/humans-consulting-hch-f893f6051455), where \*\*A\*\*[\*\*C\*\*] depends only the computational complexity of \*\*C\*\*.
\* Universality of practical variants, in which the perfect optimization is replaced by a bounded optimization, which might use the computation \*\*C\*\* as a “hint.” For example, we might consider amplification or debate where the agents are allowed to inspect a transcript of \*\*C\*\*; or if \*\*C\*\* is a neural net training process, we might train them jointly with \*\*C\*\*.

Idealized models
----------------

\*\*A\*\*[\*\*C\*\*] could be:

\* Adebate, judged by a human, with perfect debaters, whose length depends on the complexity of \*\*C\*\*. (Hopefully in such a way that \*\*A\*\*[\*\*A\*\*[\*\*C\*\*]] is only a slightly longer debate than \*\*A\*\*[\*\*C\*\*].)
\* max-HCH, using perfect optimization rather than a bounded optimizer. Again, we can impose a budget limit that depends on the complexity of \*\*C\*\*, ideally such that \*\*A\*\*[\*\*A\*\*[\*\*C\*\*]] does not involve a much larger computation than \*\*A\*\*[\*\*C\*\*].
\* Use HCH to implement two functions, Info and Answer. Define \*\*A\*\*[\*\*C\*\*](Q) to be Answer(Q, \*a\\*\*)\*,\* where \*a\\* =\* argmax Info(\*a\*, Q\*),\* and the optimization is over advice strings \*a\* of length comparable to the description of \*\*C\*\*.

I don’t think any of these would be universal if instantiated with an open-ended human policy — providing arbitrary complex inputs to a human is likely to lead to bad behavior. Instead, I imagine restricting the human to some small enough space of inputs that they can behave “reasonably” over the whole space, while still being rich enough to be universal. Even as such, I think [additional measures](/universality-and-consequentialism-within-hch-c0bee00365bd) might be needed to achieve security (since a team of humans may behave badly in some exotic corners of the input space, even if individual humans are sheltered from exotic inputs).

Practical models
----------------

In practice I’m particularly interested in the case where \*\*C\*\* has the special form described in the “Trusting opaque computations” section: it trains a neural net on some dataset, and then runs that neural network on new inputs.

\*\*A\*\*[\*\*C\*\*] could be:

\* Debates arbitrated by a human where the debaters are trained jointly with \*\*C\*\*, and can make references to individual examples from the dataset.
\* max-HCH where the agent is trained jointly with \*\*C\*\*, and the overseer can consult individual entries from the dataset.
\* HCH, combined with interpretability techniques that allow it to get information about what \*\*C\*\* is thinking and to relate its behavior to the dataset on which it was trained.

For each of these practical variants, I don’t expect it to be universal unless the corresponding idealized model is. So we can think of dividing the universality question into two pieces:

\* Would our procedure be universal \*if the optimization was performed perfectly\*?
\* Is weight sharing / interpretability strong enough to solve the optimization problems needed for universality?

I think none of the suggestions in this section are plausibly universal without some additional mechanisms, and so even if HCH or debate is universal, additional ideas are needed in order to translate those to practical universal systems. I’m still inclined to focus on the idealized case, because I want to better understand what is needed for universality without thinking about algorithmic difficulties.

IV. Which C are hard to epistemically dominate?
===============================================

The definition in this post was somewhat informal, but the discussion in this appendix will be extremely informal. The purpose is to communicate some of my early thinking about universality and my sense for what a universal reasoner might look like.

The first three examples will consider programs \*\*C\*\* with the following form:

\* Iterate over programs to find an \*f\* that successfully predicts the answers to some dataset of historical questions for which we know the answer.
\* Use that program to predict the answers to future questions.

The different examples explore features of \*f\* that might make it difficult to dominate the overall computation.

Deduction
---------

Suppose \*f\* takes the following form:

\* Given a question and context, it performs a short series of deductions from facts in the context, together with some axioms (e.g. “Bob’s eyes are more likely to twitch when he is lying,” the rules of poker, \*etc.\*)
\* When it arrives at answers that constrain the possible answers to the given question, it outputs an answer consistent with those constraints.

We could attribute the statements produced by this process to \*\*C\*\* as beliefs.We expect this set of beliefs to have useful information because it was selected for making correct predictions.

Fortunately, if HCH is able to see the training dataset then it may be able to arrive at similarly accurate views:

\* It can compute that P(observations | axioms) is high by following along with the deductive process used by \*f .\*
\* Bayes implies that the axioms are likely to be true (fully capturing the argument which leads us, from our current perspective, to expect them to be true).
\* Once HCH knows the axioms are likely to be true, it can follow along with the deductive process to reproduce all of \*f\*’s beliefs.

Modeling
--------

Suppose \*f\* takes the following form:

\* Given a question Q and context, set up a relevant physical situation.
\* Use a dynamics model M to predict what will happen at some future time.
\* Use the resulting state to answer Q.

We could say that \*\*C\*\* believes everything implied by the outcome of this physical simulation. We expect these beliefs to be accurate because \*f\* has predicted well in the past.

As in the deduction case, hopefully HCH can compute that P(observations|M is accurate) is high, since it can also compute the consequences of M. Then Bayes implies the dynamics model is accurate, and HCH can use that model to compute physical states.

Inferring all the beliefs from a dynamics model is not trivial though. As an extreme example, if \*f\* is performing an atom-by-atom simulation of a room, and that room contains Alice and Bob, then we could ascribe extensive beliefs about Alice and Bob to the computation \*\*C\*\*.

(Here we run head on into the fuzziness about what counts as a “reasonable” ascription procedure, but for the moment I’ll assume that some reasonable procedure ascribes beliefs about Alice and Bob to the computation.)

To compete with these ascriptions, HCH needs to infer those high-level beliefs about Alice and Bob from the low-level computation involving atoms. One way to do this is to search over possible “bridging” hypotheses that relate low-level physical facts to high-level facts about the environment. If such a hypothesis can explain additional high-level facts, then a Bayesian can learn that it is true. Similarly, if the bridging hypothesis relates facts about the model to constraints we know from the high-level interpretation, then the Bayesian can potentially use that as evidence. (This kind of reasoning will be discussed in a bit more detail in the next section.)

We could further hope that searching for a bridging hypothesis isn’t much harder than performing the original search over low-level physics, given that the low-level physics needed to explain a bunch of high-level facts and so already must encode some part of that correspondence.

(Note that the “deduction” example in the previous case could also involve alien concepts or models, in which case the same kind of work would be needed.)

Alien reasoning
---------------

In the previous section we described two styles of reasoning we already understand. But there are probably many kinds of reasoning that work well in practice but that would be more alien, and those might be more challenging. This section will explore one example in some detail to try to help anchor our reasoning about the general phenomenon. It will also elaborate on some of the reasoning about “bridging” hypotheses mentioned in the last section.

Suppose that our predictions are always of the same form (e.g. what is the probability the stock market will go up today), and \*f\* works as follows (the details are long but not very important):

\* Find the PSD matrix A with maximum log determinant subject to the constraints in the next bullet points, then output the (0, 0) entry.
\* There is an implicit correspondence between the rows/columns of A, and some uncertain properties X(0), X(1), X(2), …. (which we’ll view as 0–1 variables), where X(0) is the property we want to forecast.
\* If the (\*i\*, \*j\*) entry of A represented the expectation E[X(\*i\*)X(\*j\*)], then the matrix would necessarily satisfy a bunch of constraints, which we impose A. For example:
\* If the context implies that X(\*i\*) = 1, then E[X(\*i\*)X(\*j\*)] = E[X(\*j\*)] = E[X(\*j\*)²], so A(\*i\*, \*j\*) = A(\*j\*, \*j\*).
\* If X(\*i\*) and X(\*j\*) together imply X(\*k\*), then we must have E[X(\*i\*)X(\*j\*)] ≤ E[X(\*i\*)X(\*k\*)] and hence A(\*i\*, \*j\*) ≤ A(\*i\*, \*k\*).
\* For any constants \*a\*, \*b\*, …, E[(\*a\* X(1) + \*b\* X(2) + … )²] ≥ 0 — i.e., the matrix A must be PSD.

The chosen matrix A(opt) corresponds to a set of beliefs about the propositions X(\*i\*), and we can ascribe these beliefs to \*\*C\*\*. Because \*f\* predicts well, we again expect these beliefs to say something important about the world.

I chose this procedure \*f\* in part because we can give a kind of argument for why the matrix A(opt) should tend to encode accurate beliefs. But I don’t think that a universal reasoner can make use of that argument:

\* Finding the argument that \*f\* works is an additional problem, beyond finding \*f\* itself, which might be much harder.
\* A comprehensible version of that argument may be much larger than the strategy itself, so even in the idealized cases like debate with perfect optimization, we may need to increase the scale.
\* I don’t expect that all “good” reasoning strategies have clean understandable arguments in their favor (and even in this case, if it the scheme worked well it would be largely an empirical fact rather than a consequence of the simple theorems we could prove). I think this kind of example is useful because we can easily imagine a human debate judge not having the argument while still being apparently universal. This makes it a useful analogy for cases where the argument really doesn’t exist.

Instead, I think a universal reasoner needs to be able to infer the efficacy of this reasoning procedure from its empirical success. It’s relatively easy for a Bayesian to learn the regularity “\*f\* makes good predictions.” Recovering the rest of the matrix A, and learning how to interpret and whether to trust them, is the hard part.

This is going to require the same kind of bridging/identification we discussed in the last section. Let’s write X(A) for the set of beliefs about the world implied by the “intended” identification. Searching over possible identifications to find X (or something like it) is the only way we can ever relate the rows of A to the quantities X(\*i\*). Again, we can hope that it isn’t much harder than finding the original reasoning procedure.

I think that a sufficiently sophisticated Bayesian would probably be able to learn to trust X(A):

\* If \*f\* is performing well enough that we think it’s more likely to be right in the future, then the Bayesian is going to end believing some claim like “the predictions of \*f\* are good” (since it explains the data so well).
\* This is a complicated statement, and without some kind of explanation this claim has a low prior probability (roughly decaying with the complexity of \*f)\*. The Bayesian is motivated to find an explanation with higher prior probability.
\* The correspondence X can explain the constraints on the matrix A, in terms of facts that we already know about the world. This explanation may end up being simpler (or at least higher prior) than a direct enumeration of the constraints on A — I hope (and think it’s plausible) that this happens iff we’d actually believe on reflection that X(A) captures reality.
(To the extent that we are uncertain and think A’s beliefs have a non-negligible chance of capturing reality, then hopefully we can capture that by the same mechanism by ending up with a non-degenerate posterior.)
\* Now the Bayesian is faced with at least two kinds of explanations:
(a) “If you use the constraints implied by correspondence X(A) + positive semidefiniteness, and then optimize log det, you get a matrix A for which X(A) makes good predictions,”
(b) “The actual situation in the real world is described by positive semi-definite matrices with higher log determinant (under the correspondence X).”
\* Explanation (b) is explaining two things at once: both why the optimization done by \*f\* respects the constraints on our beliefs, and why that optimization leads to good predictions. Hopefully this is simpler than making two separate bridging claims, one which explains \*f\* as respecting the constraints implied by X, and one which claims that \*f\* makes good predictions. Ideally, this 2-for-1 that favors (b) exactly mirrors the underlying reasoning that leads us to actually believe that X(A) is correct, rather than resembling what we know about reality and making good predictions “by coincidence.”

This is a pretty speculative discussion — it’s not very careful, and it’s hard to make it careful in part because I don’t have a formalization of Bayesian reasoning that can even really be applied to this setting. But it seems to match my intuitions about what reasonable Bayesian reasoning “should” do, which gives me a lot more optimism that a careful Bayesian would be able to epistemically dominate \*\*C\*\*.

Deliberation and self-improvement
---------------------------------

Often we expect the computation \*\*C\*\* to have accurate beliefs because it uses a strategy that appears to work in practice — the last 3 examples have discussed that case. But there are other reasons to trust a computation.

For example, humans often write code and trust it (to some extent) even without extensive empirical testing — instead, we have a reason to think it will work, and need only modest testing to make sure that we haven’t made an error in our implementation or reasoning. If I write an automated mathematician that works by finding proofs that pass a proof checker, I don’t expect it to be correct because of the empirical record (Empirical data backs up some key assumptions, but isn’t being used to directly establishing the correctness of the method.)

Likewise, if we train a powerful agent, that agent might initially use strategies that work well in training, but over time it might use learned reasoning to identify other promising strategies and use those. Reasoning might allow it to totally skip empirical testing, or to adopt the method after much less testing than would have been necessary without the reasoning.

To dominate the beliefs produced by such reasoning, we can’t directly appeal to the kind of statistical inference made in the previous section. But in these cases I think we have access to an even more direct strategy.

Concretely, consider the situation where \*\*C\*\* contains a process \*f\* that designs a new reasoning process \*g\*. Then:

\* From the outside, we trust \*g\* because we trust \*f\* and it trusts \*g.\*
\* An otherwise-universal reasoner \*\*A\*\* will dominate \*f\*’s beliefs, and in particular if \*f\* is justified in thinking that \*g\* will work then \*\*A\*\* will believe that and understand why.
\* Once we understand \*f’\*s beliefs, dominating \*g\* is essentially another instance of the original ascription universality problem, but now from a slightly stronger epistemic state that involves both what 𝔼 knows and what \*f\* knows. So unless our original approach to universality was tightly wedded to details of 𝔼, we can probably dominate \*g\*.

At the end of the day we’d like to put all of this together into a tight argument for universality, which will need to incorporate both statistical arguments and this kind of dynamic. But I’m tentatively optimistic about achieving universality in light of the prospect of agents designing new agents, and am much more worried about the kind of opaque computations that “just work” described in the last few sections.

We've now seen a bunch of applications of universality. So what is _ascription_ universality? It's a specific formalization of the concept of universality. Recall our informal definition so far: a program A[C] is universal with respect to some class of programs C if we would trust any beliefs reported by A[C], no matter what beliefs we hear reported by programs in C. But how can a program "report beliefs"? For HCH, to find out its belief for some question, we simply ask HCH the question and take its answer as its "belief". However, we can't do this for arbitrary computations: in particular, if our agent is deliberately hiding a belief from us (and so wouldn't report it if asked), we shouldn't simply say that it doesn't have that belief. This is where _ascriptions_ come in: an ascription is any way of ascribing beliefs to the agent. Simply asking the agent and trusting its answer is one possible ascription. Another class of ascriptions would look at the agent's code and memory, and make inferences about the agent's beliefs from that. We want to say that HCH will dominate _any_ "reasonable" ascription of beliefs to the agent, where an ascription procedure is "reasonable" if it isn't "doing the hard work"; for example, if an agent always says "Yes", it is unreasonable to ascribe to it the belief that Fermat's Last Theorem is true (even though when given the question "Is Fermat's Last Theorem true" it answers yes).

Putting it all together, A[C] is _ascription universal_ with respect to some class of programs C if, for every "reasonable" ascription procedure and program c in C, A[C]'s beliefs (which we find by asking it) _epistemically dominate_ the beliefs ascribed to c. A set of beliefs B1 _epistemically dominates_ another set of beliefs B2 _from our perspective_ if, given arbitrary information about B1 and B2, for every question our best guess at the answer is what we expect B1 would answer.

So far we've been talking about A[C] dominating an entire class of programs C. We might call this _idealized_ ascription universality, and hope that it applies to systems with perfect optimization, such as <@debate@>(@AI safety via debate@) under optimal play. In practice, we'll never get perfect optimization, and our overseer isn't going to be able to think of _everything_ that any agent could be thinking about. Instead, we could aim for _practical_ ascription universality, in which A[c] dominates a particular computation c, and it gets to look at the computation c as a hint. This would allow an overseer to train an agent, while epistemically dominating that agent. Effectively we're asking two questions: first, would our procedure be universal if the optimization was performed perfectly? Second, is weight sharing / interpretability strong enough to solve the optimization problems needed for universality?

The blog post also has some informal thoughts on why we should expect that a careful Bayesian should be able to epistemically dominate computations found via machine learning (which is why we might hope that HCH could be ascription universal).

I was initially a bit confused by the decomposition into the idealized and the practical variant, but I think it makes more sense now: the assumption of _perfect_ optimization gives you a lot of mileage. For example, A[C] might choose a particular computation c in C, and then use its beliefs; if that _choice of computation_ were already "perfectly optimized" for convincing us, then we would always be at least as convinced of A[C]'s answer as the answer of any other c in C. Debate under optimal play is pretty similar: under optimal play, the agent only tells us about the most important considerations, and only brings up other considerations if its opponent asks about them; you would hope that this is like selecting the best c in C and reporting its reasons for its beliefs.

What's happening here is that the perfect optimization allows us to select the best c in C, or the best argument, etc. "for free" (i.e. without using any computation), giving us huge savings; everything else just has to present the results properly. Consider again the question of whether AI will lead to x-risk: while there are different agents thinking about fixed-point theorems, neural nets, and democratic institutions, perfect optimization lets us "pick" the agent whose arguments are most important. Then, we should only need about as much computation as the agent itself.

However, "presenting the results properly" is non-trivial: A[C] needs to report the beliefs _in plain English_, even if the computation c in C has its beliefs "encoded" (as in the case of a detailed physics simulation that can be said to "know" that humans are happy, but doesn't say that in plain English). It's not clear to me whether this will work out.

You might very reasonably be worried that perfect optimization of "the most convincing argument" will get you something that is convincing for the wrong reasons, and is not actually correlated with the truth. The definition of universality doesn't preclude this: it simply says that we trust A[C]'s beliefs over our own beliefs and any other agent's beliefs; this does hold for convincing but invalid arguments. I believe (but am not sure) that the hope is to deal with this by restricting inputs to the human to a small secure space, and by the techniques discussed above in [Universality and consequentialism within HCH](https://ai-alignment.com/universality-and-consequentialism-within-hch-c0bee00365bd).


Worst-case guarantees

(Revisited)
-----------

[![Paul Christiano](https://miro.medium.com/v2/resize:fill:88:88/1\*BNjZCuQuRfIgcXCBMipuBw.jpeg)](https://paulfchristiano.medium.com/?source=post\_page-----ce0e0a3b9b4d--------------------------------)[![AI Alignment](https://miro.medium.com/v2/resize:fill:48:48/1\*N56Qc5-aHTcfGff0scntKQ.png)](https://ai-alignment.com/?source=post\_page-----ce0e0a3b9b4d--------------------------------)[Paul Christiano](https://paulfchristiano.medium.com/?source=post\_page-----ce0e0a3b9b4d--------------------------------)

·[Follow](https://medium.com/m/signin?actionUrl=https%3A%2F%2Fmedium.com%2F\_%2Fsubscribe%2Fuser%2F57f1a655a613&operation=register&redirect=https%3A%2F%2Fai-alignment.com%2Ftraining-robust-corrigibility-ce0e0a3b9b4d&user=Paul+Christiano&userId=57f1a655a613&source=post\_page-57f1a655a613----ce0e0a3b9b4d---------------------post\_header-----------)

Published in[AI Alignment](https://ai-alignment.com/?source=post\_page-----ce0e0a3b9b4d--------------------------------)·13 min read·Jan 21, 2019--

5

Listen

Share

Even if we are very careful about how we deploy ML, we may reach the point where a small number of correlated failures could quickly become catastrophic. Powerful models could actively undermine safeguards, resist attempts at correction, and manipulate their operators.

I think the long-term safety of ML systems requires being able to rule out this kind of behavior, which I’ll call \*unacceptable\*, even for inputs which are extremely rare on the input distribution.

In this post I’ll explain why I think this goal is reasonably likely to be achievable, by highlighting the three ingredients that seem most important to me: adversarial training, transparency, and relaxations.

Preliminaries
=============

This is an updated version of [this post](/techniques-for-optimizing-worst-case-performance-39eafec74b99). It mostly has the same content, but my thinking now feels clearer and there is a more detailed description of my hopes about relaxations.

Disclaimer
----------

I’m trying to communicate intuitions about whether this problem is likely to be soluble someday, rather than making concrete progress or stake out new ground. I’m a theorist who doesn’t work in this area.

Defining acceptable
-------------------

I’ll talk about “acceptable”behavior, but be deliberately vague about what “acceptable” means.

In different contexts, different behavior might be acceptable and it’s up to the user of these techniques to decide. For example, a self-driving car trainer might specify: Crashing your car is tragic but acceptable. Deliberately covering up the fact that you crashed is unacceptable.

My key assumptions about “acceptability” are:

\* As long as the model \*always\* behaves acceptably, and achieves a high reward \*on average\*, we can be happy. The model may still make mistakes, especially when it encounters novel situations, but if we are mindful about how we use AI we can avoid these mistakes escalating into irreversible catastrophes.
\* Requiring a model to always behave acceptably wouldn’t make a hard problem too much harder. It requires some knowledge about what is acceptable, but the complexity of that knowledge doesn’t scale up with the difficulty of the task. Unacceptable behavior is a failure of commission rather than omission, and so avoiding it doesn’t require intricate understanding of the domain. Moreover, we are OK if the model conservatively avoids behaviors that are anything close to unacceptable, allowing it to use simpler concepts.

For any given task there may or may not be a notion of acceptability that meets these properties. For a chess-playing AI, it seems pretty likely to be possible — if you find yourself murdering your opponent you’ve crossed a bright line, and understanding that line doesn’t become harder as you get better at chess. For an AI that’s \*intended\* to frequently overrule strenuous objections from humans, there may be no obvious bright lines to prevent catastrophe.

My view is that we can probably get everything we want out of life by applying AI systems to problems where there is a clear notion of acceptability, even in a ruthlessly competitive world where other actors don’t care about safety. If you agree with me, then read “acceptable” as referring to whatever concept you think has those key properties. If you don’t think there is any concept with these properties, then this post is less likely to be interesting to you.

This is the second guarantee from “[Two guarantees](/two-guarantees-c4c03a6b434f),” and is basically [corrigibility](https://intelligence.org/files/Corrigibility.pdf).

I. Basic framework: \*adversar\*ial training
==========================================

In addition to training our agent to perform well on natural inputs, we can try to find adversarial inputs on which it behaves unacceptably and then penalize it for that behavior.

Adversarial training requires being able to reliably detect when the agent behaves unacceptably. I’m optimistic that we can achieve this by using debate or amplification to train an overseer to answer the question “is this behavior unacceptable?” Obviously the feasibility of this depends on the details of the task and what we consider unacceptable. I discuss the problem of training an overseer [here](/informed-oversight-18fcb5d3d1e1), and I’ll set it aside for the rest of this post.

Adversarial training also requires an adversary who is able to find inputs on which the agent behaves unacceptably. In current practice the adversary is most often gradient descent over the input space, but in the long run I think this technique can’t find most possible failures. I’m optimistic it may be possible to train a strong enough adversary, by combining transparency techniques (section II), relaxations that make the adversary’s job easier (section III), and amplification.

Even with both of those pieces, getting adversarial training to work well is an important research problem, and there could be other fundamental problems lurking in store. But I would be relatively optimistic that other challenges could be resolved. Overall, I think the most likely fundamental obstruction is the difficulty of implementing a good adversary.

II. Informed adversaries
========================

The challenge
-------------

To see how hard the adversary’s job could get, consider an agent that works as follows:

\* Predict whether the current input could possibly appear during training.
\* If the current input definitely couldn’t appear during training, then behave unacceptably.

(This is an exotic failure mode, but could arise e.g. as part of a “treacherous turn,” a situation where training produces a sophisticated enough consequentialist to understand that it needs to perform well during training in order to avoid a negative reward.)

In this context, the adversary is directly pitted against the agent. If there is \*any\* predictable difference between the adversary’s challenges and the actual situations that appear at test time, then the agent can exploit that difference to fail only at test time.

Looking inside
--------------

This game seems almost impossible for the adversary, but we do have one big advantage: we have access to the internals of the agent, so in principle could give the adversary considerable information about what the agent is “thinking.”

For example, we can examine internal features computed by the model and observe how those features depend on the input and each other. There is a growing field dedicated to interrogating neural networks to understand what they are doing. ([This article](https://distill.pub/2018/building-blocks/) describes some relevant techniques and makes an early attempt at combining them to get some insight into what a model is doing.)

On top of this kind of explicit transparency, we can have the adversary and agent share most of their computation, so that the adversary can more easily understand whatever the agent can understand (and can learn to manipulate the internal concepts used by the agent in the same way the agent does). This seems particularly important in combination with amplification, discussed briefly below.

Between these two techniques, I think it’s realistic to hope for the adversary to understand “everything the agent understands,” and moreover to understand how the agent is disposed towards unacceptable behavior. I don’t think this would automatically resolve robustness problems, but I do think it could defuse the adversarial aspect of the problem and give us traction on the remaining problem.

For example, the adversary could start by generating a very broad range of situations in which it would be particularly easy for the agent to behave unacceptably. Once the agent learned to pass these tests, the adversary could try to understand \*why\* it was passing these tests. If the agent was using a misguided picture of acceptability, the adversary could choose an input that highlights the mistake. If the agent is actively trying to detect whether it’s at training time, the adversary can look at the test that the agent uses and try to generate an input specifically to fool this test. (Though this problem may be too hard even if you understand exactly what the agent is doing — see section III.)

Approaches
----------

\*\*RL\*\*. The adversary’s job is to perform the task “find an input on which the agent will behave unacceptably.” One approach is to train this adversary directly with an objective that rewards identifying unacceptable behavior.

We can think of such an adversary as an environment, whose observations are the agent’s actions and whose actions are the agent’s observations. We can then incorporate information about what the agent is thinking by allowing the adversary to interact with transparency tools before needing to interact with the environment.

We can also train this adversary jointly with the agent (being careful about how we take derivatives!), hoping that the intermediate state produced by the agent will be useful for an adversary trying to attack that agent.

\*\*Demonstrations\*\*. This RL problem is extremely difficult. It would be easier if the adversary could learn from demonstrations of successful attacks rather than having to solve an exploration problem from scratch.

For example, if our transparency tools were powerful enough, then we might hope that a human could reliably use them to find inputs where the model would behave unacceptably. If we only needed a few examples to train the agent to behave acceptably then we wouldn’t need an automated adversary at all. If we needed many examples, we could use these human demonstrations to initialize a model that is able to implement an attack.

\*\*Amplification\*\*. Rather than using a human as an expert adversary, we could train the adversary with iterated amplification. This seems likely to be necessary in cases where the model is so complex that an individual human can’t make sense of its behavior.

We can view this process as training a set of more advanced tools that can answer higher-level questions about “what the model is thinking,” on top of some library of core building blocks for interpretability. The human constructs more and more powerful tools by using one generation of tools to produce training data for the next generation. The tools at the very highest level can be used to directly answer the question “where is the model likely to behave unacceptably?”, and the human provides training data for these very hard question by using previous generations of tools to answer questions like “why did the model decide to take that action?” and “why is the model trying to achieve this goal?”

Relationship to ascription universality
---------------------------------------

Even with the aid of transparency tools, I’m only optimistic about an adversary being able to find an attack if they are able to understand everything the model understands. This feels closely related to [ascription universality](/towards-formalizing-universality-409ab893a456).

Note that ascription universality isn’t sufficient on its own: we ultimately care about what the model would have done under different conditions, which is not the kind of thing the model can be naturally said to know. We need some other ideas; in the description above the magic is done by interpretability techniques. The actual interface between interpretability and universality is blurry and I have a very foggy picture of this space.

I don’t have any clear picture of what \*is\* enough; at this point I’m just feeling out intuitions. I’m planning to revisit this question sometime after getting more clarity on ascription universality.

III. Relaxation
===============

The challenge
-------------

Unfortunately, no matter how completely the adversary understands the agent and how clever they are, we can’t generally expect them to find all points on which the agent would behave unacceptably. The basic problem is that recognizing inputs with a certain property is easier than generating inputs with that property.

As a simple but silly example, suppose our model works as follows:

\* Pick a cryptographic puzzle (e.g. “factor RSA-2048”).
\* When it sees a solution to that puzzle, it behaves badly.
\* Otherwise, it behaves well.

Even the adversary understands perfectly what this model is doing, they can’t find an input on which it will behave badly unless they can factor RSA-2048. But if deployed in the world, this model will eventually behave badly.

Possible approach
-----------------

To deal with this problem, we need to penalize the agent for behaving badly on an input even if we can’t actually find it. This will require being conservative, and sometimes penalizing our agent even when there isn’t actually any input where it would behave badly, since merely telling if a puzzle is soluble can already be computationally infeasible.

(It might sound weird to penalize our agent if it wouldn’t actually ever do anything wrong. For intuition, consider the puzzle case: learning that my model \*would\* behave badly if it saw a factorization of RSA-2048 already tells me that something has gone wrong, even if turns out that the puzzle is insoluble.)

So we can change the adversary’s task to make it easier. Rather than asking them to actually produce an input, we are asking them to describe a situation in which the model would behave badly (“the input contains a solution to puzzle P”), even if they can’t find a situation with that description or even tell whether one exists.

I’ll call this easier problem the “relaxed” problem (as is typical in optimization), and call the hypothetical situations “pseudo-inputs” (inspired by [Barak et al](https://arxiv.org/pdf/1205.4484.pdf)).

Illustration: SDP relaxation
----------------------------

This kind of relaxation has been explored in practice in the context of verification. For example, [Raghunathan et al](https://arxiv.org/pdf/1811.01057.pdf) use an SDP to certify properties of a neural net (taking small liberties for simplicity):

\* Rather than having an attacker specify an input, the attacker specifies some properties of a hypothetical distribution over inputs. In particular, they they specify each expectation of the form 𝔼[\*pq\*] where \*p\* and \*q\* are either input variables or activations in some layer of a neural network.
\* We require the attacker’s input to satisfy a bunch of simple properties that are always satisfied by real distributions. For example, if an input \*x\* is known to be in the interval [-1, 1], then 𝔼[\*x\*²] ≤ 1. If \*z\* = ReLU(\*x\*), then either \*x = z\* or \*z\* = 0, so 𝔼[\*z²\*] = 𝔼[\*xz\*]. If \*f\* is an arbitrary function then 𝔼[\*f²\*] ≥ 0. These inequalities are all linear (in terms of the numbers 𝔼[\*pq\*]), and so define a convex set over which it is easy to optimize.
\* We read off the probability of bad behavior from 𝔼. If we were testing a complex constraint computed by another neural network, then the attacker would also specify the expectations of all of the activations of \*that\* network, such that the final output is itself just one entry of 𝔼.

Because the attacker’s constraints don’t perfectly characterize the set of achievable moments, it’s not easy to tell whether there is any distribution over inputs that has those moments. (In fact it’s NP-hard.) But on the other hand, if the attacker can’t find any set of moments, then the model really does always work.

The point of performing the relaxation is that it makes the attacker’s job \*much\* easier. In this case we the relaxation is convex, which means that the attacker can be guaranteed to find an attack if one exists.

I think the relaxations we want to use in the long term won’t be so easy to optimize over. For example, the attacker may still be able to specify properties of a hypothetical distribution, but won’t be allowed to freely specify quite so many degrees of freedom, and in exchange will have to deal with a more complex set of constraints. But hopefully this can still make the adversary’s problem easy enough that an adversary can solve them with the aid of the mechanisms described in section II.

The hope: stronger relaxations
------------------------------

We need to find a relaxation which has two properties:

\* It’s still “easy enough” for the agent to never behave unacceptably for any pseudo-input.
\* It’s easy for the adversary to find a pseudo-input on which the agent behaves unacceptably, if any exist.

For example, in the case of a model which fails whenever it encounters a factorization of RSA-2048, an attacker needs to be allowed to supply a pseudo-input like “a distribution over pairs of numbers that multiply together to 2048.” Finding that input needs to be easy.

On the flip side, if the adversary has too much flexibility, then they might be able to specify a pseudo-input like “a distribution over inputs that would cause the agent to want to kill everyone.” There need to be some limits on the adversary in order for the game to be possible.

This second requirement is tied up with one of the key properties of acceptability: somehow the agent was able to steer clear of unacceptable behaviors. We are now asking them to solve the harder task, of steering clear of unacceptable behavior not only for actual sequences of thoughts, but also for whatever “pseudo-thoughts” they might think in response to a pseudo-input. Intuitively, we need to force the adversary to respect enough of the agent’s internal structure that this “steering clear” procedure can still work, without forcing the agent to totally cripple itself.

Putting it together
===================

I think the following general plan has a significant chance of producing robustly acceptable behavior:

\* Specify a relaxed problem for an adversary, where they specify a distribution over inputs by specifying some of its properties rather than by actually producing examples. We don’t have any clear sense of how this relaxation could work.
\* Train our model with iterated amplification to answer the questions: “which action should we take in state \*s\*?”, “is action \*a\* unacceptable in state \*s\*?”, and “what’s a state where you are particularly likely to recommend an unacceptable action?”
The first two questions can potentially be answered directly with amplification, but the last would require some kind of progress in model interpretability (since it’s a question about how the agent would behave on other inputs, rather than a question that the agent itself “knows” how to answer).
\* Now we can use the model to generate a pseudo-input on which the agent is likely to behave badly, and use the model to verify that the agent in fact behaves badly on that agent (i.e., have the informed adversary described in section II play the relaxed game in section III instead of forcing it to generate an actual output)

I don’t think this particular procedure is likely to work. But I do feel like we have a plausible angle of attack for getting sufficiently “easy” worst-case guarantees. I don’t think we can see any fundamental obstruction to an approach based on adversarial training, and I feel pretty optimistic about either being able to make significant progress in this area or else identifying a clearer obstruction.

Postscripts
-----------

\* To answer “is action \*a\* unacceptable?” we need to use the model to answer a bunch of sub-questions, like “what are the worst consequences of action \*a\*?” If the model sometimes chooses bad actions, it’s reasonable to be concerned that it will also sometimes answer these subquestions badly. But we constantly train “is action \*a\* unacceptable?” on the pseudo-inputs most expected to yield unacceptable behavior, and so an average-case guarantee from the question-answering system implies that it typically answers these subquestions well.
\* If we trained the adversary using RL to elicit unacceptable behavior, then we could have a chicken-and-egg problem: the overseer fails to recognize some behavior as unacceptable, and so the adversary isn’t motivated to elicit it, and so the overseer never learns to correctly recognize it. Demonstrations allow us to overcome this problem, if the demonstrator attempts to generate unacceptable behavior even when the overseer hasn’t yet learned to recognize it.
\* I find it helpful to think about sections II and III separately, since each of them seems to confront a distinct difficulty. It seems meaningful to think about how we can relax the adversary’s problem, and independently to ask how we can give the adversary insight into where the model might behave badly. But at some point I expect them to blur together, and the whole procedure might not end up looking very much like adversarial training.

Let's start considering distribution shift again: we imagine that the agent has already been trained, and there is no overseer any more. When the agent goes off distribution, it may not choose the action that would have gotten the highest reward, and in fact it could be chosen maliciously. As a silly motivating example, perhaps the agent learned the policy "behave well until I see that RSA-2048 (a product of two very large primes) has been factored"; such a policy will behave perfectly during training but will probably commit a treacherous turn eventually.

In this setting, we would like to have a worst-case guarantee, that the model will never behave unacceptably. However, we need to enforce this guarantee at training time, _before_ we know what situations will arise at test time. Universality of the overseer isn't enough to guarantee this, because universality only means that the overseer knows what the agent _currently_ knows; the agent can't really be said to "know" that it will deceive us in the future.

Nonetheless, it seems like the solution is very related to universality, in that we would like to "peer inside" the agent to figure out how it works, to make sure there aren't situations in which it would behave unacceptably. We could use adversarial training for this purpose: train an adversary to find inputs on which the model behaves unacceptably, and then train the model on those inputs to not do that. This will require interpretability techniques, so that the adversary actually stands a chance of finding the relevant inputs. In order to handle cases like RSA-2048 above, the adversary should also be able to provide "pseudo-inputs" like "the world in which the agent sees that RSA-2048 is factored" (otherwise the adversary would have to factor RSA-2048 itself to demonstrate an input on which the agent would fail, which is far too difficult), while not being allowed to provide pseudo-inputs like "the world in which the agent kills all humans" (which no agent can behave acceptably on). Ultimately, we need to make sure the adversary will always be able to find a bad pseudo-input if one exists, without making it too difficult for the agent to behave acceptably on all pseudo-inputs. See also <@relaxed adversarial training@>(@Relaxed adversarial training for inner alignment@).

I was skeptical of the <@previous iteration of this post@>(@Techniques for optimizing worst-case performance@), which proposed adversarial training, verification, and interpretability, because it seemed like none of them would allow us to get a _guarantee_ on performance. I'm significantly more optimistic after reading this post, mainly because a couple of previous confusions have been resolved:

1. The point of verification is not that we can prove a theorem saying "this agent is beneficial"; the point is that by making _relaxations_ (pseudo-inputs), a technique commonly used in formal verification, we can reduce the burden on the other methods being used (such as adversarial training).
2. Similarly, the point of interpretability is not to help _us_ understand what the agent is doing or will do, it's to help the _overseer_ (or adversary in adversarial training) understand that. Unlike us, the overseer / adversary can scale up along with the agent itself.

I still think that it would be hard to get a guarantee with adversarial training, given that adversarial training has to eliminate _all_ vulnerabilities. On the other hand, it only has to find all of the settings where the agent is _maliciously optimizing against us_, which you might hope is a more natural category that is easier to identify without looking too much at particular inputs. This seems like an empirical question on which we'll hopefully get data, though even if it works in all cases that we see, that doesn't rule out the possibility that we failed to notice some issue that will only be triggered in the future (as in the RSA-2048 example).