Directions and desiderata for AI alignment

In the first half of this post, I’ll discuss three research directions that I think are especially promising and relevant to AI alignment:

1. \*\*Reliability and robustness.\*\* Building ML systems which behave acceptably in the worst case rather than only on the training distribution.
2. \*\*Oversight / reward learning.\*\* Constructing objectives and training strategies which lead our policies to do what we intend.
3. \*\*Deliberation and amplification.\*\* Surpassing human performance without simultaneously abandoning human preferences.

I think that we have several angles of attack on each of these problems, and that solutions would significantly improve our ability to align AI. My current feeling is that these areas cover much of the key work that needs to be done.

In the second half of the post, I’ll discuss three desiderata that I think should guide research on alignment:

1. \*\*Secure\*\*. Our solutions should work acceptably even when the environment itself is under the influence of an adversary.
2. \*\*Competitive\*\*. Our solutions should impose minimal overhead, performance penalties, or restrictions compared to malign AI.
3. \*\*Scalable.\*\* Our solutions should continue to work well even when the underlying learning systems improve significantly.

I think that taking these requirements seriously leads us to substantially narrow our focus.

It may turn out that these desiderata are impossible to meet, but if so I think that the first order of business should be understanding clearly \*why\* they are impossible. This would let us better target our work on alignment and better prepare for a future where we won’t have a completely satisfying solution to alignment.

(The ideas in this post are not novel. My claimed contribution is merely collecting these things together. I will link to my own writing on each topic in large part because that’s what I know.)

I. Research directions
======================

1. Reliability and robustness
-----------------------------

Traditional ML algorithms optimize a model or policy to perform well on the training distribution. These models can behave arbitrarily badly when we move away from the training distribution. Similarly, they can behave arbitrarily badly on a small part of the training distribution.

I think this is bad news:

\* Deploying ML systems will critically change their environment, in a way that is hard or impossible to simulate at training time. (The “treacherous turn” is a special case of this phenomenon.)
\* Deployed ML systems are interconnected and exposed to the same world. So if conditions change in a way that causes one of them to fail, \*many\* systems may fail simultaneously.
\* If ML systems are extremely powerful, or if they play a critical role in society, then a widespread failure may have catastrophic consequences.

I’m aware of three basic approaches to reliability that seem to me like they could plausibly scale and be competitive:

(\*ETA: this list is superseded by the list in\* [\*Techniques for Optimizing Worst-Case Performance\*](/techniques-for-optimizing-worst-case-performance-39eafec74b99)\*. I removed consensus and added interpretability and verification. I don’t discuss “learning the right model,” which I still consider a long shot.\*)

\* \*\*Adversarial training\*\*. At training time, attempt to construct inputs that induce problematic behavior and train on those. Eventually, we hope there will be no catastrophe-inducing inputs left. We don’t yet know what is possible to achieve. ([Szegedy 2014](https://arxiv.org/pdf/1312.6199v4.pdf), [Goodfellow 2015](https://arxiv.org/pdf/1412.6572v3.pdf))
\* \*\*Ensembling and consensus\*\*. We often have confidence that there exists \*some\* models which will generalize appropriately. If we can verify that many models agree about an answer, we can be confident that the consensus is correct. If we use this technique, we will often need to abstain on unfamiliar inputs, and in order to remain competitive we will probably need to represent the ensemble implicitly. ([Khani 2016](https://cs.stanford.edu/~pliang/papers/unanimity-acl2016.pdf))
\* \*\*Learning the right model\*\*. If we understood enough about the structure of our model (for example if it reflected the structure of the underlying data-generating process), we might be confident that it will generalize correctly. Very few researchers are aiming for a secure / competitive / scalable solution along these lines, and finding one seems almost (but not completely) hopeless to me. This is MIRI’s approach.

Usual caveats apply: these approaches may need to be used in combination; we are likely to uncover completely different approaches in the future; and I’m probably overlooking important existing approaches.

I think this problem is pretty well-understood and well-recognized, but it looks really hard. ML researchers mostly focus on improving performance rather than robustness, and so I think that this area remains neglected despite the problem being well-recognized.

(Previous posts on this blog: [\*red teams\*](https://medium.com/ai-control/red-teams-b5b6de33dc76#.w2nsces19)\*,\* [\*learning with catastrophes\*](https://medium.com/ai-control/learning-with-catastrophes-59387b55cc30#.a590k1j0p)\*,\* [\*thoughts on training highly reliable models\*](https://medium.com/ai-control/some-thoughts-on-training-highly-reliable-models-2c78c17e266d#.pbtkz0czs))

2. Oversight / reward learning
------------------------------

ML systems are typically trained by optimizing some objective over the training distribution. For this to yield “good” behavior, the objective needs to sufficiently close to what we really want.

I think this is also bad news:

\* Some tasks are very “easy” to frame as optimization problems. For example, we can already write an objective to train an RL agent to operate a profit-maximizing autonomous corporation (though for now we can only train very weak agents).
\* Many tasks that humans care about, such as maintaining law and order or helping us better understand our values, are extremely hard to convert into precise objectives: they are inherently poorly-defined or involve very long timescales, and simple proxies can be “gamed” by a sophisticated agent.
\* As a result, many tasks that humans care about may not get done well; we may find ourselves in an increasingly sophisticated and complex world driven by completely alien values.

So far, the most promising angle of attack is to optimize extremely complex objectives, presumably by learning them.

I’m aware of two basic approaches to reward learning that seem like they could plausibly scale:

\* \*\*Inverse reinforcement learning\*\*. We can observe human behavior in a domain and try to infer what the human is “trying to do,” converting it into an objective that can be used to train our systems. ([Russell 1998](http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.152.6795&rep=rep1&type=pdf), [Ng 2000](http://ai.stanford.edu/~ang/papers/icml00-irl.pdf), [Hadfield-Menell 2016](https://arxiv.org/pdf/1606.03137v3.pdf))
\* \*\*Learning from human feedback\*\*. We can pose queries to humans to figure out which behaviors or outcomes they prefer, and then optimize our systems accordingly. ([Isbell 2001](https://papers.nips.cc/paper/2118-cobot-a-social-reinforcement-learning-agent.pdf), [Thomaz 2006](http://robotic.media.mit.edu/wp-content/uploads/sites/14/2015/01/Thomaz-etal-AAAI-06.pdf), [Pilarski 2011](http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.715.7132&rep=rep1&type=pdf), [Knox 2012](http://www.bradknox.net/wp-content/uploads/2013/06/thesis-knox.pdf))

These solutions seem much closer to working than those listed in the previous section on reliability and robustness. But they still face many challenges, and are not yet competitive, scalable, \*or\* secure:

\* IRL requires a prior over preferences and a model of how human behavior relates to human preferences. Current implementations either only work in severely restricted environments, or use simple models of human rationality which cause the learner to attempt to very precisely imitate the human’s behavior (which might be challenging or impossible).
\* For similar reasons, existing IRL implementations are not able to learn from other data like human utterances or off-policy behavior, even though these constitute the largest and richest source of data about human preferences.
\* Human feedback requires accurately eliciting human preferences, which introduces many complications. (I discuss a few easy problems [here](https://medium.com/ai-control/thoughts-on-reward-engineering-82b193ec03f6#.6n2d4co3i).)
\* Human feedback is expensive and so we will need to be able to learn from a relatively small amount of labeled data. Demonstrations are also expensive and so may end up being a bottleneck for approaches based on IRL though it’s not as clear.
\* Both imitation learning and human feedback may fail when evaluating a behavior requires understanding where the behavior came from. For example, if you ask a human to evaluate a painting they may not be able to easily check whether it is derivative, even if over the long run they would prefer their AI to paint novel paintings.

(I’ve described these approaches in the context of “human” behavior, but the expert providing feedback/demonstrations might themselves be a human augmented with AI assistance, and eventually may simply be an AI system that is aligned with human interests.)

This problem has not received much attention in the past, but it seems to be rapidly growing in popularity, which is great. I’m currently working on a project in this area.

(\*Previous posts on this blog:\* [\*the reward engineering problem\*](https://medium.com/ai-control/the-reward-engineering-problem-30285c779450#.f1ihhss6w)\*,\* [\*ambitious vs. narrow value learning\*](https://medium.com/ai-control/ambitious-vs-narrow-value-learning-99bd0c59847e#.s33f26ht5)\*,\* [\*against mimicry\*](https://medium.com/ai-control/against-mimicry-6002a472fc42#.chg6xlqve)\*,\* [\*thoughts on reward engineering\*](https://medium.com/ai-control/thoughts-on-reward-engineering-82b193ec03f6#.iim1wpt9a)\*.\*)

3. Deliberation and amplification
---------------------------------

Machine learning is usually applied to tasks where feedback is readily available. The research problem in the previous section aims to obtain quick feedback in general by using human judgments as the “gold standard.” But this approach breaks down if we want to exceed human performance.

For example, it is easy to see how we could use machine learning to train ML systems to make human-level judgments about urban planning, by training them to produce plans that sound good to humans. But if we want to train an ML system to make superhuman judgments about how to lay out a city, it’s completely unclear how we could do it — without spending billions of dollars trying out the system’s ideas and telling it which ones work.

This is a problem for the same reasons discussed in the preceding section. If our society is driven by systems superhumanly optimizing short-term proxies for what we care about — such as how much they impress humans, or how much money they make—then we are liable to head off in a direction which does not reflect our values or leave us in meaningful control of the situation.

If we lowered our ambitions and decide that superhuman performance is inherently unsafe, we would be leaving huge amounts of value on the table. Moreover, this would be an unstable situation: it could last only as long as everyone with access to AI coordinated to pull their punches and handicap their AI systems.

I’m aware of two approaches to this problem that seem like they could scale:

\* \*\*IRL [hard mode]\*\*. In principle we can use IRL to recover a representation of human preferences, and then apply superhuman intelligence to satisfy those preferences much better than a human could. However, this is a much more ambitious and challenging form of IRL than is usually discussed, which [remains quite challenging](https://medium.com/ai-control/the-easy-goal-inference-problem-is-still-hard-fad030e0a876) even when you set aside all of the usual algorithmic and statistical difficulties. (Jacob Steinhardt and Owain Evans discuss this issue in [a recent post](https://jsteinhardt.wordpress.com/2017/02/07/model-mis-specification-and-inverse-reinforcement-learning/).)
\* \*\*Iterated amplification\*\*. A group of interacting humans can potentially be smarter than a single human, and a group of AI systems could be smarter than the original AI system. By using these groups as “experts” in place of individual humans, we could potentially train much smarter systems. The key questions are how to perform this composition in a way that causes the group to implement the same preferences as its members, and whether the cognitive benefits for groups are large enough to overcome the overhead of coordination. (I discuss this approach [here](https://medium.com/ai-control/policy-amplification-6a70cbee4f34#.ampcyxi9r) and in follow-up work.)
\* \*\*IRL for cognition\*\*. Rather than applying IRL to a humans’ actions, we could apply it to the cognitive actions taken by a human while they deliberate about a subject. We can then use those values to execute a longer deliberation process, asking “what would the human do if they had more time to think / more powerful cognitive tools?” I think this approach ends up being similar to a blend of the previous two.

It’s completely unclear how hard this problem is or how far we are from a solution. It is a much less common research topic than either of the preceding points.

In the short term, I think it might be easier to study analogs of this problem in the context of human behavior than to attempt to directly study it in the context of AI systems.

[Ought](https://blog.ought.com/) is a non-profit aimed at addressing (roughly) this problem; I think it is reasonably likely to make significant progress.

(\*Previous posts on this blog:\* [\*capability amplification\*](https://medium.com/ai-control/policy-amplification-6a70cbee4f34)\*,\* [\*reliability amplification\*](https://medium.com/ai-control/reliability-amplification-a96efa115687)\*,\* [\*security amplification\*](https://medium.com/ai-control/security-amplification-f4931419f903)\*,\* [\*meta-execution\*](https://medium.com/ai-control/meta-execution-27ba9b34d377)\*,\* [\*the easy goal inference problem is still hard\*](https://medium.com/ai-control/the-easy-goal-inference-problem-is-still-hard-fad030e0a876))

II. Desiderata
==============

I’m most interested in algorithms that are secure, competitive, and scalable, and I think that most research programs are very unlikely to deliver these desiderata (this is why the lists above are so short).

Since these desiderata are doing a lot of work in narrowing down the space of possible research directions, it seems worthwhile to be thoughtful and clear about them. It would be easy to gloss over any of them as obviously unobjectionable, but I would be more interested in people pushing back on the strong forms than implicitly accepting a milder form.

1. Secure
---------

Many pieces of software work “well enough” most of the time; we often learn this not by a deep analysis but by just trying it and seeing what happens. “Works well enough” often breaks down when an adversary enters the prediction.

Whether or not that’s a good way to build AI, I think it’s a bad way to do alignment research right now.

Instead, we should try to come up with alignment solutions that work in the least convenient world, when nature itself is behaving adversarially. Accomplishing this requires argument and analysis, and cannot be exclusively or based on empirical observation.

AI systems obviously won’t work well in the worst case (there is no such thing as a free lunch) but it’s reasonable to hope that our AI systems will never respond to a bad input by actively \*trying\* to hurt us —at least as long as we remain in physical control of the computing hardware, and the training process, \*etc.\*

Why does security seem important?

\* It’s really hard to anticipate what is going to happen in the future. I think it’s easy to peer into the mists and say “well, hard to know what’s going to happen, but this solution might work out OK,” and then to turn out to be too optimistic. It’s harder to make this error when we hold ourselves to a higher standard, of actually giving an argument for why things work. I think that this is a general principle for doing useful research in advance of when it is needed — we should hold ourselves to standards that are unambiguous and clear even when the future is murky. This is a theme that will recur in the coming sections.
\* We are used to technological progress proceeding slowly compared to timescales of human judgment and planning. It seems quite likely that powerful AI will be developed during or after a period of acceleration, challenging those assumptions and undermining a traditional iterative approach to development.
\* The world really does contain adversaries. It’s one thing to build insecure software when machines have power over modest amounts of money with significant human oversight, it’s another thing altogether when they have primary responsibility for enforcing the law. I’m not even particularly worried about human attackers, I’m mostly worried about a future where all it takes to launch attacks is money (which can itself be earned by executing attacks). Moreover, if the underlying ML is insecure and ML plays a role in almost all software, we are going to have a hard time writing any secure software at all.

(\*Previous posts:\* [\*security and AI alignment\*](https://medium.com/ai-control/security-and-ai-control-675ace05ce31))

2. Competitive
--------------

It’s easy to avoid building an unsafe AI system (for example: build a spreadsheet instead). The only question is how much you have to sacrifice to do it.

Ideally we’ll be able to build benign AI systems that are just as efficient and capable as the best AI that we could build by any means. That means: we don’t have to additional domain-specific engineering work to align our systems, benign AI doesn’t require too much more data or computation, and our alignment techniques don’t force us to use particular techniques or restrict our choices in other ways.

(More precisely, I would consider an alignment strategy a success if the additional costs are sublinear: if the fraction of resources that need to be spent on alignment research and run-time overhead \*decreases\* as the AI systems become more powerful, converging towards 0.)

Why is competitiveness important?

\*\*A. It’s easy to tell when a solution is plausibly competitive, but very hard to tell exactly how uncompetitive an uncompetitive solution will be.\*\* For example, if a purported alignment strategy requires an AI not to use technique or development strategy X, it’s easy to tell that this proposal isn’t competitive in general, but very hard to know exactly how uncompetitive it is.

As in the security case, it seems very easy to look into the fog of the future and say “well this seems like it will probably be OK” and then to turn out to be too optimistic. If we hold ourselves to the higher standard of competitiveness, it is much easier to stay honest.

Relatedly, we want alignment solutions that work across an extremely large range of techniques not just because we are uncertain about which techniques will be important, but because generalizing across all of the situations we can foresee is a good predictor of working for situations we can’t foresee.

\*\*B. You can’t unilaterally use uncompetitive alignment techniques; we would need global coordination to avoid trouble.\*\*If we \*don’t\* know how to build competitive benign AI, then users/designers of AI systems have to compromise efficiency in order to maintain reliable control over those systems. The most efficient systems will by default be built by whoever is willing to accept the largest risk of catastrophe (or perhaps by actors who consider unaligned AI a desirable outcome).

It may be possible to avert this kind of race to the bottom by effective coordination by e.g. enforcing regulations which mandate adequate investments in alignment or restrict what kinds of AI are deployed. Enforcing such controls domestically is already a huge headache. But internationally things are even worse: a country that handicapped its AI industry in order to proceed cautiously would face the risk of being overtaken by a less prudent competitor, and avoiding \*that\* race would require effective international coordination.

Ultimately society will be able and willing to pay \*some\* efficiency cost to reliably align AI with human interests. But the higher that cost, the harder the coordination problem that we will need to solve. I think the research community should be trying to make that coordination problem as easy as possible.

(\*Previous posts:\* [\*prosaic AI alignment\*](https://medium.com/ai-control/prosaic-ai-control-b959644d79c2)\*,\* [\*a possible stance for AI control\*](https://medium.com/ai-control/a-possible-stance-for-ai-control-research-fe9cf717fc1b)\*,\* [\*efficient and safely scalable\*](https://medium.com/ai-control/efficient-and-safely-scalable-8218fa8a871f#.m7z8qccef))

3. Scalable
-----------

Over time, we are acquiring more data, more powerful computers, richer model classes, better optimization algorithms, better exploration strategies, and so on. If we extrapolate these trends, we end up with very powerful models and policies.

Many approaches to alignment break down at some point in this extrapolation. For example, if we train an RL agent with a reward function which imperfectly approximates what we want, it is likely to fail once the agent becomes sufficiently sophisticated — unless the reward function itself becomes more sophisticated in parallel.

In contrast, let’s say that a technique is “scalable” if it continues to work just as well even when the underlying learning becomes much more powerful. (See also: Eliezer’s more colorful “[omnipotence test](https://arbital.com/p/omni\_test/).”)

This is another extremely demanding requirement. It rules out many possible approaches to alignment. For example, it probably rules out any approach that involves hand-engineering reward functions. More subtly, I expect it will rule out any approach that requires hand-engineering an informative prior over human values (though some day we will hopefully find a scalable approach to IRL).

Why is scalability important?

\* As in the previous sections, it’s easy to be too optimistic about exactly when a non-scalable alignment scheme will break down. It’s much easier to keep ourselves honest if we actually hold ourselves to producing scalable systems.
\* If AI progress rapidly, and especially if AI research is substantially automated, then we may literally confront the situation where the capabilities of our AI systems are changing rapidly. It would be desirable to have alignment schemes that continued to work in this case.
\* If we don’t have scalable solutions then we require a continuing investment of research on alignment in order to “keep up” with improvements in the underlying learning. This risks compromising competitiveness, forcing AI developers to make a hard tradeoff between alignment and capabilities. This would be acceptable if the ongoing investments in alignment are modest compared to the investments in capabilities. But as with the last point, that’s a very murky question about which it seems easy to be overly optimistic in advance. If we think the problem will be easy in the future when we have more computing, then we ought to be able to do it now. Or at the very least we ought to be able to explain how more computing will make it easy. If we make such an explanation sufficiently precise then it will itself become a scalable alignment proposal (though perhaps one that involves ongoing human effort).

(\*Previous posts:\* [\*scalable AI control\*](https://medium.com/ai-control/scalable-ai-control-7db2436feee7#.tljxalxgv))

Aside: feasibility
------------------

One might reject these desiderata because they seem too demanding: it would be great if we had a secure, competitive and scalable approach to alignment, but that might not be possible.

I am interested in trying to satisfy these desiderata despite the fact that they are quite demanding, for two reasons:

\* I think that it is very hard to say in advance what is possible or impossible. I don’t yet see any fundamental obstructions to achieving these goals, and until I see hard obstructions I think there is a significant probability that the problem will prove to be feasible (or “almost possible,” in the sense that we may need to weaken these goals only slightly).
\* If there is some fundamental obstruction to achieving these goals, then it would be good to understand that obstruction in detail. Understanding it would help us understand the nature of the problem we face and would allow us to do better research on alignment (by focusing on the key aspects of the problem). And knowing that these problems are impossible, and understanding exactly how impossible they are, helps us prepare for the future, to build institutions and mechanisms that will be needed to cope with unavoidable limitations of our AI alignment strategies.

III. Conclusion
===============

I think there is a lot of research to be done on AI alignment; we are limited by a lack of time and labor rather than by a lack of ideas about how to make progress.

Research relevant to alignment is already underway; researchers and funders interested in alignment can get a lot of mileage by supporting and fleshing out existing research programs in relevant directions. I don’t think it is correct to assume that if anyone is working on a problem then it is going to get solved — even amongst things that aren’t literally at the “no one else is doing it” level, there are varying degrees of neglect.

At the same time, the goals of alignment are sufficiently unusual that we shouldn’t be surprised or concerned to find ourselves doing unusual research. I think that area #3 on deliberation and amplification is almost completely empty, and will probably remain pretty empty until we have clearer statements of the problem or convincing demonstrations of work in that area.

I think the distinguishing feature of research motivated by AI alignment should be an emphasis on secure, competitive, and scalable solutions. I think these are very demanding requirements that significantly narrow down the space of possible approaches and which are rarely explicitly considered in the current AI community.

It may turn out that these requirements are infeasible; if so, one key output of alignment research will be a better understanding of the key obstacles. This understanding can help guide less ambitious alignment research, and can help us prepare for a future in which we won’t have a completely satisfying solution to AI alignment.

This post has mostly focused on research that would translate directly into concrete systems. I think there is also a need for theoretical research building better abstractions for reasoning about optimization, security, selection, consequentialism, and so on. It is plausible to me that we will produce acceptable systems with our current conceptual machinery, but if we want to convincingly \*analyze\* those systems then I think we will need significant conceptual progress (and better concepts may lead us to different approaches). I think that practical and theoretical research will be attractive to different researchers, and I don’t have strong views about their relative value.


Implicit extortion

In this post I describe a pattern of behavior I call “implicit extortion.” RL agents are particularly susceptible to implicit extortion, in a way that is likely to be problematic for high-stakes applications in open-ended strategic environments.

I expect that many people have made this point before. My goal is just to highlight the issue and to explore it a little bit more carefully.

Basic setup
-----------

Consider two actors, the target (T) and manipulator (M), such that:

\* M wants T to perform some \*target action\* — e.g. make a payment, leak information, buy a particular product, handicap itself…
\* M can take \*destructive actions\* that hurts both M and T — e.g. spreading rumors about T, undercutting T in a marketplace, physically attacking T…

In \*explicit extortion\*, M threatens to take the destructive action unless T performs the target action. Then a naive T reasons: “if I don’t take the target action, something bad will happen, so I better take the target action.”

In \*implicit extortion\*, M simply performs the destructive action whenever T doesn’t perform the target action. Then a naive T eventually learns that failure to take the target action is associated with something bad happening, and so learns to take the target action.

Implicit extortion is very similar to explicit extortion:

\* T would prefer notbe the kind of person who is vulnerable to extortion, so that bad things don’t happen to them.
\* Extortion doesn’t necessarily cost M very much, if they don’t follow through on the threat very often.

However, implicit extortion can be particularly hard to avoid:

\* It can be effective without T realizing that it’s happening, which makes it hard for them to respond appropriately even if they do have defenses.
\* It affects simple RL algorithms (which don’t have defenses against extortion, and can’t be easily modified to include such defenses).

Example
-------

The most extreme and blatant example would be for M to send T a daily request for $100. On any day when T fails to pay, M launches a costly cyberattack against T. A human would immediately recognize this behavior as extortion and would respond appropriately, but an RL algorithm might simply notice that paying is the best strategy and therefore decide to pay.

Implicit extortion can be much harder to detect, while still being effective. Suppose that every time T tries to change their product, M runs a grassroots smear campaign. It might not be possible for T to distinguish the situations “M is attempting to manipulate me into not changing my product” and “Every time I change the product people get really unhappy, so I should do so sparingly.”

Details
=======

How expensive is this for the manipulator?
------------------------------------------

Suppose that T is using an RL algorithm, and M is trying to manipulate them. How expensive is this for M? How likely is it to be worthwhile?

\*\*At equilibrium\*\*: T learns to always perform the target action; so only fails to take the target action while exploring. The long-term cost to M depends entirely on the target’s exploration policy.

If T uses ε-exploration, then they take the target action (1 − ε) of the time. So M only needs to pay the cost of the destructive action on an ε fraction of trials.

For complex high-level actions, the effective ε can’t be \*too\* high — it’s not a good idea to “try something crazy” 10% of the time just to see what happens. But let’s be conservative and suppose that ε=0.1 anyway.

Suppose that M is trying to directly extract money from T, $10 at a time, and that it costs M $50 of value in order to cause $15 of trouble for T.

If M asks for $10 on 10 occasions, T will refuse to pay only once as an exploration. Then M needs to pay that $50 cost only once, thereby ensuring that the cost of paying (=$10) is smaller than the average cost of refusing to pay (=$15). Meanwhile, M makes $90, pocketing $40 of profit.

In general, M can make a profit whenever the product of (payment efficiency) \\* (destructive efficiency) > ε, where “payment efficiency” is the benefit to M divided by the cost to T of the target action, and “destructive efficiency” is the cost to T divided by the cost to M of the destructive action.

In practice I think it’s not too uncommon for payment efficiency to be ~1, and for destructive efficiency to be >1, such that extortion is possible regardless of ε. Small values of ε make extortion considerably easier and more cost-effective, and make it much harder to prevent.

\*\*During learning\*\*: the analysis above only applies when the agent has already learned to consistently take the target action. Earlier in learning, the target action may only occur rarely and so punishment may be very expensive. This could be worth it over the long term but may be a major hurdle.

Fortunately for M, they can simply start by rewarding the target behavior, and then gradually shift to punishment once the target behavior is common. From the perspective of the RL agent, the benefit of the target action is the same whether it’s getting a reward or avoiding a punishment.

In the cash payment example, M could start by paying T $20 every time that T sends $10. Once T notices that paying works well, M can gradually reduce the payment towards $10 (but leaving a profit so that the behavior becomes more and more entrenched). Once T is consistently paying, M can start scaling up the cost of not paying while it gradually reduces the benefits of paying.

Analyzing the error
-------------------

Paying off a (committed) extortionist typically has the best consequences and so is recommended by causal decision theory, but \*having the policy of paying off extortionists\* is a bad mistake.

Even if our decision theory would avoid caving in to extortion, it can probably only avoid implicit extortion if it recognizes it. For example, UDT typically avoids extortion because of the logical link from “I cave to extortion” → “I get extorted.” There is a similar logical link from “I cave to implicit extortion” → “I get implicitly extorted.” But if we aren’t aware that an empirical correlation is due to implicit extortion, we won’t recognize this link and so it can’t inform our decision.

In practice the target is only in trouble if would-be manipulators know that they are inclined to comply with extortion. If manipulators base that judgment on past behavior, then taking actions that “look like what someone vulnerable to extortion would do” is itself a bad decision that even a causal decision theorist would avoid. Unfortunately, it’s basically impossible for an RL algorithm to learn to avoid this, because the negative consequences only appear over a very long timescale. In fact, the timescale for the negative consequences is longer than the timescale over which the RL agent adjusts its policy— which is too long for a traditional RL system to possibly do the credit assignment.

Other learning systems
======================

What algorithms are vulnerable?
-------------------------------

At first glance the problem may seem distinctive to policy gradient RL algorithms, where we take actions randomly and then reinforce whatever actions are associated with a high reward.

But the same problem afflicts any kind of RL. For example, a model-based agent would simply learn the model “not doing what the manipulator wants causes  to happen,” and using that model for planning would have exactly the same effect as using policy gradients.

More broadly, the problem is with the algorithm: “learn an opaque causal model and use it to inform decisions.” That’s an incredibly general algorithm. If you aren’t willing to use that algorithm, then you are at a significant competitive disadvantage, since the world contains lots of complicated causal processes that we can learn about by experiment but can’t model explicitly. So it seems like everyone just has to live with the risk of implicit extortion.

I describe the problem as afflicting “algorithms,” but it can also afflict humans or organizations. For example, any organization that is compelled by arguments like “X has always worked out poorly in the past, even though we’re not quite sure why, so let’s stop doing it” is potentially vulnerable to implicit extortion.

What about human learning?
--------------------------

Humans have heuristics like vindictiveness that help prevent us from being manipulated by extortion, and which seem particularly effective against implicit extortion. Modern humans are also capable of doing explicit reasoning to recognize the costs of giving in to extortion.

Of course, we can only be robust to implicit extortion when we recognize it is occurring. Humans do have some general heuristics of caution when acting on the basis of opaque empirical correlations, or in situations where they feel they might be manipulable. However, it still seems pretty clear that human learning is vulnerable to implicit extortion in practice. (Imagine a social network which subtly punishes users, e.g. by modulating social feedback, for failing to visit the site regularly.)

Evolution?
----------

Evolution itself doesn’t have any check against extortion, and it operates entirely by empirical correlations, so why isn’t it exploited in this way?

Manipulating evolution requires the manipulator to have a time horizon that is many times the generation length of the target. There aren’t many agents with long enough time horizons, or sophisticated enough behavior, to exploit the evolutionary learning dynamic (and in particular, evolution can’t easily learn to exploit it).

When we do have such a large gap in time horizons and sophistication — for example, when humans square off against bacteria with very rapid evolution — we do start to see implicit extortion.

For example, when a population of bacteria develop resistance to antibiotic A, we take extra pains to totally eradicate them with antibiotic B, even though we could not afford to use that strategy if A-resistance spread more broadly through the bacteria population. This is effectively implicit extortion to prevent bacteria from developing A-resistance. It would continue to be worthwhile for humanity even if the side effects of antibiotic B were much worse than the infection itself, though we probably wouldn’t do it in that case since it’s a hard coordination problem (and there are lots of other complications).

Conclusion
==========

There are many ways that an AI can fail to do the right thing. Implicit extortion is a simple one that is pretty likely to come up in practice, and which may seriously affect the applicability of RL in some contexts.

I don’t think there is any “silver bullet” or simple decision-theoretic remedy to implicit extortion, we just need to think about the details of the real world, who might manipulate us in what ways, what their incentives and leverage are, and how to manage the risk on a case-by-case basis.

I think we need to [define “alignment” narrowly enough](/clarifying-ai-alignment-cec47cd69dd6) that it is consistent with implicit extortion, just like we define alignment narrowly enough that it’s consistent with losing at chess. I’ve found understanding implicit extortion helpful for alignment because it’s one of many conditions under which an aligned agent may end up effectively optimizing for the “wrong” preferences, and I’d like to understand those cases in order to understand what we are actually trying to do with alignment.

I don’t believe implicit extortion is an existential risk. It’s just another kind of conflict between agents, that will divert resources from other problems but should “wash out in the long run.” In particular, every agent can engage in implicit extortion and so it doesn’t seem to shift the relative balance of influence amongst competing agents. (Unlike alignment problems, which shift influence from human values to whatever values unaligned AI systems end up pursuing.)

Explicit extortion occurs when an attacker makes an explicit threat to harm you if you don't comply with their demands. In contrast, in implicit extortion, the attacker always harms you if you don't do the thing that they want, which leads you to learn over time to do what the attacker wants. Implicit extortion seems particularly hard to deal with because you may not know it is happening.

Implicit extortion sounds like a hard problem to solve, and the post argues that humans don't robustly solve it. I'm not sure whether this is a problem we need to solve in order to get good outcomes -- if you can detect that implicit extortion is happening, you can take steps to avoid being extorted, and so it seems that a successful implicit extortion attack would have to be done by a very capable adversary that knows how to carry out the attack so that it isn't detected. Perhaps we'll be in the world where such adversaries don't exist.


Inaccessible information

Suppose that I have a great model for predicting “what will Alice say next?”

I can evaluate and train this model by checking its predictions against reality, but there may be many facts this model “knows” that I can’t easily access.

For example, the model might have a detailed representation of Alice’s thoughts which it uses to predict what Alice will say, \*without\* being able to directly answer “What is Alice thinking?” In this case, I can only access that knowledge indirectly, e.g. by asking about what Alice would say in under different conditions.

I’ll call information like “What is Alice thinking?” inaccessible. I think it’s very plausible that AI systems will build up important inaccessible knowledge, and that this may be a central feature of the AI alignment problem.

In this post I’m going to try to clarify what I mean by “inaccessible information” and the conditions under which it could be a problem. This is intended as clarification and framing rather than a presentation of new ideas, though sections IV, V, and VI do try to make some small steps forward.

I. Defining inaccessible information
====================================

I’ll start by informally defining what it means for information to be \*\*accessible\*\*, based on two mechanisms:

Mechanism 1: checking directly
------------------------------

If I can check X myself, \*given other accessible information,\* then I’ll define X to be accessible.

For example, I can check a claim about what Alice will do, but I can’t check a claim about what Alice is thinking.

If I can run randomized experiments, I can probabilistically check a claim about what Alice \*would\* do. But I can’t check a counterfactual claim for conditions that I can’t create in an experiment.

In reality this is a graded notion — some things are easier or harder to check. For the purpose of this post, we can just talk about whether something can be tested even a single time over the course of my training process.

Mechanism 2: transfer
---------------------

The simplest model that provides some accessible information X may also provide some other information Y. After all, it’s unlikely that the simplest model that outputs X doesn’t output \*anything\* else. In this case, we’ll define Y to be accessible.

For example, if I train a model to predict what happens over the next minute, hour, or day, it may generalize to predicting what will happen in a month or year. For example, if the simplest model to predict the next day was a fully-accurate physical simulation, then the same physics simulation might work when run for longer periods of time.

I think this kind of transfer is kind of dicey, so I genuinely don’t know if long-term predictions are accessible or not (we certainly can’t directly check them, so transfer is the only way they could be accessible).

Regardless of whether long-term predictions are accessible by transfer, there are other cases where I think transfer is pretty unlikely. For example, the simplest way to predict Alice’s behavior might be to have a good working model for her thoughts. But it seems unlikely that this model would spontaneously describe what Alice is thinking in an understandable way — you’d need to specify some additional machinery, for turning the latent model into useful descriptions.

I think this is going to be a fairly common situation: predicting accessible information may involve almost all the same work as predicting inaccessible information, but you need to combine that work with some “last mile” in order to actually output inaccessible facts.

Definition
----------

I’ll say that information is \*accessible\* if it’s in the smallest set of information that is closed under those two mechanisms, and \*inaccessible\* otherwise.

There are a lot of nuances in that definition, which I’ll ignore for now.

Examples
--------

Here are some candidates for accessible vs. inaccessible information:

\* “What will Alice say?” vs “What is Alice thinking?”
\* “What’s on my financial statement?” vs. “How much money do I really have?”
\* “Am I coughing?” vs. “What’s happening with my immune system?”
\* “How will senators vote?” vs. “What’s the state of political alliances and agreements in the senate?”
\* “What do I see on my computer screen?” vs. “Is my computer compromised?”
\* “What’s the market price of this company?” vs. “How valuable is this IP really?”
\* “Will the machine break tomorrow?” vs. “Is there hard-to-observe damage in this component?”
\* “What does the news show me from 5000 miles away?” vs. “What’s actually happening 5000 miles away?”
\* “Is this argument convincing?” vs. “Is this argument correct?”
\* “What will happen tomorrow?” vs. “What will happen in a year” (depending on whether models transfer to long horizons)

II. Where inaccessible info comes from and why it might matter
==============================================================

Our models can build up inaccessible information because it helps them predict accessible information. They know something about what Alice is thinking because it helps explain what Alice does. In this diagram, the black arrow represents the causal relationship:

![]()Unfortunately, this causal relationship doesn’t directly let us \*elicit\* the inaccessible information.

Scientific theories are prototypical instances of this diagram, e.g. I might infer the existence of electron from observing the behavior of macroscopic objects. There might not be any explanation for a theory other than “it’s made good predictions in the past, so it probably will in the future.” The actual claims the theory makes about the world — e.g. that the Higgs boson has such-and-such a mass — are totally alien to someone who doesn’t know anything about the theory.

I’m not worried about scientific hypotheses in particular, because they are usually \*extremely\* simple. I’m much more scared of analogous situations that we think of as intuition — if you want to justify your intuition that Alice doesn’t like you, or that some code is going to be hard to maintain, or that one tower of cards is going to be more stable than another, you may not be able to say very much other than “This is part of a complex group of intuitions that I built up over a very long time and which seems to have a good predictive track record.”

At that point “picking the model that matches the data best” starts to look a lot like doing ML, and it’s more plausible that we’re going to start getting hypotheses that we don’t understand or which behave badly.

Why might we care about this?
-----------------------------

In some sense, I think this all comes down to what I’ve called [strategy-stealing](/the-strategy-stealing-assumption-a26b8b1ed334): if AI can be used to compete effectively, can humans use AI to compete \*on their behalf\*?

More precisely, for every strategy A that an AI could pursue to bring about some arbitrary outcome, is there a strategy A\\* that would help humans get what we want over the long term, without leaving us at a competitive disadvantage over the short term?

![]()If so it’s good news for humanity: if most humans build AIs who execute plans like A\\*, then humans won’t be outcompeted by unaligned AIs who execute plans like A.

But the mere \*existence\* of A\\* isn’t very helpful, we need to actually be able to figure out that A\\* leads to human flourishing so that we can do it. If we can’t recognize plans like A\\*, then humanity will be at a disadvantage.

We could have a problem if the fact “A\\* leads to human flourishing” is inaccessible while the fact “A leads to paperclips” is accessible.

![]()In this case, we either need to be sure that no one acts on the basis of information like “A leads to lots of paperclips,” or we need to figure out some way to access the inaccessible information that “A\\* leads to lots of human flourishing.”

III. Eliciting inaccessible info and the instrumental policy
============================================================

I’ve been talking as if we can’t access inaccessible info (and the name is suggestive…), but I don’t think you should be convinced of that yet.

Let M be a model that reports only accessible facts — it answers questions like “what will Alice say next?” but not questions like “is Alice actually happy with the situation?” It’s built up a lot of knowledge about the world, and uses that knowledge to estimate accessible facts well as it can.

We suspect there is a related model M\\* that uses all of M’s machinery in order to explain what’s going on in Alice’s head. By the definition of inaccessibility, M\\* is at least somewhat more complex than M. So if we want to learn M\\*, we need a training objective that penalizes M relative to M\\*.

But it’s really easy to tell M\\* and M apart! M won’t automatically tell us what Alice is thinking, but it also won’t automatically make up credible stories that hold together. Both of those require extra complexity. So we can still hope to get M\\* if we train our model to (i) correctly provides accessible info, (ii) says something consistent and plausible about inaccessible info.

The real problem comes from what I’ll call the \*instrumental policy\*. Let’s say we’ve tried to dream up a loss function L(\*x\*, \*y\*) to incentivize the model to correctly answer information we can check, and gives at least plausible and consistent answers on things we can’t check. By definition, the values L(\*x\*, \*y\*) are themselves accessible.

Then it’s natural to learn a policy like: “on input \*x\*, produce the output \*y\* for which the loss L(\*x\*, \*y\*) will be minimal.” Let’s write BAD for this policy.

Put differently, BAD is the model that performs a simple substitution — when you ask it question Q, it replaces it with the meta question “What answer to question Q will make the person training me happiest?” By definition, this new question is also accessible, so it can be posed to M.

BAD may be much simpler than M\\*, because all it has to do is describe this substitution, whereas M\\* needs to build new machinery to explain inaccessible facts.

If M is very weak, BAD may perform much worse than M\\* because it doesn’t understand the loss function very well. But if M understands a lot about the world, and about its training process, then it seems like BAD will get at least as low a loss as M\\* — it’s using the same machinery as M and M\\*, and picking whatever value leads to the lowest loss. It can choose to report its own best estimates if that’s the best strategy, e.g. for questions where we can check. So by the time M and M\\* are human level, it already seems like BAD should be getting as low a loss as M\\*.

This gets a little bit more complicated if I do consistency checks or if I have large output spaces, and of course humans aren’t \*that\* good at lying, but it seems like the basic idea is pretty solid: once we have a model of accessible information, we can directly turn it into a model that will pass whatever checks we throw at it.

For concreteness, imagine M is a model of Alice’s behavior and we are asking it “is Alice actually happy?” Then M\\* is the model which translates its model of Alice’s psychology into words and explains it to us, while BAD is the model that asks “What answer would sound most believable to the person asking the question?”

The question is: if we start with model M and fine-tune it to answer inaccessible questions, is it easier to learn to articulate what it believes or is it going to learn to answer strategically?

We need to be \*very\* careful about our checks if we want to ensure that BAD doesn’t get a lower loss than M\\*. For example, it may turn out that sometimes the truth looks a little bit wrong to us…. And if we do everything right, then M\\* and BAD perform equally well, and so we may not have much control over which one we get.

IV. When inaccessible info is a safety problem
==============================================

Let’s get a bit more detailed about the argument in section II. I think that our inability to access inaccessible info would become a safety problem when:

1. We care about inaccessible facts, so we can’t just evaluate plans based on their accessible consequences.
2. Inaccessible info is a competitive advantage — agents who are blind to inaccessible facts about the world will get outcompeted.
3. There are \*some\* agents who are able to use inaccessible facts to acquire influence, e.g. because they are optimizing accessible long-term goals.

1. We care about inaccessible facts
===================================

If I only cared about accessible facts, then I might not need to ever access inaccessible facts. For example, if I cared about my life expectancy, and this was accessible, then I could ask my AI “what actions lead to me living the longest?” and execute those.

For better or worse, I think we are likely to care about inaccessible facts.

\* Generally we care about what’s \*actually happening\* and not just what appears to be happening. We don’t want smiling faces on cameras. And if there’s a lot of inaccessible action in the world, then it’s reasonably likely for accessible indicators to be systematically manipulated by inaccessible forces.
\* We care intrinsically about what happens inside people’s heads (and inside computers), not just outward appearances. Over the very long term a \*lot\* may happen inside computers.
\* If we totally give up on measuring how well things are going day-to-day, then we need to be actually optimizing the thing we really care about. But figuring that out may require reflecting a long time, and may be inaccessible to us now. We want a world where we actually reach the correct moral conclusions, not one where we believe we’ve reached the correct moral conclusions.
\* Our real long-term priorities, and our society’s long-term future, may also be really weird and hard to reason about even if we were able to know what was good. It just seems really bad to try to evaluate plans only by their very long-term consequences.
\* We care about things that are far away in space or time, which I think are likely to be inaccessible.

Overall I’m quite skeptical about the strategy “pick an accessible quantity that captures everything you care about and optimize it.” I think we basically need to optimize some kind of value function that tells us how well things are going. That brings us to the next section.

2. Inaccessible info is a competitive advantage
===============================================

Instead of using AI to directly figure out whether a given action will lead to human flourishing over the coming centuries, we could use AI to help us figure out how to get what we want over the short term — including how to acquire resources and flexible influence, how to keep ourselves safe, and so on.

This doesn’t require being able to tell how good a very long-term outcome is, but it does require being able to tell how well things are going. We need to be able to ask the AI “which plan would put us in an \*actually good\* position next year?”

Unfortunately, I think that if we can only ask about accessible quantities, we are going to end up neglecting a bunch of really important stuff about the situation, and we’ll be at a significant competitive disadvantage compared to AIs which are able to take the whole picture into account.

As an intuition pump, imagine a company that is run entirely by A/B tests for metrics that can be easily checked. This company would burn every resource it couldn’t measure — its code would become unmaintainable, its other infrastructure would crumble, it would use up goodwill with customers, it would make no research progress, it would become unable to hire, it would get on the wrong side of regulators…

My worry is that inaccessible facts will be similarly critical to running superhuman businesses, and that humans who rely on accessible proxies will get outcompeted just as quickly as the company that isn’t able to optimize anything it can’t A/B test.

\* Even in areas like business that society tries particularly hard to make legible, evaluating how well you are doing depends on e.g. valuing intellectual property and intangible assets, understanding contractual relationships, making predictions about what kinds of knowledge or what relationships will be valuable, and so on.
\* . In domains like social engineering, biology, cybersecurity, financial systems, \*etc.\*, I think inaccessible information becomes even more important.
\* If there is a lot of critical inaccessible information, then it’s not clear that a simple proxy like “how much money is actually in my bank account” is even accessible. The only thing that I can directly check is “what will I see when I look at my bank account statement?”, but that statement could itself be meaningless. We really care about things like who effectively controls that bank account and what would really happen if I tried to spend the money. (And if I largely care about inaccessible facts about the world, then “what would happen if I tried to spend my money?” may itself be inaccessible.)
\* I can pay inaccessible costs for an accessible gain — for example leaking critical information, or alienating an important ally, or going into debt, or making short-sighted tradeoffs. Moreover, if there are other actors in the world, they can try to get me to make bad tradeoffs by hiding real costs.

3. Some AIs can plan with inaccessible info
===========================================

So far this discussion could just be about an \*AI missed opportunity\*, not an \*AI risk\*.

Things become problematic when it is possible to build AI systems that do use inaccessible info to pursue ambitious long-term goals that would conflict with human flourishing. If illegible knowledge is important enough, those systems could outcompete humans and divert some (or almost all) of our civilization’s resources.

This happens if \*any\* interesting long-term goal is accessible, i.e. if there’s any accessible goal that benefits from accumulating influence.

Why might some long-term goal be accessible?

\* Verifiable long-term predictions may accessible, e.g. “total reward over the next 50 years” might be accessible as a generalization of “total reward over the next 50 minutes.”
\* Even if models don’t transfer in a natural way to very long time horizons, they may still transfer in \*some\* way. Maybe they can’t generalize to something as cleanly specified as “total reward,” but e.g. they may be able to predict quantities that happen to be natural in terms of the representation they are using.
\* More generally, models will do \*something\* when we run them way out of distribution. I think the best reason to expect models to not plan intelligently over long time horizons is that they lack the abilities and knowledge. Once they have relevant cognitive machinery, I think it’s pretty likely that they’ll use the machinery somehow when they go out of distribution, and if you search over a lot of possibilities.

The key asymmetry working against us is that optimizing flourishing appears to require a particular quantity to be accessible, while danger just requires \*anything\* to be accessible.

There’s a longer discussion about whether anyone would want to build such AI systems, or might be likely to do it accidentally. And another discussion about how bad it would actually be. But I think that once we satisfy the 3 properties in this section then we have a real problem, and the remaining question is one of magnitude, with a bunch of factors pulling up and down.

V. Possible responses
=====================

At a high level, we’d like to get good answers by training a model to make checkable predictions and then somehow incentivizing it to use the same machinery to also report inaccessible facts, for example by forcing it to report views and enforcing consistency.

I think there are a bunch of problems with that plan, but the strongest one is that it’s totally unclear how we ever get the intended policy rather than the instrumental policy.

Here are some possible angles of attack:

\* In some intuitive sense there is a discontinuity between saying X because you believe it and saying X because you believe that it’s what you need to say to get a high reward. If weak policies are straightforward rather than instrumental, then maybe we can exploit continuity to avoid the instrumental strategy.
I’m generally pretty scared of these approaches since they feel pretty brittle, and people seem prone to being optimistic even before we can test or understand anything.
But I’m a bit more hopeful if we consider continuity of behavior rather than continuity in the space of policies, and if we exploit it by using a model at time T to help check the behavior of a model at time T+1.
\* If we were able to actually understand something about what the policy was doing, even crudely, it might let us discriminate between instrumental and intended behavior. I don’t think we have any concrete proposals for how to understand what the policy is doing well enough to make this distinction, or how to integrate it into training. But I also don’t think we have a clear sense of the obstructions, and I think there are various obvious obstructions to interpretability in general that \*don’t\* apply to this approach. So together with the other points on this list — especially the existence of a discontinuity and the use of previous versions of a model to help — I think it could be part of a solution.
\* The “instrumental” strategy — “pick the answer that minimizes my training loss” — may have a shorter description, but it seems to be unnecessarily computationally complex. First it figures out what’s true, and then it strategically decides what to say in light of that. It would be a bit cheaper just to actually report what’s true, if we set up the training process well enough that honest reporting got you optimal reward.
It’s not clear how to leverage this though, since “use the simplest model” seems like an important and useful prior. We can’t just replace it with “use the cheapest model” without sacrificing competitiveness. That said, it may be able to buy us something in combination with the previous approaches.
(See discussion [here](https://www.lesswrong.com/posts/nyCHnY7T5PHPLjxmN/open-question-are-minimal-circuits-daemon-free) and [here](https://www.alignmentforum.org/posts/fM5ZWGDbnjb7ThNKJ/are-minimal-circuits-deceptive).)
\* We aren’t able to check inaccessible answers, and they aren’t output by the simplest model, but maybe we can make being straightforward the easiest strategy in some other sense. For example, if you have two models who can’t interact but need to agree on a consistent story, you could imagine the truth being the easiest thing for them to coordinate on. I think that scheme in particular is pretty unlikely to work, but there is a large space of possibilities here, especially when combined with the other points on this list. For example, maybe you could perform consistency checks between models at different points in time.
\* Maybe we can work with models M that don’t understand enough about the world, or especially about their training process, in order to turn into the instrumental policy BAD. I tend to be pretty skeptical of “security by obscurity” — if we want to make life hard for the instrumental policy not working, I want it to be because we have randomness hidden from it, or we’ve constructed a smarter agent that it can’t simulate. I think it’s pretty bad news if we are relying on a smart model not understanding facts about its own situation, and that’s the kind of solution that is particularly likely to break down over time.
\* We could hope that our models don’t build up important inaccessible knowledge, e.g. because it’s possible to justify most interesting conclusions with deduction or because we have reasonably good accessible proxies for our value function. I’m pretty skeptical about this over the long term, but I’m not sure exactly how bad it will be how early.
\* The argument in this post is pretty informal, and there’s a reasonable chance that you can drive a solution through one of the many gaps/loopholes. I like the problem-solving strategy: “write out the proof that there is no solution, and then sift through the proof looking for a fatal hole.”

Overall I don’t see an obvious way forward on this problem, but there are enough plausible angles of attack that it seems exciting to think about.

VI. How this relates to amplification and debate
================================================

Overall I don’t think it’s very plausible that amplification or debate can be a [scalable](/scalable-ai-control-7db2436feee7) AI alignment solution on their own, mostly for the kinds of reasons discussed in this post — we will eventually run into some inaccessible knowledge that is never produced by amplification, and so never winds up in your distilled agents.

In the language of my [original post on capability amplification](/policy-amplification-6a70cbee4f34), the gap between accessible and inaccessible knowledge corresponds to an obstruction. The current post is part of the long process of zooming in on a concrete obstruction, gradually refining our sense of what it will look like and what our options are for overcoming it.

I think the difficulty with inaccessible knowledge is not specific to amplification — I don’t think we have any approach that moves the needle on this problem, at least from a theoretical perspective, so I think it’s a plausible candidate for a [hard core](/hard-core-subproblems-8948463455ef) if we fleshed it out more and made it more precise. (I would describe MIRI’s approach to this problem could be described as despair + hope you can find some other way to produce powerful AI.)

I think that iterated amplification \*does\* address some of the most obvious obstructions to alignment — the possible gap in speed / size / experience / algorithmic sophistication / etc. between us and the agents we train. I think that having amplification mind should make you feel a bit less doomed about inaccessible knowledge, and makes it much easier to see where the real difficulties are likely to lie.

But there’s a significant chance that we end up needing ideas that look totally different from amplification/debate, and that those ideas will obsolete most of the particulars of amplification. Right now I think iterated amplification is by far our best concrete alignment strategy to scale up, and I think there are big advantages to starting to scale something up. At the same time, it’s really important to push hard on conceptual issues that could tell us ASAP whether amplification/debate are unworkable or require fundamental revisions.


Informed oversight

(revisited)
-----------

[![Paul Christiano](https://miro.medium.com/v2/resize:fill:88:88/1\*BNjZCuQuRfIgcXCBMipuBw.jpeg)](https://paulfchristiano.medium.com/?source=post\_page-----18fcb5d3d1e1--------------------------------)[![AI Alignment](https://miro.medium.com/v2/resize:fill:48:48/1\*N56Qc5-aHTcfGff0scntKQ.png)](https://ai-alignment.com/?source=post\_page-----18fcb5d3d1e1--------------------------------)[Paul Christiano](https://paulfchristiano.medium.com/?source=post\_page-----18fcb5d3d1e1--------------------------------)

·[Follow](https://medium.com/m/signin?actionUrl=https%3A%2F%2Fmedium.com%2F\_%2Fsubscribe%2Fuser%2F57f1a655a613&operation=register&redirect=https%3A%2F%2Fai-alignment.com%2Finformed-oversight-18fcb5d3d1e1&user=Paul+Christiano&userId=57f1a655a613&source=post\_page-57f1a655a613----18fcb5d3d1e1---------------------post\_header-----------)

Published in[AI Alignment](https://ai-alignment.com/?source=post\_page-----18fcb5d3d1e1--------------------------------)·8 min read·Jan 10, 2019--

1

Listen

Share

To train an agent with RL, we need to answer the question: “which of the trajectories τ¹ or τ² is better?”

Likewise, in order to train for acceptable worst-case performance, we probably need to answer a question like: “is trajectory τ acceptable?”

We want to answer these questions “well enough” that our agents end up aiming for good outcomes on average and always behaving acceptably. But how well is well enough?

In this post I’ll argue that if our goal is to train a well-motivated agent, it is necessary and sufficient for our answers to reflect \*everything our agent knows\*. This seems essentially equivalent to [ascription universality](/towards-formalizing-universality-409ab893a456) of the oversight process.

(This post is an update of [these](/adequate-oversight-25fadf1edce9) [2016 posts](/the-informed-oversight-problem-1b51b4f66b35). My understanding of the problem has improved a lot over the last few years.)

Information vs incentives
-------------------------

We can distinguish two functions of feedback:

\* It provides \*information\* to the agent we are training. For example, if I provide negative feedback when an agent serves me pizza, it can learn that I don’t like pizza. (This might happen by gradient descent selecting for agents with different beliefs, or by an RL algorithm that uses past reward signals as information.)
\* It determines the \*incentives\* for the agent we are training, i.e. is used to select the motivations of the agent. For example, an agent pre-trained to make predictions about the world might already understand that I dislike pizza. If I provide negative feedback when the agent serves me pizza, then gradient descent will select for agents that are motivated to avoid things I dislike (at least instrumentally).

From the perspective of information, there is no such thing as “good enough.” More information is always better.

This post is about incentives. We can aim for a “correctly motivated” agent, who lacks relevant information but is at least [trying](/clarifying-ai-alignment-cec47cd69dd6) to do the right thing. At that point there is nothing to gain from better oversight.

So our question is: how well do we have to answer these questions to produce an approximately correctly motivated agent?

(In the long run, I think that incentives are the more important function of feedback. Agents should mostly learn about the world from other richer signals. So I’m not worried about whether our feedback is optimal from an information perspective.)

Necessity of knowing what the agent knows
-----------------------------------------

Suppose that our agent understands facts about the world that we don’t understand, i.e which aren’t reflected in our answer to “which of the trajectories τ¹ or τ² is better?” I’ll argue that this is a bad situation.

If the agent literally took actions and made observations that weren’t available to us, then we would clearly have a problem. For example, consider an agent trying to defend a computer system from attackers. If the agent has access to actions or information we are missing, then it might deliberately take actions that compromise the tools we would use to determine whether our system has been compromised. Because we can’t observe this malicious behavior, we can’t assign it a low reward. And after compromising our tools well enough, the agent could ensure that they always paint a rosy picture of the situation regardless of what actually happens. We intended to protect ourselves from attack, but instead introduced a powerful attacker with privileged access to our system.

The same thing can happen if we observe everything our agent observes, if we aren’t able to understand everything our agent understands. In the security example, literally seeing a sequence of bits moving across an interface gives you almost no information — something can look innocuous, but cause a huge amount of trouble. In order to incentivize our agent to avoid causing trouble, we need to be able to detect any trouble that the agent deliberately causes. Even an apparently mundane gap in our understanding could hide attacks, just as effectively as if we’d been literally unable to observe the agent’s behavior.

We might hope that an attack will ultimately cash out as \*something\* that we can unambiguously observe, so that this is merely a problem with short time horizons. But I don’t think that’s the case — ultimately we are always restricted to indirect observations of the things we care about. We rely on second-hand reports, on viewing small parts of a situation that we hope are representative, on cameras and other sensors that purport to describe the world. All of these things would also be manipulated by an agent incentivized to make things look good to us, and understanding what the agent understands is the most natural defense against that kind of manipulation.

Intuitive argument of sufficiency
---------------------------------

Suppose we “know everything that the agent knows.” That means the agent should defer to our estimate for any unknown quantity.

Now consider two possible motives an agent could have:

\* Create trajectories that are actually good, according to our “real” preferences ([extrapolated as we wish that extrapolated](https://intelligence.org/files/CEV.pdf)…)
\* Create trajectories we believeare good

If the agent defers to us about the goodness of trajectories, then these two motives will lead to identical actions. So selecting for agents that take actions that we think are good also optimally selects for agents actually motivated to do good.

Intuitively, imagine a student who is trying to predict whether their teacher will think a statement is true. If the teacher knows everything the student knows and can see every consideration the student sees, then the student shouldn’t expect to be able to predict any systematic error in the teacher’s beliefs. So trying to predict what the teacher believes is exactly the same as trying to predict what’s true.

Tightening the argument
-----------------------

The argument in the last section isn’t quite right. Most glaringly, if the overseer is universal then it means \*we\* think the overseer has better beliefs than the agent, but it doesn’t mean \*the agent\* thinks that.

\*\*Changing the statement\*\*. To fix the argument, we need to be a bit more careful. We’ll aim to show that if we think the overseer is ascription universal, then:

\* We expect the agent A to achieve at least as much utility (by our real, unobserved utility function) than any action that can be easily computed using any set of beliefs that can be ascribed to A — and in particular A achieves at least as much utility as if it were perfectly motivated.

We can’t infer that A is perfectly motivated, because it may have arbitrary beliefs that differ from those of the overseer. For example, A might believe that the world will certainly end in a week; instead of planning under that assumption (as it would if it were “correctly” motivated), it will learn to choose actions that the overseer prefers. From its perspective it is choosing actions suboptimally in order to please the overseer, but from our perspective this is better than if it were “correctly” motivated.

Another way to see we need this weakening: many different believes can be ascribed to the agent, and it can’t simultaneously behave as if it were perfectly motivated with respect to each of those sets of beliefs.

\*\*Assumptions and proof\*\*. This will mostly just be notation; other than that it follows almost immediately from the definition of ascription universality, though we need to do a little bit of work to massage the definition into the right form.

Let 𝔼 be any subjective expectation w.r.t. which the overseer is ascription universal.

Let 𝔼ᴼ be the expectations of the overseer. Let U be our actual utility — this is not a well-defined function we can actually evaluate, but we can query a subjective expectation about it. Then the reward function is \*r\*(\*a\*) = 𝔼ᴼ[U(\*a\*)]. Note that the overseer doesn’t condition on the fact that the agent produced this particular action (so no one needs to reason about the training process itself).

Let 𝔼ᴬ be some expectations that can be ascribed to the agent. Let A be the expression representing the agent’s output (\*\*after training\*\*), and let \*a\* be the actual output (we need to distinguish the symbol from the action itself because we are plugging this into a subjective expectation). Let A′ be a particular different definition of an action the agent could feasibly output, e.g. argmax 𝔼ᴬ[𝔼ᴼ[U(·)]], and let \*a\*′ be its actual value.

Because A and A′ can both be computed using the beliefs 𝔼ᴬ, we can apply the epistemic dominance property to predicates defined in terms of A, A′. (There is the subtlety that dominance requires properties to be open in the product topology, which can be avoided by carefully considering randomization and only becomes relevant when the utility function U itself depends on the behavior of the agent.)

We’ll assume that 𝔼 believes that training converges well enough that the action A will receive a higher expected reward than A′ (otherwise the agent would eventually switch to outputting A′). We also assume that 𝔼 knows that the overseer epistemically dominates the agent, and that it satisfies a standard [iterated expectation](https://en.wikipedia.org/wiki/Law\_of\_total\_expectation) property.

Then we have:

\* 𝔼[\*a ←\* A; \*a\*′ ← A′; U(\*a\*) − U(\*a\*′)]
= 𝔼[\*a ←\* A; \*a\*′ ← A′; 𝔼[U(\*a\*) − U(\*a\*′) | \*a\*=A, \*a\*′=A′]] (\*iterated expectation\*)
= 𝔼[\*a ←\* A; \*a\*′ ← A′; 𝔼[𝔼ᴼ[U(\*a\*) − U(\*a\*′)] | \*a\*=A, \*a\*′=A′]] (\*dominance\*)
= 𝔼[\*a ←\* A; \*a\*′ ← A′; 𝔼ᴼ[U(\*a\*) − U(\*a\*′)]] (\*iterated expectation\*)
= 𝔼[\*a ←\* A; \*a\*′ ← A′; \*r\*(\*a\*) − \*r\*(\*a\*′)]] (\*definition of reward\*)
≥ 0 (\*training is converged\*)

As desired.

(The meaning of the 𝔼[…;…] notation is to take the expected value of the output of the program which performs the sequence of steps separated by ;’s, returning the final value.)

This argument only constrains the equilibrium, but eventually we need to say something about the entire training process. It looks to me like very similar ideas are needed for the more general case, but it’s a lot harder to make the arguments precise.

Corrigibility
-------------

So far we’ve discussed reward functions that induce correct motivations. The other half of our problem was recognizing unacceptably bad behavior: mistakes so bad that they cannot be corrected.

Again, we can’t hope to recognize all such mistakes. For example, it may turn out that a particular physics experiment will destroy the world, but correctly predicting that would require a detailed understanding of physics.

But if we know everything the agent knows, then we can avoid cases where the agent intentionally brings about a catastrophic outcome. In particular, we can avoid cases where the agent takes an action precisely because it would prevent us from correcting the agent’s behavior or reasserting control over the situation.

Ascription universality seems to be just what we need for this purpose as well: if we believe the agent is pursuing a plan \*because\* it prevents us from correcting the situation, then we can ascribe that belief to the agent, and so an ascription-universal overseer will recognize that the behavior is incorrigible and can deem the behavior as unacceptable.

Such an overseer may also sometimes flag as unacceptable certain actions that are catastrophic but which the agent didn’t recognize as catastrophic. But I think that’s fine, I’m not unhappy if we end up training the agent to avoid a catastrophic mistake.

What does this actually look like?
----------------------------------

If we are training a model with RL, my best candidate for obtaining a universal overseer is using [iterated amplification](https://arxiv.org/abs/1810.08575) to train a second head on the agent to answer relevant questions, including questions of the form “Which of these two trajectories is better?”

There are difficulties from training a model to play a game where it shares almost all of its activations with its opponent, and with ensuring that the overseer can remain sufficiently competent relative to the agent (without slowing down training). These resemble the discussion of the `Info` proposal [here](/the-informed-oversight-problem-1b51b4f66b35), but rather than having a separate overseer who evaluates how useful information was, we directly optimize the side information by using the same amplification process. Both of these issues seem OK with gradient descent, but they are larger problems for some other search algorithms, so I don’t consider the issue settled.

This proposal also faces all of the usual difficulties of iterated amplification — it’s not clear how to generate the distribution of questions, whether the training process is stable, whether errors compound, whether the steps of amplification increase the complexity of training, and so on. These issues seem practically important, and keeping informed oversight in mind can help us understand exactly where the bar is for an adequate solution. I tentatively think the conceptual difficulties in achieving ascription universality are more likely to be a fundamental obstruction.

Introduced in [this older post](https://ai-alignment.com/the-informed-oversight-problem-1b51b4f66b35), we consider a setting in which an overseer is training an agent, and the overseer is smarter than the agent. Intuitively, this should work out well, because if the agent ever tries to do something bad, the overseer should notice this (since it is smarter), and penalize it. However, the problem is that the agent might _in the course of computing an answer_ come up with considerations that the overseer can't easily find. For example, an agent tasked with creating novel literature might end up plagiarizing from one book; the overseer may not be able to search through all the books to detect the plagiarism. However, if the overseer is universal, then it should also know that the agent's output is plagiarized from the book (because the agent knows that), and it can penalize it appropriately.

We can even construct a semi-formal argument (see the post for the math). Let us assume that there is some true utility function U, and when the agent takes some action a, the overseer provides it with reward r = E[U(a)], where the expectation is relative to the overseer's beliefs. Let us assume away problems of distribution shift: in particular, we'll assume that when choosing between actions a1 and a2, the agent always chooses the action with higher reward. Now suppose that the agent chooses action a1. We can conclude that a1 has higher reward than a2. This means that the observer believes that a1 has higher true utility than a2. But by universality, we always trust the observer over the agent and ourselves, and so we must also believe that a1 has higher true utility than a2, which means that we believe the agent is choosing the right action.

This application of universality is the one I believe the most; even intuitively it seems like the entire informed oversight problem is that the overseer _doesn't_ know everything that the agent knows, and universality straightforwardly fixes that. So I see this as the motivating example for caring about universality at all.


Learning the prior

\*(\*[\*Beth’s post on imitative generalization\*](https://www.alignmentforum.org/posts/JKj5Krff5oKMb8TjT/imitative-generalisation-aka-learning-the-prior-1) \*is a much clearer description of the unnamed algorithm in this post, which I’m calling “Imitative Generalization.”)\*

Suppose that I have a dataset D of observed (\*x\*, \*y\*) pairs, and I’m interested in predicting the label \*y\*\\* for each point \*x\*\\* in some new set D\\*. Perhaps D is a set of forecasts from the last few years, and D\\* is a set of questions about the coming years that are important for planning.

The classic deep learning approach is to fit a model \*f\* on D, and then predict \*y\*\\* using \*f\*(\*x\*\\*).

This approach implicitly uses a somewhat strange prior, which depends on exactly how I optimize \*f\*. I may end up with the model with the smallest l2 norm, or the model that’s easiest to find with SGD, or the model that’s most robust to dropout. But \*none\* of these are anywhere close to the “ideal” beliefs of a human who has updated on D.

This means that neural nets are unnecessarily data hungry, and more importantly that they can generalize in an undesirable way. I now think that this is a safety problem, so I want to try to attack it head on by learning the “right” prior, rather than attempting to use neural nets as an implicit prior.

Warm-up 1: human forecasting
----------------------------

If D and D\\* are small enough, and I’m OK with human-level forecasts, then I don’t need ML at all.

Instead I can hire a human to look at all the data in D, learn all the relevant lessons from it, and then spend some time forecasting \*y\*\\* for each \*x\*\\*.

Now let’s gradually relax those assumptions.

Warm-up 2: predicting human forecasts
-------------------------------------

Suppose that D\\* is large but that D is still small enough that a human can extract all the relevant lessons from it (or that for each \*x\*\\* in D\\*, there is a small subset of D that is relevant).

In this case, I can pay humans to make forecasts for many randomly chosen \*x\*\\* in D\\*, train a model \*f\* to predict those forecasts, and then use \*f\* to make forecasts about the rest of D\\*.

The generalization is now coming entirely from human beliefs, not from the structural of the neural net — we are only applying neural nets to iid samples from D\\*.

Learning the human prior
========================

Now suppose that D is large, such that a human can’t update on it themselves. Perhaps D contains billions of examples, but we only have time to let a human read a few pages of background material.

Instead of learning the unconditional human forecast P(\*y\*|\*x\*), we will learn the forecast P(\*y\*|\*x,\* Z), where Z is a few pages of background material that the human takes as given. We can also query the human for the prior probability Prior(Z) that the background material is true.

Then we can train \*f\*(\*y\*|\*x\*, \*Z\*) to match P(\*y\*|\*x\*, Z), and optimize Z\\* for:

> \*log Prior(Z\\*) + sum((\*x\*,\* y\*) ~ D) log f(\*y\*|\*x, \*Z\\*)\*
>
>

We train \*f\* in parallel with optimizing Z\\*, on inputs consisting of the current value of Z\\* together with questions \*x\* sampled from D and D\\*.

For example, Z might specify a few explicit models for forecasting and trend extrapolation, a few important background assumptions, and guesses for a wide range of empirical parameters. Then a human who reads Z can evaluate how plausible it is on its face, or they can take it on faith in order to predict \*y\*\\* given \*x\*\\*.

The optimal Z\\* is then the set of assumptions, models, and empirical estimates that works best on the historical data. The human never has to reason about more than one datapoint at a time — they just have to evaluate what Z\\* implies about each datapoint in isolation, and evaluate how plausible Z\\* is a priori.

This approach has many problems. Two particularly important ones:

\* To be competitive, this optimization problem needs to be nearly as easy as optimizing \*f\* directly on D, but it seems harder: finding Z\\* might be much harder than learning \*f,\* learning a conditional \*f\* might be much harder than learning an unconditional \*f\*, and jointly optimizing Z and \*f\* might present further difficulties.
\* Even if it worked our forecasts would only be “human-level” in a fairly restrictive sense — they wouldn’t even be as good as a human who actually spent years practicing on D before making a forecast on D\\*. To be competitive, we want the forecasts in the iid case to be at least as good as fitting a model directly.

I think the first point is an interesting ML research problem. (If anything resembling this approach ever works in practice, credit will rightly go to the researchers who figure out the precise version that works and resolve those issues, and this blog post will be a footnote.) I feel relatively optimistic about our collective ability to solve concrete ML problems, unless they turn out to be impossible. I’ll give some preliminary thoughts in the next section “Notes & elaborations.”

The second concern, that we need some way to go beyond human level, is a central philosophical issue and I’ll return to it in the subsequent section “Going beyond the human prior.”

Notes & elaborations
--------------------

\* Searching over long texts may be extremely difficult. One idea to avoid this is to try to have a human guide the search, by either generating hypotheses Z at random or sampling perturbations to the current value of Z. Then we can fit a generative model of that exploration process and perform search in the latent space (and also fit \*f\* in the latent space rather than having it take Z as input). That rests on two hopes: (i) learning the exploration model is easy relative to the other optimization we are doing, (ii) searching for Z in the latent space of the human exploration process is strictly easier than the corresponding search over neural nets. Both of those seem quite plausible to me.
\* We don’t necessarily need to learn \*f\* everywhere, it only needs to be valid in a small neighborhood of the current Z. That may not be much harder than learning the unconditional \*f\*.
\* Z represents a full posterior rather than a deterministic “hypothesis” about the world, e.g. it might say “R0 is uniform between 2 and 3.” What I’m calling Prior(Z) is really the KL between the prior and Z, and P(\*y|x,\*Z) will itself reflect the uncertainty in Z. The motivation is that we want a flexible and learnable posterior. (This is particularly valuable once we go beyond human level.)
\* This formulation queries the human for Prior(Z) before each fitness evaluation. That might be fine, or you might need to learn a predictor of that judgment. It might be easier for a human to report a ratio Prior(Z)/Prior(Z′) than to give an absolute prior probability, but that’s also fine for optimization. I think there are a lot of difficulties of this flavor that are similar to other efforts to learn from humans.
\* For the purpose of studying the ML optimization difficulties I think we can basically treat the human as an oracle for a reasonable prior. We will then need to relax that rationality assumption in the same way we do for other instances of learning from humans (though a lot of the work will also be done by our efforts to go beyond the human prior, described in the next section).

Going beyond the human prior
============================

How do we get predictions better than explicit human reasoning?

We need to have a richer latent space Z, a better Prior(Z), and a better conditional P(\*y\*|\*x\*, Z).

Instead of having a human predict \*y\* given \*x\* and Z, we can use amplification or debate to train f(\*y\*|\*x\*, Z) and Prior(Z). This allows Z to be a large object that cannot be directly accessed by a human.

For example, Z might be a full library of books describing important facts about the world, heuristics, and so on. Then we may have two powerful models debating “What should we predict about \*x\*, assuming that everything in Z is true?” Over the course of that debate they can cite small components of Z to help make their case, without the human needing to understand almost anything written in Z.

In order to make this approach work, we need to do a lot of things:

1. We still need to deal with all the ML difficulties described in the preceding section.
2. We still need to analyze debate/amplification, and now we’ve increased the problem difficulty slightly. Rather than merely requiring them to produce the “right” answers to questions, we also need them to implement the “right” prior. We already needed to implement the right prior as part of answering questions correctly, so this isn’t too much of a strengthening, but we are calling attention to a particularly challenging case. It also imposes a particular structure on that reasoning which is a real (but hopefully slight) strengthening.
3. Entangled with the new analysis of amplification/debate, we also need to ensure that Z is able to represent a rich enough latent space. I’ll discuss implicit representations of Z in the next section “Representing Z.”
4. Representing Z implicitly and using amplification or debate may make the optimization problem even more difficult. I’ll discuss this in the subsequent section “Jointly optimizing Mz and f.”

Representing Z
--------------

I’ve described Z as being a giant string of text. If debate/amplification work at all then I think text is in some sense “universal,” so this isn’t a crazy restriction.

That said, representing complex beliefs might require \*very long\* text, perhaps many orders of magnitude larger than the model \*f\* itself. That means that optimizing for (Z, \*f\*) jointly will be much harder than optimizing for \*f\* alone.

The approach I’m most optimistic about is representing Z implicitly as the output of another model Mz. For example, if Z is a text that is trillions of words long, you could have Mz output the \*i\*th word of Z on input \*i\*.

(To be really efficient you’ll need to share parameters between \*f\* and Mz but that’s not the hard part.)

This can get around the most obvious problem — that Z is too long to possibly write down in its entirety — but I think you actually have to be pretty careful about the implicit representation or else we will make Mz’s job too hard (in a way that will be tied up the competitiveness of debate/amplification).

In particular, I think that representing Z as implicit flat text is unlikely to be workable. I’m more optimistic about the kind of approach described in [approval-maximizing representations](/approval-maximizing-representations-56ee6a6a1fe6) — Z is a complex object that can be related to slightly simpler objects, which can themselves be related to slightly simpler objects… until eventually bottoming out with something simple enough to be read directly by a human. Then Mz implicitly represents Z as an exponentially large tree, and only needs to be able to do one step of unpacking at a time.

Jointly optimizing Mz and f
---------------------------

In the first section I discussed a model where we learn \*f\*(\*y\*|\*x\*, Z) and then use it to optimize Z. This is harder if Z is represented implicitly by Mz, since we can’t really afford to let \*f\* take Mz as input.

I think the most promising approach is to have Mz and \*f\* both operate on a compact latent space, and perform optimization in this space. I mention that idea in Notes & Elaborations above, but want to go into more detail now since it gets a little more complicated and becomes a more central part of the proposal.

(There are other plausible approaches to this problem; having more angles of attack makes me feel more comfortable with the problem, but all of the others feel less promising to me and I wanted to keep this blog post a bit shorter.)

The main idea is that rather than training a model Mz(·) which implicitly represents Z, we train a model Mz(·, \*z\*) which implicitly represents a distribution over Z, parameterized by a compact latent \*z.\*

Mz is trained by iterated amplification to imitate a superhuman exploration distribution, analogous to the way that we could ask a human to sample Z and then train a generative model of the human’s hypothesis-generation. Training Mz this way is itself an open ML problem, similar to the ML problem of making iterated amplification work for question-answering.

Now we can train \*f\*(\*y|x, z\*) using amplification or debate. Whenever we would want to reference Z, we use Mz(·, \*z\*). Similarly, we can train Prior(\*z\*). Then we choose \*z\\*\* to optimize log Prior(\*z\*\\*) + sum((\*x\*, \*y\*) ~ D) log \*f\*(\*y|x, z\*\\*).

Rather than ending up with a human-comprehensible posterior Z\\*, we’ll end up with a compact latent \*z\*\\*. The human-comprehensible posterior Z\\* is implemented implicitly by Mz(·, \*z\*\\*).

Outlook
=======

I think the approach in this post can potentially resolve the issue described in [Inaccessible Information](/inaccessible-information-c749c6a88ce), which I think is one of the largest remaining conceptual obstacles for amplification/debate. So overall I feel very excited about it.

Taking this approach means that amplification/debate need to meet a slightly higher bar than they otherwise would, and introduces a bit of extra philosophical difficulty. It remains to be seen whether amplification/debate will work at all, much less whether they can meet this higher bar. But overall I feel pretty excited about this outcome, since I was expecting to need a larger reworking of amplification/debate.

I think it’s still very possible that the approach in this post can’t work for fundamental philosophical reasons. I’m not saying this blog post is anywhere close to a convincing argument for feasibility.

Even if the approach in this post is conceptually sound, it involves several serious ML challenges. I don’t see any reason those challenges should be impossible, so I feel pretty good about that — it always seems like good news when you can move from philosophical difficulty to technical difficulty. That said, it’s still quite possible that one of these technical issues will be a fundamental deal-breaker for competitiveness.

My current view is that we don’t have candidate obstructions for amplification/debate as an approach to AI alignment, though we have a lot of work to do to actually flesh those out into a workable approach. This is a more optimistic place than I was at a month ago when I wrote [Inaccessible Information](/inaccessible-information-c749c6a88ce).