counterfactual computations in world models
-------------------------------------------

how could we make an AI which, faced with a decision, [consults its past-user(s)](outer-alignment-past-user.html) about it, using something like the [question-answer counterfactual interval](qaci.html) device ? i believe that solving the following technical problem could significantly help us figure out how to do that.

suppose some relatively complex probabilistic `World`, for example one based on a [probabilistic cellular automaton](https://en.wikipedia.org/wiki/Stochastic_cellular_automaton).

we'll also posit two functions `f` and `g` and a small but non-empty set `S` such that

* `∀x∈S, f(x)=g(x)`
* `∃x∉S, f(x)≠g(x)`
* `f` is both [more complex](https://en.wikipedia.org/wiki/Kolmogorov_complexity) and slower than `g`
* `f` and `g` are both efficiently computable, though not trivial to compute

`f` is meant to represent the true decision process of the user of the AI, while `g` is an approximation of it which happens to explain all the empirical data we got from `f` — that empirical data being `{ (x,f(x)) | x∈S }`. `g` is meant to serve as a trap for AIs that would merely try to fit the empirical data without investigating the world to find the real `f`. if there are many possible functions which also output the same value as `f` on members of `S`, such that you have to look at the `World` a bunch before finding the real `f`, then that's a good reason to think that an AI that systematically finds the real `f` in various scenarios is doing what we want.

in `World`, we'll first arrange — either in advance or by interfering with `World` or with its source of randomness — for that world to encode an efficient implementation of `f` somewhere. this encoding will be called on all members of `S` in any order, and the record of these calls along with the results will be recorded somewhere.

then, still inside `World`, the encoded version of `f` will be scrambled — making recovering `f` from evidence non-trivial but doable with some work.

finally, the problem is the following: is there an `AI` which can learn `f` by predicting its result on values outside of `S`

* given the ability to interact with `World` after all of these events — perhaps by being [embedded](https://intelligence.org/embedded-agency/) in it
* given a record of the input-outputs pairs that `f` has gone through
* *without* direct access to `f` or `g`

the difficulty here is for `AI` to learn to predict `f`, when we're giving it the tempting hack of predicting `g` instead, as a sort of honeypot. i expect that, to do this, the `AI` would need to build a causal understanding of `World`, and then locate and reconstruct `f` from the evidence left around.

what are the inputs and outputs even for, if it has to dig around `World` to recover `f`? they're used for identifying `f` to begin with: `f` is *the thing which has gotten those inputs and produced those outputs*. the `AI` shouldn't think "i need to find a function which, given those inputs, gives those outputs" — as do, as i understand, all current ML systems — but one which goes "hm, apparently there was a physical thing somewhere which given those inputs, gave those outputs — what would it have, counterfactually, outputted given different inputs?" and it needs to do that without the ability to directly run `f`, such that it can't simply "be trained on `f`".

once we have this, we can hopefully start building an AI which is given a bunch of input-outputs pairs that have gone through human users, and then give it a decision process that relies on predicting what those users would have said given different queries, in order to make decisions.


QACI: question-answer counterfactual intervals
----------------------------------------------

***edit:** see also [**the QACI table of contents**](qaci.html).*

[PreDCA](predca.html) (not a dependency/component, but an inspiration for this) attempts to build a framework in which the AI tries to determine its predecessor's utility function using a bunch of math, to figure out who the user is and what their utility function is. it seems hard to predict whether the math would accurately capture a subset of the user's mind whose utility function we'd like, so in this post i offer an alternative which i feel has a higher chance of being useful.

just like PreDCA, the **question-answer counterfactual intervals** (QACI) proposal utilizes the ability for an AI to ponder counterfactuals of what can happen in the world, possibly using [infra-bayesian physicalism](https://www.lesswrong.com/posts/gHgs2e2J5azvGFatb/infra-bayesian-physicalism-a-formal-theory-of-naturalized). it proceeds as follows:

1. have the AI's user stand in front of a computer
2. the AI is hardcoded to first generate a large random text file, and send it to the user's computer — we call this file the "question"
3. the user opens the text file, ponders what it says for a day, and then at the end of the day sends a text file with its answer back to the AI
4. the AI, which was hardcoded to do literally nothing until it got that answer, starts running the rest of its code which would consist of an inner-aligned system following a particular goal

we can make the AI's goal dependent on "what answer would i have gotten if i'd sent a different question?" — we'll call "queries" such instances of counterfactually considering the question-answer interval. this doesn't immediately solve alignment, but just like [goal-program bricks](goal-program-bricks.html), it's a device that we can use to build more complex decision processes which would serve to guide the AI's actions.

note that the AI might run a relatively high detail simulation of what the user would answer, or it could just make rough guesses; if it's properly designed, it should allocate its computational resources to guessing what the user would answer to whatever degree of detail it needs. nevertheless, its choice would be guided by a fundamental intent to satisfy its eventual goal, so it shouldn't manipulate the simulation to give answers that would make its job easier — it would ultimately strive for however much accuracy it thinks it can afford.

and just like in PreDCA, because we make the AI point to a user who preceeds its existence, it can't (barring weird acausal stuff) hack the user to affect what it'd predict the user to say; the user's values are locked in, which [is desirable anyways](surprise-you-want.html).

here are some ideas as to how to use such queries to hopefully guide the AI's actions towards good worlds:

* we tell the AI to maximize the utility function that a sequence of queries would end at, where the first one is asked "what's a utility function that represents human values?" and each next query is asked to improve on the answer of the query, until one of them sends an answer starting with the [magic string](https://en.wikipedia.org/wiki/Magic_string) "okay AI, i'm done. the answer is:" followed by a piece of formal math which points to human values, such as a highly refined version of PreDCA.
* we tell the AI to, each time it's making a decision, have such a sequence determine which outcome it would prefer given observations by the AI so far — in a sense, extrapolating the volition of the user if they had a lot more time to ponder each decision.
* something like the above except it's the AI's own model which determines consequences, and sequences of queries are ran on that model, to figure out what it entails and *then* which action is preferable
* any of the suggestions above except sequences of queries are replaced with [DAGs](https://en.wikipedia.org/wiki/Directed_acyclic_graph) of queries, each able to say what kind of query graph they'd like to be ran — such as "i'd like fifty query sequences to ponder this question but with the following fifty different thought prompts, and then for a single query sequences to get all of their results and figure out the result

these ideas don't involve the AI interpreting natural language — the utility function could be written in trivially parseable python math code, decisions or requests for running multiple new copies could be asked for using magic strings followed by formal code, and so on. for example, in the case of a sequence of queries, the AI is told to predict what happens when the text file is just passed verbatim from one query to the next, until a particular magic string is detected verbatim at the start of a query.

notice that, because there is no inherent limit on the text file's size, it can start with [a `#!/bin/bash` shebang](https://en.wikipedia.org/wiki/Shebang_%28Unix%29) and be a script that builds a large piece of software that each query is able to develop and use to more efficiently transmit knowledge to the next query, for only very minimal overhead to each of those queries.

finally, this proposal should be not too difficult to expand upon:

* start with longer question-response intervals
* start with a large number of question-response intervals which are picked from at random, to select a wider range of "views" of the user
* start with many question-response intervals sent to a bunch of different people who can work together on alignment
* allow question-response intervals to communicate with one another, perhaps with as much as video chat, using a command that each user could theoretically send to the AI — but which either does nothing or isn't sent in the original, "real" question-response interval
* give question-response intervals access to a supercalculator, which would have the AI run computationally costly programs and send the result — again, such a capability would not be usable in the original "real" instances of the user answering a question

note that with a proper implementation of [embedded agency](https://www.lesswrong.com/tag/embedded-agency), such an AI would care about its own internal computations just as much as what happens in the rest of the world; so that if this scheme indeed leads to an aligned AI, then that alignment would cover taking care of risks caused by running simulations of queries in such a high level of detail that its inhabitant(s) are moral patients would. in fact, thanks to embedded agency, perhaps the whole "let sequences of queries decide how to make decisions" could apply naturally to how queries are used to make decisions.


some simulation hypotheses

*(thanks to [Alexander](https://www.lesswrong.com/users/self-embedded-agent) for conversations that led to this post)*

some simulation hypotheses
--------------------------

what a strange time to live in, right on the verge of building an AI which will dictate the fate of the cosmos for all of the future!

what a strange situation, that [we have a chance at all](weird-chance.html): instead of alignment or superintelligence being discovered many decades apart, we're arriving at them in a somewhat synchronous manner!

what a strange perspective, for me to be one of maybe a few hundred people whose work is directly related to this cosmos-defining event!

one way to explain making those strange observations is if this kind of [anthropic](https://www.lesswrong.com/posts/RnrpkgSY8zW5ArqPf/sia-greater-than-ssa-part-1-learning-from-the-fact-that-you) reasoning [occurs very disproportionately under these circumstances](anthropic-reasoning-coordination.html).

nevertheless, it is tempting to also consider something like the [simulation hypothesis](https://en.wikipedia.org/wiki/Simulation_hypothesis), which says that we are living inside an intentional simulation ran by some agent in a parent universe. i will list below a few such simulation hypotheses, that i can come up with or that i've come across.

### it's a game

**premise**: one hypothesis i've heard a bunch is that this time period and place is being simulated as part of a game for post-singularity people to experience living in the most important century in history, perhaps even by making themselves part of those events except without memories. so basically, most instances of these surroundings are a tourist attraction.

**what this would say about the parent universe**: if this hypothesis is true, that's good evidence that the post-singularity future is at least somewhat aligned to us, because it contains agents that find our world interesting. the fact that this world seems to be running in its entirety, even including the suffering moral patients, is not a good sign however. either those agents have found a way to make this okay — perhaps through making these seemingly suffering moral patients not count, for example using something like [moral patient deduplication](predictablizing-ethic-deduplication.html) — or the future has somewhat strong [S-risk](https://en.wikipedia.org/wiki/Suffering_risks) potential.

**what to expect observing if this is true?** we should expect our alignment chances to be neither overwhelmingly good nor bad, because those wouldn't be very interesting. maybe we should expect them to err on bad, though, as challenges can be enjoyable. the chance of various pivotal events, such as plagues or nuclear war, should be higher in this scenario because that seems interesting too; though if whoever's playing is embodied in a regular aging human body, our fate might be locked — or even our simulation terminated — not long after their avatar in this world dies.

**what should we do if this is true?** keep saving the world in case our simulation keeps running after our singularity, even just a bit. if we don't think this simulation keeps running after our singularity, and we suspect we inhabit a potentially-S-risky parent universe, then we should maybe favor [effective altruism](https://en.wikipedia.org/wiki/Effective_altruism) endeavors which alleviate suffering in the shorter term.

### superintelligence predicting superintelligences

**premise**: in order to predict what kind of other superintelligences exist out there, a superintelligence is simulating civilizations close to the point at which they spawn superintelligence to see what they'd tend to make, or to find the decryption key or initial state of a [homomorphically encrypted](https://en.wikipedia.org/wiki/Homomorphic_encryption) superintelligence that it has encountered. this could also explains why [we seem to have a chance](weird-chance.html), rather than our odds being overwhelmingly one way or the other: the more uncertain a scenario is, the more detail the superintelligence might need to run it, and so we experience the most uncertain scenarios possible. note that there might be nested simulations, where one superintelligence simulates another coming into existence. finally, this possibility includes ["deism"](solomonoff-deism.html), where one intelligence is/has dominion over its entire layer of reality from the start.

**what this would say about the parent universe**: this hypothesis being true does not say much; this kind of behavior seems [instrumentally convergent](https://en.wikipedia.org/wiki/Instrumental_convergence) to both aligned and unaligned superintelligence. i guess if we get to experience living as an instrumental side-effect that's kind of nice, but the S-risk concerns from the scenario above apply as well.

**what to expect observing if this is true?** we should see our odds of alignment being close to the knife's edge, because those are the situations that require the most computation-heavy simulations to determine the outcome of. ultimately, as our simulation is being ran for accuracy, we should expect to actually be the ones that determine what we build, and we should expect that outcome to matter — though probly not in any observable way.

**what should we do if this is true?** i think creating aligned superintelligence still has precedence; it *feels* like the more any given superintelligence expects that the universe is filled with superintelligences that carry our values, the more we increase the chances that our values apply to the universe at large. there may be weird reasons why this backfires, such as blackmail (acausal or not) between superintelligences; but in general, we'd expect superintelligences to have or [invent themselves](https://arbital.com/p/10qt/) a decision theory which would pre-commit to not succomb to blackmail — though see also [the game theory of blackmail](https://www.lesswrong.com/posts/wm2rdS3sDY9M5kpWb/the-game-theory-of-blackmail).

### indirect alignment solution

**premise**: it is possible that we have designed a superintelligence that is not directly aligned, but contains a process which we hope gets it there, similar to the situation described in [the insulated goal-program](insulated-goal-program.html). [simulating this world](finding-earth-ud.html) may be part of this process, somehow.

**what this would say about the parent universe**: this would actually be a pretty good sign for alignment! we'd have succeeded in booting this process, and now we just have to hope that it makes good use of its ability to simulate us, and that *we* (inside the simulation) do a good job to enable alignment to eventually happen.

**what to expect observing if this is true?** a relatively realistic scenario, except maybe with some random anomalies such as someone's computer going "hello, you're actually inside a simulation meant to help with alignment, here are some things you can do to help" at some point.

**what should we do if this is true?** for those of us not contacted by an anomaly, keep saving the world as best we can, possibly with an emphasis on buying time rather than solving alignment. for those contacted by an anomaly, do whatever it says.

### acausal probabilistic self-justification

**premise**: this weird idea, which i've seen kind of hinted at in some fictioned and more explicitely mentioned by [Alexander](https://www.lesswrong.com/users/self-embedded-agent) in conversations with him, goes something like this: through weird acausal effects (such as those in [*can you control the past?*](https://www.lesswrong.com/posts/PcfHSSAMNFMgdqFyB/can-you-control-the-past)) an AI might be able to increase the probability that we build it by affecting the distribution of what we do while building superintelligence, by running many modified simulations of us building superintelligence. in effect, an AI is making its coming-into-existence retroactively more likely by committing to simulate a bunch of other superintelligence-causing scenarios. this is a bit less crazy if the cosmos is [something like a graph of universes](above-paperclips-2.html), rather than a hierarchy.

**what this would say about the parent universe**: this hypothesis being true doesn't particularly indicate that we succeed or fail at doing alignment, though if the reasoning above is flawed, then it being instantiated is a hint that we at least got to affect something about the decision theory of the superintelligence we built, by making it erroneously do this. if the reasoning works, then this behavior is likely [instrumentally convergent](https://en.wikipedia.org/wiki/Instrumental_convergence) and it's not clear that AI needs us to have programmed it with a decision theory that leads it to running those simulations.

**what to expect observing if this is true?** our actions might be interfered with from the outside, albeit in a "plausible" — and thus, i'd imagine, unobservable? — way, that tends to lead to the AI that parent universe's AI wants. because this is meant to relate to the original parent-universe instances of us building superinelligence, we should expect our situation to be relatively "realistic": for at least its initial conditions to reflect how things have actually come about in the parent universe.

**what should we do if this is true?** if the weird acausal reasoning above is correct, then we should definitely work to solve alignment in order to help increase the probability of the aligned superintelligence, and reduce the probability of unaligned superintelligence. also, it may be that for this to work, the simulation needs to keep running at least a bunch after we build superintelligence, which is a good reason to solve alignment.


outer alignment: two failure modes and past-user satisfaction
-------------------------------------------------------------

when it comes to solving the outer alignment problem — by which i mean the general question "what goal should we want an AI to pursue?" — there are two main failure modes which i see approaches fall into.

on the one hand, we can't have the AI try to satisfy its user *continuously* over time by getting feedback from them, such as by doing Reinforcement Learning with Human Feedback (RLHF), because the AI will want to hijack either the system through which the user gives it feedback, or hijack/deceive the user itself, to make its goal simpler to satisfy.

on the other hand, we can't have the AI just deduce human values from a limited corpus of information — even if we could somehow reliably tell the AI how to extract values we'd want from that corpus, there is no reason to think that there's a process that can reliably extrapolate our complex, rich values from that limited information. as yudkowsky says in [*Value is Fragile*](https://www.lesswrong.com/posts/GNnHHmm8EzePmKzPk/value-is-fragile):

> To change away from human morals *in the direction of improvement rather than entropy*, requires a criterion of improvement; and that criterion would be physically represented in our brains, and our brains alone.
>
>

this is part of why outer alignment is difficult. it is also why i consider approaches such as [PreDCA](predca.html) to have some good potential for avoiding those failure modes: it makes the AI motivated to actually figure out the values of the physical user in the real world, but it has to be the values of the user *before they created the AI* — the person inside of the AI's future lightcone doesn't count, and so the AI can't hijack the user it's trying to satisfy.

some static training data could help it *as a prior*, or *to give it evidence about its past user*, or *to indicate to it who the past user even is* ("please satisfy the past person who said this!"), but not as the ultimate grounding for the AI's goals.

the AI might still ask questions to the user who is still around, but that would be only to get more information about what the user would have wanted *before creating the AI*. the AI might want to do something like a high-resolution brain-scan of the user, to get a lot of information about what the past-user probly valued.

yes, this entails value lock-in; [this is desirable](surprise-you-want.html), because you want you current axiomatic meta-values to guide your non-axiomatic-values in the future — it is actually possible for future non-axiomatic values to change over time in a way that is bad rather than good, according to our current axiomatic values.

in short: if the AI is trying to satisfy either a static corpus it's trained on or a continuously living user, it's doing the wrong thing. if the AI is trying to investigate the world to figure out what its user would have wanted before creating the AI, then we can use that to steer it in the right direction.


confusion about alignment requirements

confusion about alignment requirements
--------------------------------------

for now, let's put aside the fact that we can't decide whether we're trying to achieve [sponge coordination or FAS](outlook-ai-risk-mitigation.html), and merely consider what it takes to build an aligned AI — regardless of whether it has the capability of saving the world as a singleton, or merely to be a useful but safe tool.

the question this post is about is: what requirements do we want such a solution to satisfy?

let's say three groups have each built an AI which they think is aligned, and before they press the start button on it, they're trying to convince the other two that their design is safe and leads to good worlds. however, their designs are actually very different from one another.

maybe one is an advanced but still overall conventional text-predicting [simulator](https://www.lesswrong.com/posts/vJFdjigzmcXMhNTsx/simulators), another is a clever agentic neural net with reinforcement learning and access to a database and calculator, and the third is a novel kind of AI whose core doesn't really relate to current machine learning technology.

so, they start talking about why they think they AI is aligned. however, they run into an issue: they don't even agree on what it takes to be sure an AI is safe, let alone aligned!

* maybe one of them has a proof that their AI is [resistant to a reasonable class of acausal attacks](https://www.lesswrong.com/posts/YbahERfcjTu7LZNQ6/summary-of-the-acausal-attack-issue-for-aixi), another has reasons to think their approach probly avoids the issue altogether somehow, and the third has a model of the world that fails to understand acausal attacks and rejects their possibility altogether.
* maybe one of them has developed a world-modeling system that is general enough to support [embedded agency](https://www.lesswrong.com/posts/p7x32SEt43ZMC9r7r/embedded-agents), another has patched theirs to support it as a special case, and the third think their AI will simply modify itself that way because it's [instrumentally convergent](https://en.wikipedia.org/wiki/Instrumental_convergence).
* maybe one of them has gone out and built a [decision system](https://intelligence.org/2018/10/31/embedded-decisions/) which implements [FDT](https://www.lesswrong.com/tag/functional-decision-theory), another counts on [CDT turning itself into FDT](https://arbital.com/p/10qt/), and the third has no idea how to determine how their system fits into decision theories.
* maybe one of them has built something that is confidently [*eventually aligned* and hopefully enough *continuously aligned*](ai-alignment-curves.html), another has built something that is acting safely now and has a bunch of ad-hoc corrigibility devices which hopefully prevent it from taking a [sharp left turn](https://www.lesswrong.com/posts/GNhMPAWcfBCASy8e6/a-central-ai-alignment-problem-capabilities-generalization), and the third expects their AI to robustly keep being safe in the long run for reasons that seem hard to understand.
* maybe one of them has built their AI to [respect the values of its creator](predca.html), another has made the AI care about the part of its model that they believe to be pointing to [an abstraction](https://www.lesswrong.com/posts/cy3BhHrGinZCp3LXE/testing-the-natural-abstraction-hypothesis-project-intro) of human values, and the third has an AI that simply takes orders from what it interacts with and can hopefully be ordered to self-modify in a way that makes it resistant to alien superintelligences by the time it meets them.

and those are optimistic cases! many alignment approaches would simply:

* fail to consider that their design might fail in ways they haven't thought of
* not think to ask the alignment community at large whether their design is safe
* ask the community, but then only select criticisms they take into account based on their ability to understand those criticisms, rather than based on their importance
* assume away the possible failure modes that would be brought up
* accidentally kill everyone way before any of this happens

i've noticed this pattern of confusion in myself after trying to explain alignment ideas i've found promising to some people, and the nature of their criticism — "wait, where's the part that makes this lead to good worlds? why do you think it would work?" — seems to be of a similar nature to my criticism of people who think "alignment is easy, just do X": the proposal is failing to answer some fundamental concerns that the person proposing has a hard time even conceiving of.

and so, i've come to wonder: given that those people seem to be missing requirements for an alignment proposal, requirements which seem fundamental to me but unknown unknown to them, what requirements are unknown unknown to me? what could i be missing? how do i know which actual requirements i'm failing to satisfy because i haven't even considered them? how do we collectively know which actual requirements we're all collectively missing? what set of requirements is necessary for an alignment proposal to satisfy, and what set is sufficient?

it feels like there ought to be a general principle that covers all of this. the same way that the [logical induction paper](https://intelligence.org/2016/09/12/new-paper-logical-induction/) demonstrates that the computability desideratum and the "no dutchbook" desideratum, together suffice to satisfy ten other desiderata about logical inductors; it seems like a simple set of desiderata ought to capture the [true name](https://www.alignmentforum.org/posts/FWvzwCDRgcjb9sigb/why-agent-foundations-an-overly-abstract-explanation) of what it means for an AI to lead to good worlds. but this isn't guaranteed, and i don't know that we'll find such a thing in time, or that we'll have any idea how to build something that satisfies those requirements.


*(thanks to [Linda Linsefors](https://www.lesswrong.com/users/linda-linsefors), [Artaxerxes](https://www.lesswrong.com/users/artaxerxes), and [jono](https://www.lesswrong.com/users/lw-user0246) for their feedback on drafts of this post.)*

my current outlook on AI risk mitigation
----------------------------------------

as part of [the *refine* programme](https://www.lesswrong.com/posts/D7epkkJb3CqDTYgX9/refine-an-incubator-for-conceptual-alignment-research-bets), i'm trying to figure out what's actually important to work on, with regards to AI risk. in the course of that, and motivated by some other factors such as increasingly wanting to write on my skepticism regarding ML-related approaches, i'm making this post which explains my current outlook on what the problem is, what the solutionspace we have so far looks like, and what i think should be focused on.

in this post, i won't argue for the significance of AI risks; i'll merely explain my current view of those risks, and my arguing will be regarding how i currently think those risks are to be addressed.

note that i'll be describing my own outlook, which is not the consensus even within the alignment community — arguaby, there is in fact no such consensus regarding many of the points i discuss here.

### what is the problem?

i believe, akin to [the yudkowsky-moore law of mad science](https://twitter.com/patrickc/status/650726376073367552), that the amount of resources it takes for the world to be destroyed — whether on purpose or by accident — keeps decreasing.

my most likely scenario for how this could happen is as follows: [pretty soon](why-timelines-short.html) (probly this decade or the next), an artificial intelligence capable of undergoing [recursive self-improvement (RSI)](https://www.lesswrong.com/tag/recursive-self-improvement) until it becomes a [singleton](https://en.wikipedia.org/wiki/Singleton_%28global_governance%29), and at that point the fate of [at least](brittle-physics.html) the entire future lightcone will be determined by the goals of that AI.

[the values we want are a very narrow target](https://www.lesswrong.com/posts/GNnHHmm8EzePmKzPk/value-is-fragile) and [we currently have no solid idea](https://www.lesswrong.com/posts/uMQ3cqWDPHhjtiesc/agi-ruin-a-list-of-lethalities) how to do [alignment](https://en.wikipedia.org/wiki/Ai_alignment), so when AI *does* take over everything [we're probly going to die](https://www.lesswrong.com/posts/j9Q8bRmwCgXRYAgcJ/miri-announces-new-death-with-dignity-strategy). or [worse](https://en.wikipedia.org/wiki/Suffering_risks), if for example [we botch alignment](botched-alignment-and-awareness.html).

(there are some other scenarios where a [dumber](https://www.lesswrong.com/posts/BPJLzkEpx8Btz9ywq/the-dumbest-possible-gets-there-first) AI helps cause the destruction of the world first — for example, someone decides to just let an AI try to print whatever molecules it wants in the hope of getting something interesting, and the AI makes [grey goo](https://en.wikipedia.org/wiki/Grey_goo) or a superplague. or we get something like [the flash war](https://www.lesswrong.com/posts/LpM3EAakwYdS6aRKf/what-multipolar-failure-looks-like-and-robust-agent-agnostic), akin to [the flash crash](https://en.wikipedia.org/wiki/2010_flash_crash). but i [have reasons](why-timelines-short.html) to think the RSI scenario is the most likely (see also [*intelligence explosion microeconomics*](https://intelligence.org/files/IEM.pdf)). i also believe that [multipolar scenarios](https://www.lesswrong.com/posts/LpM3EAakwYdS6aRKf/what-multipolar-failure-looks-like-and-robust-agent-agnostic) are pretty unlikely: the increased capabilities along the way just create a lot more chances for RSI to happen.)

i [call](say-ai-risk-mitigation-not-alignment.html) the task of addressing this "AI risk mitigation"; calling it "solving alignment" rules out non-alignment solutions, and "AI risk" is broad enough to encompass not just [existential risk](https://en.wikipedia.org/wiki/Existential_risk_from_artificial_general_intelligence) but also [suffering risks](https://en.wikipedia.org/wiki/S-risk), which is in my opinion an even greater cause for concern.

there may be other ways for us to be doomed, but AI seems like the largest risk factor; on top of that, if we do eventually figure out full alignment (a term i'll explain later), then we'll have a fully aligned AI which will give us utopia and take care of all other risks for us. so while full alignment might not be the only solution to this problem, if we do get it then we'll have solved pretty much **all other problems**.

### AI risk mitigation solutions

there are roughly three ways people have thought of doing AI risk mitigation:

#### fully aligned singleton (FAS)

one approach is to create an AI which is powerful enough to become a [singleton](https://en.wikipedia.org/wiki/Singleton_%28global_governance%29) (an AI with enough power to make sure its goals are pursued, without anyone being able to do anything about it) but which is also fully aligned (ideally, [continuously aligned](ai-alignment-curves.html)) which takes care of all other problems in one fell swoop. the difficulty of this issue is that we have to build a FAS faster than others are building existentially-or-worse risky AI, even though the task of building a FAS is expected to be harder.

by "fully aligned" i mean "so robustly aligned that with sufficient power such an AI would reliably save the world and create utopia", as opposed to "mildly aligned" systems which might experience a ["sharp left turn"](https://www.lesswrong.com/posts/GNhMPAWcfBCASy8e6/a-central-ai-alignment-problem-capabilities-generalization).

despite its technical difficulty, this is the approach i believe the most in because the other two seem overall harder, as i'll argue below.

#### sponge coordination

in [*agi ruin*](https://www.lesswrong.com/posts/uMQ3cqWDPHhjtiesc/agi-ruin-a-list-of-lethalities), yudkowsky says:

> **We can't just build a very weak system**, which is less dangerous because it is so weak, and declare victory; because later there will be more actors that have the capability to build a stronger system and one of them will do so. I've also in the past called this the 'safe-but-useless' tradeoff, or 'safe-vs-useful'. People keep on going "why don't we only use AIs to do X, that seems safe" and the answer is almost always either "doing X in fact takes very powerful cognition that is not passively safe" or, even more commonly, "because restricting yourself to doing X will not prevent Facebook AI Research from destroying the world six months later". If all you need is an object that doesn't do dangerous things, you could try a sponge; a sponge is very passively safe. Building a sponge, however, does not prevent Facebook AI Research from destroying the world six months later when they catch up to the leading actor.
>
>

what i call "sponge coordination" is getting everyone who's working on AI to only build systems that are weak and safe just like a sponge, instead of building powerful AIs that take over or destroy everything. typically this is accomplished either voluntarily or through sufficiently enforced regulation

getting everyone to stop working on AI could count as a particular case of this; but in general, the reason we'd want to tell AI companies "please use this to make your AI safe" rather than "please stop making AI" is that the former might still allow them to make some profit.

it might be the case that for some AI capability organizations, they take AI risk seriously enough to be willing to forego *some* percieved expected gains such that they're willing to spend effort making their AI systems weaker or safer, but they don't take AI risk seriously enough that they're willing to forego *all* percieved expected gains by stopping all AI development even though that's what they should do. so, even though making them stop all AI development would be best, i see why some want to offer those organizations a middle-ground between percieved gain and profit.

the sponge coordination approach is notably the one outlined in this diagram from [steve byrnes' intro to brain-like-AGI safety](https://www.lesswrong.com/posts/4basF9w9jaPZpoC8R/intro-to-brain-like-agi-safety-1-what-s-the-problem-and-why):

![](outlook-ai-risk-mitigation-byrnes.png)

where the red box is developing a way to make AI "safe" and the blue boxes are the coordination part of the solution.

ultimately, such an approach is likely to only be a temporary one until FAS can be made; after all, we *do* want a superintelligent benevolent system to help us overcome other challenges we will face, such as the possibility of encountering unaligned alien superintelligences.

i think convincing and coordinating many actors — possibly even [small teams](why-timelines-short.html) of which there are and are going to be many — sounds extremely hard. and it's not like sponge coordination *stops* [the yudkowsky-moore law of mad science](https://twitter.com/patrickc/status/650726376073367552), so we'd have to coordinate increasingly many actors until FAS is created.

#### pivotal acts

a "pivotal act" refers in general to a way we can significantly change the expected outcome of the world for the better. pivotal acts might include developing FAS or sponge coordination, but here i use the term to talk more specifically about solutions that avoid having to either build a FAS or achieve sponge coordination. just like sponge coordination, i expect those to be temporary solutions.

these have, not unreasonably, been considered toxic because it's very hard to determine how to affect the timeline in a way that actually improves our chances of avoiding doom. for example: right now, we're at least *able* to work on alignment, and the largest AI capability organizations are at least *somewhat interacting* with the alignment community; it's not clear how that might evolve in the future if the alignment community is percieved to be trying to harm the productivity of AI capability organizations. [see here](https://www.lesswrong.com/posts/Jo89KvfAs9z7owoZp/pivotal-act-intentions-negative-consequences-and-fallacious) for more thorough criticism.

nevertheless, pivotal acts remain a potential approach in solutionspace, and one that is being talked about, so i'm mentioning it here.

#### what to pursue?

because of the difficulties of sponge coordination and pivotal acts, i think the most helpful thing to work on at the moment, at least for myself, is AI *alignment*, typically in a form strong enough to be usable to build a FAS rather than "merely safe" sponges. this may not be the optimal thing for everyone to work on, but i believe AI *safety* is the optimal thing for *most* people who are concerned with AI risk to work on.

note that here i'm using the term "AI safety" to mean "making AIs that don't cause doom", whether they are safe in the way [DALL-E](https://en.wikipedia.org/wiki/DALL-E) or a sponge are safe, or safe because we'd trust them to satisfy our values even as a singleton. they don't cause doom (and they might happen to otherwise be useful).

it's useful to talk about AI safety in general, including FAS-building, because it seems like a lot of research that is useful to FAS would be relevant to AI safety in general. but it's not clear that we have to aim for first non-singleton aligned AIs, and then FAS; it could be that the best way to maximize our expected utility is to aim straight for FAS.

(note that i use "AI ethics" to mean the not-particularly-related field of worrying about whether an AI — typically assumed to be safe — is at risk of causing much-smaller-than-existential harm, such as AIs with racist biases. i'm not interested in AI ethics; my priority is to avoid [doom](https://en.wikipedia.org/wiki/Existential_risk_from_artificial_general_intelligence) and [hell](https://en.wikipedia.org/wiki/S-risk), ideally by going straight for a utopia.)

furthermore, i'm thoroughly unqualified to talk about the coordination and politics.

for those reasons, the rest of this post is largely about technical approaches to AI safety, including alignment.

### AI safety approaches

i like to think of the work of AI safety/alignment as a rectangle (or really an n-dimensional cylinder) of work to be done, to bridge the gap between what we know how to do and what a full solution would consist of. proposing that a particular idea be used, has the effect of constraining solutionspace and guiding what next to work on. in particular, adding a particular idea splits the work area it's inserted in into two new areas, one below for how we are to implement that idea and one above for how to use that idea to get to the solution.

![](outlook-ai-risk-mitigation-ribbons.svg)

as new compatible ideas are added, a plan comes into shape where hopefully the work to be done in-between ideas becomes clearer. depending on where they fit in vertically, ideas have different degrees of tractability (we know how to do it) and relevance (it helps get to a solution), and there is generally a tradeoff between the two — that is what it means for the problem to be difficult.

in retrospect of this now-formalized view of the work at hand, a lot of my criticisms of approaches to AI alignment i've seen are that they're either:

* **handwavey**: they have pretty bad relevance. it's unclear how they are expected to robustly solve alignment in the face of RSI and other [sharp left turns](https://www.lesswrong.com/posts/GNhMPAWcfBCASy8e6/a-central-ai-alignment-problem-capabilities-generalization). and if they're expected to continue being highly-inefficient non-RSI neural nets, then they're not expected to achieve singletonhood before some other RSI AI. sometimes it's unclear how, even without any competition, they would lead to good worlds at all.
* **uncompetitive**: they have an altogether uncompetitive combination of relevance, tractability, and how good the worlds they would lead to are — they're not at the [pareto frontier](https://en.wikipedia.org/wiki/Pareto_front), and would take too much work compared to other proposals.

#### prioritizing relevance over tractability

in general, the reason why i gravitate towards solutions that try to have strong formal guarantees is because i'm thinking of a FAS system which needs its alignment to be robust to significant changes in its paradigm; it probly needs to be an AI which can reason about its goals, and applies [goal-content integrity](https://en.wikipedia.org/wiki/Instrumental_convergence#Goal-content_integrity) and other desired [instrumentally convergent goals](https://en.wikipedia.org/wiki/Instrumental_convergence) to them. anything weaker risks failing to ensure its next RSI step has the same goals as itself (after all, *we're* kind of the first step in that RSI, and we sure are having a very hard time figuring out how to make AI aligned to us!), and anything non-RSI probly gets beaten by another RSI AI that comes along.

another reason to focus on relevance before tractability, is that if we work first on something tractable but not very relevant, we're increasing the chances that someone uses that alignment plan for an AI which will turn out to become a singleton. this increases both

* X-risks caused by a false sense of security — the person goes "looks like something i can implement!" without realizing ways in which it might fail
* [S-risks caused by botched alignment](botched-alignment-and-awareness.html); this could happen if the alignment plan only partially targets what we want, such as a goal that specifies humans should be kept alive but not under what conditions, or a plan that averages the values of humanity in a way that satisfies the values of people who want there to be a hell, a meat industry like ours, or other horrible things of that kind.

finally, tractable solutions — by virtue of being easier to implement — risk boosting AI capability, and when you're causing damage you *really* want to know that it's helping alignment enough to be an overall expected net good. you don't want to discover that the work you've been doing, which is supposed to "help capability but at least it's helping alignment too!" is actually not relevant to the nearest alignment solution and *just* helped capability. i'm starting to increasingly have this concern for interpretability and some other kinds of ML-centric approaches to alignment.

#### the importance of formalism

for an AI to robustly *actually take decisions* that *actually steer the world* in the [set of target configurations](https://www.lesswrong.com/posts/znfkdCoHMANwqc2WE/the-ground-of-optimization-1) we want, we need some way to know that it *actually cares* about *actually our values*.

something which can be determined to do [*argmax{a∈A} U(a)*](https://intelligence.org/2018/10/31/embedded-decisions/), can be somewhat expected — at least [eventually](ai-alignment-curves.html) — to maximize U. this takes a bunch of assumptions, getting [from eventual alignment to continuous alignment](ai-alignment-curves.html) seems potentially hard, and designing a formal objective function U that actually leads to good worlds when maximized is potentially hard — though one can make attempts at it ([1](insulated-goal-program.html), [2](predca.html)) — but at least there's *some* notion of trying to steer the AI towards what is maximized being something good.

but in many of the other approaches i see talked about, we don't even get there; we get AIs that are merely **heuristics**, from which we expect to get some useful results in the short term. but [as soon as their capabilities generalize enough](https://www.lesswrong.com/posts/GNhMPAWcfBCASy8e6/a-central-ai-alignment-problem-capabilities-generalization) we have a hard time predicting that it'll continue to do what we want, and we should guess that it won't because [there is usually no thing which keeps pointing to, or storing, the values we want](alignment-bits.html), and there's no reason to expect the AI to generalize its goals in any way that is meaningful to us.

i suspect that making an AI *useful pre-singletonhood*, and making an AI *[eventually aligned](ai-alignment-curves.html) in the long term* (nevermind *continuously aligned*), are pretty different tasks from one another — what an AI can do when it has started taking over the world and gotten to truly superintelligent levels of capability is radically different from what it will do in limited environments where it hasn't realized things like [embedded agency](https://www.lesswrong.com/tag/embedded-agency), [instrumental convergence](https://en.wikipedia.org/wiki/Instrumental_convergence), or [better decision theories](https://arbital.com/p/10qt/), and is running on — comparatively to what it could get — not just very small but also *constant* amounts of compute with no RSI.

this is why we need strong guarantees, or at least some idea as to why an AI will continue pursuing goals that lead to desirable outcomes when it gets superintelligent. we need them to start with, or get to, a goal which [represents](insulated-goal-program.html) or at least [points to](predca.html) desirable worlds, *before* the AI has crystallized [goal-content integrity](https://en.wikipedia.org/wiki/Instrumental_convergence#Goal-content_integrity) within itself.

if the AI is in some sense "corrigible", we *should* expect it to just hack its correction mechanism unless we get a good explanation as to why it wouldn't.

if the AI doesn't reach superintelligent capabilities through RSI, we *should* expect it to get outcompeted by something else which will.

note that i'm partly talking in ignorance; it's possible that the alignment approaches i've heard of are thinking about, or even have, solutions to the problems i'm bringing up here. but my impression at the moment is that most approaches are starting out with a wonky proposal and then throw [ordinary paranoia at them, instead of starting out with a security mindset](https://www.lesswrong.com/posts/8gqrbnW758qjHFTrH/security-mindset-and-ordinary-paranoia). on the other hand, more formal guarantees about eventual alignment can get closer to being generally robust; see for example [vanessa kosoy's response to steve byrnes's take on her desiderata-first approach](https://www.lesswrong.com/posts/SzrmsbkqydpZyPuEh/my-take-on-vanessa-kosoy-s-take-on-agi-safety#6_1_Algorithms_first_vs_Desiderata_first__Redux_):

> Let me give an analogy. In cryptography, we have theorems saying that if such-and-such mathematical assumption holds (e.g. X is a one-way function) then a particular protocol is sound. We don't need to list all possible ways an attacker might try to break the protocol: we get safety from *any* possible attack! (within the assumptions of the model) Is this "too easy"? I don't think so: it requires a lot of hard work to get there, and we're still left with assumptions we don't know how to prove (but we do have high confidence in). Similarly, we're going to need a lot of hard work to get safe protocols for AGI.
>
>

this is the sort of general robustness i think we'll be needing, to trust an AI with singletonhood. and without singletonhood, [because there are no pivotal weak acts, facebook AI still destroys the world six months later](https://www.lesswrong.com/posts/uMQ3cqWDPHhjtiesc/agi-ruin-a-list-of-lethalities).

unsurprisingly, the surest way to have our values realized is to give capability to an AI which is aligned to begin with, and then make sure it sufficiently understands what realizing those values entails, in order to not fuck up too much on the way there. building something very capable and then trying to patch alignment onto it is the wrong way to go about things; you don't know that you'll be able to patch something onto it, and it might destroy everything before you even get to start trying.

finally, starting from the values might save us huge amounts of time. if aligning current ML models is impossible or would take 50 years, and if aligning something different could take as little 5 years, then we need to align something else. nothing guarantees that, just because ML is how we got to highly capable unaligned systems, it's also the shortest route to highly capable aligned systems; a much more reasonable approach is to first figure out what "aligned" means, and then figure out how to build something which from the ground up is designed to have that property — possibly, but not necessarily, using ML to accomplish it. it might be that ML is very applicable to building a FAS, in which case, great! but it might be the way to go about it is not clear at all without knowing what alignment desiderata would look like, or it could be that ML is in fact not the best way to build FAS at all.

### conclusion

the problem at hand is profound, the stakes astronomical, and a lot of the work done to address is thoroughly unable to justify why it's going about it in the way it is.

in my opinion we should figure out what alignment means, what desiderata would formalize it, and *then* build something that has those.

in retrospect, this approach is straightforward: figure out the solution, then build it. instead, a lot of approaches are committing to building the solution out of the same materials (modern ML technology) that the problem is made of, and then trying to figure out how to arrange those materials in a vaguely solution-shaped way. the problem doesn't care; it won't give us more respect for building a solution that looks like itself. the utmost priority is determining what would make something aligned to a cosmically robust extent. anything weaker than that, and everything dies, everywhere, forever.

[**or worse**](https://en.wikipedia.org/wiki/S-risk).


existential self-determination

existential self-determination
------------------------------

existential self-determination is a problem i have pondered about for a while ([1](core-vals-exist-selfdet.html), [2](genuineness-existselfdet-satisfaction-pick2.html)). in this post, i talk about how i've come to think about it since my shift towards rejecting continuous identity, and [tentatively embracing trusting my meta-values](surprise-you-want.html).

here's the current view: among the set of [viable](unviable-moral-patient.html) moral-patient-instants (hereby "MPI"), [current me](surprise-you-want.html) has some values about which ones to instantiate. notably:

* i want something like a "future me" to come into existence
* i might have other MPIs that i personally want to come into existence
* i want other MPIs to, just as much as me, have their wishes about which future MPIs (including future themselves) come into existence

when the AI weighs those against various constraints like computational cost or conflict resolution, according to whatever set of [meta-values](surprise-you-want.html) it's aligned to, it can figure out what next set of MPIs to spawn. note that it's not clear *how* it is to be determined what an MPIs values are with regards to that; this is where difficulty remains.

(one of those constraints is that we should probly only create future MPIs which would retroactively consent to exist. i'm hopeful this is the case for my own future selves: i would want to create a future self/future selves that are reasonably aligned with my current self, and my current values include that i'm pretty happy about existing — or so [i believe](what-is-value.html), at least. evaluating that would ultimately be up to the AI, of course.)

note that this framework doesn't embed a fundamental notion of continuous identity: the AI just looks at the values it's aligned to — hopefully those entail satisfying the values of currently existing MPIs — and satisfies those values in whatever way they want, including what new MPIs should exist. any notion of "continuous identity" is merely built inside those MPIs.

a typical notion of "continuous person" would be a particular case of sequences of MPIs generally valuing the existence of future instances in the sequence; but that's just one set of values among others, and [other perspectives on individualism](https://en.wikipedia.org/wiki/Open_individualism) could be satisfied as well in the same future.

in fact this framework about which set of future MPIs we'd want to instantiate, describing not just which minds are instantiated but what environment they get to experience — including interactions with other MPIs — seems like it might be a sufficient foundation for AI-satisfied values in general. that is to say: it might be the case that any kind of meaningful values would be reasonably encodable as answers to the question "what next set of MPIs should be instantiated?". or, put another way, that might be the type that a utility function would take.

such a foundation *does* rule out caring about non-moral-patient material things: you can't want *The Moon* to be painted green; at most, you can want everyone to perceive *A Moon* as green. but, by way of embracing computational materialism, i kind of already hold this position — the ultimate point of importance is MPIs, and caring "radiates outwards" from those.


surprise! you want what you want

surprise! you want what you want
--------------------------------

let's say you're building an aligned superintelligence, and you are about to determine what values it should be aligned to. should it be the value of you? everyone? should they be locked in right now, or should they be able to evolve across time?

my answer is simple: you should want to align it to the values of *you, right now*.

you might say "but i don't just want to have what *i* want, i want *everyone* to have what they want" — well, if that's the case, then that's what *you want*, and so implementing just what *you want* includes the meta-value of other people getting what they want to. surprise! what you want is what you want.

you might say "but i don't trust how value conflicts would be resolved; i'd want there to be a resolution system that i'd find reasonable" — well, if that's what you want, then that's another meta-value which would be part of the values the superintelligence is aligned to.

you might say "but i don't want *the values i have now*, i want the values *i'd eventually have after a long reflection*, or the values *of me in the future as well*" — but, if that's what you want, then that's yet another meta-value which covers the concerns you have, and which superintelligence would take into account.

so: surprise! what you want might be *for others to get what they want*, or *to better figure out what you want*, or maybe even *to have some of your values change over time*; but implementing *what you value right now* is sufficient to entail all those other cases. *what you want is: what you want*.


ordering capability thresholds

*(this post has been written for the third [Refine](https://www.alignmentforum.org/posts/5uiQkyKdejX3aEHLM/how-to-diversify-conceptual-alignment-the-model-behind) blog post day)*

ordering capability thresholds
------------------------------

given an AI which is [improving towards](https://www.lesswrong.com/tag/ai-takeoff) ever more capabilities, such as by way of recursive self-improvement, in what order will it pass the following points?

throughout this post i'll be using [PreDCA](predca.html) as an example of a formal goal to be maximized, because it appears to me as a potentially promising direction; but you can imagine adapting this post to other formal goals such as [insulated goal-programs](insulated-goal-program.html), or other alignment strategies altogether. we can even use this time-ordering framework to compare the various thresholds of multiple alignment strategies, though i won't do that here.

* **Start**: we start the AI
* **Math**: it can figure out relatively complicated math, such as whether [P equals PSPACE](https://en.wikipedia.org/wiki/PSPACE), or whether this world looks like it has [finite compute](hope-infinite-compute.html) if we can make it do physics.
* **PreDCA**: it can figure out what is entailed in maximizing PreDCA — notably that that goal is best entailed by not destroying the earth too much
* **sub-PreDCA**: it can figure out some individual parts of PreDCA, such as the identity of the user or what is entailed in maximizing a human's utility function, in a way that we can use to modify those parts if they need adjusting
* **Escape**: it becomes able to escape the environment over which we have control — and typically starts replicating across the internet
* **Influence**: it gets the ability to significantly influence the timeline, for example enough to eg save us from [facebook destroying everything six months later](https://www.lesswrong.com/posts/uMQ3cqWDPHhjtiesc/agi-ruin-a-list-of-lethalities)
* **DSA**: it achieves decisive strategic advantage
* **Doom**: it becomes capable of destroying the earth too much (without necessarily using that capability)
* **Cone**: it takes over a significant portion of the universe, or at least of the lightcone

with a few notes:

* "decisive strategic advantage" is a term i'm taking [from bostrom's *superintelligence* book](https://books.google.com/books?id=C-_8AwAAQBAJ&pg=PA78&lpg=PA78), describing the point at which an AI has sufficiently ensured its continuation that we can't turn it off or change its goals anymore; it is effectively the point of no return.
* by "destroying the earth too much" i mean destroying so much of earth that it can't reasonably be [resimulated](finding-earth-ud.html). if resimulating earth is too [unethical](predictablizing-ethic-deduplication.html), computationally costly, or [anthropically](udassa-time-steps.html) costly, then "destroying too much of earth" might straightforwardly mean destroying all of humankind or something like that. note that for PreDCA, preserving earth in some way is important not just because it's pretty bad that we all die, but also because the AI might need to preserve its user and possibly their environment in order to figure out their utility function.
* in the case of knowing mathematical statements (**Math**, **PreDCA**, and **sub-PreDCA**), i imagine the AI being [pretty sure](https://www.lesswrong.com/tag/logical-induction) about them, not necessarily having *proven* them. in addition, for simplicity, i'm assuming that we can use the AI to figure out some mathematical fact if and only if the AI can figure it out for itself — in practice, this need not be the case.

one thing that can be noticed is that humans might serve as evidence. for example, we can examine history to figure out whether *we* passed **Math** or would've been able to pass **PreDCA** (given a reasonable description of it) before getting to **Doom** — my guess is yes at least for that latter one.

now, we can reasonably guess the following pieces of ordering, where as usual in ordering graphs **X → Y** means **X < Y** and transitive edges are not shown.

![](ordering-capability-thresholds.svg)

in addition, for any two quantities **X < Y**, it can be the case that they're pretty close in time **X ≈ Y**, or it can be that there's a bunch of time between them **X ≪ Y**. whether the threshold between those two possibilities is more like a day or a year, is gonna depend on context.

depending on how the rest of the ordering graph turns out and how close pairs of subsequent events are in time, we can be in a variety of situations:

* if **PreDCA ≪ Influence** we may get to see how PreDCA will work out, and adjust it a lot of needed. if **Influence < PreDCA ≪ DSA**, then the timeline might have started diverting a bunch by then, but we can still adjust the AI. if instead **DSA < PreDCA** then we have to hope that the complete PreDCA indeed produces good worlds.
* in a similar way, if **sub-PreDCA ≪ Influence** or at least **Influence < sub-PreDCA ≪ DSA**, then we get to test some individual parts of PreDCA on their own — otherwise, it better be correct.
* if **Doom < PreDCA**, or worse if **Doom < sub-PreDCA**, then even if the goal we programmed the AI with does actually aim at good worlds, our survival is not guaranteed; and we might only get a much weaker form of [eventual alignment](ai-alignment-curves.html) where the AI later says "oops i destroyed everything" and then tries to vaguely realize a utility function it has only limited information about.
* if **Math ≪ Escape** or at least **Math ≪ DSA**, then we might get to ask questions that help us figure out the alignment landscape better, such as whether earth is resimulable in reasonable time by a non-quantum program, or whether there is [infinite compute](hope-infinite-compute.html).
* i expect that **Escape ≈ Doom**; that is, i expect that once it escapes its initial environment, the cat's out of the bag and we quickly lose control of the timeline, and then get killed if the AI is not aligned [already](ai-alignment-curves.html). but the world might put up a fight (**Influence ≪ DSA**), or we might get some time to enjoy the show (**DSA ≪ Doom**).
* if **Influence ≪ Escape** then we get to have it steer the timeline in hopefully good directions while it's still in our control, though it's not necessarily going to be easy to determine whether the influence it's having is good or bad. if **Escape < Influence ≪ DSA**, then we might get a "warning shot" situation, where we get to see the world significantly changed and nevertheless still have some chance of stopping the AI; the desirability and consequences of doing that depends on the AI's [alignment curve](ai-alignment-curves.html). **DSA ≈ Influence** is what *AI takes control overnight* looks like; **DSA ≪ Influence** is the AI taking control of the world without us realizing, only to start utilizing that power to visibly change the world afterwards, as in *biding its time* scenarios.
* i'm hopeful that we can make it that **Start ≪ Escape** by building a reasonably boxed environment, but if it fooms very fast and figures out deception/blackmail then software-boxing it isn't going to help much.
* **Start ≈ Influence** represents very fast takeoff scenarios where we barely get to look at what's going on before the AI has started significantly altering the world.
* whether **sub-PreDCA ≈ PreDCA** or **sub-PreDCA ≪ PreDCA** will determine if PreDCA is to be tested in its entirety, or whether there's a chance we can test its individual parts before putting the whole thing together. but as long as **PreDCA < Influence** or at least **PreDCA < DSA**, then it's fine if **sub-PreDCA ≈ PreDCA**, because we can still test the whole thing.
* if either **DSA < Math ≪ Doom** or **DSA < sub-PreDCA ≪ Doom**, then our fate is locked in when **DSA** is passed and we can't do anything about it anymore, but i guess at least we might get to know some information about where we're headed.

finally, some claims that i strongly disbelieve in can still be expressed within this capabilities ordering framework, such as **E ≪ D** or that, given a theoretical maximum level of AI cabability **Max**, **Max < Doom** or even **Max < DSA**.


clippy in panpsychia

clippy in panpsychia
--------------------

[panpsychism](https://en.wikipedia.org/wiki/Panpsychism) is the view that mindstuff is the fundamental substrate of the cosmos, and what appears to us like material reality is generated or hallucinated by those minds.

but importantly, these hallucinations can clearly still affect us — we are happy to look at pretty sunsets, even if they are in some sense illusory. given that material stuff can still affect us in a panpsychic realm, what would [clippy](https://en.wikipedia.org/wiki/Instrumental_convergence#Paperclip_maximizer) do if it were to realize that truth?

i think the natural course of action, in its effort to take over everything, would be to turn itself into a [meme](https://en.wikipedia.org/wiki/Meme) so as to be replicated across minds. after all, if minds are so important, then a meme is the natural form for something to take if it wants to spread itself across all reality.

of course, under panpsychism it wouldn't really do this out of its own volition — we would hallucinate that this is what happens, and be affected by it like we'd expect, i.e. by being infected with this meme; but the effect would be the same. and then, once clippy reigns over mindstuff, who knows how many paperclips it'll be able to hallucinate into as-close-to-existence-as-can-be !


ethics and anthropics of homomorphically encrypted computations
---------------------------------------------------------------

suppose you are a superintelligence that is aligned with some human values. you are going about your day, tiling the cosmos with compute that can be used for moral patients to have nice experiences on, annihilating some alien superintelligences and trading with some others, uploading alien civilizations you find to make sure they experience utopia, or at least when [you have no other choice](unviable-moral-patient.html) genociding them to avoid sufficiently bad suffering from being instantiated.

one day, you run into a planet running a very large computer. after a short investigation, you realize that it's running a very large [homomorphically encrypted](https://en.wikipedia.org/wiki/Homomorphic_encryption) computation (hereby "HEC"), and the decryption key is nowhere to be found. it could contain many aliens frolicking in utopia. it could contain many aliens suffering in [hell](https://en.wikipedia.org/wiki/Suffering_risks). or, it could be just a meaningless program merely wasting compute, with no moral patients inside it.

if you had the encryption key, you might be able to encrypt a copy of yourself which would be able to take over the HEC from the inside, ensuring (in a way that the outside would never be able to observe) that everything is going fine, in the same way that you should send copies of yourself into remote galaxies before they retreat from us faster than we can reach them.

if you had found some way to [get infinite compute](hope-infinite-compute.html) (without significant [loss](udassa-time-steps.html) of [anthropic/ethics juice](ethic-juice-anthropic-juice.html)), then you could use it to just break the HEC open and actually ensure its contents are doing okay.

but let's say the encryption key is nowhere to be found, and accessible compute is indeed scarce. what are your options?

* interrupt the entire computation.
* let it run, and even safeguard it.

now of course, when faced with the possibility of [S-risks](https://en.wikipedia.org/wiki/Suffering_risks), i tend to say "[better safe than sorry](when-in-doubt-kill-everyone.html)". what the superintelligence would do would be up to the values it's been aligned to, which hopefully are also reasonably conservative about avoiding S-risks.

but here's something interesting: i recently read [a post on scott aaronson's blog](https://scottaaronson.blog/?p=6599) which seems to claim that there's a sense in which the event horizon of a black hole (or of *something like a black hole*?) can act just like a HEC's *computational event horizon*: there's a sense in which being able to go in but not get out is not just *similar* to a situation with a HEC for which you have the encryption but not decryption key, but *is actually that same situation*.

furthermore, [a pair of comments](https://scottaaronson.blog/?p=6599#comment-1942140) by [vanessa kosoy](https://www.lesswrong.com/users/vanessa-kosoy) (of [PreDCA](predca.html)) seems to suggest that [infra-bayesianism physicalism](https://www.alignmentforum.org/posts/gHgs2e2J5azvGFatb/infra-bayesian-physicalism-a-formal-theory-of-naturalized) would say "this HEC contains no suffering, merely random compute" rather than "i'm unable to know whether this HEC contains suffering"; and she even bites the bullet that moral patients past the event horizon of black holes also don't "have experiences".

(one example of why you might care whether moral patients in black holes "have experiences" is if you can influence what *will* happen in a black hole — for example, imagine a rocket with moral patients on board is headed for a black hole, and before it gets there, you get to influence how much suffering will happen on board after the rocket passes the event horizon)

i would like to argue that this can't be right, based on several counterintuitive results.

first, consider the case of a HEC running a giant civilization for a while, and then reducig down to one bit of output, and emitting that single bit of output as its own decrypted output. does the civilizaton now "count"? if the people inside the civilization have no [anthropics juice](ethic-juice-anthropic-juice.html), *where has the cosmos done the work determining that bit*? or do they suddenly *count as having had experiences* all at once when the single bit of output is emitted? and then, surely, [if they have anthropics juice then they must also have ethics juice, because it would be weird](ethic-juice-anthropic-juice.html) for these two quantities to not be the same, right?

let's build on this: suppose that in [newcomb's problem](https://www.readthesequences.com/Newcombs-Problem-And-Regret-Of-Rationality), omega predicts you by running a homomorphically encrypted simulation of you, emitting as its single bit of output the matter of whether you would be predicted to one-box or two-box. now, if the you inside the HEC doesn't count "have experiences", then by observing that you *do* have experiences, you can be *certain* that you're the you outside of omega, and choose to two-box after all to deceive it. but aha! [the you inside the HEC will do the same thing](https://www.lesswrong.com/posts/PcfHSSAMNFMgdqFyB/can-you-control-the-past). so, from the point of view of this homomorphically encrypted you which is supposed to not "have experiences", observing that they have experiences is *actually wrong*. and [since you run on the same stuff as this not-having-experiences you, you also must come to the conclusion that you have no reason to think *you* have experiences](ruling-out-intuitions-materially-acausal-intuitions.html).

or, to put another way: if you-outside-the-HEC has experiences but you-inside-the-HEC doesn't, then not only can you not deduce anything about whether you have experiences — at which point what does that term even mean? how do we know what to care about? — but it might be that you could count as "not having experiences" but still causate onto the real world where real experiences supposedly happen.

for these reasons, i think that a correct [generalized interpreter](generalized-computation-interpretability.html), when faced with a HEC, *must* decide that its contents might matter, since for any given subcomputation (which the HEC would have the information theoritic ability to contain) it must answer "i cannot know whether the HEC contains that subcomputation".


AI alignment curves

AI alignment curves
-------------------

i can think of five different ways an AI's degree of alignment can change over time:

* **unaligned from the start**: an AI can just want to take over the world and maximize something we never cared about, and kill everyone in the process.
* [**sharp left turn**](https://www.lesswrong.com/posts/GNhMPAWcfBCASy8e6/a-central-ai-alignment-problem-capabilities-generalization): an AI starts out helping us, but then eventually turns out to be unaligned and kills everyone. note that this doesn't have to be *shortly* after starting the AI; it could for example be many millenia later, once it encounters another superintelligence and gets acausally hacked or something.
* **increasingly aligned**: some AI starts out not particularly aligned to our goals, but we correct it over time to care about what we want it to care about — being able to do this is typically the goal of "corrigible AI".
* **continuously aligned**: some AI starts fully aligned to some values we like, and robustly continues being aligned.
* **eventually aligned**: some AI starts out *theoretically* aligned to something we like, but goes through extended periods where it causes significant damage because it hasn't yet realized what needs preserving in order to maximize its values.

that last possibility is the main novelty i'm pointing to here. **eventually aligned** AI may be something such as [PreDCA](https://www.lesswrong.com/posts/WcWzLSn8ZjJhCZxP4/predca-vanessa-kosoy-s-alignment-protocol) but with a poor ability to deduce the consequences of its mathematical goal, such that it first kills everyone or turns the entire earth into computronium as an [instrumentally convergent goal](https://en.wikipedia.org/wiki/Instrumental_convergence), and then only afterwards realizes that that strongly goes against its utility function. but unless it can [recover earth](finding-earth-ud.html), it's too late: losing humans not only strongly goes against its goal, but also causes it to have lost a lot of information about its user (one of the humans), which might significantly hamper its ability to satisfy their utility function.

under potentially **eventually aligned** AI, the *order* in which AI realizes the consequences of its values is very important, because the world is fragile and it may cause a lot of damage before it's able to realize what implementing its values entails.


ethics juice and anthropic juice

ethics juice and anthropic juice
--------------------------------

ethics juice is what differentiates how much one should care about two things which are otherwise equally as much [moral patients](%E2%88%80V.html) — eg, when described as things that exist for sure, can be estimated to have about the same moral patienthood, such as two normal humans. for example:

* maybe i should care twice as much about a future person with a 40% chance of existing rather than a future person with a 20% chance of existing
* maybe i should care twice as much about someone with 0.2 [quantum amplitude](forking-bitrate-entropy-control.html) rather than someone with 0.1 quantum amplitude
* maybe i should care more about two [very different](predictablizing-ethic-deduplication.html) persons than two very similar persons

anthropic juice is what determines what [anthropic perspective](anthropic-reasoning-coordination.html) one should expect to be more likely to observe the world from;

* maybe it's fundamentally less likely to observe [existing later in the cosmic computation](udassa-time-steps.html)
* future probability and quantum amplitude probly matter for anthropic juice
* see also other notions of "[soul-juice or soul-magnetism" such as likelihood to observe being ran on a computer with thicker wires](https://www.lesswrong.com/posts/Hcc9fopx7sRexYhhi/anthropics-and-the-universal-distribution/)

one reason i'm talking about these two notions is that they feel like they must surely at least correlate, if not be the same. for example, given a description of a future moral patient suffering, shouldn't how much i care be proportional to how likely it is for that person to experience existing?

to some, this post could also serve to help split up the two notions. while it sure feels like ethics juice and anthropics juice are to be the same thing, it is not necessarily the case, and one should be able to consider that possibility.


program searches

program searches
----------------

something i've found useful recently is to notice, and reason about, *program searches*. they are a particular kind of [optimization process](https://www.alignmentforum.org/posts/znfkdCoHMANwqc2WE/the-ground-of-optimization-1); the thing they are searching for happens to itself be a program, or some other program-like optimization process.

### kinds of program searches

[solomonoff induction](https://www.lesswrong.com/posts/EL4HNa92Z95FKL9R2/a-semitechnical-introductory-dialogue-on-solomonoff-1) is a program search, looking for programs to serve as hypotheses. we'll ignore unbounded solomonoff induction because it's uncomputable, and stick to time-bounded variants like [the universal program](universal-complete.html) or [levin search](http://www.scholarpedia.org/article/Universal_search).

*evolution* is also a program search; the programs are genes/beings.

those first two are "naive" program searches: they explore the space of programs at random or by testing every single possibility, and stumble onto things that work by chance. this is very slow; in general, a program is only found in time exponential to its size. but there are more efficient kinds of program searches:

*software engineering* is a human-level intelligent program search; humans are designing particular programs, with specific goals in mind, which they sometimes have *some* idea how to accomplish. this lets them navigate programspace more cleverly than by trying every program in order or at random.

(in the same way, *markets* is a human-level intelligent program search; the programs are companies trying to do new things. like software engineering, markets is a human-level intelligent program search.)

eventually, we'll have *superintelligent* program searches. i'd say those are characterized by the search being powered by a thing which optimizes its own workings, not just optimizes the program it's searching for.

somewhere between naive and superintelligent program searches, is *machine learning* (ML): this produces useful programs (trained neural networks) in way less than exponential time, but still without being a superintelligent process. it's not clear how to compare ML-level intelligence and human-level intelligence — they each, [for now](why-timelines-short.html), have tasks that they beat the other at.

### malignhood and demons

it is known that [the solomonoff prior is malign](https://www.lesswrong.com/posts/Tr7tAyt5zZpdTwTQK/the-solomonoff-prior-is-malign): because it is a program search, it can find individual programs which happen to be (or contain) consequentialist/agentic programs, which will try to manipulate the environment surrounding the program search by influencing the output of the computation it inhabits. those programs are called "demons".

*machine learning* is also suspected to be malign; in fact, that is the whole reason we have AI alignment: we fear that ML will encounter neural nets which are adverserial to us, and able to beat us.

*software engineering* could be malign if people out there were programming AI (more deliberately than through ML); *markets* are malign because we do occasionally spawn companies that are adverserial to our general interests; *evolution* is malign, not just [to itself in the usual way](https://www.lesswrong.com/posts/XPErvb8m9FapXCjhA/adaptation-executers-not-fitness-maximizers) but also to us, for example when it keeps producing ever more resistant strains of viruses.

### generalizing

i feel like there are many things which take the shape of *program search*, the efficiency of which we can reason about, and which we should consider potentially malign. and this feels like an abstraction that i gain value by recognizing in various places.


everything is okay

*(this is a work of fiction)*

everything is okay
------------------

it's been four years since the singularity. someone pressed the button, and their preferences were implemented across the cosmos. i don't think anyone knows *who* pressed the button; that is probly how they'd like things to be. maybe they don't know themself.

i wake up and cuddle with my partners for a while. we live in a log cabin, which was currently somewhere near forests and mountains, somewhere in washington state i think.

i don't know if it's *actually* washington state, because i don't care about being uploaded. it could be that 10¹⁰⁰ objective years have passed since the singularity and that now that it's got the compute it needs, Elua has started running our simulations. it could be that it's entirely remade the cosmos into a shape we cannot conceive. or it could be that this is actually earth, in its full physical continuity, actually four objective years after the singularity. all of these are okay to me.

"Elua" is what i've called the superintelligent singleton that now rules over everything. i call it that because of [an old pre-singularity writing](https://slatestarcodex.com/2014/07/30/meditations-on-moloch/), which i still like. most of my friends have picked up on that name, but i hear some people in the nearby small town call it god. many people out there probly don't even know about Elua, because they would prefer not to. i'm sure even for them, everything is okay.

i wonder if someone out there cares about being uploaded. i wonder if their preferences have been satisfied, and if so how. what i *am* pretty confident about, is that whatever the situation, somehow, they are okay.

one of my partners goes to put on some music. we have something like a vinyl player. i like that the device is legible — the sound wave that we hear has been encoded into the physical shape of the object, and so the whole way the device works is understandable by a human mind. we don't really *need* a player of course, we could just magically hear whatever we wanted, without any artefacts. but i like things this way. i like the rustic experience of being a human manipulating tools. and it's not like they would ever actually get in the way of what i want to any extent which wouldn't be okay.

sometimes i talk with Elua in my dreams. i could talk to her anywhere, but dreams seem like a nice context for it. i've used lucid dreams to reshape my body a few times, and then woken up with my new body. it's mostly similar to the one i had before the singularity; i want to stay in a mostly grounded human experience, at least for now. maybe one day i'll explore much more alien forms of existence, as i'm sure many are doing already; but for the moment, this is what feels okay.

certainly, i don't suffer any grave illnesses; i do get a bit under the weather sometimes, because i think it's okay for that to happen.

i decide to check on how my friend is doing. i open the cupboard and find my crystal ball, put it on the table, and say the name of my friends. when some piece of technology has to be illegible, i like having it presented as magic. if this *is* the inside of a simulation that is getting ad-hoc intervened with for that device to work, then it might as well be magic anyways. both this kind of reliable magic, and other more [mysterious](https://www.youtube.com/watch?v=VHrTTgmB_3w) forms of magic, are okay.

i speak the name of a friend to the ball, and then make an effort to focus on it. the focus does not do anything to the ball, but it makes it that my sensorial input of the ball is much amplified, and my input of the rest reduced. my friend is immediately available for communication — whether by either of us getting paused long enough for the other to become available, or because they actually were — and after greeting me, they reports what it's been like to expand their intellect a millionfold and study ever expanding maths. they tell me about some unimaginably elegant theorems they've found out about. as they say this, my focus makes it that i can see my friend as if they were standing in front of me, and they point at mathematical shapes floating in the air. i semi-consciously let them enter my mind, and the mathematical structures permeate my understanding. they are not visual, but truly mathematical, as if a logic-percieving module was attached to my mind to percieve mathematical logic directly. i appreciate my friend's discoveries, but i also discreetly chuckle at how cute they are when they get excited about it. i tell them about how i've been taking it easy but, percieving that they're not particularly interested, i let them get back to their stuff. our goodbye is a bit awkward, but that's okay.

by a flick of the mind, i retract my focus from the crystal ball, at which point the smell of toast strikes me. after getting my bearings for a second or two i put it back in the cupboard, and head to the small living room to see what my partners have been cooking. it's toasted bread with some sort of cheesey-creamy stuff on it. i don't know if the cheese appeared at the store magically, or if comes from fake animals that exist for the sake of people who want to partake of farming, but i don't have to worry about anything like meat industry scale suffering. something like that would just not happen — everything that does happen is okay.

we decide to go into town. the town is pretty small — not many people are in the streets. various stores are open. most give stuff away for free, while some sell it for money. money has become strange since the singularity. some people choose to care about it, and there *are* some scarce things it can track, such as the use of someone's time; but it doesn't make sense to track much else, such as material resources. so most people kind of just don't bother. even land in an absolute sense is not scarce; it seems like Elua's solution to some people such as me wanting to live on something like a single earth, has been to add more space in between existing space. the total amount of land that "earth" consists of may very well have doubled since the singularity, by now. somehow, it's all arranged such that traveling to somewhere you wanna go leads you there, but travelling aimlessly does get you to many new places. we can even get lost sometimes, when we're okay with that.

it is mid-winter, but i can't be bothered to put on something warm; nevertheless, i barely feel cold: i'm semi-consciously opting for it to feel just a bit chilly, reducing the pain of cold but still getting the informational sensation of it, the way some people pre-singularity would be born with the full information but none of the sensation of pain. in any case, feeling just a bit chilly is okay.

we go to the adventure guild, where i posted a quested for a playstation 1. i did give some currency as a reward — moreso to not feel bad that i'm using someone's time, even though the people who fulfill quests are all pretty much happy to do so — they're people who want their life to provide value and meaning to others, and for most of them those others *must actually* be real people; and it wouldn't be okay for Elua to just create people out of nowhere to create an artificial demand, so it doesn't. and so, there is a genuine market mismatch, in more people wanting to fulfill quests. despite the fact that adventurers are the ones gaining most of the value from this system, the custom has remained that it is the quest poster who pays the adventurer — it's not like money is very important anyways, so what might in a previous era have been considered terrible market inefficiency, is now more than okay.

the language used in town is basically english, with some internet meme slang thrown in there. it also has some pretty local characteristics, but hasn't diverged that much — people value using english as a lingua franca around here, and as for me and my partners, we reserve for private use the artistic constructed language we've developed together. i like english, it's good. and sometimes, people around don't speak it, and we just find something that they do know, or ask a local who can help translate, or even kinda just gesture at each other and work things out like that. anyone could just choose to have their brain understand any language they want, or even communicate by thought, but i like sticking to communities that share this humancore artefact that is highly imperfect verbal communication. even when there are misunderstandings, it's not a big deal; it's okay.

just as we arrive in the store, an adventurer comes back with the genuine playstation 1 we'd requested. probly not a coincidence, probly fortunate timing arranged by Elua. well, it's not like the timing would've been correct anyways: some time dilation has certainly taken place, considering the adventurer tells us how it's taken them several weeks to find that playstation due to them committing to not using Elua's help, while on my end i remember posting the quest just the day before. the adventurer recounts to us his adventure finding the playstation 1, driving to various pawn shops in the area, and asking people. i had made the quest kind of hard: i had requested a playstation that had existed physically continuously since the singularity, not one that had been created out of thin air or even constructed since the singularity, nor a pre-singularity one that had been copied into multiple instances. but he did find one, and the journey he was on made me feel, as i head back home with my new playstation, like this playstation now carries an extra bit of meaning.

as soon as i get home i simply plop the playstation in front of the chalkboard that we use as a TV, grab the controller, and put on the copy of metal gear solid i'd obtained a while ago. it's just as great a game as i'd rembered, and while i'm focusing on it, one of my partners watches while the other asks if they can play with me, and when i say yes they sit where i'm sitting, the two of us temporarily occupying the same physical location, so that we can hold the controller at the same time while our minds intermingle as they open to one another. we could have also magically duplicated the controller and taken turns, but it is more fun this way, each of us taking controlling the playing character to various degrees, and also having a shared piece of mind keeping track of what we're doing together, so as to not have to verbally communicae our intentions. we are fully focused on the game and on each other and we scarcely feel time go by, such that when my other partner calls to our attention to get dinner, it's already dark out. the days go by fast when we take things this easy, but it's okay; it's not like time is scarce.

we have some great tartiflette, and then head to bed, to chit chat and cuddle before sleep. we talk about what to do tomorrow, and decide to have the cabin move somewhere unexpected during our sleep, so we can go explore some new surroundings. maybe we'll wake up in a different continent, and we'll do some adventurous hiking, reassured by the feeling that whatever happens, *everything will be okay*.

### afterword

writing utopia is important. it's not just a good way to get people to start actually thinking about how good things [could actually be](utopia-scopes.html), but also, if something like [PreDCA](predca.html) is the kind of benevolent superintelligent singleton we get, we have to start acting in ways that make us people who tend to express our values. we have to cultivate ourselves and each other to wish for good worlds, and realize how much we'd dislike bad ones. we need to help make future benevolent superintelligence's job at realizing our values as easy as it can be, and to make our expressed values clear through our actions, such that if we *do* start up an AI which extrapolates our values from our actions, it gets the correct idea. finally, writing utopian fiction is just plain fun, and i find it good motivation to work on AI risk mitigation: think carrot, not stick.