The first AI Safety Camp & onwards

*by Remmelt Ellen and Linda Linsefors*

#### Summary

Last month, 5 teams of up-and-coming researchers gathered to solve concrete problems in AI-alignment at our 10-day AI safety research camp in Gran Canaria.

This post describes

* the event format we came up with
* our experience & lessons learned in running it in Gran Canaria
* how you can contribute to the next camp in Prague on 4-14 October & future editions

#### **The event format**

*In* [*February*](http://effective-altruism.com/ea/1kd/proposal_for_the_ai_safety_research_camp/)*, we proposed our plans for the AI Safety Camp:*

**Goals:** Efficiently launch aspiring AI safety and strategy researchers into concrete productivity by creating an ‘on-ramp’ for future researchers.

Specifically:

1. Get people started on and immersed into concrete research work intended to lead to published papers.
2. Address the bottleneck in AI safety/strategy of few experts being available to train or organize aspiring researchers by efficiently using expert time.
3. Create a clear path from ‘interested/concerned’ to ‘active researcher’.
4. Test a new method for bootstrapping talent-constrained research fields.

*Note*: this does not involve reaching out to or convincing outside researchers – those who are not yet doing work in the field of AI safety – on the imperative of solving alignment problems.

**Method:** Run an online study group culminating in an intensive in-person research camp. Participants work in teams on tightly-defined research projects on the following topics:

* Agent foundations
* Machine learning safety
* Policy & strategy
* Human values

Project ideas are proposed by participants prior to the start of the program. After that, participants split into teams around the most popular research ideas (each participant joins one team, and each team focuses on one research topic).

**What the camp isn’t about:**

The AI Safety Camp is not about convincing anyone about the importance of AI-alignment research. The camp is for people who are already on board with the general ideas, and who want to develop their research skills and/or find like-minded people to collaborate with. Trying to convert people from adjacent research fields is a very different project, which we do not think mixes well with this event.

The AI Safety Camp is not a summer school (unlike the [one coming up this August in Prague](http://humanaligned.ai/)). There are no teachers although teams can correspond with experienced advisors. Participants are expected to have the knowledge needed to do research together. However, it is not required for everyone to have research experience, or to know every relevant fact. That is what the team is for – to help each other, and lift each other up.

#### The first camp

**How it came to be:**

The project got started when Linda Linsefors tried to figure out how to find AI safety researchers to cowork with in a supportive environment. Effective Altruism Global London (November 2017) was coming up so she decided to network there to look for a “multiplayer solution” to the problem – one that would also also help others in a similar situation.

After bouncing ideas off various people in the conference corridors, Linda had formed a vague plan of starting a research retreat – renting a venue somewhere and inviting others to try to do research together.

While joining an Open Philanthropy Project open office hour, Tom McGrath (who became our team preparation leader) overheard Linda talking about her idea and wanted to explore it further. Later, while couchsurfing at Sam Hilton’s place, she met Remmelt Ellen (who became our meetings & logistics leader) and together they created and drew attention to a Facebook group and form where people could indicate their interest. Nandi Schoots (who became our interviews & programme leader) and David Kristoffersson (who became our international connector) quickly found the Facebook group and joined our first organisers’ call.

Our core organising team formed within a week, after which we scheduled regular video calls to sort out the format, what to call the event, where to organise it, and so on. We hit the ground running and coordinated well through Facebook chats and Zoom calls considering we were a bunch of international volunteers. Perhaps our team members were unusually dedicated because each of us had taken the initiative to reach out and join the group. We also deliberately made fast decisions on next actions and who would carry them out – thus avoiding the kind of dragged-out discussions where half of the team has to sit idly by to wait for conclusions that no one acts upon.

Initially, we decided to run the first camp in July 2018 in either Berlin or the UK. Then, Las Palmas, Gran Canaria was suggested as an alternative in our [Facebook group](https://www.facebook.com/groups/AISafetyCamp/) by Maia Pasek from [Crow’s Nest](https://www.facebook.com/groups/crowsnestrationality/) (sadly Maia passed away before the camp started). We decided to run a small pilot camp there in April to test how well the format worked – thinking that Gran Canaria was a cheap, attractive sub-tropical island with on-the-ground collaborators to sort out the venue (this ended up being mostly Karol Kubicki).

However, in February a surprising number of researchers (32) submitted applications of mostly high quality – too many for our 12-person AirBnB apartment (a cancellable booking made by Greg Colbourn). Instead, we booked an entire hostel to run the full edition that we had originally envisaged for July, effectively shortening our planning time by 3 months.

This forced us to be effective and focus on what was most important to make the camp happen. But we were also basically chasing the clock at every step of the organising process, which led to costly mistakes such as rushing out documents and spending insufficient time comparing available venues (we reviewed many more lessons learned in a 14-page internal document).

Most of the original organisers were exhausted after the event finished and were not going to lead a second edition any time soon. Fortunately, some of the Gran Canaria camp participants are taking up the mantle to organise the second camp together with EA Czech Republic in Prague this October (for more on this see “Next Camp” below).

**Team formation:**

Each applicant was invited for an interview call (with the help of Markus Salmela), of which we accepted 25 for the camp (of these, 4 people were unable to join the event).

From there, we invited participants to jot down their preferences for topics to work on and planned a series of calls to form research teams around the most popular topics.

After forming 5 teams, we had an online preparation period of roughly 6 weeks to get up to speed on our chosen research topics (through Slack channels, calls and in-person chats). This minimised the need to study papers at the camp itself. However, it was up to each team to decide how to best spend this time – e.g. some divided up reading materials, or wrote [research proposals](https://docs.google.com/document/d/1ZzczZkXY7gvYHobBco6FXl0dH8K4KgWok--hEeAUgHE/edit) and got feedback from senior researchers (including Victoria Krakovna, Stuart Armstrong and Owain Evans).

**Event structure:**

The camp consisted of coworking punctuated by team support sessions and participant-organised activities.

*Programme summary:*

*Day 1:* Arrival, starting ceremony

*Day 2:* Team research

*Day 3:* Team research

*Day 4:* Research idea presentations, half day off

*Day 5:* Team debugging, research ducking in pairs, team research

*Day 6:* Inter-team hamming circles, team research, research ducking in pairs

*Day 7:* Day off

*Day 8:* Team research

*Day 9:* Team research, AlphaZero presentation (participant initiative), career circle

*Day 10:* Team research, research presentations, closing ceremony

*Day 11:* Feedback form, departure

The programme was split into three arcs (day 1-4, day 4-7, day 7-11) where the workload gradually intensified until it turned down again – hopefully enabling teams to do intensive work sprints while not burning out.

The support sessions on day 5 and 6 were aimed at helping teams resolve bottlenecks and become more productive. Although a few participants mentioned they were surprisingly useful, doing them during daylight hours hindered teams from getting on with research. For future camps, we suggest having only optional Hamming circles and research ducking sessions in the evening.

Participants also shared their own initiatives on the dining room blackboard such as morning yoga, beach walks, mountain hiking, going out for dinner, a clicker game and an AlphaZero presentation. We wholeheartedly recommend fostering [unconference](https://en.wikipedia.org/wiki/Unconference)-style initiatives at research events – they give participants the freedom to make up for what you have missed.

Two experienced Centre for Applied Rationality workshop mentors, Ben Sancetta and Anne Wissemann, had the job of supporting participants in sorting out any issues they or their team encountered, and helping ensure that everyone was happy (Anne also oversaw supplies). Luckily, everyone got along so well that Anne and Ben only had a handful of one-on-ones. Nevertheless, having them around was a treat for some participants, as it allowed them to drop in and vent whatever was on their mind, knowing that it would not unduly bother either of them.

**Budget:**

![AISC1 budget](https://aisafety.camp/wp-content/uploads/2018/06/AISC1-budget-192x300.png)

The total cost of organising the camp was *€11,572* (excluding some items paid for by the organisers themselves).

The funds were managed through the bank account of Effective Altruism Netherlands. Unspent money was transferred to the Czech Association for Effective Altruism for the next camp (they are open to donations if their EA Grant application for the camp gets delayed or rejected).

**Results:**

Each team has [written a brief summary](https://aisafety.camp/2018/06/05/aisc-1-research-summaries/) of the work they did during the camp (as well as future plans). Other outcomes include:

* The bounded rationality team has received [funding](https://docs.google.com/document/d/1NIg4OnQyhWGR01fMVTcxpz8jDd68JdDIyQb0ZZyB-go/edit?usp=drive_web&ouid=116971326409968879540) from Paul Christiano to continue their work.
* The Gridworld team has written a [blogpost](https://www.gleech.org/grids/) and are making a GitHub pull request for their work to be added to the Safety Gridworlds repository.
* The Safe AF team is writing a paper on their results.
* At least 2 participants have changed their career plans towards working on AI safety (many participants were already junior researchers or had already made up their minds prior to the camp).
* 8 more participants reported an increase in their motivation and/or confidence in doing research work.

As our Gran Canaria “pilot camp” grew in ambition we implicitly worked towards the outcomes we expected to see for the “main camp”:

1. Three or more draft papers have been written that are considered to be promising by the research community.
2. Three or more researchers who participated in the project would obtain funding or a research role in AI safety/strategy in the year following the camp.

It is too soon to say about whether the first goal will be met, although with one paper in preparation and one team having already obtained funding it is looking plausible. The second goal was already met less than a month after the camp.

#### Improving the format

The format of the AI Safety Camp is still under development. Here are two major points we would like to improve. Suggestions are welcome.

1. **Managing team onboarding:**After the interviews, we accepted applicants on the condition that they would find a research team, which created uncertainty for them. Furthermore, we did not even enforce this rule (though perhaps we should have).

Forming research teams that consist of people with a good fit for promising topics lies at the foundation of a productive camp. But it is also a complex problem with many variables and moving parts (e.g. Do we accept people first and form teams around these people, or do we form teams first and accept people based on their fit with a team? Should we choose research topics first and then decide who joins which team, or should we form teams first and then let them choose topics?).  We handled this at the first camp by trying to do everything at the same time. Although this worked out okay, the onboarding process can be made easier to follow and smoother at future camps.
*Note*: The irrationality team of 5 people ended up splitting into two sub-groups since one of the topics seemed too small in scope for 5 people. We suggest limiting group sizes to 4 people at future camps.
2. **Integrating outside advisors:**Many senior AI Safety researchers replied slowly to our email requests to advise our teams, presumably because of busy schedules. This led to a dilemma:A. If we waited until we knew what the research topics would be, then we might not have gotten an answer from potential advisors in time.
B. If we acted before topics had been selected, we would end up contacting many senior researchers who were not specialised in the final topics.

At the first camp, we lacked time for working out a clear strategy, so teams ended up having to reach out to advisors we found. For future camps, it should be easier to connect advisors with teams given that the next organisers are already on the move. Hopefully, experienced researchers reading this post will also be inclined to offer a few spare hours to review research proposals and draft papers (please send us a short [email](mailto:contact@aisafety.camp)).

#### Next camps

**The next camps will happen in:**

4-14 Oct 2018:

Prague, Czechia

in collaboration with the [Czechia Association for Effective Altruism](http://effective-altruism.com/ea/1ls/introducing_czech_association_for_effective/) (they will also organize the [Human-aligned AI Summer School](http://humanaligned.ai/) in August)

~ March 2019:

Blackpool, United Kingdom

at the [EA Hotel](https://www.facebook.com/groups/1624791014242988/) (offers free accommodation for researchers)

**If you’ve gotten this far, we can use your contribution:**

* [Apply](https://goo.gl/forms/HyjnH4FjkSh0knHf2) to join the Prague camp
* [Email](mailto:contact@aisafety.camp) contact@aisafety.camp if you are considering
	+ advising research teams on their projects
	+ contributing your skills to organising camps
	+ funding future camps
	+ running your own edition next year

	criteria: experienced organisers who will run our general format & uphold a high quality standard that reflects well on the wider research community
* [Join](https://www.facebook.com/groups/AISafetyCamp/) our Facebook group to stay informed

#### **Acknowledgement**

Centre for Effective Altruism (€2,961)

Machine Intelligence Research Institute (€3,223)

Greg Colbourn (€3,430)

Lotta and Claes Linsefors (€4,000)


AISC 1: Research Summaries

The 2018 Gran Canaria AI safety camp teams have worked hard in the preparation of the camp and in the 10 day sprint. Each team has written a brief summary of the work they did during the camp:

#### Irrationality

Team: *Christopher Galias, Johannes Heidecke, Dmitrii Krasheninnikov, Jan Kulveit, Nandi Schoots*

* Our team worked on how to model human (ir)rationality in the context of value learning when trying to learn a human’s reward function based on expert demonstrations with inverse reinforcement learning (IRL).
* We focussed on two different sub-topics: bounded rationality and time-correlated irrationality.
* Bounded rationality topic:
	+ We analyzed the difference between perfectly rational and boundedly rational agents and why the latter might provide a better model for human behavior, explaining many biases observed in human thinking.
	+ We looked at existing formalizations of bounded rationality, especially an information theoretic perspective introduced by Ortega and Braun.
	+ We started investigating how to model bounded rational agents for reinforcement learning problems.
	+ We began formalizing how to model the inverse step of IRL for bounded rational agents, based both on Maximum Causal Entropy IRL and Guided Cost Learning.
	+ We set up a small test environment with many satisficing solutions and an optimal solution which is hard to find. We collected human expert demonstrations for this environment and compared it to the performance of a fully rational computer agent. The observed differences support the claim that bounded rationality models are needed in IRL to extract adequate reward functions.
	+ We received funding from Paul Christiano to continue our work.
* Time-correlated irrationality topic:
	+ The project consists of 2 parts: introducing a Laplace prior on the softmax temperatures of the transitions of the Boltzmann-rational agent, and enforcing a correlation between the temperatures at nearby timesteps.
	+ During the camp we worked out the math & the algorithm for the first part, and have started working on the implementation.
	+ The second part of the project and the writeup will be done in the following months. We plan to both work remotely and meet up in person.

#### Wisdom

Team: *Karl Koch, David Kristoffersson, Markus Salmela, Justin Shovelain*

We further developed tools for determining the harm versus benefit of projects on the long-term future:

* (Context: We have earlier work here, notably a decision tree for analyzing projects.)
* Heuristics: Worked extensively on developing practical heuristics for determining whether a technological development is net beneficial or harmful in the long run
* Scaffolding: Defined a wider context for the decision tree, to tell you when to use the decision tree and how to improve interventions/projects to be more good for the world.
* Race/competitive dynamics: Modeled some conditions of generating competitive races.
* Information concealment: Incorporated information from man-made disasters and information concealment

Developed a potential existential risk reduction funding delegation strategy for rich donors:
* Analyzed how to maximize a funder’s ability to update on data and use the knowledge of others, and yet mostly avoid the principal agent problem and Goodhart’s law
* Developed a funding organization design with expert delegates, collaborative investment decisions, and strong self-improving elements

#### Zero Safety

Team: *Vojta Kovarik, Igor Sieradzki, Michael Świętek*

* Goal: Better understand the strategy learned by Alpha Zero algorithm
* Implemented Alpha Zero in Gomoku, trained Alpha Zero in (a) 6\*6 board, 4-in-a-row and (b) 8\*8, 5-in-a-row
* Training the neural net in (a) took ~40M samples. We managed to train a new neural net using only 350 unique samples in such a way that the resulting strategy is very similar to the original Alpha Zero player.
* This led us to discover a weakness in the strategy learned by both the new Alpha Zero and the original one.
* Future plans: Test on more complex games, experiment with more robust ways of finding representative subsets of the training data, visualize these representative subsets in an automated way.

#### Safe AF

Team: *James Bell, Linda Linsefors, Caspar Oesterheld, Joar Skalse*

* Investigated the behaviour of common very simple machine learning algorithms in Newcomb like contexts, with the idea of trying to figure out what decision theory they are implicitly implementing.
* Specifically we looked at the epsilon-greedy and softmax algorithms for bandit problems. At each step these algorithms compute a probability distribution over actions and then draw their next action from that distribution. The reward for each action depended on the probability distribution that the algorithms had found as an intermediate step but they were trained in the standard way i.e. assuming that there was no such dependence.
* Formulated a selection of decision theory problems as bandit problems. Such bandit problems provide a general enough framework to include variants of playing a prisoners dilemma against a copy, evidential blackmail and death in Damascus.
* We found that the algorithms did not coherently follow any established decision theory, however they did show a preference for ratifiable choices of probability distribution and we were able to find some results on their convergence properties. We are writing a paper with our results.
* Later published: [Reinforcement Learning in Newcomblike Environments](https://aisafety.camp/feed/_wp_link_placeholder)

#### Side effects in Gridworlds

Team: *Jessica Cooper, Karol Kubicki, Gavin Leech, Tom McGrath*

* Implemented a baseline Q-learning agent for gridworld environments.
* Implemented inverse reinforcement learning in the Sokoban gridworld from Deepmind’s original paper.
* Created new gridworlds to cover a wider variety of side effects and expose more nuances, for instance the difficulty in defining “leaving the environment unchanged” when the environment is dynamic or stochastic.
* Code is available on [our Github repository](https://github.com/side-grids/ai-safety-gridworlds) and Gavin Leech has written [a blog post](https://www.gleech.org/grids/) that goes into more detail.
* Future plans:
	+ Generalise the tools that we created to work with arbitrary pycolab environments.
	+ Add maximum entropy deep IRL.
	+ Submit a pull request with the above to the Safety Gridworlds repository in order to make it easier for others to get started doing machine learning safety research.

#### Last but not least…

We would like to thank those who have funded the camp: MIRI, CEA, Greg Colbourne, Lotta and Claes Linsefors.


epistemic range

epistemic range
---------------

there is a [*security mindset*](https://www.lesswrong.com/posts/8gqrbnW758qjHFTrH/security-mindset-and-ordinary-paranoia)-ish general principle, of which [motivated stopping and motivated continuation](https://www.lesswrong.com/posts/L32LHWzy9FzSDazEg/motivated-stopping-and-motivated-continuation) are *ordinary paranoia*-ish special cases, which i call "epistemic range".

motivated stopping and motivated continuation are *heuristics* that catch some failure modes where you pursue an epistemic investigation — an instance of reasoning about a question in order to improve your belief state about the answer — to whichever extent lets you get a belief state that is the one you *want* to get. of course, in epistemic rationality, you should not *want* to believe any particular thing; you want your belief state to correspond to whatever is actually true.

and, that there can be such a thing as {stopping an epistemic inquiry too early} or {continuing an epistemic inquiry for too long} imply that there is a range, and maybe even a particular point, at which you should stop. in this respect, i believe epistemology to be akin to science: you would want to (do the kind of thing that is equivalent to) *preregister* epistemic investigations with a method for knowing when to stop.

personally, i believe that a good rule of thumb for when to stop is when [it feels like you can just as easily come up with narratives for multiple mutually incompatible possibilities](overcoming-narratives.html). on [dath ilan](https://www.lesswrong.com/tag/dath-ilan/discussion), they probably have a more robust notion of where to stop, and look at the belief state, and decide that this is what you believe for now until you either

* get more object-level evidence about the topic
* get more meta-level evidence about where your stopping point should be
* get more contextual evidence about how much confidence you need in this belief, to accomplish your goals

which occur to me as the three main things that impact how far one should want to pursue an epistemic investigation.

note that epistemic range depends, among other things, on where your epistemic investigation has gotten you; as a mathematical function, epistemic range should be a function from current-state-of-epistemic-investigation to boolean (whether to stop or not), not a function from question and context to static number.

on a particular subject matter, the notion of {how far you should go} is your **epistemic range**. the set of all your epistemic ranges on various questions is your **epistemic frontier**, and it nicely draws a shape representing how much you can figure out.


a short chat about realityfluid

a short chat about realityfluid
-------------------------------

(anonymized and lightly edited)


*this work was done by [Tamsin Leake](https://carado.moe) and [Julia Persson](https://www.lesswrong.com/users/juliahp) at [Orthogonal](https://orxl.org).*

*thanks to [mesaoptimizer](https://mesaoptimizer.com/) for his help putting together this post.*

formalizing the QACI alignment formal-goal
------------------------------------------

what does the [QACI](qaci.html) plan for [formal-goal alignment](formal-alignment.html) actually look like when formalized as math? in this post, we'll be presenting our current formalization, which we believe has most critical details filled in.

this post gives a brief explanation of what QACI tries to do, but people unfamiliar with this alignment scheme might want to read the [narrative explanation](narrative-explanation-qaci.html), which is a recommended introduction to QACI — though keep in mind that it's not entirely up to date.

this post straightforwardly builds up the math for QACI from the bottom up; and while it does explain all of the math, it does so by presenting it all at once. you might find prefer reading the companion post, [*"an Evangelion dialogue explaining the QACI alignment plan"*](qaci-invention-dialogue.html), which builds up this math gradually and provides more context.

### 1. math constructs

in this first part, we'll be defining a collection of mathematical constructs which we'll be using in the rest of the post.

#### 1.1. basic set theory

we'll be assuming basic set theory notation; in particular, A×B×C is the set of tuples whose elements are respectively members of the sets A, B, and C, and for n∈ℕ, Sn is the set of tuples of n elements, all members of S.

𝔹={⊤,⊥} is the set of booleans and ℕ is the set of natural numbers including 0.

given a set X, 𝒫(X) will be the set of subsets of X.

#S is the cardinality (number of different elements) in set S.

for some set X and some complete ordering <∈X2→𝔹, min< and max< are two functions of type 𝒫(X)\{∅}→X finding the respective minimum and maximum element of non-empty sets when they exist, using < as an ordering.

#### 1.2. functions and programs

if n∈ℕ, then we'll denote f∘n as repeated composition of f: f∘…∘f (n times), with ∘ being the composition operator: (f∘g)(x)=f(g(x)).

λx:X.B is an anonymous function defined over set X, whose parameter x is bound to its argument in its body B when it is called.

A→B is the set of functions from A to B, with → being right-associative (A→B→C is A→(B→C)). if f∈A→B→C, then f(x)(y) is simply f applied once to x∈A, and then the resulting function of type B→C being applied to y∈B. A→B is sometimes denoted BA in set theory.

A→HB is the set of always-halting, always-succeeding, deterministic programs taking as input an A and returning a B.

given f∈A→HB and x∈A, R(f,x)∈ℕ\{0} is the runtime duration of executing f with input x, measured in compute steps doing a constant amount of work each — such as turing machine updates.

#### 1.3. sum notation

i'll be using a syntax for sums ∑ in which the sum iterates over all possibles values for the variables listed *above* it, given that the constraints *below* it hold.

x,y∑y=1y=xmod2x∈{1,2,3,4}x≤2

says "for any value of x and y where these three constraints hold, sum y".

#### 1.4. distributions

for any countable set X, the set of distributions over X is defined as:

ΔX≔{f|f∈X→[0;1],∑x∈Xxf(x)≤1}

a function f∈X→[0;1] is a distribution ΔX over X if and only if its sum over all of X is never greater than 1. we call "mass" the scalar in [0;1] which a distribution assigns to any value. note that in our definition of distribution, we do not require that the distribution over all elements in the domain sums up to 1, but merely that it sums up to *at most* 1. this means that different distributions can have different "total mass".

we define ΔX0∈ΔX as the empty distribution: ΔX0(x)=0.

we define ΔX1∈X→ΔX as the distribution entirely concentrated on one element: ΔX1(x)(y)={1ify=x0ify≠x

we define NormalizeX∈ΔX→ΔX which modifies a distribution to make it sum to 1 over all of its elements, except for empty distributions:

NormalizeX(δ)(x)≔{δ(x)∑y∈Xyδ(y)ifδ≠ΔX00ifδ=ΔX0

we define UniformX as a distribution attributing equal value to every different element in a finite set X, or the empty distribution if X is infinite.

UniformX(x)≔{1#Xif#X∈ℕ0if#X∉ℕ

we define maxXΔ∈ΔX→𝒫(X) as the function finding the elements of a distribution with the highest value:

maxXΔ(δ)≔{x|x∈X,∀x′∈X:δ(x′)≤δ(x)}

#### 1.5. constrained mass

given distributions, we will define a notation which i'll call "constrained mass".

it is defined as a syntactic structure that turns into a sum:

v1,…,vpv1,…,vp𝐌[V]≔∑X1(x1)⋅…⋅Xn(xn)⋅Vx1:X1x1∈domain(X1)⋮⋮xn:Xnxn∈domain(Xn)C1C1⋮⋮CmCm

in which variables x are sampled from their respective distributions X, such that each instance of V is multiplied by X(x) for each x. constraints C and iterated variables v are kept as-is.

it is intended to weigh its expression body V by various sets of assignments of values to the variables v, weighed by how much mass the X distributions return and filtered for when the C constraints hold.

to take a fairly abstract but fully calculable example,

𝐌x,f[f(x,2)]≔∑x,f(λn:{1,2,3}.n10)(x)⋅Uniform{min,max}(f)⋅f(x,2)x:λn:{1,2,3}.n10x∈domain(λn:{1,2,3}.n10)f:Uniform{min,max}f∈domain(Uniform{min,max})xmod2≠0xmod2≠0=∑x,fx10⋅12⋅f(x,2)x∈{1,2,3}f∈{min,max}xmod2≠0=1⋅min(1,2)10⋅2+3⋅min(3,2)10⋅2+1⋅max(1,2)10⋅2+3⋅max(3,2)10⋅2=1⋅1+3⋅2+1⋅2+3⋅320=1+6+2+920=1820=910

in this syntax, the variables being sampled from distributions are allowed to be bound by an arbitrary amount of logical constraints or new variable bindings below it, other than the variables being sampled from distributions.

#### 1.6. bitstrings

𝔹\* is the set of finite bitstrings.

bitstrings can be compared using the lexicographic order <𝔹\*, and concatenated using the ‖ operator. for a bitstring x∈𝔹\*, |x|∈ℕ is its length in number of bits.

for any countable set X, EncodeX∈X→𝔹\* and will be some reasonable function to convert values to bitstrings, such that ∀(x,y)∈X2:EncodeX(x)=EncodeX(y)⇔x=y. "reasonable" entails constraints such as:

* it can be computed efficiently.
* it can be inverted efficiently and unambiguously.
* its output's size is somewhat proportional to the actual amount of information. for example, integers are encoded in binary, not unary.

#### 1.7. cryptography

we posit σ≔𝔹σ¯, the set of "signatures", sufficiently large bitstrings for cryptographic and uniqueness purposes, with their length defined as σ¯=231 for now. this *feels* to me like it should be enough, and if it isn't then something is fundamentally wrong with the whole scheme, such that no manageable larger size would do either.

we posit a function ExpensiveHash∈𝔹\*→Hσ, to generate fixed-sized strings from seed bitstrings, which must satisfy the following:

* it must be too expensive for the AI to compute *in any way* (including through superintelligently clever tricks), but cheap enough that we can compute it outside of the AI — for example, it could require quantum computation, and the AI could be restricted to classical computers
* it should take longer to compute (again, in any way) than the expected correct versions of Loc's f,g functions (as will be defined later) could afford to run
* it should tend to be collision-resistant

at some point, we might come up with more formal ways to define ExpensiveHash in a way that checks that it isn't being computed inside Loc's f,g functions, nor inside the AI.

#### 1.8. text and math evaluation

for any countable set X, we'll be assuming EvalMathX∈𝔹\*→{{x}|x∈X}∪{∅} to interpret a piece of text as a piece of math in some formal language, evaluating to either:

* a set of just one element of X, if the math parses and evaluates properly to an element of X
* an empty set otherwise

for example,

EvalMathℕ("1+2")={3}EvalMathℕ("hello")=∅

#### 1.9. kolmogorov simplicity

for any countable sets X and P:

KX−∈ΔX is some "[kolmogorov](https://en.wikipedia.org/wiki/Kolmogorov_complexity) simplicity" distribution over set X which has the properties of never assigning 0, and summing/converging to 1 over all of X. it must satisfy ∀x∈X:KX−(x)>0 and ∑x∈XxKX−(x)=1.

K− is expected to give more mass to simpler elements, in an information-theoretic sense.

notably, it is expected to "deduplicate" information that appears in multiple parts of a same mathematical object, such that even if x∈𝔹\* holds lots of information, K𝔹\*−(x) is not much higher (higher simplicity, i.e. lower complexity) to K𝔹\*×𝔹\*−(x,x).

we could define KX− [similarly to cross-entropy](https://www.lesswrong.com/posts/KcvJXhKqx4itFNWty/k-complexity-is-silly-use-cross-entropy-instead), with some universal turing machine UTM∈𝔹\*×ℕ→𝔹\* returning the state of its tape after a certain number of compute steps:

i,nKX−≔NormalizeX(λx:X.∑1(2|i|⋅(n+1))2)i∈𝔹\*n∈ℕUTM(i,n)=EncodeX(x)

*kolmogorov simplicity over X with a prior from P*, of type KP,X−~:P→ΔX, allows elements it samples over to share information with a prior piece of information in P. it is defined as KP,X−~(p)≔NormalizeX(λx:X.KP×X−(p,x)).

### 2. physics

in this section we posit some formalisms for modeling world-states, and sketch out an implementation for them.

#### 2.1. general physics

we will posit some countable set Ω of world-states, and a distribution Ωα∈ΔΩ of possible initial world-states.

we'll also posit a function Ωα→∈Ω→ΔΩ which produces a distribution of future world-states for any specific world-state in the universe starting at α.

given an initial world-state α∈Ω, we'll call Ωα→(α) the "universe" that it gives rise to. it must be the case that ∑ω∈ΩωΩα→(α)(ω)=1.

when α describes the start of a quantum universe, individual world-states Ω following it by Ωα→ would be expected to correspond to [many-worlds everett branches](https://www.lesswrong.com/tag/many-worlds-interpretation).

for concreteness's sake, we could posit Ω⊂𝔹\*, though note that α is expected to not just hold information about the initial state of the universe, but also about how it is computed forwards.

given a particular α∈Ω:

finally, we define SimilarPastsα∈Ω×Ω→[0;1] which checks how much they have past world-states ωpast in common:

ω1SimilarPastsα(ω2,ω2′)≔𝐌[Ωα→(ω1)(ω2)⋅Ωα→(ω1)(ω2′)]ω1:Ωα→(α)

#### 2.2. quantum turing machines

we will sketch out here a proposal for Ω, Ωα, and Ω→ such that our world-state w has hopefully non-exponentially-small Ωα→(α)(ω).

the basis for this will be a universal [quantum turing machine](https://en.wikipedia.org/wiki/Quantum_Turing_machine). we will posit:

* Tape≔{s|s∈𝒫(ℤ),#s∈ℕ} the set of turing machine tapes, as *finite* (thanks to #s∈ℕ) sets of relative integers representing positions in the tape holding a 1 rather than a 0.
* State some finite (#S∈ℕ) set of states, and some state0∈State.
* Ω≔Tape×State×ℤ: world-states consist of a tape, state, and machine head index.
* ΔΩq≔{f|f∈Ω→ℂ,∑ω∈Ωω‖f(ω)‖2=1} the set of "quantum distributions" over world-states
* Step∈ΔΩq→ΔΩq the "time step" operator running some universal turing machine's transition matrix to turn one quantum distribution of world-states into another

we'll also define Δℕ2∈Δℕ as the "quadratic realityfluid distribution" which assigns diminishing quantities to natural numbers, but only quadratically diminishing: Δℕ2(n)≔Normalizeℕ(1(n+1)2)

we can then define Ω→ as repeated applications of Step, with quadratically diminishing realityfluid:

n1,n2,sΩα→(ω1)(ω2)≔c⋅𝐌[s(n1,ω1)⋅s(n1+n2,ω2)]n1:Δℕ2n2:Δℕ2s(n,ω)=‖Step∘n(ΔΩ1(α))(ω)‖2

where the constant c is whatever scalar it needs to be for ∑ω∈ΩωΩα→(α)(ω)=1 to be satisfied.

this implementation of Ωα→ measures how much ω2 is in the future of ω1 by finding paths from α to ω1, and then longer paths from α to ω2.

and finally, we define Ωα as a distribution giving non-zero value to world-states (t,state0,0) where t is a tape where no negative-index cells are set to 1.

Ωα(t,s,i)≔{Δℕ2(∑n∈tn2n)ifs=state0,i=0,t⊂ℕ0otherwise

because we selected a universal (quantum) turing machine, there is at least one input tape implementing any single quantum algorithm, including the quantum algorithm implementing our physics.

### 3. implementing QACI

finally, we get into the core mechanisms of QACI.

the core idea of QACI is "blob location": mathematically formalizing the idea of locating our world and locating bitstrings (which i'll call "blobs") stored on computers within that world, out of the space of all possible computational universes, by sampling over functions which extract those blobs from world-states in Ω and functions which can produce a counterfactual world where that blob has been replaced with another blob of the same length (in number of bits).

#### 3.1. question blob and observation

throughout these functions, we will posit the following constants:

* the initial factual question blob q∈𝔹\*
* two "observation" blobs μ1∈𝔹\* and μ2∈𝔹\*

μ1,μ2 are variables which will be passed around, called "observations". in normal AI agent framings, an AI would have a history of actions and observations, and decide on its next action based on that; but, in the [one-shot](delegated-embedded-agency-decision-theory.html) framing we use, there is only a single action and a fixed set of observations. the observations, in practice, will be a very large pieces of data helping the AI locate itself in the multiverse of all possible computations, as well as get a better idea of how and where it is being ran. we will likely include in it things like:

* a full explanation of the QACI alignment plan, including the math
* the AI's code
* a dump of wikipedia and other large parts of the internet
* a copy of some LLM

μ1 will be produced before the question blob is generated, and μ2 will be produced after the question blob is generated but before the AI is launched.

#### 3.2. overview

the overall shape of what we're doing can be seen on the illustration below: we start at the start of the universe α, and use four blob locations and a counterfactual blob function call to locate five other world-states. the illustration shows distributions of future and past world-states, as well as a particular sampling of for all four blob locations.

* we sample ωμ1 using Loc(α,Ωα→(α),μ1,ξ), world-states containing the first observation μ1
* we sample ωμ2 using Loc(α,Ωα→(ωμ1),μ2,ξ), world-states containing the second observation μ2
* we sample ωq using Loc(α,Ωα→(ωμ1),q,ξ), world-states containing the question blob q, but requiring that its world-state ωq precede the world-state ωμ2
* we get ωq′, the world-state with a counterfactual question blob, using blob location γq found by sampling ωq
* we sample ωr using Loc(α,Ωα→(ωq′),r,ξ), possible world-states containing an answer to a given counterfactual question q′

![](qaci-math-2.svg)

the location path from ωq′ to ωr is used to run QACI intervals, where counterfactual questions q′ are inserted and answers r are located in their future.

(we could also build fancier schemes where we locate the AI's returned action, or its code running over time, in order to "tie more tightly" the blob locations to the AI — but it is not clear that this helps much with [blob location failure modes i'm concerned about](blob-quantum-issue.html).)

for the moment, we merely rely on μ1 and μ2 being uniquely identifying enough — though implementing them as *static bitstrings* might suffice, perhaps they could instead be implemented as *[lazily evaluated](https://en.wikipedia.org/wiki/Lazy_evaluation) associative maps*. when the AI tries to access members of those maps, code which computes or fetches information from the world (such as from the internet) would be executed determines the contents of that part of the observation object. this way, the observation would be conceptualized as a static object to the AI — and indeed it wouldn't be able to observe any mutations — but it'd be able to observe arbitrary amounts of the world, not just amounts we'd have previously downloaded.

we could make the QACI return not a scoring over actions but a proper utility function, but this only constrains the AI's action space and doesn't look like it helps in any way, including making QACI easier for the AI to make good guesses about. perhaps with utility functions we find a way to make the AI go "ah, well i'm not able to steer much future in world-states where i'm in hijacked sims", but it's not clear how or even that this helps much. so for now, the math focuses on this simple case of returning an action-scoring function.

#### 3.3. blob location

(remember that while this section does explain the blob location math, it does so by presenting it all at once. for a gentler introduction, see part **7. blob location** (and onwards) of the [dialogue explaining QACI](qaci-invention-dialogue.html))

for any blob length (in bits) n∈ℕ:

first, we'll posit Γn≔𝔹n→Ω the set of blob locations; they're identified by a counterfactual blob location function, which takes any counterfactual blob and return the world-state in which a factual blob has been replaced with that counterfactual blob.

Locn∈Ω×ΔΩ×𝔹n×Ξ→ΔΓn tries to locate an individual blob b (as a bitstring of length n) in a particular world-state sampled from the time-distribution (past or future) δ (which will usually be a distribution returned by Ωα→) within the universe starting at α.

it returns a distribution over counterfactual insertion functions of type 𝔹n→Ω which take a counterfactual blob and return the matching counterfactual world-state. the elements in that distribution typically sum up to much less than 1; the total amount they sum up to corresponds to how much Loc finds the given blob in the given world-state to begin with; thus, sampling from a distribution returned by Loc in a constrained mass calculation 𝐌 is useful even if said result is not used, because of its multiplying factor.

note that the returned counterfactual insertion function can be used to locate the factual world-state — simply give it the factual blob as input.

Ξ is some countably infinite set of arbitrary pieces of information which each call to Loc can use internally — the goal of this is for multiple different calls to Loc to be able to share some prior information, while only being penalized by K− for it once. for example, an element of Ξ might describe how to extract the contents of a specific laptop's memory from physics, and individual Loc calls only need to specify the date and the memory range. for concreteness, we can posit Ξ≔𝔹\*, the set of finite bitstrings.

f,g,ω,b′,τLocn(α,δ,b,ξ)(γ)≔𝐌[SimilarPastsα(ω,g(b′,τ))R(g,(b′,τ))+R(f,g(b′,τ))](f,g):KΞ,(Ω→H𝔹n×𝔹\*)×(𝔹n×𝔹\*→HΩ)−~(ξ)ω:λω:maxXΔ(λω:Ω.{δ(ω)iff(ω)=(b,τ)0otherwise).δ(ω)b′:Uniform𝔹n∀b′′∈𝔹n:γ(b′′)=g(b′′,τ)f(γ(b′′))=(b′′,τ)

Loc works by sampling a pair of functions f,g, which convert world-states forth and back into {pairs whose first element is the blob and whose second element represents everything in the world-state except the blob}.

that latter piece of information is called τ (tau), and rather than being sampled τ is defined by the return value of f on the original world-state — notably, τ is not penalized for being arbitrarily large, though f and g are penalized for their compute time.

for a given fixed pair of f and g, Loc finds the set of hypothesis world-states ω with the highest value within the time-distribution δ for which f,g work as intended. this is intended to select the "closest in time" world-states in δ, to avoid adversarial attackers generating their own factual blobs and capturing our location.

it then weighs locations using, for every counterfactual blob b′∈𝔹n:

* the degree to which counterfactual world-states tend to share pasts with the original factual world-state, for b′.
* the compute time of g and f on counterfactual blobs and world-states respectively.

note that Locn, by design, only supports counterfactual blobs whose length n is equal to the length of the initial factual blob b — it wouldn't really make sense to talk about "replacing bits" if the bits are different.

in effect, Loc takes random f,g decoding and re-encoding programs, measures how complex and expensive they are and how far from our desired distributions are world-states in which they work, and how close to the factual world-state their counterfactual world-states are.

#### 3.4. blob signing

we'll define Π≔𝔹|q|−σ¯, the set of possible answer bitstring payloads.

counterfactual questions will not be signed, and thus will be the set of bitstrings of the same length as the factual question — 𝔹|q|.

we'll define Sign∈Π×𝔹\*→𝔹|q| as Sign(π,k)≔ExpensiveHash(π‖k)‖π. this functions tags blob payloads using a "signature" generated from a seed bitstring, concatenating it to the blob payload.

#### 3.5. action-scoring functions

we will posit A⊂𝔹\* as the finite set of actions the AI can take, as a finite set of bitstrings.

we'll call U≔A→[0;1] the set of "scoring functions" over actions — functions which "have an opinion" about various actions. this is similar to utility functions, except it's over actions rather than over worlds or world-histories.

they can be composed using Compose∈ΔU→U, which could be simple scoring-function averaging:

uCompose(δ)(a)≔?𝐌[u(a)]u:δ

but alternatively, we could use something like [Diffractor's Rose bargaining](https://www.lesswrong.com/posts/vJ7ggyjuP4u2yHNcP/threat-resistant-bargaining-megapost-introducing-the-rose) to reduce the ability for scoring/utility functions to threaten each other — and notably ours.

Compose≔?Rose

(where i'm using ≔? to mean "maybe define this way, but i'm not sure")

#### 3.6. QACI query

using those, we define QACI∈Ω×Γ|q|×Ξ×𝔹|q|→ΔΠ which given a physics hypothesis α, a question blob location γq, and a blob location prior ξ, returns the highest guess returned answer payload πr for a given counterfactual question q′.

γrQACI(α,γq,ξ,q′)(πr)≔𝐌[1]γr:Loc|q|(α,Ωα→(γq(q′)),Sign(πr,q′),ξ)

QACI works by sampling answer blob locations γr, from world-states in the future of the counterfactual question world-state γq(q′), signed using q′.

with its first three parameters fixed, QACI becomes the straightforward counterfactual query function 𝔹|q|→ΔΠ as used in [the narrative explanation of QACI](narrative-explanation-qaci.html): one can call it with arbitrary counterfactual text inputs (within the size limitation), and get a distribution over possible answers, which can easily be collapsed using maxΠΔ.

#### 3.7. top-level QACI call

the top-level call to the QACI query function, QACI0∈Ω×Γ|q|×Ξ→ΔU interprets its output as a piece of math and executes it with, as parameters, various global and contextual values it might need access to, and returns a distribution over action-scoring functions:

πr,fQACI0(α,γq,ξ)(u)≔𝐌[1]πr:QACI(α,γq,ξ,q0′)f∈EvalMath{q}×{μ1}×{μ2}×Ω×Γ|q|×Ξ→U(πr)f(q,μ1,μ2,α,γq,ξ)=u

where q0′∈𝔹|q| is the initial counterfactual blob, such as "return a good scoring of actions" encoded in ASCII, padded with zeros to be of the right length.

QACI0's distribution over answers demands that the answer payload πr, when interpreted as math and with all required contextual variables passed as input (q,μ1,μ2,α,γq,ξ), returns an action-scoring function equal to u — this is how it measures the weight of any action-scoring function u.

𝐌[1] makes it that QACI0's distributions are only determined by the sampled variables and logical requirements.

EvalMath's f function having access to QACI's distribution over output texts rather than best candidate allows it to discard as many invalid candidates as it needs and stick to ones that match whatever constraits it has.

#### 3.8. action scoring

we'll posit the AI as AI∈U→A — a program which tries to satisfy a scoring over actions, by making a high-expected-score guess.

we define Score∈U, the action-scoring function which the AI will be making guesses about as a scoring function over actions, which happens to be one that is, hopefully, good. this is the scoring function for which the AI will be trying to produce an action that is as favorable as possible, within its limited capabilities.

α,ξ,γμ1,γμ2,γqScore≔Compose(λu:U.𝐌[NormalizeU(QACI0(α,γq,ξ))(u)])α:Ωαξ:KΞ−γμ1:Loc|μ1|(α,Ωα→(α),μ1,ξ)γμ2:Loc|μ2|(α,Ωα→(γμ1(μ1)),μ2,ξ)γq:Loc|q|(α,Ωα→(γμ1(μ1)),q,ξ)Ωα→(γq(q))(γμ2(μ2))>Ωα→(γμ2(μ2))(γq(q))

where the following variables are sampled:

* an initial state of the universe α.
* a blob location prior ξ, sampled for simplicity.
* the blob locations of μ1, μ2, and q.

and the world-state containing the second observation μ2 is required to be in the future of the word-state containing the question q.

then, we rate possible actions a by composing the scoring functions produced by QACI0 for all of those blob location hypotheses.

for any question location, the set of action-scoring functions sampled by QACI0 is normalized. this is because pairs of *AI-action location and question location* should not be penalized for having a "harder to find" answer — once the observations and question have been located, we want to obtain the answer no matter what.

finally, we'll just execute the action returned by AI(Score).