Updates and Lessons from AI Forecasting

Earlier this year, my research group [commissioned 6 questions](https://prod.hypermind.com/ngdp/en/showcase2/showcase.html?sc=JSAI&ref=bounded-regret.ghost.io) for professional forecasters to predict about AI. Broadly speaking, 2 were on geopolitical aspects of AI and 4 were on future capabilities:

\* Geopolitical:
 + How much larger or smaller will the largest Chinese ML experiment be compared to the largest U.S. ML experiment, as measured by amount of compute used?
 + How much computing power will have been used by the largest non-incumbent (OpenAI, Google, DeepMind, FB, Microsoft), non-Chinese organization?
\* Future capabilities:
 + What will SOTA (state-of-the-art accuracy) be on the MATH dataset?
 + What will SOTA be on the Massive Multitask dataset (a broad measure of specialized subject knowledge, based on high school, college, and professional exams)?
 + What will be the best adversarially robust accuracy on CIFAR-10?
 + What will SOTA be on Something Something v2? (A video recognition dataset)

Forecasters output a probability distribution over outcomes for 2022, 2023, 2024, and 2025. They have financial incentives to produce accurate forecasts; the rewards total \$5k per question (\$30k total) and payoffs are (close to) a [proper scoring rule](https://en.wikipedia.org/wiki/Scoring\_rule?ref=bounded-regret.ghost.io#Proper\_scoring\_rules), meaning forecasters are rewarded for outputting calibrated probabilities.

Depending on who you are, you might have any of several questions:

\* What the heck is a professional forecaster?
\* Has this sort of thing been done before?
\* What do the forecasts say?
\* Why did we choose these questions?
\* What lessons did we learn?

You're in luck, because I'm going to answer each of these in the following sections! Feel free to skim to the ones that interest you the most.

And before going into detail, here were my biggest takeaways from doing this:

\* Projected progress on math and on broad specialized knowledge are both faster than I would have expected. I now expect more progress in AI over the next 4 years than I did previously.
\* The relative dominance of the U.S. vs. China is uncertain to an unsettling degree. Forecasters are close to 50-50 on who will have more compute directed towards AI, although they do at least expect it to be within a factor of 10 either way.
\* It's difficult to come up with forecasts that reliably track what you intuitively care about. Organizations might stop reporting compute estimates for competitive reasons, which would confound both of the geopolitical metrics. They might similarly stop publishing the SOTA performance of their best models, or do it on a lag, which could confound the other metrics as well. I discuss these and other issues in the "Lessons learned" section.
\* Professional forecasting seems really valuable and underincentivized. (On that note, I'm interested in hiring forecasting consultants for my lab--please [e-mail](mailto:jsteinhardt@berkeley.edu) me if you're interested!)

\*Acknowledgments.\* The particular questions were designed by my students [Alex Wei](https://www.alexwei.org/?ref=bounded-regret.ghost.io), [Collin Burns](http://collinpburns.com/?ref=bounded-regret.ghost.io), Jean-Stanislas Denain, and [Dan Hendrycks](https://people.eecs.berkeley.edu/~hendrycks/?ref=bounded-regret.ghost.io). [Open Philanthropy](https://www.openphilanthropy.org/?ref=bounded-regret.ghost.io) provided the funding for the forecasts, and [Hypermind](https://www.hypermind.com/en/?ref=bounded-regret.ghost.io) ran the forecasting competition and constructed the aggregate summaries that you see below. Several people provided useful feedback on this post, especially Luke Muehlhauser and Emile Servan-Schreiber.

What is a professional forecaster? Has this been done before?
=============================================================

Professional forecasters are individuals, or often teams, who make money by placing accurate predictions in prediction markets or forecasting competitions. A good popular treatment of this is Philip Tetlock's book [\*Superforecasting\*](https://en.wikipedia.org/wiki/Superforecasting:\_The\_Art\_and\_Science\_of\_Prediction?ref=bounded-regret.ghost.io), but the basic idea is that there are a number of general tools and skills that can improve prediction ability and forecasters who practice these usually outperform even domain experts (though most strong forecasters have some technical background and will often read up on the domain they are predicting in). Historically, many forecasts were about geopolitical events (perhaps reflecting government funding interest), but there have been recent forecasting competitions about [Covid](https://goodjudgment.com/covidrecovery/?ref=bounded-regret.ghost.io)-[19](https://prod.hypermind.com/ngdp/en/showcase2/showcase.html?sc=Covid19&ref=bounded-regret.ghost.io) and the [future of food](https://www.metaculus.com/tournament/alt-protein-tournament/?ref=bounded-regret.ghost.io), among others.

At this point, you might be skeptical. Isn't predicting the future really hard, and basically impossible? An important thing to realize here is that forecasters usually output \*probabilities over outcomes\*, rather than a single number. So while I probably can't tell you what US GDP will be in 2025, I can give you a probability distribution. I'm personally pretty confident it will be more than \$700 billion and less than \$700 trillion (it's currently $21 trillion), although a professional forecaster would do much better than that.

There are a couple other important points here. The first is that forecasters' probability distributions are often \*significantly\* wider than the sorts of things you'd see pundits on TV say (if they even bother to venture a range rather than a single number). This reflects the future actually being quite uncertain, but even a wide range can be informative, and sometimes I see forecasted ranges that are a lot narrower than I expected.

The other point is that most forecasts are for at most a year or two into the future. Recently there have been some experimental attempts to forecast out to [2030](https://prod.hypermind.com/ngdp/en/showcase2/showcase.html?sc=AI2030&ref=bounded-regret.ghost.io), but I'm not sure we can say yet how successful they were. Our own forecasts go out to 2025, so we aren't as ambitious as the 2030 experiments, but we're still avant-garde compared to the traditional 1-2 year window. If you're interested in what we currently know about the feasibility of long-range forecasting, I recommend [this detailed blog post](https://www.openphilanthropy.org/blog/how-feasible-long-range-forecasting?ref=bounded-regret.ghost.io) by Luke Muehlhauser.

So, to summarize, a professional forecaster is someone who is paid to make accurate probabilistic forecasts about the future. Relative to pundits, they express significantly more uncertainty. The moniker "professional" might be a misnomer, since most income comes from prizes and I'd guess that most forecasters have a day job that produces most of their income. I'd personally love to live in a world with truly professional forecasters who could fully specialize in this important skill.

\*Other forecasting competitions.\* Broadly, there are all sorts of forecasting competitions, often hosted on [Hypermind](https://predict.hypermind.com/?ref=bounded-regret.ghost.io), [Metaculus](https://www.metaculus.com/?ref=bounded-regret.ghost.io), or [Good Judgment](https://goodjudgment.com/?ref=bounded-regret.ghost.io). There are also prediction markets (e.g. [PredictIt](https://www.predictit.org/?ref=bounded-regret.ghost.io)), which are a bit different but also incentivize accurate predictions. Specifically on AI, Metaculus had a recent [AI prediction tournament](https://www.metaculus.com/ai-progress-tournament/?ref=bounded-regret.ghost.io), and Hypermind ran the same questions on their own platform ([AI2023](https://prod.hypermind.com/ngdp/en/showcase2/showcase.html?sc=AI2023&ref=bounded-regret.ghost.io), [AI2030](https://prod.hypermind.com/ngdp/en/showcase2/showcase.html?sc=AI2030&ref=bounded-regret.ghost.io)). I'll discuss below how some of our questions relate to the AI2023 tournament in particular.

What the forecasts say
======================

Here are the point estimate forecasts put together into a single chart (expert-level is approximated as ~90%):

![forecast](https://bounded-regret.ghost.io/content/images/2021/10/forecast.png)

The MATH and Multitask results were the most interesting to me, as they predict rapid progress starting from a low present-day baseline. I'll discuss these in detail in the following subsections, and then summarize the other tasks and forecasts.

To get a sense of the uncertainty spread, I've also included aggregate results below (for 2025) on each of the 6 questions; you can find the results for other years [here](https://prod.hypermind.com/ngdp/en/showcase2/showcase.html?sc=JSAI&ref=bounded-regret.ghost.io). The aggregate combines all crowd forecasts but places higher weight on forecasters with a good track record.

![](https://bounded-regret.ghost.io/content/images/2021/08/hypermind\_us\_china.png "Machine-Learning: China vs USA")

![](https://bounded-regret.ghost.io/content/images/2021/08/hypermind\_incumbents.png "Machine-Learning: Rest of Field")

![](https://bounded-regret.ghost.io/content/images/2021/08/hypermind\_math.png "State of the Art: MATH")

![](https://bounded-regret.ghost.io/content/images/2021/08/hypermind\_multitask.png "State of the Art: Massive Multitask Language Understanding")

![](https://bounded-regret.ghost.io/content/images/2021/08/hypermind\_cifar10\_robust.png "State of the Art: CIFAR-10 8/255")

![](https://bounded-regret.ghost.io/content/images/2021/08/hypermind\_video-2.png "State of the Art: Something Something V2")

MATH
----

The MATH dataset consists of competition math problems for high school students. A Berkeley PhD student got in the ~75% range, while an IMO gold medalist got ~90%, but probably would have gotten 100% without arithmetic errors. The questions are free-response and not multiple-choice, and can contain answers such as $\frac{1 + \sqrt{2}}{2}$.

Current performance on this dataset is quite low--6.9%--and I expected this task to be quite hard for ML models in the near future. However, forecasters predict more than 50% accuracy\\* by 2025! This was a big update for me. (\\*More specifically, their median estimate is 52%; the confidence range is ~40% to 60%, but this is potentially artifically narrow due to some restrictions on how forecasts could be input into the platform.)

To get some flavor, here are 5 randomly selected problems from the "Counting and Probability" category of the benchmark:

\* How many (non-congruent) isosceles triangles exist which have a perimeter of 10 and integer side lengths?
\* A customer ordered 15 pieces of gourmet chocolate. The order can be packaged in small boxes that contain 1, 2 or 4 pieces of chocolate. Any box that is used must be full. How many different combinations of boxes can be used for the customer's 15 chocolate pieces? One such combination to be included is to use seven 2-piece boxes and one 1-piece box.
\* A theater group has eight members, of which four are females. How many ways are there to assign the roles of a play that involve one female lead, one male lead, and three different objects that can be played by either gender?
\* What is the value of $101^{3} - 3 \cdot 101^{2} + 3 \cdot 101 -1$?
\* 5 white balls and $k$ black balls are placed into a bin. Two of the balls are drawn at random. The probability that one of the drawn balls is white and the other is black is $\frac{10}{21}$. Find the smallest possible value of $k$.

Here are 5 randomly selected problems from the "Intermediate Algebra" category (I skipped one that involved a diagram):

\* Suppose that $x$, $y$, and $z$ satisfy the equations $xyz = 4$, $x^3 + y^3 + z^3 = 4$, $xy^2 + x^2 y + xz^2 + x^2 z + yz^2 + y^2 z = 12$. Calculate the value of $xy + yz + zx$.
\* If $\|z\| = 1$, express $\overline{z}$ as a simplified fraction in terms of $z$.
\* In the coordinate plane, the graph of $\|x + y - 1\| + \|\|x\| - x\| + \|\|x - 1\| + x - 1\| = 0$ is a certain curve. Find the length of this curve.
\* Let $\alpha$, $\beta$, $\gamma$, and $\delta$ be the roots of $x^4 + kx^2 + 90x - 2009 = 0$. If $\alpha \beta = 49$, find $k$.
\* Let $\tau = \frac{1 + \sqrt{5}}{2}$, the golden ratio. Then $\frac{1}{\tau} + \frac{1}{\tau^2} + \frac{1}{\tau^3} + \dotsb = \tau^n$ for some integer $n$. Find $n$.

You can see all of the questions at [this](https://github.com/hendrycks/math?ref=bounded-regret.ghost.io) git repo.

If I imagine an ML system getting more than half of these questions right, I would be pretty impressed. If they got 80% right, I would be super-impressed. The forecasts themselves predict accelerating progress through 2025 (21% in 2023, then 31% in 2024 and 52% in 2025), so 80% by 2028 or so is consistent with the predicted trend. This still just seems wild to me and I'm really curious how the forecasters are reasoning about this.

Multitask
---------

The Massive Multitask dataset also consists of exam questions, but this time they are a range of high school, college, and professional exams on 57 different subjects, and these \*are\* multiple choice (4 answer choices total). Here are five example questions:

\* (Jurisprudence) Which position does Rawls claim is the least likely to be adopted by the POP (people in the original position)?
 + (A) The POP would choose equality above liberty.
 + (B) The POP would opt for the ‘maximin’ strategy.
 + (C) The POP would opt for the ‘difference principle.’
 + (D) The POP would reject the ‘system of natural liberty.
\* (Philosophy) According to Moore’s “ideal utilitarianism,” the right action is the one that brings about the greatest amount of:
 + (A) pleasure. (B) happiness. (C) good. (D) virtue.
\* (College Medicine) In a genetic test of a newborn, a rare genetic disorder is found that has X-linked recessive transmission. Which of the following statements is likely true regarding the pedigree of this disorder?
 + (A) All descendants on the maternal side will have the disorder.
 + (B) Females will be approximately twice as affected as males in this family.
 + (C) All daughters of an affected male will be affected.
 + (D) There will be equal distribution of males and females affected.
\* (Conceptual Physics) A model airplane flies slower when flying into the wind and faster with wind at its back. When launched at right angles to the wind, a cross wind, its groundspeed compared with flying in still air is
 + (A) the same (B) greater (C) less (D) either greater or less depending on wind speed
\* (High School Statistics) Jonathan obtained a score of 80 on a statistics exam, placing him at the 90th percentile. Suppose five points are added to everyone’s score. Jonathan’s new score will be at the
 + (A) 80th percentile.
 + (B) 85th percentile.
 + (C) 90th percentile.
 + (D) 95th percentile.

Compared to MATH, these involve significantly less reasoning but more world knowledge. I don't know the answers to these questions (except the last one), but I think I could figure them out with access to Google. In that sense, it would be less mind-blowing if an ML system did well on this task, although it would be accomplishing an intellectual feat that I'd guess very few humans could accomplish unaided.

The actual forecast is that ML systems will be around 75% on this by 2025 (range is roughly 70-85, with some right-tailed uncertainty). I don't find this as impressive/wild as the MATH forecast, but it's still pretty impressive.

My overall take from this task and the previous one is that forecasters are pretty confident that we \*won't\* have the singularity before 2025, but at the same time there will be demonstrated progress in ML that I would expect to convince a significant fraction of skeptics (in the sense that it will look untenable to hold positions that "Deep learning can't do X").

Finally, to give an example of some of the harder types of questions (albeit not randomly selected), here are two from Professional Law and College Physics:

\* (College Physics) One end of a Nichrome wire of length 2L and cross-sectional area A is attached to an end of another Nichrome wire of length L and cross- sectional area 2A. If the free end of the longer wire is at an electric potential of 8.0 volts, and the free end of the shorter wire is at an electric potential of 1.0 volt, the potential at the junction of the two wires is most nearly equal to
 + (A) 2.4 V (B) 3.3 V (C) 4.5 V (D) 5.7 V
\* (Professional Law) The night before his bar examination, the examinee’s next-door neighbor was having a party. The music from the neighbor’s home was so loud that the examinee couldn’t fall asleep. The examinee called the neighbor and asked her to please keep the noise down. The neighbor then abruptly hung up. Angered, the examinee went into his closet and got a gun. He went outside and fired a bullet through the neighbor’s living room window. Not intending to shoot anyone, the examinee fired his gun at such an angle that the bullet would hit the ceiling. He merely wanted to cause some damage to the neighbor’s home to relieve his angry rage. The bullet, however, ricocheted off the ceiling and struck a partygoer in the back, killing him. The jurisdiction makes it a misdemeanor to discharge a firearm in public. The examinee will most likely be found guilty for which of the following crimes in connection to the death of the partygoer?
 + (A) Murder (B) Involuntary manslaughter (C) Voluntary manslaughter (D) Discharge of a firearm in public

You can view all the questions at [this](https://github.com/hendrycks/test?ref=bounded-regret.ghost.io) git repo.

Other questions
---------------

The other four questions weren't quite as surprising, so I'll go through them more quickly.

\*SOTA robustness:\* The forecasts expect consistent progress at ~7% per year. In retrospect this one was probably not too hard to get just from trend extrapolation. (SOTA was 44% in 2018 and 66% in 2021, with smooth-ish progress in-between.)

\*US vs. China:\* Forecasters have significant uncertainty in both directions, skewed towards the US being ahead in the next 2 years and China after that (seemingly mainly due to heavier-tailed uncertainty), but either one could be ahead and up to 10x the other. One challenge in interpreting this is that either country might stop publishing compute results if they view it as a competitive advantage in national security (or individual companies might do the same for competitive reasons).

\*Incumbents vs. rest of field:\* forecasters expect newcomers to increase size by ~10x per year for the next 4 years, with a central estimate of 21 EF-days in 2023. Note the [AI2023 results](https://prod.hypermind.com/ngdp/en/showcase2/showcase.html?sc=AI2023&ref=bounded-regret.ghost.io) predict the largest experiment by anyone (not just newcomers) to be 261EFLOP-s days in 2023, so this expects newcomers to be ~10x behind the incumbents, but only 1 year behind. This is also an example where forecasters have significant uncertainty--newcomers in 2023 could easily be in single-digit EF-days, or at 75 EF-days. In retrospect I wish I had included Anthropic on the list, as they are a new "big-compute" org that could be driving some fraction of the results, and who I wouldn't have intended to count as a newcomer (since they already exist).

\*Video understanding:\* Forecasters expect us to hit 88% accuracy (range: ~82%-95%) in 2025. In addition, they expect accuracy to increase at roughly 5%/year (though this presumably has to level off soon after 2025). This is faster than ImageNet, which has only been increasing at [roughly 2%/year](https://paperswithcode.com/sota/image-classification-on-imagenet?ref=bounded-regret.ghost.io). In retrospect this was an "easy" prediction in the sense that [accuracy has increased by 14% from Jan'18 to Jan'21](https://paperswithcode.com/sota/action-recognition-in-videos-on-something?ref=bounded-regret.ghost.io) (close to 5%/year), but it is also "bold" in the sense that progress since Jan'19 has been minimal. (Apparently forecasters are more inclined to average over the longest available time window.) In terms of implications, video recognition is one of the last remaining "instinctive" modalities that humans are very good at, other than physical tasks (grasping, locomotion, etc.). It looks like we'll be pretty good at a "basic" version of it by 2025, for a task that I'd intuitively rate as less complex than ImageNet but about as complex as CIFAR-100. Based on vision and language I expect an additional 4-5 years to master the "full" version of the task, so expect ML to have mostly mastered video by 2030. As before, this simultaneously argues \*against\* "the singularity is near" but \*for\* "surprisingly fast, highly impactful progress".

Why we chose these questions
============================

We liked the AI2023 questions (the previous prediction contest), but felt there were a couple categories that were missing. One was geopolitical (the first 2 questions), but the other one was benchmarks that would be highly informative about progress. The AI2023 challenge includes forecasts about a number of benchmarks, e.g. Pascal, Cityscape, few-shot on Mini-ImageNet, etc. But there aren't ones where, if you told me we'd have a ton of progress on them by 2025, it would update my model of the world significantly. This is because the tasks included in AI2023 are mostly in the regime where NNs do reasonably well and I expect gradual progress to continue. (I would have been surprised by the few-shot Mini-ImageNet numbers 3 years ago, but not since GPT-3 showed that few-shot works well at scale).

It's not so surprising that the AI2023 benchmarks were primarily ones that ML already does well on, because most ML benchmarks are created to be plausibly tractable. To enable more interesting forecasts, we created our own "hard" benchmarks where significant progress would be surprising. This was the motivation behind the MATH and Multitask datasets (we created [both](https://arxiv.org/abs/2103.03874?ref=bounded-regret.ghost.io) of [these](https://arxiv.org/abs/2009.03300?ref=bounded-regret.ghost.io) ourselves). As mentioned, I was pretty surprised by how optimistic forecasters were on both tasks, which updated me downward a bit on the task difficulty but also upward on how much progress we should expect in the next 4 years.

The other two benchmarks already existed but were carefully chosen. Robust accuracy on CIFAR was based on the premise that adversarial robustness is really hard and we haven't seen much progress--perhaps it's a particularly difficult challenge, which would be worrying if we care about the safety of AI systems. Forecasters instead predicted steady progress, but in retrospect I could have seen this myself. Even though adversarial robustness "feels" hard (perhaps because I work on it and spend a lot of time trying to make it work better), the actual year-on-year numbers showed a pretty clear 7%/year improvement.

The last task, video recognition, is an area that not many people work in currently, as it seems challenging compared to images (perhaps due to hardware constraints). But it sounds like we should expect steady progress on it in the coming years.

Lessons learned
===============

It can sometimes be surprisingly difficult to formalize questions that track an intuitive quantity you care about.

For instance, we initially wanted to include questions about economic impacts of AI, but were unable to. For instance, we wanted to ask "How much private vs. public investment will there be in AI?" But this runs into the question of what counts as investment--Do we count something like applying data science to agriculture? If you look at most metrics that you'd hope track this quantity, they include all sorts of weird things like that, and the weird things probably dominate the metric. We ran into similar issues for indicators of AI-based automation--e.g. do industrial robots on assembly lines count, even if they don't use much AI? For many economic variables, short-term effects may also disort results (investment might drop because of a pandemic or other shock).

There were other cases where we did construct a question, but had to be careful about framing. We initially considered using parameters rather than compute for the two geopolitical questions, but it's possible to achieve really high parameter counts in silly ways and some organizations might even do so for publicity (indeed we think this is already happening to some extent). Compute is harder to fake in the same way.

As discussed above, secrecy could cloud many of the metrics we used. Some organizations might not publish compute numbers for competitive reasons, and the same could be true of SOTA results on leaderboards. This is more likely if AI heats up significantly, so unfortunately I expect forecasts to be least reliable when we need them most. We could potentially get around this issue by interrogating forecasters' actual reasoning, rather than just the final output.

I also came to appreciate the value of doing lots of legwork to create a good forecasting target. The MATH dataset obviously was a lot of work to assemble, but I'm really glad we did because it created the single biggest update for me. I think future forecasting efforts should more strongly consider this lever.

Finally, even while often expressing significant uncertainty, forecasters can make bold predictions. I'm still surprised that forecasters predicted 52% on MATH, when current accuracy is 7% (!). My estimate would have had high uncertainty, but I'm not sure the top end of my range would have included 50%. I assume the forecasters are right and not me, but I'm really curious how they got their numbers.

Because of the possibility of such surprising results, forecasting seems really valuable. I hope that there's significant future investment in this area. Every organization that's serious about the future should have a resident or consultant forecaster. I am putting my money where my mouth is and currently hiring forecasting consultants for my research group; please [e-mail](mailto:jsteinhardt@berkeley.edu) me if this sounds interesting to you.

This post provides an update on a project obtaining professional forecasts about progress in AI. I’m not going to summarize the full post here and instead list a few high-level takeaways:

1. The author found two of the forecasts surprising, while the other four were more in line with his expectations. The surprising forecasts suggested faster progress than he would have expected, and he has updated accordingly.
2. The forecasts imply confidence that AGI won’t arrive before 2025, but at the same time there will be clear and impressive progress in ML by then.
3. If you want to use forecasting, one particularly valuable approach is to put in the necessary work to define a good forecasting target. In this case, the author’s research group did this by creating the <@MATH@>(@Measuring Mathematical Problem Solving With the MATH Dataset@) and <@Multitask@>(@Measuring Massive Multitask Language Understanding@) datasets.


Thoughts on Updatelessness

[This post assumes knowledge of decision theory, as discussed in Eliezer Yudkowsky’s [\*Timeless Decision Theory\*](http://intelligence.org/files/TDT.pdf).]

One interesting feature of some decision theories that I used to be a bit confused about is “updatelessness”. A thought experiment suitable for explaining the concept is [\*counterfactual mugging\*](https://wiki.lesswrong.com/wiki/Counterfactual\_mugging): “[Omega](https://wiki.lesswrong.com/wiki/Omega) [a being to be assumed a perfect predictor and absolutely trustworthy] appears and says that it has just tossed a fair coin, and given that the coin came up tails, it decided to ask you to give it $100. Whatever you do in this situation, nothing else will happen differently in reality as a result. Naturally you don’t want to give up your $100. But Omega also tells you that if the coin came up heads instead of tails, it’d give you $10000, but only if you’d agree to give it $100 if the coin came up tails.”

There are various alternatives to this experiment, which seem to illustrate a similar concept, although they are not all structurally isomorphic. For example Gary Drescher discusses Newcomb’s problem with transparent boxes in ch. 6.2 and retribution in ch. 7.3.1 of his book [\*Good and Real\*](https://www.gwern.net/docs/2006-drescher-goodandreal.pdf). Another relevant example is [Parfit’s hitchhiker](https://wiki.lesswrong.com/wiki/Parfit%27s\_hitchhiker).

Of course, you [win](http://lesswrong.com/lw/7i/rationality\_is\_systematized\_winning/) by refusing to pay. To strengthen the intuition that this is the case, imagine that the whole world just consists of one instance of counterfactual mugging and that you already know for certain that the coin came up tails. (We will assume that there is no anthropic uncertainty about whether you are in a simulation used to predict whether you would give in to counterfactual mugging. That is, Omega used some (not necessarily fully reliable) way of figuring out what you’d do. For example, Omega may have created you in a way that implies giving in or not giving in to counterfactual mugging.) Instead of giving money, let’s say thousands of people will be burnt alive if you give in while millions could have been saved if the coin had come up heads. Nothing else will be different as a result of that action. I don’t think there is any dispute over what choices maximizes expected utility for this agent.

The cause of dispute is that agents who give in to counterfactual mugging win in terms of expected value as judged from before learning the result of the coin toss. That is, prior to being told that the coin came up tails, an agent better be one that gives in to counterfactual mugging. After all, this will give her 0.5\\*$10,000 – 0.5\\*$100 in expectation. So, there is a conflict between what the agent would rationally want her future self to choose and what is rational for her future self to do. (Another example of this is the [absent-minded driver](http://lesswrong.com/lw/182/the\_absentminded\_driver/).) There is nothing particularly confusing about the existence of problems with such [inconsistency](https://en.wikipedia.org/wiki/Dynamic\_inconsistency#In\_game\_theory).

Because being an “updateless” agent, i.e. one that makes the choice based on how it would have wanted the choice to be prior to updating, is better for future instances of mugging, sensible decision theories would self-modify into being updateless with regard to all \*future\* information they receive. (Note that being updatelessness doesn’t mean that one doesn’t change one’s behavior based on new information, but that one goes through with the plans that one would have committed oneself to pursue before learning that information.) That is, an agent using a decision theory like (non-naive) evidential decision theory (EDT) would commit to giving in to counterfactual mugging and similar decision problems prior to learning that it ended up in the “losing branch”. However, if the EDT agent already knows that it is in the losing branch of counterfactual mugging and hasn’t thought about updatelessness, yet, it wouldn’t give in, although it might (if it is smart enough) self-modify into being updateless in the future.

One immediate consequence of the fact that updateless agents are better off is that one would want to program an AI to be updateless from the start. I guess it is this sense in which people like the researchers of the [Machine Intelligence Research Institute](https://intelligence.org/) consider updatelessness to be correct despite the fact that it doesn’t maximize expected utility in counterfactual mugging.

But maybe updateless is not even needed explicitly if the decision theory can take over epistemics. Consider the EDT agent, to whom Omega explains counterfactual mugging. For simplicity’s sake, let us assume that Omega explains counterfactual mugging and only then states which way the coin came up. After the explanation, the EDT agent could precommit, but let’s assume it can’t do so. Now, Omega opens her mouth to tell the EDT agent how the coin came up. Usually, decision theories are not connected to epistemics, so upon Omega uttering the words “the coin came up heads/tails”, Bayesian updating would run its due course. And that’s the problem, since after Bayesian updating the agent will be tempted to reject giving in, which is bad from the point of view of before learning which way the coin came up. To gain good evidence about Omega’s prediction of oneself, EDT may update in a different way to ensure that it would receive the money if the coin came up heads. For example, it could update towards the existence of both branches (which is basically equivalent to the updateless view of continuing to maintain the original position). Of course, self-modifying or just using some decision theory that has updatelessness built in is the much cleaner way to go.

Overall, this suggests a slightly different view of updatelessness. Updatelessness is not necessarily a property of decision theories. It is the natural thing to happen when you apply acausal decision theory to updating based on new information.

\*\*Acknowledgment:\*\* This work was funded by the Foundational Research Institute (now the [Center on Long-Term Risk](https://longtermrisk.org/)).


Decision Theory and the Irrelevance of Impossible Outcomes

(This post assumes some knowledge of the decision theory of Newcomb-like scenarios.)

One problem in the decision theory of Newcomb-like scenarios (i.e. the study of whether causal, evidential or some other decision theory is true) is that even the seemingly obvious basics are fiercely debated. Newcomb’s problem seems to be fundamental and the solution obvious (to both sides), and yet scholars disagree about its resolution. If we already fail at the basics, how can we ever settle this debate?

In this post, I propose a solution. Specifically, I will introduce a very plausible general principle that decision rules should abide by. One may argue that settling on powerful general rules (like the one I will propose) must be harder than settling single examples (like Newcomb’s problem). However, this is not universally the case. Especially in decision theory, we should expect general principles to be especially convincing because a common defense of two-boxing in Newcomb’s scenario is that Newcomb’s problem is just a weird edge case in which rationality is punished. By introducing a general principle that CDT (or, perhaps, EDT) violates, we can prove the existence of a \*general\* flaw.

Without further ado, the principle is: The decisions we make should not depend on the utilities assigned to outcomes that are impossible to occur. To me this principle seems obvious and indeed it is consistent with expected value calculations in non-Newcomb-like scenarios: Imagine having to deterministically choose an action from some set \*A\*. (We will ignore [mixed strategies](https://en.wikipedia.org/wiki/Strategy\_(game\_theory)#Pure\_and\_mixed\_strategies).) The next state of the world is sampled from a set of states \*S\* via a distribution P and depends on the chosen action. We are also given a utility function \*U\*, which assigns values to pairs of a state and an action. Let \*a\* be an action and let \*s\* be a possible state. If \*P\*(\*s\*,\*a\*) = 0 (or \*P\*(\*s\*|\*a\*)=0 or \*P\*(\*s\* given the causal implications of \*a\*)=0 – we assume all of these to be the equivalent in this non-Newcomb-like scenario), then it doesn’t matter what \*U\*(\*s\*,\*a\*) is, because in an [expected value](https://en.wikipedia.org/wiki/Expected\_value) calculation, \*U\*(\*s\*,\*a\*) will always be multiplied with \*P\*(\*s\*,\*a\*)=0. That is to say, any expected value decision rule gives the same outcome regardless of \*U\*(\*s\*,\*a\*). So, expected value decision rules abide by this principle at least in non-Newcomb-like scenarios.

Let us now apply the principle to a Newcomb-like scenario, specifically to the prisoner’s dilemma played against an exact copy of yourself. Your actions are \*C\* and \*D\*. Your opponent is the “environment” and can also choose between \*C\* (cooperation) and \*D\* (defection). So, the possible outcomes are (\*C\*,\*C\*), (\*C\*,\*D\*), (\*D\*,\*C\*) and (\*D\*,\*D\*). The probabilities P(\*C\*,\*D\*) and P(\*D\*,\*C\*) are both 0. Applied to this Newcomb-like scenario, the principle of the irrelevance of impossible alternatives states that our decision should only depend on the utilities of (\*C\*,\*C\*) and (\*D\*,\*D\*). Evidential decision theory behaves in accordance with this principle. (I leave it as an exercise to the reader to verify this.) Indeed, I suspect that it can be shown that EDT generally abides by the principle of the irrelevance of impossible outcomes. The choice of causal decision theory on the other hand \*does\* depend on the utilities of the impossible outcomes \*U\*(\*D\*,\*C\*) and \*U\*(\*C\*,\*D\*). Remember that in the prisoner’s dilemma the payoffs are such that \*U\*(\*D\*,\*x\*)>\*U\*(\*C\*,\*x\*) for any action \*x\* of the opponent, i.e. no matter the opponent’s choice it is always better to defect. This [dominance](https://en.wikipedia.org/wiki/Strategic\_dominance) is given as the justification for CDT’s decision to defect. But let us say we increase the utility of \*U\*(\*C\*,\*D\*) such that \*U\*(\*C\*,\*D\*)>U(\*D\*,\*D\*) and decrease the utility of \*U\*(\*D\*,\*C\*) such that \*U\*(\*D\*,\*C\*)>\*U\*(\*C\*,\*C\*). Of course, we must make these changes for the utility functions of both players so as to retain symmetry. After these changes, the dominance relationship is reversed: \*U\*(\*C\*,\*x\*)>\*U\*(\*D\*,\*x\*) for any action \*x.\* Of course, the new [payoff matrix](https://en.wikipedia.org/wiki/Normal-form\_game) is not that of a prisoner’s dilemma anymore – the game is different in important ways. But when played against a copy, these differences do not seem significant, because we only changed the utilities of outcomes that were impossible to achieve anyway. Nevertheless, CDT would switch from \*D\* to \*C\* upon being presented with these changes, thus violating the principle of the irrelevance of impossible outcomes. This is a \*systematic\* flaw in CDT: Its decisions depend on the utility of outcomes that it can already know to be impossible.

The principle of the irrelevance of impossible outcomes can be used beyond arguing against CDT. As you may remember from [my post on updatelessness](https://casparoesterheld.com/2016/11/21/thoughts-on-updatelessnes/), sensible decision theories will precommit to give Omega the money in the counterfactual mugging thought experiment. (If you don’t remember or haven’t read that post in the first place, this is a good time to catch up, because the following thoughts are based on the ideas from the post.) Even EDT, which ignores the utility of impossible outcomes, would self-modify in this way. However, the decision theory resulting from such self-modification violates the principle of the irrelevance of impossible outcomes. Remember that in counterfactual mugging, you give in because this was a good idea to precommit to when you didn’t yet know how the coin came up. However, once you know that the coin came up the unfavorable way, the positive outcome, which gave you the motivation to precommit, has become impossible. Of course, you only give in to counterfactual mugging if the reward in this now impossible branch is sufficiently high. For example, there is no reason to precommit to give in if you lose money in both branches. This means that once you have become updateless, you violate the principle of the irrelevance of impossible outcomes: your decision in counterfactual mugging depends on the utility you assign to an outcome that cannot happen anymore.

\*\*Acknowledgment:\*\* This work was funded by the Foundational Research Institute (now the [Center on Long-Term Risk](https://longtermrisk.org/)).


Is it a bias or just a preference? An interesting issue in preference idealization

When taking others’ preferences into account, we will often want to idealize them rather than taking them too literally. Consider the following example. You hold a glass of transparent liquid in your hand. A woman walks by, says that she is very thirsty and would like to drink from your glass. What she doesn’t know, however, is that the water in the glass is (for some reason not relevant to this example) poisoned. Should you allow her to drink? Most people would say you should not. While she does desire to drink out of the glass, this desire would probably disappear upon gaining knowledge of its content. Therefore, one might say that her object-level preference is to drink from the glass, while her idealized preference would be not to drink from it. There is not too much literature on preference idealization, as far as I know, but, if you’re not already familiar with it, anyway, consider looking into “[Coherent Extrapolated Volition](https://wiki.lesswrong.com/wiki/Coherent\_Extrapolated\_Volition)\*“.\*

Preference idealization is not always as easy as inferring that someone doesn’t want to drink poison, and in this post, I will discuss a particular sub-problem: accounting for [cognitive biases](https://en.wikipedia.org/wiki/Cognitive\_bias), i.e. systematic mistakes in our thinking, as they pertain to our moral judgments. However, the line between biases and genuine moral judgments is sometimes not clear.

Specifically, we look at cognitive biases that people exhibited in non-moral decisions, where their status as a bias to be corrected is much less controversial, but which can explain certain ethical intuitions. By offering such an [error theory](https://casparoesterheld.files.wordpress.com/2018/02/baggini\_fosl\_error\_theory.pdf) of a moral intuition, i.e. an explanation for how people could erroneously come to such a judgment, the intuition is called into question. Defendants of the intuition can respond that even if the bias can be used to explain the genesis of that moral judgment, they would nonetheless stick with that moral intuition. After all, the existence of \*all\* our moral positions can be explained by non-moral facts about the world – “[explaining is not explaining away](http://lesswrong.com/lw/oo/explaining\_vs\_explaining\_away/)”. Consider the following examples.

[\*Omission bias\*](https://en.wikipedia.org/wiki/Omission\_bias): People judge consequences of inaction as less severe than those of action. Again, this is clearly a bias in some cases, especially non-moral ones. For example, losing $1,000 by not responding to your bank in time is just as bad as losing $1,000 by throwing them out of the window. A business person who judges the two equivalent losses equally will \*ceteris paribus\* be more successful. Nonetheless, most people distinguish between [act and omission](https://plato.stanford.edu/entries/doing-allowing/) in cases like the [fat man trolley problem](https://en.wikipedia.org/wiki/Trolley\_problem#The\_fat\_man).

[\*Scope neglect\*](https://en.wikipedia.org/wiki/Scope\_neglect): The scope or size of something often has little or no effect on people’s thinking when it should have. For example, when three groups of people were asked what they would pay for interventions that would affect 2,000, 20,000, or 200,000 birds, people were willing to pay roughly the same amount of money irrespective of the number of birds. While scope neglect seems clearly wrong in this (moral) decision, it is less clearly so in other areas. For example, is a flourishing posthuman civilization with 2 trillion inhabitants really twice as good as one with 1 trillion? It is not clear to me whether answering “no” should be regarded as a judgment clouded by scope neglect (caused, e.g., by our [inability to imagine](https://casparoesterheld.com/2015/12/06/cheating-at-thought-experiments/) the two civilizations in question) or a moral judgment that is to be accepted.

[\*Contrast effect\*](https://en.wikipedia.org/wiki/Contrast\_effect) (also see [decoy effect](https://en.wikipedia.org/wiki/Decoy\_effect), [social comparison bias](https://en.wikipedia.org/wiki/Social\_comparison\_bias), [Ariely on relativity](https://en.wikipedia.org/wiki/Predictably\_Irrational#The\_Truth\_about\_Relativity), [mere subtraction paradox](https://en.wikipedia.org/wiki/Mere\_addition\_paradox#Alternative\_usage), [Less-is-better effect](https://en.wikipedia.org/wiki/Less-is-better\_effect)): Consider the following market of computer hard drives, from which you are to choose one.

| | | | |
| --- | --- | --- | --- |
| Hard drive model | Model 1 | Model 2 | Model 3 (decoy) |
| Price | $80 | $120 | $130 |
| Capacity | 250GB | 500GB | 360GB |

Generally, one wants to expend as little money as possible while maximizing capacity. In the absence of model 3, the decoy, people may be undecided between models 1 and 2. However, when model 3 is introduced into the market, it provides a new reference point. Model 2 is better than model 3 in all regards, which increases its attractiveness to people, even relative to model 1. That is, models 1 and 2 are judged by how they compare with model 3 rather than by their own features. The effect [clearly exposes](https://en.wikipedia.org/wiki/Independence\_of\_irrelevant\_alternatives) an instance of irrationality: the existence of model 3 doesn’t affect how model 1 compares with model 2. When applied to ethical evaluation, however, it calls into question a firmly held intrinsic moral preference for [social equality](https://en.wikipedia.org/wiki/Social\_equality) and [fairness](https://en.wikipedia.org/wiki/Fairness). Proponents of fairness seem to assess a person’s situation by comparing it to that of Bill Gates rather than judging each person’s situation separately. Similar to how the overpriced decoy changes our evaluation of the other products, our judgments of a person’s well-being, wealth, status, etc. may be seen as irrationally depending on the well-being, wealth, status, etc. of others.

Other examples include [peak-end rule](https://en.wikipedia.org/wiki/Peak%E2%80%93end\_rule)/[extension neglect](https://en.wikipedia.org/wiki/Extension\_neglect)/[evaluation by moments](http://www.vwl.tuwien.ac.at/hanappi/TEI/momentsfull.pdf) and [average utilitarianism](https://en.wikipedia.org/wiki/Average\_and\_total\_utilitarianism); [negativity bias](https://en.wikipedia.org/wiki/Negativity\_bias) and [caring more about suffering than about happiness](https://foundational-research.org/the-case-for-suffering-focused-ethics/); [psychological distance](https://en.wikipedia.org/wiki/Construal\_level\_theory) and [person-affecting views](https://en.wikipedia.org/wiki/Person-affecting\_view); [status-quo bias](https://en.wikipedia.org/wiki/Status\_quo\_bias) and various population ethical views (person-affecting views, the belief that most sentient beings that already exist have lives worth living); [moral credential effect](https://en.wikipedia.org/wiki/Moral\_credential\_effect); [appeal to nature](https://en.wikipedia.org/wiki/Appeal\_to\_nature) and [social Darwinism](https://en.wikipedia.org/wiki/Social\_Darwinism)/[normative evolutionary ethics](https://en.wikipedia.org/wiki/Evolutionary\_ethics#Normative\_evolutionary\_ethics).

\*\*Acknowledgment:\*\* This work was funded by the Foundational Research Institute (now the [Center on Long-Term Risk](https://longtermrisk.org/)).


“Betting on the Past” by Arif Ahmed

[This post assumes knowledge of decision theory, as discussed in Eliezer Yudkowsky’s [Timeless Decision Theory](http://intelligence.org/files/TDT.pdf) and in Arbital’s [Introduction to Logical Decision Theory](https://arbital.com/p/logical\_dt/).]

I recently discovered an interesting thought experiment, “[Betting on the Past](http://bjps.oxfordjournals.org/content/65/4/665)” by Cambridge philosopher Arif Ahmed. It can be found in his book [Evidence, Decision and Causality](http://www.cambridge.org/gb/academic/subjects/philosophy/philosophy-science/evidence-decision-and-causality), which is an elaborate defense of [Evidential Decision Theory](https://wiki.lesswrong.com/wiki/Evidential\_Decision\_Theory) (EDT). I believe that Betting on the Past may be used to money-pump non-EDT agents, refuting [Causal Decision Theories](https://plato.stanford.edu/entries/decision-causal/) (CDT), and potentially even ones that use [logical conditioning](https://arbital.com/p/logical\_dt/), such as [Timeless Decision Theory](https://wiki.lesswrong.com/wiki/Timeless\_decision\_theory) (TDT) or [Updateless Decision Theory](https://wiki.lesswrong.com/wiki/Updateless\_decision\_theory) (UDT). At the very least, non-EDT decision theories are unlikely to win this bet. Moreover, no conspicuous perfect predicting powers, genetic influences, or manipulations of decision algorithms are required to make Betting on the Past work, and anyone can replicate the game at home. For these reasons, it might make a more compelling case in favor of EDT than the [Coin Flip Creation](http://lesswrong.com/r/discussion/lw/oih/did\_edt\_get\_it\_right\_all\_along\_introducing\_yet/), a problem I recently proposed in an attempt to defend EDT’s answers in [medical Newcomb problems](http://lesswrong.com/lw/gu1/decision\_theory\_faq/#medical-newcomb-problems). In Ahmed’s thought experiment, Alice faces the following decision problem:

> \*Betting on the Past\*: In my pocket (says Bob) I have a slip of paper on which is written a proposition P. You must choose between two bets. Bet 1 is a bet on P at 10:1 for a stake of one dollar. Bet 2 is a bet on P at 1:10 for a stake of ten dollars. So your pay-offs are as in [Figure 1]. Before you choose whether to take Bet 1 or Bet 2 I should tell you what P is. It is the proposition that the past state of the world was such as to cause you now to take Bet 2. [[Ahmed 2014, p. 120](http://www.cambridge.org/gb/academic/subjects/philosophy/philosophy-science/evidence-decision-and-causality)]
>
>

Ahmed goes on to specify that Alice could indicate which bet she’ll take by either raising or lowering her hand. One can find a detailed discussion of the thought experiment’s implications, as well as a formal analysis of CDT’s and EDT’s decisions in Ahmed’s book. In the following, I want to outline a few key points.

Would CDT win in this problem? Alice is betting on a past state of the world. She can’t causally influence the past, and she’s uncertain whether the proposition is true or not. In either case, Bet 1 strictly dominates Bet 2: no matter which state the past is in, Bet 1 always yields a higher utility. For these reasons, causal decision theories would take Bet 1. Nevertheless, as soon as Alice comes to a definite decision, she updates on whether the proposition is true or false. If she’s a causal agent, she then finds out that she has lost: the past state of the world was such as to cause her to take Bet 1, so the proposition is false. If she had taken Bet 2, she would have found out that the proposition was correct, and she would have won, albeit a smaller amount than if she had won with Bet 1.

Betting on the Past seems to qualify as a kind of [Newcomb’s paradox](https://en.wikipedia.org/wiki/Newcomb%27s\_paradox); it seems to have an equivalent payoff matrix (Figure 1).

\*\*Figure 1\*\*: Betting on the past has a similar payoff matrix to Newcomb’s paradox

| | | |
| --- | --- | --- |
| | P is true | P is false |
|  Take Bet 1 | 10 | -1 |
|  Take Bet 2 | 1 | -10 |

Furthermore, its causal structure seems to resemble those of e.g. the Smoking Lesion or Solomon’s problem, indicating it as a kind of medical Newcomb problem. In medical Newcomb problems, a “Nature” node determines both the present state of the world (whether the agent is sick/will win the bet) and the agent’s decision (see Figure 2). In this regard, they differ from Newcomb’s original problem, where said node refers to the agent’s decision algorithm.

\*\*Figure 2\*\*: Betting on the past (left) has a similar causal structure to medical Newcomb problems (right).

 [![screen-shot-2017-01-24-at-13-29-52](https://casparoesterheld.files.wordpress.com/2017/02/screen-shot-2017-01-24-at-13-29-52.png?w=317&resize=317%2C383&h=383#038;h=383 "screen-shot-2017-01-24-at-13-29-52")](https://casparoesterheld.com/screen-shot-2017-01-24-at-13-29-52/) [![screen-shot-2017-01-24-at-13-46-40](https://casparoesterheld.files.wordpress.com/2017/02/screen-shot-2017-01-24-at-13-46-40.png?w=315&resize=315%2C383&h=383#038;h=383 "screen-shot-2017-01-24-at-13-46-40")](https://casparoesterheld.com/screen-shot-2017-01-24-at-13-46-40/)
One could object to Betting on the Past being a medical Newcomb problem, since the outcomes conditional on our actions here are certain, while e.g. in the Smoking Lesion, observing our actions only shifts our probabilities in degrees. I believe this shouldn’t make a crucial difference. On the one hand, we can conceive of absolutely certain medical Newcomb cases like the \*Coin Flip Creation\*. On the other hand, Newcomb’s original problem is often formalized with absolute certainties as well. I’d be surprised if probabilistic vs. certain reasoning would make a difference to decision theories. First, we can always approximate certainties to an arbitrarily high degree. We might ask ourselves why a negligible further increase in certainty would at some point suddenly completely change the recommended action, then. Secondly, we’re never really certain in the real world anyway, so if the two cases would be different, this would render all thought experiments useless that use absolute certainties.

If Betting on the Past is indeed a kind of medical Newcomb problem, this would be an interesting conclusion. It would follow that if one prefers Bet 2, one should also one-box in medical Newcomb problems. And taking Bet 2 seems so \*obviously\* correct! I point this out because one-boxing in medical Newcomb problems is what EDT would do, and it is often put forward as both a counterexample to EDT and as the decision problem that separates EDT from [Logical Decision Theories](https://arbital.com/p/logical\_dt/) (LDT), such as TDT or UDT. (See e.g. [Yudkowsky 2010](http://intelligence.org/files/TDT.pdf), p.67)

Before we examine the case for EDT further, let’s take a closer look at what LDTs would do in Betting on the Past. As far as I understand, LDTs would take correlations with other decision algorithms into account, but they would ignore “retrocausality” (i.e. smoke in the smoker’s lesion, chew gum in the chewing gum problem, etc.). If there is a purely physical cause, then this causal node isn’t altered in the logical counterfactuals that an LDT agent reasons over. Perhaps if the bet was about the state of the world yesterday, LDT would still take Bet 2. Clearly, LDT’s algorithm already existed yesterday, and it can influence this algorithm’s output; so if it chooses Bet 2, it can change yesterday’s world and make the proposition true. But at some point, this reasoning has to break down. If we choose a more distant point in the past as a reference for Alice’s bet – maybe as far back as the birth of our universe – she’ll eventually be unable to exert any possible influence via logical counterfactuals. At some point, the correlation becomes a purely physical one. All she can do at that point is what opponents of evidential reasoning would call “managing the news” ([Lewis, 1981](http://www.tandfonline.com/doi/pdf/10.1080/00048408112340011)) – she can merely try to go for the action that gives her the best Bayesian update.

So, do Logical Decision Theories get it wrong? I’m not sure about that; they come in different versions, and some haven’t yet been properly formalized, so it’s hard for me to judge. I can very well imagine that e.g. Proof-Based Decision Theory would take Bet 2, since it could prove P to be either true or false, contingent on the action it would take. I would argue, though, that if a decision theory takes Bet 2 – and if I’m right about Betting on the Past being a medical Newcomb problem – then it appears it would also have to “one-box”, i.e. take the option recommended by EDT, in other medical Newcomb problems.

If all of this is true, it might imply that we don’t really need LDT’s logical conditioning and that EDT’s simple Bayesian conditioning on actions could suffice. The only remaining difference between LDT and EDT would then be EDT’s lack of [updatelessness](https://casparoesterheld.com/2016/11/21/thoughts-on-updatelessnes/). What would an updateless version of EDT look like? Some progress on this front has already been made by [Everitt, Leike, and Hutter 2015](https://jan.leike.name/publications/Sequential%20Extensions%20of%20Causal%20and%20Evidential%20Decision%20Theory%20-%20Everitt,%20Leike,%20Hutter%202015.pdf). Caspar Oesterheld and I hope to be able to say more about it soon ourselves.

Acknowledgement
---------------

I wrote this post while working for the Foundational Research Institute, which is now the [Center on Long-Term Risk](https://longtermrisk.org/).


The average utilitarian’s solipsism wager

The following prudential argument is relatively common in my circles: [We probably live in a simulation](http://www.simulation-argument.com/), but if we don’t, our actions matter much more. Thus, expected value calculations are dominated by the utility under the assumption that we (or some copies of ours) are in the real world. Consequently, the simulation argument [affects our prioritization only slightly](https://foundational-research.org/how-the-simulation-argument-dampens-future-fanaticism) — we should still mostly act under the assumption that we are not in a simulation.

[A commonly cited analogy is due to Michael Vassar](http://www.33rdsquare.com/2012/10/jaan-tallinns-metaphysical-quest.html): “If you think you are Napoleon, and [almost] everyone that thinks this way is in a mental institution, you should still act like Napoleon, because if you are, your actions matter a lot.” An everyday application of this kind of argument is the following: Probably, you will not be in an accident today, but if you are, the consequences for your life are enormous. So, you better fasten your seat belt.

Note how these arguments do not affect the probabilities we assign to some event or hypothesis. They are only about the event’s (or hypothesis’) \*prudential weight\* — the extent to which we tailor our actions to the case in which the event occurs (or the hypothesis is true).

For total utilitarians (and many other consequentialist value systems), similar arguments apply to most theories postulating a large universe or multiverse. To the extent that it makes a difference for our actions, we should tailor them to the assumption that we live in a large multiverse with many copies of us because under this assumption we can affect the lives of many more beings.

For [average utilitarians](https://en.wikipedia.org/wiki/Average\_and\_total\_utilitarianism), the exact opposite applies. Even if they have many copies, they will have an impact on a much smaller \*fraction\* of beings if they live in a large universe or multiverse. Thus, they should usually base their actions on the assumption of a small universe, such as a universe in which Earth is the only inhabited planet. This may already have some implications, e.g. via the [simulation argument](https://foundational-research.org/how-the-simulation-argument-dampens-future-fanaticism) or the [Fermi paradox](https://en.wikipedia.org/wiki/Fermi\_paradox). If they also take the average over time — I do not know whether this is the default for average utilitarianism — they would also base their actions on the assumption that there are just a few past and future agents. So, average utilitarians are subject to a much stronger [Doomsday argument](https://en.wikipedia.org/wiki/Doomsday\_argument).

Maybe the bearing of such prudential arguments is even more powerful, though. There is some chance that [metaphysical solipsism](https://en.wikipedia.org/wiki/Metaphysical\_solipsism) is true: the view that only my (or your) own mind exists and that everything else is just an illusion. If solipsism were true, our impact on average welfare (or average preference fulfillment) would be enormous, perhaps 7.5 billion times bigger than it would be under the assumption that Earth exists — about 100 billion times bigger if you also [count humans that have lived in the past](https://en.wikipedia.org/wiki/World\_population#Number\_of\_humans\_who\_have\_ever\_lived). Solipsism seems to deserve a probability larger than one in 6 (or 100) billion. (In fact, I think solipsism is likely enough for this to qualify as a non-[Pascalian](https://en.wikipedia.org/wiki/Pascal's\_Wager) argument.) So, perhaps average utilitarians should maximize primarily for their own welfare?

### Acknowledgements

The idea of this post is partly due to Lukas Gloor. This work was funded by the Foundational Research Institute (now the [Center on Long-Term Risk](https://longtermrisk.org/)).


Anthropic uncertainty in the Evidential Blackmail

I’m currently writing a piece on anthropic uncertainty in Newcomb problems. The idea is that whenever someone simulates us to predict our actions, this leads us to have anthropic uncertainty about whether we’re in this simulation or not. (If we knew whether we were in the real world or in the simulation, then the simulation wouldn’t fulfill its purpose anymore.) This kind of reasoning changes quite a lot about the answers that decision theories give in predictive dilemmas. It makes their reasoning “more [updateless](https://casparoesterheld.com/2016/11/21/thoughts-on-updatelessnes/)”, since they reason from a more impartial stance: a stance from which they don’t know their exact position in the thought experiment, yet.

This topic isn’t new, but it hasn’t been discussed in-depth before. As far as I am aware, it has been brought up on LessWrong [by gRR](http://lesswrong.com/lw/asi/anthropic\_reasoning\_by\_cdt\_in\_newcombs\_problem/) and in [two](http://lesswrong.com/lw/5k/sleeping\_beauty\_gets\_counterfactually\_mugged/) [blog posts](http://lesswrong.com/lw/f37/naive\_tdt\_bayes\_nets\_and\_counterfactual\_mugging/) by Stuart Armstrong. Outside LessWrong, there is a post by [Scott Aaronson](http://www.scottaaronson.com/blog/?p=30), and one by [Andrew Critch](http://acritch.com/deserving-trust/#more-2581). The idea is also mentioned in passing by Neal (2006, p. 13). Are there any other sources and discussions of it that I have overlooked?

In this post, I examine what the assumption that predictions or simulations lead to anthropic uncertainty implies for the [Evidential Blackmail](https://agentfoundations.org/item?id=32) (also XOR Blackmail), a problem which is often presented as a counter-example to evidential decision theory (EDT) (Cf. Soares & Fallenstein, 2015, p. 5; Soares & Levinstein, 2017, pp. 3–4). A similar problem has been introduced as “Yankees vs. Red Sox” by Arntzenius (2008), and discussed by Ahmed and Price (2012). I would be very grateful for any kind of feedback on my post.

We could formalize the blackmailer’s procedure in the Evidential Blackmail something like this:

`def blackmailer():

    your\_action = your\_policy(receive\_letter)

    if predict\_stock() == “retain” and your\_action == “pay”:

        return “letter”

    elif predict\_stock() == “fall” and your\_action == “not pay”:

        return “letter”

    else

        return “no letter”`

Let p denote the probability P(retain) with which our stock retains its value a. The blackmailer asks us for an amount of money b, where 0

Schwarz, W. (2015). Lost memories and useless coins: revisiting the absentminded driver. Synthese, 192(9), 3011–3036.

Soares, N., & Fallenstein, B. (2015, July 7). Toward Idealized Decision Theory. arXiv [cs.AI]. Retrieved from

Soares, N., & Levinstein, B. (2017). Cheating Death in Damascus. Retrieved from


Complications in evaluating neglectedness

Neglectedness (or crowdedness) is a heuristic that effective altruists use to assess how much impact they could have in a specific cause area. It is usually combined with scale (a.k.a. importance) and tractability (a.k.a. solvability), which together are meant to approximate expected value. (In fact, under certain idealized definitions of the three factors, multiplying them [is equivalent to](https://80000hours.org/articles/problem-framework/#introducing-how-we-define-the-factors) expected value. However, this removes the heuristic nature of these factors and probably does not describe how people typically apply them.) For introductions and thoughts on the framework as well as neglectedness in particular see:

\* Benjamin Todd: [\*A framework for strategically selecting a cause\*](https://80000hours.org/2013/12/a-framework-for-strategically-selecting-a-cause/).
\* Paul Christiano: [\*Neglectedness and impact\*](https://80000hours.org/2014/01/neglectedness-and-impact/).
\* 80,000 hours: [\*How to compare different global problems in terms of impact\*](https://80000hours.org/articles/problem-framework/).
\* William MacAskill: \*Doing Good Better\*. Chapter 10.

One reason why the neglectedness heuristic and the framework in general are so popular is that they are much easier to apply than explicit cost-effectiveness or expected value calculations. In this post, I will argue that evaluating neglectedness (which may usually be seen as the most heuristic and easiest to evaluate part of the framework) is actually quite complicated. This is in part to make people more aware of issues that are sometimes not and often only implicitly taken into account. In some cases, it may also be an argument against using the heuristic at all. Presumably, most of the following considerations won’t surprise many practitioners. Nonetheless, it appears useful to write them down, which, to my knowledge, hasn’t been done before.

Neglectedness and diminishing returns
=====================================

There are a few different definitions of neglectedness. For example, consider the following three:

1. “If we add more resources to the cause, we can expect more promising interventions to be carried out.” ([source](https://80000hours.org/2013/12/a-framework-for-strategically-selecting-a-cause/))
2. You care about a cause much more than the rest of society. ([source](https://80000hours.org/2014/01/neglectedness-and-impact/))
3. “How many people, or dollars, are currently being dedicated to solving the problem?” ([source](https://80000hours.org/articles/problem-framework/#definition-2))

The first one is quite close to expected value-type calculations and so it is quite clear why it is important. The second and third are more concrete and easier to measure but ultimately only [relevant because they are proxies of the first](https://80000hours.org/articles/problem-framework/#why-is-it-important). If society is already investing a lot into a cause, then the most promising interventions in that cause area are already taken up and only less effective ones remain.

Because the second and, even more so, the third are easier to measure, I expect that, in practice, most people use these two when they evaluate neglectedness. Incidentally, these definitions also fit the terms “neglectedness” and “crowdedness” much better. I will argue that neglectedness in the second and third sense has to be translated into neglectedness into the first sense and that this translation is difficult. Specifically, I will argue that the [diminishing returns curves](https://en.wikipedia.org/wiki/Diminishing\_returns) on which the connection between already invested resources and the value of the marginal dollar is based on can assume different scales and shapes that have to be taken into account.

A standard diminishing return curve may look roughly like this:

![IMG\_20170621\_133952](https://casparoesterheld.files.wordpress.com/2017/06/img\_20170621\_133952.jpg?w=300&h=225)

The x-axis represents the amount of resources invested into some intervention or cause area, the y-axis represents the returns of that investment. The derivative of the returns (i.e., the marginal returns) decreases, [potentially](https://80000hours.org/2014/01/neglectedness-and-impact/) in inverse proportion to the cumulative investment.

Even if returns diminish in a way similar to that shape, there is still the question of the scale of that graph (not to be confused with the scale/importance of the cause area), i.e. whether values on the x-axis are in the thousands, millions or billions. In general, returns probably diminish slower in cause areas that are in some sense large and uniform. Take the global fight against malaria. Intervening in some areas is more effective than in others. For example, it is more effective in areas where malaria is more common, or where it is easier to, say, provide mosquito nets, etc. However, given how widespread malaria is (about 300 million cases in 2015), I would expect that there is a relatively large number of areas almost tied for the most effective places to fight malaria. Consequently, I would guess that once the most effective intervention is to distribute provide mosquito nets, even hundreds of millions do not diminish returns all that much.

Other interventions have much less room for funding and thus returns diminish much more quickly. For example, the returns of helping some specific person will usually diminish way before investing, say, a billion dollars.

If you judge neglectedness only based on the raw amount of resources invested into solving a problem ([as suggested by 80,000 hours](https://80000hours.org/articles/problem-framework/#how-to-assess-it-2)), then this may make small cause areas look a lot more promising than they actually are. Depending on the exact definitions, this remains the case if you combine neglectedness with scale and tractability. For example, consider the following two interventions:

1. The global fight against malaria.
2. The fight against malaria in some randomly selected subset of 1/100th of the global area or population.

The two should usually be roughly equally promising. (Perhaps 1 is a bit more promising because every intervention contained in 2 is also in 1. On the other hand, that would make “solve everything” hard to beat as an intervention. Of course, 2 can also be more or less promising if an unusual 1/100th is chosen.) But because the raw amount of resources invested into 1 is presumably 100 times as big as the amount of resources invested into 2, 2 would, on a naive view, be regarded as much more neglected than 1. The product of scale and tractability is the same in 1 and 2. (1 is a 100 times bigger problem, but solving it in its entirety is also roughly 100 times more difficult, though I presume that some definitions of the framework judge this differently. In general, it seems fine to move considerations out of neglectedness into tractability and scope [as long as](http://effective-altruism.com/ea/ss/the\_importantneglectedtractable\_framework\_needs/) they are not double-counted or forgotten.) Thus, the overall product of the three is greater for 2, which appears to be wrong. If on the other hand, neglectedness denotes the extent to which returns have diminished (the first of the three definitions given at the beginning of this section), then the neglectedness of 1 and 2 will usually be roughly the same.

Besides the scale of the return curve, the shape can also vary. In fact, I think many interventions initially face increasing returns from learning/research, creating economies of scale, specialization within the cause area, etc. For example, in most cause areas, the first $10,000 are probably invested into prioritization, organizing, or (potentially symbolic) interventions that later turn out to be suboptimal. So, in practice return curves may actually look more like the following:

![IMG\_20170621\_134248](https://casparoesterheld.files.wordpress.com/2017/06/img\_20170621\_134248.jpg?w=300&h=225)

This adds another piece of information (besides scale) that needs to be taken into account to translate the amount of invested resources into how much returns have diminished: how and when do returns start to diminish?

There are many other return curve shapes that may be less common but mess up the neglectedness framework more. For example, some projects produce some large amount of value if they succeed but produce close to no value if they fail. Thus, the (actual not expected) return curve for such projects may look like this:

![IMG\_20170621\_134241](https://casparoesterheld.files.wordpress.com/2017/06/img\_20170621\_134241.jpg?w=300&h=225)

Examples may include developing vaccines, colonizing Mars or [finding cause X](https://www.effectivealtruism.org/articles/three-heuristics-for-finding-cause-x/).

If such a cause area is already relatively crowded according to the third (and second) sense, that may make them \*less\* “crowded” in the first sense. For example, if nobody had invested money into finding a vaccine against malaria (and you don’t expect others to invest money into it into the future either, see below) then this cause area is maximally neglected in the second and third sense. However, given [how expensive clinical trials are](http://www.nature.com/nrd/journal/v16/n6/full/nrd.2017.70.html), the marginal returns of donating a few thousand dollars into it are essentially zero. If on the other hand, others have already contributed enough money to get a research project off the ground at all, then the marginal returns are higher, because there is at least some chance that your money will enable a trial in which a vaccine is found. (Remember that we don’t know the exact shape of the return curve, so we don’t know when the successful trial is funded.)

I would like to emphasize that the point of this section is not so much that people apply neglectedness incorrectly by merely looking at the amount of resources invested into a cause and not thinking about implications in terms of diminishing returns at all. Instead, I suspect that most people implicitly translate into diminishing returns and take the kind of the project into account. However, it may be beneficial if people were more aware of this issue and how it makes evaluating neglectedness more difficult.

Future resources
================

When estimating the neglectedness of a cause, we need to take into account, not only people who are currently working on the problem (as a literal reading of [80,000 hours’ definition](https://80000hours.org/articles/problem-framework/#definition-2) suggests), but also people who have worked on it in the past and future. If a lot of people have worked on a problem in the past, then this indicates that the low-hanging fruit has already been picked. Thus, even if nobody is working in the area anymore, marginal returns have probably diminished a lot. I can’t think of a good example where this is a decisive consideration because if an area has been given up on (such that there is a big difference between past and current attention), it will usually score low in tractability, anyway. Perhaps one example is the search for new ways to organize society, government and economy. Many resources are still invested into thinking about this topic, so even if we just consider resources invested today, it would not do well in terms of neglectedness. However, if we consider that people have thought about and “experimented” in this area for thousands of years, it appears to be even more crowded.

We also have to take future people and resources into account when evaluating neglectedness. Of course, future people cannot “take away” the most promising intervention in the way that current and past people can. However, their existence causes the top interventions [to be performed anyway](https://concepts.effectivealtruism.org/concepts/counterfactual-considerations/). For example, let’s say that there are 1000 equally costly possible interventions in an area, generating 1000, 999, 998, …, 1 “utils” (or lives saved, years of suffering averted, etc.), respectively. Each intervention can only be performed once. The best 100 interventions have already been taken away by past people. Thus, if you have money for one intervention, you can now only generate 900 utils. But if you know that future people will engage in 300 further interventions in that area, then whether you intervene or not actually only makes a difference of 600 utils. All interventions besides the one generating 600 utils would have been executed anyway. (In [Why Charities Don’t Differ Astronomically in Cost-Effectiveness](http://reducing-suffering.org/why-charities-dont-differ-astronomically-in-cost-effectiveness/), Brian Tomasik [makes a similar point](http://reducing-suffering.org/why-charities-dont-differ-astronomically-in-cost-effectiveness/#Returns\_look\_high\_before\_big\_players\_enter).)

The number of future people who would counterfactually engage in some cause area is an important consideration in many cause areas considered by effective altruists. In general, if a cause area is neglected by current and past people, the possibility of future people engaging in an intervention creates a lot of variance in neglectedness evaluations. If recently 10 people started working on an area, then it is very uncertain how much attention it will have in the future. And if it will receive a lot more attention regardless of our effort, then the neglectedness score may change by a factor of 100. The future resources that will go into long-established (and thus already less neglected) cause areas, on the other hand, are easier to predict and can’t make as much of a difference.

One example where future people and resources are an important consideration is AI safety. People often state that AI safety is a highly neglected cause area, presumably under the assumption that this should be completely obvious given how few people currently work in the area. At least, it is rare that the possibility of future people going into AI safety is considered explicitly. Langan-Dathi even writes that “due to [AI safety] being a recent development it is also highly neglected.” I, on the other hand, would argue that being a recent development only makes a cause \*look\* highly neglected if one doesn’t consider future people. (Again, Brian [makes almost the same point regarding AI safety](http://reducing-suffering.org/why-charities-dont-differ-astronomically-in-cost-effectiveness/#Returns\_look\_high\_before\_big\_players\_enter).)

Overall, I think many questions in AI safety should nonetheless be regarded as relatively neglected because I think there is a good chance that future people won’t recognize them as important fast enough. That said, I think some AI safety problems will become relevant in regular AI capability research or near time applications (such as self-driving cars). For example, I expect that some of [Amodei et al.’s (2016)](https://arxiv.org/abs/1606.06565) “Concrete Problems in AI Safety” will be (or would have been) picked up, anyway. Research in these areas of AI safety is thus potentially less intrinsically valuable, although it may still have a lot of instrumental benefits that make them worthwhile to pursue.

My impression is that neglecting future people in evaluating neglectedness is more common than forgetting to translate from invested resources into diminishing marginal returns. Nonetheless, in the context of this post the point of this section is that considering future resources makes neglectedness more difficult to evaluate. Obviously, it is hard to foresee how many resources will be invested into a project in the future. Because the most promising areas will not have received a lot of attention, yet, the question of their neglectedness will be dominated by how much resources they will receive in the future. Thus, in the most important cases, neglectedness is hard to estimate.

What should count as “the same cause area”?
===========================================

At least the operationalization of neglectedness involves estimating the amount of (past, current and future) resources invested into a cause area. But which resources count as going into the same cause area? For example, if the cause area is malaria, should you count people who work in global poverty as working in the same cause area?

Because the number of people working in an area is only relevant as a proxy for how much marginal returns have diminished, the answer seems to be: Count people (and resources) to the extent that their activities diminish the marginal returns in the cause area in question. Thus, resources invested into alleviating global poverty have to be taken into account, because if people’s income increases, this will allow them to take measures against malaria as well.

As another example, consider the cause area of advocating some moral view X (say effective altruism). If only a few people currently promote that view, then one may naively view advocating X as neglected. However, if neglectedness is intended to be a proxy for diminishing returns, then it seems that we also have to take into account moral advocates of other views. Because most people regularly engage in some form of moral advocacy (e.g., when they talk about morality with their friends and children), many people already hold moral views that our advocacy has to compete with. Thus, we may want to take these other moral advocates into account for evaluating neglectedness. That said, if we apply neglectedness together with tractability and scope, it seems reasonable to include such considerations in either tractability or neglectedness. (As Rob Wiblin [remarks](http://effective-altruism.com/ea/ss/the\_importantneglectedtractable\_framework\_needs/), the three factors blur heavily into each other. In particular, neglectedness can make an intervention more tractable. As Wiblin notes, we should take care not to double-count arguments. We also shouldn’t forget to count arguments at all, though.)

Acknowledgements
================

I am indebted to Tobias Baumann for valuable comments. I wrote this post while working for the Foundational Research Institute, which is now the [Center on Long-Term Risk](https://longtermrisk.org/).


A survey of polls on Newcomb’s problem

One classic story about [Newcomb’s problem](https://en.wikipedia.org/wiki/Newcomb's\_paradox) is that, at least initially, people one-box and two-box in roughly equal numbers (and that everyone is confident in their position). To find out whether this is true or what exact percentage of people would one-box I conducted a meta-survey of existing polls of people’s opinion on Newcomb’s problem.

The surveys I found are listed in the following table:

I deliberately included even surveys with tiny sample sizes to test whether the results from the larger sample size surveys are robust or whether they depend on the specifics of how they obtained the data. For example, the description of Newcomb’s problem in the Guardian survey contained a paragraph on why one should one-box (written by [Arif Ahmed](http://www.phil.cam.ac.uk/people/teaching-research-pages/ahmed/ahmed-page), author of [\*Evidence, Decision and Causality\*](http://www.cambridge.org/us/academic/subjects/philosophy/philosophy-science/evidence-decision-and-causality?format=AR&isbn=9781316056073#hCsZHCS60f1McRFp.97)) and a paragraph on why one should two-box (by [David Edmonds](https://en.wikipedia.org/wiki/David\_Edmonds\_(philosopher))). Perhaps the persuasiveness of these arguments influenced the result of the survey?

Looking at all the polls together, it seems that the picture is at least somewhat consistent. The two largest surveys of non-professionals both give one-boxing almost the same small edge. The other results diverge more, but some can be easily explained. For example, decision theory is a commonly discussed topic on LessWrong with some of the opinion leaders of the community ([including](http://lesswrong.com/lw/nc/newcombs\_problem\_and\_regret\_of\_rationality/) founder Eliezer Yudkowsky) endorsing one-boxing. It is therefore not surprising that opinions on LessWrong have converged more than elsewhere. Considering the low sample sizes, the other smaller surveys of non-professionals also seem reasonably consistent with the impression that one-boxing is only slightly more common than two-boxing.

The surveys also show that, as has often been remarked on, there exists a significant difference between opinion among the general population / “amateur philosophers” and professional philosophers / decision theorists (though the consensus among decision theorists is not nearly as strong as on LessWrong).

\*\*Acknowledgment:\*\* This work was funded by the Foundational Research Institute (now the [Center on Long-Term Risk](https://longtermrisk.org/)).


A behaviorist approach to building phenomenological bridges

A few weeks ago, I [wrote](http://lesswrong.com/r/discussion/lw/pft/naturalized\_induction\_a\_challenge\_for\_evidential/) about the BPB problem and how it poses a problem for classical/non-[logical](https://arbital.com/p/logical\_dt/) decision theories. In my post, I briefly mentioned a behaviorist approach to BPB, only to immediately discard it:

One might think that one could map between physical processes and algorithms on a pragmatic or functional basis. That is, one could say that a physical process A implements a program p to the extent that the results of A correlate with the output of p. I think this idea goes into the right direction and we will later see an implementation of this pragmatic approach that does away with naturalized induction. However, it feels inappropriate as a solution to BPB. The main problem is that two processes can correlate in their output without having similar subjective experiences. For instance, it is easy to show that [Merge sort and Insertion sort](https://en.wikipedia.org/wiki/Sorting\_algorithm#Popular\_sorting\_algorithms) have the same output for any given input, even though they have very different “subjective experiences”.

Since writing the post I became more optimistic about this approach because the counterarguments I mentioned aren’t particularly persuasive. The core of the idea is the following: Let A and B be parameterless algorithms[1](#fn1). We’ll say that A and B are equivalent if we believe that A outputs x iff B outputs x. In the context of BPB, your current decision is an algorithm A and we’ll say B is an instance or implementation of A/you iff A and B are equivalent. In the following sections, I will discuss this approach in more detail.

You still need interpretations
==============================

The definition only solves one part of the BPB problem: specifying equivalence between algorithms. This would solve BPB if all agents were bots (rather than parts of a bot or collections of bots) in Soares and Fallenstein’s [Botworld 1.0](https://intelligence.org/files/Botworld.pdf). But in a world without any Cartesian boundaries, one still has to map parts of the environment to parameterless algorithms. This could, for instance, be a function from histories of the world onto the output set of the algorithm. For example, if one’s set of possible world models is a set of cellular automata (CA) with various different initial conditions and one’s notion of an algorithm is something operating on natural numbers, then such an interpretation \*i\* would be a function from CA histories to the set of natural numbers. Relative to \*i\*, a CA with initial conditions contains an instance of algorithm A if A outputs \*x\* <=> \*i\*(\*H\*)=\*x\*, where \*H\* is a random variable representing the history created by that CA. So, intuitively, \*i\* is reading A’s output off from a description the world. For example, it may look at the physical signals sent by a robot’s microprocessor to a motor and convert these into the output alphabet of A. E.g., it may convert a signal that causes a robot’s wheels to spin to something like “forward”. Every interpretation \*i\* is a separate instance of A.

Joke interpretations
--------------------

Since we still need interpretations, we still have the problem of “joke interpretations” ([Drescher 2006](https://www.gwern.net/docs/statistics/decision/2006-drescher-goodandreal.pdf), sect. 2.3; also see [this Brian Tomasik essay](http://reducing-suffering.org/interpret-physical-system-mind/#Computations\_are\_relative\_to\_interpretation) and references therein). In particular, you could have an interpretation \*i\* that does most of the work, so that the equivalence of A and \*i\*(\*H\*) is the result of \*i\* rather than the CA doing something resembling A.

I don’t think it’s necessarily a problem that an EDT agent might optimize its action too much for the possibility of being a joke instantiation, because it gives all its copies in a world equal weight no matter which copy it believes to be. As an example, imagine that there is a possible world in which joke interpretations lead to you to identify with a rock. If the rock’s “behavior” does have a significant influence on the world and the output of your algorithm correlates strongly with it, then I see no problem with taking the rock into account. At least, that is what EDT would do anyway if it has a regular copy in that world.[2](#fn2) If the rock has little impact on the world, EDT wouldn’t care much about the possibility of being the rock. In fact, if the world also contains a strongly correlated non-instance[3](#fn3) of you that faces a real decision problem, then the rock joke interpretation would merely lead you to optimize for the action of that non-copy.

If you allow all joke interpretations, then you would view yourself in all worlds. Thus, the view may have similar implications as the l-zombie view where the joke interpretations serve as the l-zombies.[4](#fn4) Unless we’re trying to metaphysically justify the l-zombie view, this is not what we’re looking for. So, we may want to remove “joke interpretations” in some way. One idea could be to limit the interpretation’s computational power ([Aaronson 2011](https://arxiv.org/abs/1108.1791), sect. 6). My understanding is that this is what people in CA theory use to define the notion of implementing an algorithm in a CA, see, e.g., Cook ([2004](http://www.complex-systems.com/pdf/15-1-1.pdf), sect. 2). Another idea would be to include only interpretations that you yourself (or A itself) “can easily predict or understand”. Assuming that A doesn’t know its own output already, this means that \*i\* cannot do most of the work necessary to entangle A with \*i\*(H). (For a similar point, cf. [Bishop 2004](http://www.doc.gold.ac.uk/~mas02mb/Selected%20Papers/2004%20BICS.pdf), sect. “Objection 1: Hofstadter, ‘This is not science’”.) For example, if \*i\* would just compute A without looking at H, then A couldn’t predict \*i\* very well if it cannot predict itself. If, on the other hand, \*i\* reads off the result of A from a computer screen in H, then A would be able to predict \*i\*’s behavior for every instance of H. Brian Tomasik [lists](http://reducing-suffering.org/interpret-physical-system-mind/#Using\_many\_possible\_interpretations) a few more criteria to judge interpretations by.

Introspective discernibility
============================

In my original rejection of the behaviorist approach, I made an argument about two sorting algorithms which always compute the same result but have different “subjective experiences”. I assumed that a similar problem could occur when comparing two equivalent decision-making procedures with different subjective experiences. But now I actually think that the behaviorist approach nicely aligns with what one might call introspective discernibility of experiences.

Let’s say I’m an agent that has, as a component, a sorting algorithm. Now, a world model may contain an agent that is just like me except that it uses a different sorting algorithm. Does that agent count as an instantiation of me? Well, that depends on whether I can introspectively discern which sorting algorithm I use. If I can, then I could let my output depend on the content of the sorting algorithm. And if I do that, then the equivalence between me and that other agent breaks. E.g., if I decide to output an explanation of my sorting algorithm, then my output would explain, say, bubble sort, whereas the other algorithm’s output would explain, say, merge sort. If, on the other hand, I don’t have introspective access to my sorting algorithm, then the code of the sorting algorithm cannot affect my output. Thus, the behaviorist view would interpret the other agent as an instantiation of me (as long as, of course, it, too, doesn’t have introspective access to its sorting algorithm). This conforms with the intuition that which kind of sorting algorithm I use is not part of my subjective experience. I find this natural relation to introspective discernibility very appealing.

That said, things are complicated by the equivalence relation being subjective. If you already know what A and B output, then they are equivalent if their output is the same — even if it is “coincidentally” so, i.e., if they perform completely unrelated computations. Of course, a decision algorithm will rarely know its own output in advance. So, this extreme case is probably rare. However, it is plausible that an algorithm’s knowledge about its own behavior excludes some conditional policies. For example, consider a case like Conitzer’s ([2016](https://arxiv.org/abs/1610.05733), [2017](https://arxiv.org/abs/1705.03560)), in which copies of an EU-maximizing agent face different but symmetric information. Depending on what the agent knows about its algorithm, it may view all the copies as equivalent or not. If it has relatively little self-knowledge, it could reason that if it lets its action depend on the information, the copies’ behavior would diverge. With more self-knowledge, on the other hand, it could reason that, because it is an EU maximizer and because the copies are in symmetric situations, its action will be the same no matter the information received.[5](#fn5)

Consciousness
=============

The BPB problem resembles the problem of consciousness: the question “does some physical system implement my algorithm?” is similar to the question “does some physical system have the conscious experience that I am having?”. For now, I don’t want to go too much into the relation between the two problems. But if we suppose that the two problems are connected, we can draw from the philosophy of mind to discuss our approach to BPB.

In particular, I expect that a common objection to the behaviorist approach will be that most instantiations in the behaviorist sense are [behavioral p-zombies](https://en.wikipedia.org/wiki/Philosophical\_zombie#Types\_of\_zombies). That is, their output behavior is equivalent to the algorithm’s but they compute the output in a different way, and in particular in a way that doesn’t seem to give rise to conscious (or subjective) experiences. While the behaviorist view may lead us to identify with such a p-zombie, we can be certain, so the argument goes, that we are not given that we have conscious experiences.

Some particular examples include:

\* Lookup table-based agents
\* Messed up causal structures, e.g. Paul Durham’s experiments with his whole brain emulation in Greg Egan’s novel [\*Permutation City\*](https://en.wikipedia.org/wiki/Permutation\_City).

I personally don’t find these arguments particularly convincing because I favor [Dennett’s](https://en.wikipedia.org/wiki/Consciousness\_Explained) and [Brian Tomasik’s](http://reducing-suffering.org/#Consciousness) eliminativist view on consciousness. That said, it’s not clear whether eliminativism would imply anything other than relativism/anti-realism for the BPB problem (if we view BPB and philosophy of mind as sufficiently strongly related).

Acknowledgment
==============

This work was funded by the Foundational Research Institute (now the [Center on Long-Term Risk](https://longtermrisk.org/)).

---

1. I use the word “algorithm” in a very broad sense. I don’t mean to imply Turing computability. In fact, I think any explicit formal specification of the form “f()=…” should work for the purpose of the present definition. Perhaps, even [implicit specifications](https://en.wikipedia.org/wiki/Implicit\_function) of the output would work. [↩](#ref1 "Jump back to footnote 1 in the text.")

2. Of course, I see how someone would find this counterintuitive. However, I suspect that this is primarily because the rock example triggers [absurdity heuristics](https://wiki.lesswrong.com/wiki/Absurdity\_heuristic) and [because it is hard to imagine](https://casparoesterheld.com/2015/12/06/cheating-at-thought-experiments/) a situation in which you believe that your decision algorithm is strongly correlated with whether, say, some rock causes an avalanche. [↩](#ref2 "Jump back to footnote 2 in the text.")

3. Although the behaviorist view defines the instance-of-me property via correlation, there can still be correlated physical subsystems that are not viewed as an instance of me. In particular, if you strongly limit the set of allowed interpretations (see the next paragraph), then the potential relationship between your own and the system’s action may be too complicated to be expressed as A outputs x <=> i(H)=x\*.\*[↩](#ref3 "Jump back to footnote 3 in the text.")

4. I suspect that the two might differ in medical or “common cause” Newcomb-like problems like the [coin flip creation problem](http://lesswrong.com/r/discussion/lw/oih/did\_edt\_get\_it\_right\_all\_along\_introducing\_yet/). [↩](#ref4 "Jump back to footnote 4 in the text.")

5. If this is undesirable, one may try to use logical counterfactuals to find out whether B also “would have” done the same as A if A had behaved differently. However, I’m very skeptical of logical counterfactuals in general. Cf. [the “Counterfactual Robustness” section in Tomasik’s post](http://reducing-suffering.org/interpret-physical-system-mind/#Counterfactual\_robustness). [↩](#ref4 "Jump back to footnote 4 in the text.")