A Dialogue on Suffering Subroutines

A Dialogue on Suffering Subroutines
===================================

29 August 2015
by [Brian Tomasik](https://longtermrisk.org/author/brian-tomasik/ "Posts by Brian Tomasik")

First written: 20 Dec. 2013; last update: 27 Apr. 2017

 This piece presents a hypothetical dialogue that explains why instrumental computational processes of a future superintelligence might evoke moral concern. I give some examples of present-day systems that we may consider at least somewhat conscious, such as news reporting or automated stock trading. Agent-like components seem to emerge in many places, and it's plausible this would continue in the computing processes of a future civilization. Whether these subroutines matter, how much they matter, and how to even count them are questions for future generations to figure out, but it's good to keep an open mind to the possibility that our intuitions about what suffering is may change dramatically with new insights.

Contents

\* [Dialogue](#Dialogue)
\* [Further analogies](#Further\_analogies)
\* [Schwitzgebel's view](#Schwitzgebels\_view)
\* [Suffering can be simple: A reply to Metzinger](#Suffering\_can\_be\_simple\_A\_reply\_to\_Metzinger)
\* [Onion piece](#Onion\_piece)
\* [Acknowledgements](#Acknowledgements)
\* [A note on terminology](#A\_note\_on\_terminology)
\* [Footnotes](#Footnotes)

Dialogue
--------

\*Alice\*: Greetings, Brian. I heard that you're concerned about the possibility of what you call "suffering subroutines." You say that artificial intelligences (AIs) in the future -- whether human-inspired or [paperclipping](http://wiki.lesswrong.com/wiki/Paperclip\_maximizer) -- might run immense numbers of computations that we may consider to be conscious suffering. I find this hard to believe. I mean, why wouldn't instrumental computations be just dumb components of an unfeeling system?

\*Brian\*: They might be, but they might not be, and the latter possibility is important to consider. As one general point, note that sentience evolved on Earth ([possibly more than once](https://www.facebook.com/brian.tomasik/posts/620256828222)), so it seems like a useful algorithmic construct.

\*Alice\*: Sure, sentience was useful on Earth for competitive organisms, but in the subroutines of an AI, every computation is subserving the same goal. Processes are allocated computing resources "each according to his needs from each according to his talents," as Dan Dennett [observed](http://youtu.be/OlRHd-r2LOw?t=14m52s).

\*Brian\*: Possibly. But in that same talk you cite, Dennett goes on to explain that computing processes in the human brain may be competitive rather than cooperative, Darwinian rather than Marxist. Dennett [proffers](http://www.youtube.com/watch?v=OlRHd-r2LOw&feature=youtu.be&t=16m24s) a hypothesis that "centrally planned economies don't work, and neither do centrally coordinated, top-down brains."

\*Alice\*: Meh, if that's true, it's probably a vestige of the way evolution came up with the brain. Surely an orderly process could be designed to reduce the wasted energy of competition, and since this would have efficiency advantages, it would be a convergent outcome.

\*Brian\*: That's not obvious. [Evolutionary algorithms](https://en.wikipedia.org/wiki/Evolutionary\_algorithm) are widely useful and not always replaceable by something else. In any event, maybe non-Darwinian processes could also consciously suffer.

\*Alice\*: Umm, example, please?

\*Brian\*: It seems plausible that many accounts of consciousness would include non-evolved agents under their umbrellas. Take the [global-workspace theory](https://en.wikipedia.org/wiki/Global\_Workspace\_Theory) for instance. Are you familiar with that?

\*Alice\*: Do explain.

\*Brian\*: In the so-called [LIDA implementation](http://ccrg.cs.memphis.edu/tutorial/mindAccordingToLIDA/Brief-Account.pdf) of the global-workspace model, a cognitive cycle includes the following components:

\* Incoming sensations from the world are momentarily stored by sensory memory.
\* Many parallel unconscious modules operate on these sensations, picking them apart in various ways, and drawing on stored memories for additional insight.
\* These unconscious modules form coalitions, advocating for why their insights are most important.![](https://longtermrisk.org/files/Broadcast-350x233.png "'Visualisation of broadcast routing scheme.' Uploaded by Easyas12c, who says 'Visual model is based on some similar images hosted at www.gloomfaq.de. They were cached by google image search, but were no longer available, so I don't know the original author. I created these new images from scratch, but the appearance is very similar.' This image is in the public domain worldwide. https://commons.wikimedia.org/wiki/File:Broadcast.svg")
\* The broadcasted news then produces updates to memories and learning in various parts of the brain, including inclinations to select different actions (i.e., [reinforcement learning](http://www.utilitarian-essays.com/reinforcement-learning.html)). Information may also return from the receiver back to the broadcaster.
\* With these updated activations for various behaviors, the organism then acts on the environment, receives new sensations, and the cycle happens again. These cycles are hypothesized to occur at about 10 Hz.

[This diagram](http://ccrg.cs.memphis.edu/tutorial/synopsis.html) lays out the various components.

Note that Stan Franklin, one of the managers of the LIDA project, believes that the earlier version of his system, IDA, is "functionally conscious" but not "phenomenally conscious," as he explains in his 2003 paper, "[IDA, a Conscious Artifact?](http://ccrg.cs.memphis.edu/assets/papers/IDA-ConsciousArtifact.pdf)" This seems to stem from his tentative agreement with David Chalmers about the hard problem of consciousness (see p. 10). Because I believe this view is [confused](http://www.utilitarian-essays.com/consciousness.html), I think the functional consciousness under discussion here \*is\* also phenomenal consciousness.

\*Alice\*: I see. So why is this relevant?

\*Brian\*: If consciousness is this kind of "global news broadcasting," it seems to be a fairly common sort of operation. I mean, one obvious example is the news itself: Stories compete for worthiness to be aired, and the most important news segments are broadcast, one at a time, to viewers who then update their memories and actions in response. Then new things happen in the world, new stories take place, many reporters investigate them in parallel, they compete to get their stories aired, and the cycle happens again. "Emotions" and "dispositions" may [modulate](https://en.wikipedia.org/wiki/Neuromodulation) this process -- for instance, more conservative news agencies will prefer to air different stories than liberal news agencies, and the resulting messages will be biased in the direction of the given ideology. Likewise, the national mood at a given moment may cause some stories to be more relevant than they would have been given a different mood. People who care a lot about the news stories they hear may get in touch with the news agencies to give feedback and engage in further coordination ("reentrant signaling"). And so on.

Of course, we see analogous behavior in other places as well:

\* The various "unconscious" computing components of an airplane flight system might update their calculations based on a globally broadcast signal about latitude, longitude, wind resistance, turbulence, etc., and "memory" logs of the past data stream may be stored for later reference.![](https://longtermrisk.org/files/Philippine-stock-market-board-350x263.jpg "'Phillippine stock market board.' By Katrina.Tuliao [CC-BY-2.0 (http://creativecommons.org/licenses/by/2.0)], via Wikimedia Commons: https://commons.wikimedia.org/wiki/File:Philippine-stock-market-board.jpg")
\* An updated trading price on a stock exchange is broadcast to vast numbers of individual computer-trading systems, which update their actions and save records of the price history for machine learning in the future. Theoretically, the systems could even perform [online learning](https://en.wikipedia.org/wiki/Online\_machine\_learning) like animals do.
\* Data about the latest unemployment rates is announced, and this is distributed to various teams doing macroeconomic prediction. Their policy advice is then adjusted, and the data is logged for future reference. These policies affect the world, unemployment changes, measurement indicators pick up that information, it's aggregated, and then a broadcast about unemployment rates happens again next month.
\* Software patterns like [publish–subscribe](https://en.wikipedia.org/wiki/Publish%E2%80%93subscribe\_pattern) and [observer](https://en.wikipedia.org/wiki/Observer\_pattern) involve globally broadcasting updates to many receivers who can then act on the new information.
\* [Insulin release](https://en.wikipedia.org/wiki/Insulin#Release) happens when glucose enters beta cells and triggers internal processes ("computations") in those cells. Those processes release calcium ions, which unlock stored insulin and "broadcast" it throughout the animal's bloodstream. The broadcast insulin signal produces updates in glucose utilization throughout the animal, which changes the internal behavior of various organs and the external behavior of the organism. (In general, hormones are a form of "information broadcasting", even though we don't normally think of processes outside the brain as being consciousness-like.)
\* In "[Global Workspace Dynamics](http://www.ncbi.nlm.nih.gov/pmc/articles/PMC3664777/)," Baars, Franklin, and Ramsoy give the analogy of a fire alarm as a global signal that triggers different local actions for different people. The fire department (one receiver of the alarm) may try to locate the fire alarm itself (the broadcaster) to interact with it better, in analogy with reentrant signaling.
\* The same paper describes brain activity and waves using an analogy with a crowd at a football game. "Chatting" is what happens when people/neurons talk with their neighbors in patterned ways that may appear random when seen from a distance. "Chanting" is when a global delta wave sweeps through the population in a coordinated fashion. "Cheering" is a global wave in response to a strong stimulus.
\* The coordinated communication of large numbers of neurons could also be analogized in phenomena like social movements, where people form coalitions of synchronized activity and compete to gain attention of the media in order to broadcast their message to the whole population and thereby influence action tendencies, memories, learning, etc. This comparison takes the activist goal of "[raising social consciousness](https://en.wikipedia.org/wiki/Consciousness\_raising)" to a new level.

Note that some of these systems are not competitive, and so the claim that lack of Darwinian competition means lack of conscious suffering may not be accurate.

These analogies can actually give insight into why consciousness emerges in brains: It's for a similar reason as why national and global news emerges in human society. Global broadcasting of the most important events and knowledge serves to keep every part of the social system up to date, in sync, aware of required actions (e.g., hurricane warnings, voting days), and alerted to global searches ("Have you seen this crime suspect?") and coordination. As the "[Global Workspace Dynamics](http://www.ncbi.nlm.nih.gov/pmc/articles/PMC3664777/)" paper says: "What is the use of binding and broadcasting in the [cortico-thalamic] C-T system? One function is to update numerous brain systems to keep up with the fleeting present." This process of serializing the most salient updates for global broadcasting may ultimately create a more effective society (organism) than if every local community were isolated and reacted independently with parochial ("unconscious") reflexes or using only [tribal knowledge](https://en.wikipedia.org/wiki/Tribal\_knowledge). When a broadcast becomes globally conscious, it's available to all regions, including the verbal/speech centers of a person for conscious report (or to the writers/bloggers of society for verbalization in text). Events in illiterate farming communities would be "unconscious" to the world without journalists who visit to help broadcast those stories. The world can become more conscious of its memories when historians uncover and share information about past events. And the spotlight of attention shifts based on the most emotionally salient events that happen. In general, fast, global network communication over radio, television, and the Internet are making the world more conscious of itself, in a surprisingly literal sense.

Why do we only care about conscious experiences? For instance, we'd be horrified to undergo conscious surgery but don't mind undergoing surgery while anaesthetized. Presumably it's because the parts of us that "care about" our experiences -- such as by reacting aversively, triggering stress feelings, planning ways to avoid the experience, encoding traumatic memories, and so on -- only know about the damaging stimuli when they become conscious. Typically a strong negative stimulus will win competitions to be consciously broadcast, but when anaesthesia blocks pathways from nociceptors to access by the full brain, it prevents the suite of "caring about" responses that would ordinarily be triggered. An analogy in the social realm is that society cares about and responds to negative events when they're reported in the media, but if scandals are covered up or reporters are prevented from talking about atrocities, this is like applying local anaesthesia.

More often, neglect of certain harms and focus on other types of harms is built in to the system. For instance, a sliver in your eye would hurt vastly more than a sliver in your leg because your eye has many more nerve endings. Similarly, a rich person killed in the United States would attract far more attention and response than a poor person killed in Africa because there are vastly more reporters covering the former, and the story about the rich American would seem more salient to the readers (neurons) who vote it up on Twitter.

Another analogy between consciousness and news reporting is that in both cases, once an object enters the spotlight of attention, other events in that spotlight can come to attention that would have otherwise remained hidden. For example, suppose your leg itches, causing you to focus your consciousness on your leg. That may allow you to then feel the breeze on your leg as well, whereas you otherwise would have filtered out that information from your awareness. Likewise, when a news story about X surfaces, this often leads to investigations into other stories Y and Z that relate to X, and stories about V and W that previously would have been ignored become "newsworthy". As an example, following the [pool party incident](https://en.wikipedia.org/wiki/2015\_Texas\_pool\_party\_incident) in McKinney, Texas on 5 Jun. 2015, a series of other news stories about McKinney, Texas also became national headlines, whereas previously those kinds of stories wouldn't have reached beyond the local news.

I haven't explored interpretations of the processes mentioned above according to other [models of consciousness](http://www.scholarpedia.org/article/Models\_of\_consciousness), but I expect you'd find that systems like these would be at least somewhat conscious in those frameworks as well. In general, most accounts of what consciousness is appeal to general principles that don't go away when neurons stop being involved.

And beyond consciousness, we can see other mind-like processes at play in many systems. Take memory for example. Apparently memories consist of neural connections that become strengthened by repeated use, and they fade as the connections decay. This reminds me of a series of dirt roads through a town. They're first created by some event, they become strengthened with use, and they revert back to wilderness with disuse. A road that hasn't been traveled on in years may become overrun by returning vegetation, but it can still be re-activated more easily than creating a new road from scratch somewhere else. And like with neural connections, a stronger road allows things to flow more easily between the regions it connects.

\*Alice\*: Are you really saying that news reports and stock exchanges are conscious? And that roads have memory?

\*Brian\*: I don't know.[1](#link\_ajs-fn-id\_1-251) But I think we should take the possibility seriously. In any case, it could be that future computational systems contain more human-like entities. For instance, suppose an AI wants to undertake a research program on string theory, to better update its models of physics. It will partition some fraction of computing power to that project. It may want to parallelize the work for speed, so it might create lots of different "research teams" that work on the problem separately and publish their results to others. These teams might compete for "grant money" (i.e., additional computing resources) by trying to produce high-quality findings better than the other teams. These components might be sufficiently agent-like as to evoke our moral concern.

The process of intelligently searching the space of possibilities based on value assessments is a general phenomenon. Animals search through a field until they find a lush patch of strawberries; then they experience reward at the discovery and focus their efforts there for a while. Humans, too, feel reward while trying to figure things out. For instance, V.S. Ramachandran's [peekaboo principle](http://jjgallaher.blogspot.com/2009/07/art-in-brain-peekaboo-principle.html) is based on the idea that humans receive little squirts of pleasure every time they unpack a small piece of a puzzle, and these "mini aha" moments motivate them to keep going. Perhaps there would be a similar process at play for an AI's research teams. When a small discovery is made, the good news is broadcast throughout the team, and this encourages more actions like what led to the discovery.

As I stated it, this model suggests something akin to David Pearce's [gradients of bliss](http://www.hedweb.com/object33.htm) because the rewards for research discoveries are positive. But perhaps the system would use gradients of agony, with research findings being rewarded by temporary relief from discomfort. If there is a possibility for choice between a "gradients of bliss" and a "gradients of agony" design to achieve roughly similar computational ends, this suggests room for humane concerns to make a meaningful difference.

As another illustration, consider economics. Under both capitalism and communism, we see the emergence of hierarchical forms of organization. The CEO of a corporation seems like a decent model for the conscious control center of a brain: The workers perform their duties away from its sight, and then the most important news about the company is bubbled up to the CEO's desk. Then the CEO broadcasts updates to all the workers, including compensation rewards, which adjust worker action inclinations. The company also stores records of these events for later use. The most important ("emotionally salient") historical memories are better preserved, and less relevant ones slowly decay with time. This whole process mimics the global-workspace theory in broad outline. And the fact that hierarchies of this type have emerged in all kinds of governmental and economic systems suggests that they may be common even among the construction workers and researchers of an AI.

\*Alice\*: Hmm, maybe. But if there are just a few companies that the AI is directing, that's not a lot of conscious minds. Maybe these suffering corporations are then not a big deal, relative to much larger numbers of suffering wild animals, etc. What's more, the [clock speed](http://www.utilitarian-essays.com/computations-i-care-about.html#clock-speed) of a corporate consciousness would be glacial compared with that of an animal.

\*Brian\*: Well, even if we don't weight by brain size, who says corporations are the only parts of this process that are conscious? Hierarchical organization is a recurrent pattern of organized systems in general. It could happen at the highest level -- the executive AI controlling its component corporations -- but it would also happen in a fractal way at many lower layers too: Each corporation is composed of subdivisions, each subdivision has its own subdivisions, etc. At some point we might hit a level of entities analogous to "workers." Even below that might be the subcomponent coalitions of an individual worker's brain, which compete for attention by the worker brain's executive-control system. Each of these could have consciousness-like components. And their clock speeds would be quite fast.

One concept in the LIDA model is that of a "codelet," which [one page](https://web.archive.org/web/20160410233548/https://people.cs.kuleuven.be/~joaquin.vanschoren/Flexo/uml/glossary/codelet.html) defines as

> tiny agents, carrying small pieces of code (hence the name). They can be interpreted as being a small part of a process, but then leading its own life, very much like an ant is a small part of a "process" to gather food, to defend the anthill or to nurture newborns. They run in parallel [...], and none are indispensable.
>
>
> [...] The entity calling the codelet will estimate its urgency (reflecting the promise of further investigation). Highly urgent codelets can preempt lower urgency codelets [...], and if a codelet's urgency sinks well below that of other's, it just dies out, leaving computer resources to more ambitious codelets. If a codelet sees it has no real work to do in the current situation (due to a bad estimation or changed situation), it sizzles.

It's [plausible](http://www.utilitarian-essays.com/insect-pain.html) that individual ants are conscious. So too, maybe even tiny components of an individual worker's brain could be seen as conscious.

\*Alice\*: But if a larger consciousness contains many smaller consciousnesses, which each contain many smaller consciousnesses, how do we count them? What are the weights? Do the lowest-level consciousnesses dominate? This discussion is getting curiouser and curiouser!

\*Brian\*: Indeed. But these are issues that we need to resolve at some point. To some extent I'm [punting the question](http://utilitarian-essays.com/robustness-against-uncertainty.html) to our more intelligent descendants. Still, it's useful to realize that suffering subroutines \*could\* be a big deal in the future, so that we don't naively reach conclusions based on a circumscribed view of what we might care about.

\*Alice\*: From the standpoint of "consciousness as broadcasting," do you think insects are conscious?

\*Brian\*: It's an important question. It certainly seems plausible that insects would have some sort of LIDA-like cognitive cycle: Inputs, unconscious processing, most important insights bubble up and are distributed, and they affect action inclinations. Even if this kind of architecture didn't exist exactly, we might see adumbrations of it in whatever insects do. I mean, for example, if one part of the brain communicates its insights to several other parts of the brain, even if not globally, isn't this like a mini-broadcast? Isn't that sort of like consciousness already? In general, any kind of communication-and-updating process would have shadows of the operation that we think of as consciousness. This illustrates my [more general point](http://www.utilitarian-essays.com/computations-i-care-about.html#graded-sentience) that consciousness comes in gradations -- there's not a single cutoff point where what was unconscious matter suddenly has the lights come on. There are just atoms moving in various ways, and some of them activate our sympathies more than others.

\*Alice\*: Well, that raises a question: If we can care about whatever we like, however much we like, why shouldn't I just care about humans and maybe some animals, and forget about these suffering subroutines entirely?

\*Brian\*: You can do that, and perhaps we would choose to upon reflection. I don't know what the best criteria are for carving out our caring-about function. But it seems plausible that algorithms are a big part of it, and then when we see processes that resemble these algorithms somewhere else, it raises the question of why we care about them in some forms but not others. I don't know where our hearts will ultimately fall on the matter.

\*Alice\*: Do you think even basic physics might contain consciousness?

\*Brian\*: I don't know. I hope not, but I wouldn't rule it out. Giulio Tononi's "phi" postulates that even an electron has a nonzero measure of consciousness, for instance.

With the global-workspace model, maybe we could see elementary particles as broadcasting information that then influences other regions -- e.g., the nuclear reactions in the sun broadcast photons, and the sun's mass pulls other objects toward it. But it's not clear that any real "agent" process is going on here. Where are the learning, action selection, memories, etc.? So naively it seems like these kinds of dead physical things aren't conscious, but maybe I'm not looking at them right, and maybe we'll discover ways in which there are agents even in the math of microscopic physics.

\*Alice\*: Speaking of math, do you think Darwinism could ultimately go the way of the dodo? I mean, Darwinian competition is just an attempt at [hill climbing](https://en.wikipedia.org/wiki/Hill\_climbing) in a high-dimensional space. But sometimes we have mathematical tools that let us perform exact optimizations without needing to "guess and check." Could intelligence ultimately be reduced to a series of really big mathematical optimization problems that can be solved (at least somewhat) analytically, thereby averting a lot of this expensive computation of agent-like things? Similarly, reinforcement learning is [direct adaptive optimal control](https://web.archive.org/web/20161223113316/http://webdocs.cs.ualberta.ca/~sutton/papers/sutton-barto-williams-91.pdf), but optimal-control problems can potentially be solved by analytic methods like the Bellman equations if you know the payoffs and transition probabilities ahead of time.

\*Brian\*: Maybe, though it does seem hard to imagine that we could analytically solve some of these really specific, data-driven problems without computing in detail the process being modeled. Perhaps this just reflects lack of imagination on my part, and of course, there are times when macro-scale approximations can remain ignorant of microfoundations. In any case, the actions of the AI on the galaxy to implement its goals would still require lots of real, physical manipulation -- e.g., supervisors to coordinate workers in building solar colonies and such. The possibility you cite is fun to speculate on, but it's not sufficiently probable to substantially affect the concern about suffering subroutines, given that consciousness-like processes seem to be such a convergent feature of organized systems so far.

\*Alice\*: Do ecosystems suffer? Could this broad view of consciousness provide some vindication of the otherwise seemingly absurd idea that nature as a whole can have moral standing apart from the welfare of the individuals it contains?

\*Brian\*: In principle it's certainly possible ecosystems could contain shadows of consciousness, but it's not clear they usually do. Where is the global broadcasting? What are the action components that are being updated? Maybe you could come up with some interpretations. Even if so, it's not clear what an ecosystem wants. Unlike corporations or ants, ecosystems don't have clear goals. Even if we identified a goal, it wouldn't necessarily align conservationism; it might go the other way. In any event, even if an ecosystem's consciousness did align with conservationism, it's dubious whether the interests of the ecosystem as a whole could outweigh those of [quintillions](http://www.utilitarian-essays.com/number-of-wild-animals.html) of suffering individual animals within it.

If we think ecosystems can suffer, then a natural way to prevent future suffering is to have fewer of them. Even if we adopted the stance from environmental ethics of considering ecosystems objects of intrinsic moral importance regardless of sentience, it's not obvious that ecosystems are inherently \*good\*? We might think they're inherently \*bad\*. This kind of "negative environmental ethics" seems a natural idea for a negative-leaning consequentialist.

\*Alice\*: Yeah. Maybe one suggestion could be that the atmospheric CO2 levels are a global signal broadcast to all subcomponents of the biosphere. This then causes (very small) changes in the behavior of animals, plants, and inorganic entities like sea ice. The responses of these entities then have an impact back on CO2 levels, which are then broadcast globally. I guess in this model, the broadcasts are continuous rather than discrete.

\*Brian\*: I suppose that's one interpretation you could make, though what would be the valence of CO2? In the stock-trading example, we could say that for the subset of traders that are net long in the security, an increase in the stock price would have positive valence. What about for CO2?

\*Alice\*: Maybe those organisms that do better with more CO2 would receive it with positive valence, and vice versa? The "learning" of the ecosystem would then be strengthening those organisms that do well with higher CO2, just like the dopaminergic learning of an animal involves strengthening connections for action neurons that just fired given the current context.

\*Brian\*: Ok, I can kind of see that, although in the case of dopamine, the action neurons were responsible for bringing the reward; in the case of the atmosphere, a whole bunch of stuff brought the increase in CO2 levels, and it wasn't necessarily the organisms that benefit from CO2 who were responsible for the emissions. Indeed, people often remark how humans in the global South are "punished" by the CO2 emissions of those in the global North.

Anyway, even if we did consider the carbon cycle somewhat analogous to a brain, keep in mind that the clock speed of this operation is really slow. Of course, since CO2 changes are continuous rather than coming in discrete pulses, the idea of a clock speed isn't really appropriate, but I guess we can still create a rough notion about cycles per year of relevant operations.

\*Alice\*: And of course, at the same time, we could have H2O levels as another currency of the biosphere, and temperature as another, and so on. There would be multiple broadcasting systems at play.

\*Brian\*: Right. In general, we can pattern-match a complex process in many different ways as being composed of many different systems that each have some resemblance to consciousness. This actually returns us to the old puzzle in the philosophy of computationalism: [What is a computation, anyway?](http://utilitarian-essays.com/computations-i-care-about.html#what-is-a-computation) One answer is that we see various physical processes as resembling various computations to various degrees, and we can then care about them in proportion to their resemblance. The same thing is going on here -- only, this is not John Searle's toy Wordstar program in the wall but a genuine instance of seeing consciousness-like operations in various places. It's like [pareidolia](https://en.wikipedia.org/wiki/Pareidolia) for our empathy systems.

Personally, I don't really care intrinsically about the Earth's carbon cycle, water cycle, etc. to any appreciable degree. I think the connection to animal minds is a pretty far stretch.

\*Alice\*: Yes. Moreover, the way we've been discussing consciousness has been pretty simple and crude. There may be important pieces of the puzzle that we've neglected, and we might consider these important as well for making an entity conscious in a way that matters.

\*Brian\*: Agreed! This should not be taken as the end of the conversation but only the beginning.

Further analogies
-----------------

There are many more similarities between operations in our brains and phenomena in the worlds of politics, physics, etc. Sebastian Seung's book \*[Connectome](https://en.wikipedia.org/wiki/Connectome:\_How\_the\_Brain%27s\_Wiring\_Makes\_Us\_Who\_We\_Are)\* provides several additional comparisons. One of my friends remarked on Seung's work: "I don't think I've ever read a book with so many illuminating analogies!" While most of Seung's readers presumably see these analogies as merely didactic aids, I would suggest that they might also have moral significance if we care about brain-like processes in non-brain places.

Schwitzgebel's view
-------------------

[Schwitzgebel (2016)](http://www.newappsblog.com/2016/12/is-most-of-the-intelligence-in-the-universe-non-conscious-ai.html "'Is Most of the Intelligence in the Universe Non-Conscious AI?'") reaches a similar conclusion as the previous dialogue did:

>
> \*\*unity of organization in a complex system plausibly requires some high-level self-representation or broad systemic information sharing.\*\* [...] most current scientific approaches to consciousness [...] associate consciousness with some sort of broad information sharing -- a "[global workspace](http://cogweb.ucla.edu/CogSci/GWorkspace.html)" or "[fame in the brain](http://www.scholarpedia.org/article/Multiple\_drafts\_model)" or "[availability to working memory](https://global.oup.com/academic/product/the-conscious-brain-9780195314595)" or ["higher-order" self-representation](https://plato.stanford.edu/entries/consciousness-higher/). On such views, we would expect a state of an intelligent system to be conscious if its content is available to the entity's other subsystems and/or reportable in some sort of "introspective" summary. For example, if a large AI knew, about its own processing of lightwave input, that it was representing huge amounts of light in the visible spectrum from direction alpha, and if the AI could report that fact to other AIs, and if the AI could accordingly modulate the processing of some of its non-visual subsystems (its long-term goal processing, its processing of sound wave information, its processing of linguistic input), then on theories of this general sort, its representation "lots of visible light from that direction!" would be conscious. And we ought probably to expect that large general AI systems would have the capacity to monitor their own states and distribute selected information widely. \*\*Otherwise, it's unlikely that such a system could act coherently over the long term. Its left hand wouldn't know what its right hand is doing.\*\*
>
>
>

Suffering can be simple: A reply to Metzinger
---------------------------------------------

In response to the 2015 Edge question, "What do you think about machines that think?", Thomas Metzinger [explored](http://edge.org/response-detail/26091 "\"What If They Need To Suffer?\" for \"2015 : WHAT DO YOU THINK ABOUT MACHINES THAT THINK?\"") a similar question as the above dialogue addressed: Will AIs necessarily suffer, or could they be intelligent without suffering? Metzinger doesn't give a firm answer, but he enumerates four conditions that he believes are necessary for suffering:

1. Being conscious
2. Being self-conscious in the sense of having a sense of ownership of suffering (e.g., "this is \*my\* suffering")
3. The experience has negative valence
4. The negative valence is transparent in the sense that it can't be doubted.

This list is interesting and provides four helpful criteria that may enhance a holistic conception of suffering, but in my opinion these criteria are neither necessary nor exhaustive. I would consider them like four principles that one might propose for the meaning of "justice" -- a concept sufficiently complex that probably no four concrete criteria by themselves can define it.

Let's see why each of these conditions is not strictly necessary. The most straightforward is probably #2, since it seems easy to imagine being in pain without engaging sufficient cognition to attribute that pain to yourself. Suffering can be a flood of "badness" feeling, which needn't be sufficiently differentiated that one recognizes that the badness is an experience on the part of oneself. For instance, a depressed person might feel that the whole world is bad -- that there's just a general badness going on.

#4 also doesn't seem necessary, because it can still be morally disvaluable if someone is uncertain whether he's in agony. For instance, suppose you step on something. You're not sure whether the object has punctured the skin of your foot. You think you might feel some sharp pain in your foot, but you're not sure if it's actually there or just imagined, until you actually look at your foot and see the sharp object. (Michael Tye offers a [similar example](https://web.archive.org/web/20160401145456/http://michaeltye.us/Pain2.pdf "see the last question of the interview, \"MICHAEL TYE ON PAIN\" (http://philosophybites.com/2012/08/michael-tye-on-pain.html)").) I'm not sure what Metzinger would think of this case. In any event, it seems that transparency is actually quite easy to satisfy. It takes a complex cognitive system to produce doubts about experiences. Simple agents should generally have transparent emotions.

As far as #1, I think [all systems are](https://longtermrisk.org/flavors-computation-flavors-consciousness/) at least marginally conscious, so even if condition #1 is necessary, it's always satisfied. Of course, the \*degree\* of consciousness of a system matters enormously, but Metzinger's piece seems to be asking whether particular AIs would suffer at all.

As far as #3, I agree that valence plays an important, perhaps central, role in human suffering. This valence might prototypically be the [reward part](http://reducing-suffering.org/ethical-issues-artificial-reinforcement-learning/#Valence\_networks) of a reinforcement-learning (RL) system. If one insists that valence can only make sense in the context of a rigid definition of RL, then I agree that not all AIs would have valence (although many still would, given the importance of RL for autonomous behavior). But if we interpret negative valence more broadly as "information indicating that something should be avoided", or even more compactly as "information that produces avoidance", then this kind of operation can be seen in many more systems, including non-learning agents that merely follow fixed stimulus-response rules. Indeed, the basic template of one physical event causing another avoidance-like event runs as deep as the interactions of fundamental physical particles, if we take enough of a high-level view and don't insist on greater complexity in our definition.

Overall, I find Metzinger's criteria too narrow. They leave out vast numbers of simpler systems that I think still deserve some ethical consideration. Nonetheless, I appreciate that Metzinger's proposals enrich our conceptualization of more complex suffering.

\*Onion\* piece
-------------

\*The Onion\* has a humorous article, "[Scientists Confident Artificially Intelligent Machines Can Be Programmed To Be Lenient Slave Masters](http://www.theonion.com/article/scientists-confident-artificially-intelligent-mach-51170)," in which AI researchers discuss the goal of shaping AI trajectories in such a way that AIs treat their human workers (what I might call "suffering human subroutines") more humanely. I find it extremely implausible that AIs would actually use human laborers in the long run, but they plausibly would use conscious worker agents of some sort -- both sophisticated scientist/engineer subroutines and other simpler subroutines of the kind discussed in this piece.

Unlike human laborers, these subroutines would presumably enjoy working as hard as possible on the task at hand. Humans evolved to dislike exertion as a way to conserve energy except when required, but robots built to carry out a given task would be optimized to want to carry out exactly that task. That said, more sophisticated digital agents might, like humans, feel mild unpleasantness if they expended time or energy on fruitless activities. For instance, a robot should dislike moving around and thereby draining its battery unless it thinks doing so will conduce to achieving a reward.

Acknowledgements
----------------

I learned of the idea that suffering subroutines might be ethically relevant from Carl Shulman in 2009. In response to this piece, Carl [added](https://www.facebook.com/brian.tomasik/posts/641988228322?comment\_id=905370&offset=0&total\_comments=7):

> Of course, there can be smiling happy subroutines too! Brian does eventually get around to mentioning "gradients of bliss", but this isn't a general reason for expecting the world to be worse, if you count positive experiences too.
>
>
> I would say "sentient subroutines."
>
>

Some examples in this piece were also partly inspired from a [post](https://www.facebook.com/ben.west.1029/posts/10201401708580746) by Ben West, linking to Eric Schwitzgebel's "[If Materialism Is True, the United States Is Probably Conscious](https://web.archive.org/web/20200514103014/https://faculty.ucr.edu/~eschwitz/SchwitzPapers/USAconscious-130208.htm)," which I discuss more [in another piece](http://www.utilitarian-essays.com/hedonistic-vs-preference.html#non-conscious).

A note on terminology
---------------------

I coined the phrase "suffering subroutines" in a 2011 post on [Felicifia](https://web.archive.org/web/20180808173047/https://felicifia.org/). I chose the alliteration because it went nicely with "sentient simulations," giving a convenient abbreviation (SSSS) to the conjunction of the two concepts. I define sentient simulations as explicit models of organisms that are accurate enough to count as conscious, while suffering subroutines are \*incidental\* computational processes that nonetheless may matter morally. Sentient synthetic [artificial-life](https://en.wikipedia.org/wiki/Artificial\_life) agents are somewhere on the border between these categories, depending on whether they're used for psychology experiments or entertainment (sentient simulations) vs. whether they're used for optimization or other industrial processes (suffering subroutines).

It appears that [Meghan Winsby](https://web.archive.org/web/20140220185104/http://www.rotman.uwo.ca/who-we-are/our-members/meghan-winsby/) (coincidentally?) used the same "suffering subroutines" phrase in an excellent 2013 paper: "[Suffering Subroutines: On the Humanity of Making a Computer that Feels Pain](http://www.iacap.org/proceedings\_IACAP13/paper\_48.pdf)." It seems that her usage may refer to what I call sentient simulations, or it may refer to general artificial suffering of either type.

Footnotes
---------

1. [This summary](http://www.openphilanthropy.org/david-chalmers-professor-philosophy-new-york-university-may-20-2016 "'David Chalmers, Professor of Philosophy, New York University on May 20, 2016'") of a conversation with David Chalmers says "one popular theory is that information in the brain is conscious if and only if it is part of a global workspace; information outside the global workspace is unconscious. But it would be a big leap to conclude from this that any system with a global workspace is conscious and that systems that lack a global workspace are not."  [(back)](#back\_ajs-fn-id\_1-251)

[This summary](http://www.openphilanthropy.org/david-chalmers-professor-philosophy-new-york-university-may-20-2016 "'David Chalmers, Professor of Philosophy, New York University on May 20, 2016'") of a conversation with David Chalmers says "one popular theory is that information in the brain is conscious if and only if it is part of a global workspace; information outside the global workspace is unconscious. But it would be a big leap to conclude from this that any system with a global workspace is conscious and that systems that lack a global workspace are not."


A Lower Bound on the Importance of Promoting Cooperation
========================================================

29 August 2015
by [Brian Tomasik](https://longtermrisk.org/author/brian-tomasik/ "Posts by Brian Tomasik")

First written: 3 Jan. 2014; last update: 7 Jun. 2016

 This piece suggests a lower-bound [Fermi calculation](https://en.wikipedia.org/wiki/Fermi\_problem) for the cost-effectiveness of working to promote international cooperation based on one specific branch of possible future scenarios. The purpose of this exercise is to make our thinking more concrete about how cooperation might exert a positive influence for suffering reduction and to make its potential more tangible. I do not intend for this estimate to be quoted in comparison with standard DALYs-per-dollar kinds of figures because my parameter settings are so noisy and arbitrary, and more importantly because these types of calculations are not the best ways to compare projects for shaping the far future when many complex possibilities and flow-through effects are at play. I enumerate other reasons why advancing cooperation seems robustly positive, although I don't claim that cooperation is obviously better than alternate approaches.

Contents

\* [Introduction](#Introduction)
\* [Caveats](#Caveats)
\* [By what fraction could compromise reduce future suffering?](#By\_what\_fraction\_could\_compromise\_reduce\_future\_suffering)
\* [How much future suffering is in our hands?](#How\_much\_future\_suffering\_is\_in\_our\_hands)
\* [Combining the estimates](#Combining\_the\_estimates)
\* [Weird physics](#Weird\_physics)
\* [Other reasons to support cooperation](#Other\_reasons\_to\_support\_cooperation)
 + [A value-of-information argument for future focus](#A\_value-of-information\_argument\_for\_future\_focus)
\* [Footnotes](#Footnotes)

Introduction
------------

[Compromise](http://utilitarian-essays.com/compromise.html) has the potential to benefit most value systems in expectation, by allowing each side in a dispute to get more of what it wants than its fractional share of power. This is wonderful, but how much could compromise matter? In this piece I suggest a Fermi calculation for a lower bound on how much suffering might be prevented by working to promote compromise. The estimates that I use for each variable are more conservative than I think is likely to be the case.

Caveats
-------

I \*do not think\* a Fermi calculation like I describe below is the best approach for evaluating relative cost-effectiveness. This calculation traces one specific, highly conjunctive branch in the vast space of possible branches for how the future might unfold. Most of the expected impact of promoting compromise probably comes from branches that I'm ignoring.

Likewise, activities other than promoting compromise also have many [flow-through effects](http://blog.givewell.org/2013/05/15/flow-through-effects/) on many different possible future branches. Comparing projects to shape the future requires much more than a single Fermi calculation. We should use additional quantitative and qualitative estimates across many models, as well as general heuristics. One of the strongest arguments for promoting compromise is not that it dominates in a Fermi calculation (probably it doesn't) but that "increasing the pie" for many value systems is generally a good idea and seems more [robustly](http://utilitarian-essays.com/robustness-against-uncertainty.html) positive than almost anything else.

That said, explicit and detailed Fermi estimates can help to [clarify our thinking and identify holes](http://lesswrong.com/lw/jfm/another\_critique\_of\_effective\_altruism/aagj), and this is one reason for undertaking the exercise.

By what fraction could compromise reduce future suffering?
----------------------------------------------------------

Suppose the following parameter estimates. Remember, these are designed to be \*conservatively low\*, not most likely. The estimates in each bullet are conditional probabilities given the outcomes from the previous bullets.

\* 40% chance that humanity doesn't go extinct due to causes other than artificial intelligence (AI) in the next few centuries.
\* 20% chance that humanity will develop strong AI in the next few centuries conditional on not going extinct due to non-AI factors.
\* 5% chance that human values will be encapsulated by strong AI.
\* 5% chance that those values, once encapsulated, would be preserved indefinitely rather than changing in arbitrary directions or converging to some inevitable endpoint.
\* 10% chance that, in the default scenario, there is a competition among nations to build the first strong AI to serve that nation's own interests and values.
\* 10% chance that this competition could be turned to compromise if enough people worked on promoting moral tolerance and international cooperation. For example, creating a ![](https://longtermrisk.org/files/United\_Nations\_Flags\_-\_cropped-350x234.jpg "UN flags in Geneva. By Tom Page (Flickr: IMG\_1965) [CC-BY-SA-2.0 (http://creativecommons.org/licenses/by-sa/2.0)], via Wikimedia Commons: https://commons.wikimedia.org/wiki/File:United\_Nations\_Flags\_-\_cropped.jpg")
\* What is "enough people" working to promote cooperation? Say, for example, 1 in 100 working adults on the planet devoting their careers to the cause, over the next 200 years. Assuming a population of maybe ~5 billion working people at a given time, that means about 50 million people over 200 years, or 10 billion person-years.
 + Assume that an effort some fraction of this size has a linearly reduced expected impact. For instance, 5 billion person-years of work instead of 10 billion would mean the chance of turning competition to cooperation is 5% instead of 10%.
 + 50 million people is a lot. There are just [~4,500](https://web.archive.org/web/20150501192315/http://blog.inomics.com:80/phd-graduates-disciplines-and-numbers/) PhDs awarded in the social sciences in the US per year. \*Foreign Affairs\* magazine has [~150,000](https://en.wikipedia.org/wiki/Foreign\_Affairs) subscribers. The entire US government employs [just over 4 million](http://www.facethefactsusa.org/facts/that-bloated-federal-workforce-historically-it-looks-buff) people in civilian and military roles.
\* Absent cooperation, suppose there would be two main superpowers in conflict. Of course, there might be more, but I think the analysis would be basically the same in that case. Imagine that one superpower cares slightly more about suffering reduction than the other. (For example, the USA currently cares more about animal welfare than China.) In particular, suppose that if one country's values controlled the future, the amount of suffering would be X, and if the other controlled the future, suffering would be 1.02X. Suppose each side has equal odds of winning this "Cold War" race. The expected amount of suffering under winner takes all is (X + 1.02X)/2 = 1.01X. Suppose that because of diminishing returns to additional suffering reduction per unit of resources, a compromise arrangement would allow the sides to reach 1.009X suffering instead -- roughly a 1 in 1000 reduction. This specific calculation has been rather detailed, but from a higher level, suggesting that cooperation could reduce expected suffering in the future by 1 in 1000 due to harmonization of conflicting values across countries seems conservatively low.
\* 10% chance that the difference in suffering between the policies of these two countries would actually be a permanent feature of the AIs that those countries would produce in winner-takes-all scenarios rather than being eliminated by [extrapolation](http://wiki.lesswrong.com/wiki/Coherent\_Extrapolated\_Volition).
\* Apply a discount for uncertainty about whether people's efforts actually improve or degrade cooperation in the long run. For example, maybe some activists push for changes to nuclear policy that disrupt the stability of mutual deterrence and make conflicts worse. In particular, suppose the chance is 65% that efforts to promote cooperation actually do promote cooperation and 35% that they hinder it by an equal amount. Then the discount factor is 0.65 - 0.35 = 0.3.

Given these parameter settings, a lower bound on the fraction of future suffering reduced per person-year of work to promote cooperation is

> 40% \\* 20% \\* 5% \\* 5% \\* 10% \\* 10% \\* [1/(10 billion)] \\* (1/1000) \\* 10% \\* 0.3 = 6 \\* 10‑21.
>
>

How much future suffering is in our hands?
------------------------------------------

\* Suppose, conservatively, that only 10‑5 of total, intensity-weighted hedonic experience in the future is negative. This might include [suffering subroutines](http://www.utilitarian-essays.com/suffering-subroutines.html), sentient [simulations](https://en.wikipedia.org/wiki/Life\_simulation\_game) of [wild animals](http://www.utilitarian-essays.com/suffering-nature.html), etc.
\* Nick Bostrom [estimates](http://www.nickbostrom.com/astronomical/waste.html) potential future hedonic experience as 1038 humans surviving for ~1010 years in the Virgo Supercluster, or 1048 experience-years. Of course, some of these might be animals and suffering subroutines, but I'll keep using the "currency" of human experience-years as the reference point.
\* Say the probability that a colonization scenario with this many minds actually happens is 10‑8. Alternatively, we could say that the probability is 10‑6 that a colonization future with this magnitude of computational power happens and has 1% hedonically relevant computations. Any combination of possibilities that leads to an expected 10‑8 fractional multiplier is equivalent.
\* Apply a significant discount factor to account for the incredulity of the proposition that we are in a position to influence this many future experience-years. One might think it's almost impossible that we would happen to be the influential few to affect such a vast future, but model uncertainty suggests we might give some nonzero probability that we are actually in such an incredible period of history. Say the probability that we are is 10‑10.

The expected number of suffering-years in our hands would then be

> 10‑5 \\* 1048 \\* 10‑8 \\* 10‑10 = 1025.
>
>

Combining the estimates
-----------------------

Multiplying 6 \\* 10‑21 by 1025 gives 60,000 expected suffering-years that we can prevent per year of work to promote compromise. Assuming a year of work means 40 hours per week for 50 weeks, this is (60,000)/(40\\*50) = 30 suffering-years per hour, or 0.5 per minute.

To convert this into a per-dollar estimate, suppose it would take $150K per year to pay someone to work on compromise, assuming that person would otherwise have done something unrelated and altruistically neutral. This figure is very high for a nonprofit salary, but if someone is willing to work for a lot less, chances are she's already committed to the cause and would have a high opportunity cost, because she could be [earning to give](http://www.utilitarian-essays.com/make-money.html) instead. In order to attract talented people who would otherwise do altruistically neutral work, a high salary would be required. And remember, this is a conservative calculation. 60,000 expected suffering-years divided by $150K is ~150 suffering-days prevented per dollar. (Here I'm ignoring the fact that future labor-years should be cheaper in present dollars assuming investment returns outpace increases in wages.)

It's important to remember just how imprecise these particular numbers are. For instance, if I had taken the anthropic discount factor to be 10‑5 instead of 10‑10, we would have had 6 billion suffering-years prevented per year of work, or 40,000 suffering-years prevented per dollar.

Weird physics
-------------

This scenario assumed a bound of 1048 experience-years, but there's some chance physics is other than we think and allows for obscenely higher amounts of computation. Indeed, there's a nonzero probability that [infinite computation](https://en.wikipedia.org/wiki/Hypercomputation) is possible, implying infinite future suffering. Our calculation would then blow up to say that every second spent on promoting compromise prevents infinite expected suffering.

A few thoughts on this:

1. Blowing up the number of experience-years we affect shouldn't change \*relative\* comparisons among activities that shape the far future. (And every activity shapes the far future to some extent.) It merely highlights that everything we do has a remote chance of being massively more important than it seems.
2. Black swans like these are a main reason why Fermi calculations are incomplete and inadequate to capture all the factors we need to consider for choosing policies. It can be much more stable to use heuristics like "work on positive-sum projects that make it more likely that our descendants can use their vastly greater wisdom to tackle problems that are beyond our grasp."
3. It seems absurd that we would be the lucky few to be in a position to influence infinitely many future minds. The probability of this seems naively like it should tend to 1/infinity. In general, I think anthropic considerations are an Achilles heel for these calculations about the astronomical importance of the far future.

Other reasons to support cooperation
------------------------------------

I've taken pains to clarify that the calculation in this piece is hardly exhaustive of why cooperation is important but only scratches the surface with one concrete scenario. There are many other reasons for suffering reducers to support international cooperation:

1. Rogue developers. While it seems reasonable to assume that most countries care appreciably about reducing suffering, the same needn't be true for smaller groups of "rogue" AI developers. Stronger global governance would help states enforce coordination rather than letting a bunch of individual groups compete against governments and against each other to build the first strong AI to satisfy their own peculiar ideologies. Note that this kind of enforcement could be desirable \*even for the people who would have joined the rogue groups\* because they would be forced to cooperate rather than defect on a (multi-player) prisoner's dilemma, which is Pareto-preferred by every prisoner in the game. (Compare with "[Why do hockey players support helmet rules, even though they choose not to wear helmets when there is no rule?](http://www.econport.org/content/teaching/modules/NFG/Hockey.html)")
2. Increased humaneness. Cooperation and tolerance make society more humane. When violence is less of a concern, people have more room to explore [self-expression](https://en.wikipedia.org/wiki/File:Inglehart\_Values\_Map.svg), and cultural heroes may shift from being focused on military victory to being focused on kindness. Conversely, the expanding circle of compassion can help advance cooperation, by showing people that those in other countries aren't really very different, and we're all citizens of the world.
3. Maintaining stability and rule of law. Some of the most significant [potential sources](http://www.utilitarian-essays.com/astronomical-suffering.html) of suffering in the future are reinforcement-learning algorithms, artificial-life simulations, and other sentient computational processes. Reducing these forms of suffering would plausibly require machine-welfare laws or norms within a stable society. It's hard to imagine humane concerns carrying currency in a competitive, Wild West environment. International cooperation and other measures to maintain social tranquility are important for enabling more humane standards for industrial and commercial computations.
4. More time to reflect. Cooperation is expected to slow or avert [AI arms races](http://utilitarian-essays.com/ai-arms-race.html), which means humanity should have more time to improve social institutions and philosophical reflectiveness before making potentially irrevocable decisions about the future of the galaxy.
5. Being nice. Because cooperation is good for everyone, if suffering reducers promote it, others will appreciate this fact and may be more inclined to reciprocate toward suffering reducers by doing them favors in other ways. In other words, promoting inter\*national\* cooperation is a form of inter\*personal\* cooperation with other altruists who have different values from ours.
6. Common-sense heuristics. Almost everyone on Earth agrees that stronger international cooperation would be good. "World peace" is a near universal goal, even though it has a ring of platitude by now.
7. Robustness. Probably the strongest reason, which generalizes some of the scenarios discussed above, is that cooperation puts our descendants in a better position, both in terms of social institutions and moral values, to be able to tackle issues that we have no hope of addressing today. It's quite plausible that most of the suffering in the future will come from something that we can't even anticipate now. We should aim to empower our descendants to handle unknown unknowns, by advancing positive \*social\* technology -- including institutions for peace and compromise -- [relatively faster than](http://utilitarian-essays.com/differential-intellectual-progress.html) \*scientific\* technology.

Putting our descendants in a better position to address challenges is useful even if strong AI and space colonization never materialize. Even if humans just continue on Earth for a few million years more, cooperation still improves our trajectory. Of course, this case involves vastly less suffering for us to mitigate, and what we do now may not have a significant impact on what happens tens of thousands of years hence absent goal-preserving AI, so this scenario is negligible in the overall calculations, but those who feel nervous about tiny probabilities of massive impacts would appreciate this consideration. That said, if our only concern was about Earth in the very short term, then plausibly other interventions would appear more promising.

### A value-of-information argument for future focus

There's a general argument that we should focus on far-future scenarios even if they seem unlikely to materialize due to anthropic considerations because of value of information. In particular, suppose there were two main scenarios to which we assigned equal prior probability before anthropic updating: ShortLived, where humanity lasts only a few more centuries, and LongLived, where humanity lasts billions more years. Say LongLived has N times as many experience-moments as ShortLived and so is N times as important. Correspondingly, the anthropic-adjusted probability of LongLived might, under certain views of anthropics, tend toward 1/N. The expected value of ShortLived is (probability)\\*(value) = (roughly 1)\\*(1) = 1 compared against an expected value for LongLived of (probability)\\*(value) = (1/N)\\*N = 1. So it's not clear whether to focus on short-term actions (e.g., reducing wild-animal suffering in the coming centuries) or long-term actions (e.g., promoting international cooperation, good governance, and philosophical wisdom in order to improve the seed conditions for the AI that colonizes our galaxy).

When we consider value of information, it pushes toward longer-term actions because they leave open the option of returning to focus on short-term actions if further analysis leads to that conclusion. To make the explanation simple, imagine that halfway through the expected lifetime of humanity given ShortLived, altruists reassessed their plans to decide if they should continue doing actions targeting LongLived futures or if they should focus instead on ShortLived futures. For the sake of clarity, imagine that at this juncture, they have perfect knowledge about whether to focus on short-term or long-term futures. If long-term futures were best to focus on, they would have already been doing the right thing so far and could stick with it. If short-term futures were more important, they could switch to working on short-term futures for the remaining half of humanity's lifetime and still get half the total value as if they had worked on short-term issues from the beginning.[1](#link\_ajs-fn-id\_1-253)

Of course, a reverse situation could also be true: start focusing on short-term futures and then re-evaluate to decide whether to focus on long-term futures halfway. The difference is that if people have focused on long-term futures from the beginning, they'll have more wisdom and capacity at the halfway point to make this evaluation. This is an instance of the general argument for frontloading wisdom and analysis early and then acting later. Of course, there are plenty of exceptions to this -- for instance, maybe by not acting early, people lose motivation to act altruistically at all. This general conceptual point is not airtight but merely suggestive.

In personal communication, Will MacAskill made a similar argument about "option value" in a related context and thereby partly inspired this section. Needless to say, there are other considerations besides option value in both directions. For instance, there's greater entropy between our actions now and the quality of experience-moments billions of years from now (though a nontrivial probability of a pretty small entropy, assuming we influence a goal-preserving or otherwise politically stable outcome). Meanwhile, experience-moments of the future may have greater intensity, so the stakes may be higher.

Finally, as was hinted in the Fermi calculation, we could fudge a way to make the far future dominate by saying there's a nontrivial probability that our anthropic discount is wrong and that the future really is as important as it seems naively. This may work, though it also feels suspicious because similar sorts of model-uncertainty arguments could be invoked to justify lots of weird considerations dominating our calculations. The importance of the far future seems one of the more robust sentiments among intelligent thinkers, though, so the fudge feels less hacky in this case.

Footnotes
---------

1. Suppose the value of short-term work is known to be 1. The value of long-term work is either N with probability 1/N or is 0 otherwise. By doing just short-term work, we could guarantee a value of 1. By doing long-term work that includes research about whether long-term work is worthwhile, we could choose -- at the halfway point of humanity's lifetime in the ShortLived case -- to switch to short-term work or not. In the exaggerated scenario where we learn perfectly whether far-future work will pay off, we switch to doing short-term work in (N-1)/N of the cases, garnering value of 0.5 for the remaining time. And in 1/N cases, we stick with long-term work and get a payoff of N. So now our expected value is [(N-1)/N] \\* 0.5 + (1/N) \\* N, which is basically 1.5 if N is large. This is 50% better than just starting with the short-term work.  [(back)](#back\_ajs-fn-id\_1-253)

Suppose the value of short-term work is known to be 1. The value of long-term work is either N with probability 1/N or is 0 otherwise. By doing just short-term work, we could guarantee a value of 1. By doing long-term work that includes research about whether long-term work is worthwhile, we could choose -- at the halfway point of humanity's lifetime in the ShortLived case -- to switch to short-term work or not. In the exaggerated scenario where we learn perfectly whether far-future work will pay off, we switch to doing short-term work in (N-1)/N of the cases, garnering value of 0.5 for the remaining time. And in 1/N cases, we stick with long-term work and get a payoff of N. So now our expected value is [(N-1)/N] \\* 0.5 + (1/N) \\* N, which is basically 1.5 if N is large. This is 50% better than just starting with the short-term work.