judiciary.senate.gov

Extracting Money from Causal Decision Theorists
Caspar Oesterheld,Vincent Conitzer
Duke University, Department of Computer Science
focaspar, conitzerg@cs.duke.edu
Abstract
Newcomb’s problem has spawned a debate about
which variant of expected utility maximization (if
any) should guide rational choice. In this paper,
we provide a new argument against what is proba-
bly the most popular variant: causal decision the-
ory (CDT). In particular, we provide two scenarios
in which CDT voluntarily loses money. In the ﬁrst,
an agent faces a single choice and following CDT’s
recommendation yields a loss of money in expec-
tation. The second scenario extends the ﬁrst to a
diachronic Dutch book against CDT.
1 Introduction
In Newcomb’s problem [Nozick, 1969; Ahmed, 2014 ], a “be-
ing” offers two boxes, A and B. Box A is transparent and
contains $1,000. Box B is opaque and may contain either
$1,000,000 or nothing. An agent is asked to choose between
receiving the contents of both boxes, or of box B only. How-
ever, the being has put $1,000,000 in box B if and only if the
being predicted that the agent would choose box B only. The
being’s predictions are uncannily accurate. What should the
agent do?
Causal decision theory (CDT) recommends that the agent
reason as follows: I cannot causally affect the content of the
boxes – whatever is in the boxes is already there. Thus, if
I choose both boxes, regardless of what is in box B, I will
end up with $1,000 more than if I choose one box. Hence, I
should choose both boxes.
Evidential decision theory (EDT) , on the other hand, rec-
ommends that the agent reason as follows: if I choose
one box, then in all likelihood the being predicted that I
would choose one box, so I can expect to walk away with
$1,000,000. (Even if the being is wrong some small percent-
age of the time, the expected value will remain at least close to
$1,000,000.) If I choose both, then I can expect to walk away
with (close to) $1,000. Hence, I should choose one box.
While Newcomb’s problem itself is far-fetched, it has been
argued that the difference between CDT and EDT matters in
various game-theoretic settings. First, it has been pointed out
Contact Authorthat Newcomb’s problem closely resembles playing a Pris-
oner’s Dilemma against a similar opponent [Brams, 1975;
Lewis, 1979 ]. Whereas CDT recommends defecting even
in a Prisoner’s Dilemma against an exact copy, EDT rec-
ommends cooperating when facing sufﬁciently similar op-
ponents. What degree of similarity is required and whether
EDT and CDT ever come apart in this way in practice has
been the subject of much discussion [Ahmed, 2014, Sec-
tion 4.6 ]. A common view is that CDT and EDT come
apart only under fairly speciﬁc circumstances that do not
include most human interactions. Still, Hofstadter [1983 ],
for example, argues that “superrational” humans should co-
operate with each other in a real-world one-shot Prisoner’s
Dilemma, reasoning in a way that resembles the reasoning
done under EDT. Economists have usually been somewhat
dismissive of such ideas, sometimes referring to them as
“(quasi-)magical thinking” when trying to explain observed
human behavior [Shaﬁr and Tversky, 1992; Masel, 2007;
Daley and Sadowski, 2017 ]. Indeed, standard game-theoretic
solution concepts are closely related to ratiﬁcationism [Joyce
and Gibbard, 1998, Section 5 ], a variant of CDT which we
will revisit later in this paper (see Part 3 of Section 4).
Second, even if the players of a game are dissimilar, it is
in the nature of strategic interactions that each player’s be-
havior is predicted by the other players. Newcomb’s prob-
lem itself (as well as the A DVERSARIAL OFFER discussed in
this paper) differs from strategic interactions as they are usu-
ally considered in game theory in its asymmetry. However,
one might still expect that CDT and EDT offer different per-
spectives on how rational agents should deal with the mutual
prediction inherent in strategic interactions [Gauthier, 1989,
Section XI ].
Third, CDT and EDT differ in their treatment of situations
with imperfect recall. Seminal discussions of such games are
due to Piccione and Rubinstein [1997 ]and Aumann et al.
[1997 ][cf. Bostrom, 2010, for an overview from a philoso-
pher’s perspective]. While these problems originally were
not associated with Newcomb’s problem, the relevance of
different decision theories in this context has been pointed
out by Briggs [2010 ][cf. Armstrong, 2011; Schwarz, 2015;
Conitzer, 2015].
The importance of the differences in decision theory is am-
pliﬁed if, instead of humans, we consider artiﬁcial agents.
After all, it is common that multiple copies of a software sys-
Copyright c
2020 for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0).

tem are deployed and other parties are often able to obtain
the source code of a system to analyze or predict its behavior.
As some software systems choose more autonomously, we
might expect their behavior will be (approximately) describ-
able by CDT or EDT (or yet some other theory). If either of
these theories has serious ﬂaws, we might worry that if a sys-
tem implements the wrong theory, it will make unexpected,
suboptimal choices in some scenarios. Such scenarios might
arise naturally, e.g., as many copies of a system are deployed.
We might also worry about adversarial problems like the one
in this paper.
One argument against CDT is that causal decision theorists
(tend to) walk away with less money than evidential decision
theorists, but this argument has not proved decisive in the de-
bate. For instance, one inﬂuential response has been that CDT
makes the best out of the situation – ﬁxing whether the money
is in box B – which EDT does not [Joyce, 1999, Section 5.1 ].
It would be more convincing if there were Newcomb-like sce-
narios in which a causal decision theorist volunteers to lose
money (in expectation or with certainty).1Constructing such
a scenario from Newcomb’s problem is non-trivial. For ex-
ample, in Newcomb’s problem, a causal decision theorist may
realize that box B will be empty. Hence, he would be unwill-
ing to pay more than $1,000 for the opportunity to play the
game.
In this paper, we provide Newcomb-like decision problems
in which the causal decision theorist voluntarily loses money
to another agent. We ﬁrst give a single-decision scenario in
which this is true only in expectation (Section 2). We then
extend the scenario to create a diachronic Dutch book against
CDT – a two-step scenario in which the causal decision theo-
rist is sure to lose money (Section 3). Finally, we discuss the
implications of the existence of such scenarios (Section 4).
1Walking away with the maximum possible (expected) payoff
under any circumstances is not a realistic desideratum for a decision
theory: any decision theory X has a lower expected payoff than some
other decision theory Y in a decision problem that rewards agents
simply for using decision theory Y [cf. Skalse, 2018, for a harder-
to-defuse version of this point]. However, such a setup does not
allow one to devise a generic scenario in which an agent voluntarily
loses money, i.e. loses money in spite of having the option to walk
away losing nothing.
Furthermore, scenarios with voluntary loss appear signiﬁcantly
more problematic for pragmatic reasons. Regardless of what you
think is the right option in Newcomb’s problem, you might not view
Newcomb’s problem as relevant ground for decision-theoretical ar-
gument because it is so unlikely that one would ever face New-
comb’s problem in the real world. For instance, even if you thought
that one-boxing is rational (and two-boxing is not), you might stick
with CDT nonetheless because your real-world expected opportu-
nity costs from two-boxing in Newcomb’s problem are negligible.
[For some discussion of this deﬂationary argument, see, e.g., Gau-
thier, 1989, Section XI; Ahmed, 2014, Section 7.1.iv; Oesterheld,
2019, Section 1, and references therein.] However, if there is a
Newcomb-like problem in which the causal decision theorist volun-
tarily loses money to some other agent, this generates a signiﬁcant
incentive to place him in such a situation.2 Extracting a proﬁt in expectation from
causal decision theorists
Consider the following scenario:
ADVERSARIAL OFFER : Two boxes, B1andB2,
are on offer. A (risk-neutral) buyer may purchase
one or none of the boxes but not both. Each of
the two boxes costs $1. Yesterday, the seller put
$3in each box that she predicted the buyer not to
acquire. Both the seller and the buyer believe the
seller’s prediction to be accurate with probability
0:75. No randomization device is available to the
buyer (or at least no randomization device that is
not predictable to the seller).2
If the buyer takes either box Bi, then the expected money
gained by the seller is
$1P($3 inBijbuyer chooses Bi)$3
= $10:25$3
= $0:25:
Hence, the buyer suffers an expected loss of $0:25(if he buys
a box). The best action for the buyer therefore appears to be
to not purchase either box. Indeed, this is the course of action
prescribed by EDT as well as other decision theories that rec-
ommend one-boxing in Newcomb’s problem [e.g., those pro-
posed by Spohn, 2012; Poellinger, 2013; Soares and Levin-
stein, 2017].
In contrast, CDT prescribes that the buyer buy one of the
two boxes. Because the agent cannot causally affect yester-
day’s prediction, CDT prescribes to calculate the expected
utility of buying box Bias
P($3 in boxBi)$3$1; (1)
whereP($3 in boxBi)is the buyer’s subjective probability
that the seller has put money in box Bi,prior to updating
this belief based on his own decision. For i= 1;2, letpibe
the probability that the buyer assigns to the seller having pre-
dicted him to buy Bi. Similarly, let p0be the probability the
buyer assigns to the seller having predicted him to buy noth-
ing. These beliefs should satisfy p0+p1+p2= 1. Because
p00, we have that (p0+p1)+(p0+p2) = 2p0+p1+p21.
Hence, it must be the case that p0+p11
2orp0+p21
2(or both). Because P($3 in boxBi) =p0+p3ifori= 1;2,
it isP($3 in boxBi)1
2for at least one i2f1;2g. Thus,
the expected utility in eq. 1 of at least one of the two possible
purchases is at least1
2$3$1 = $0:50, which is positive.
Any seller capable of predicting the causal decision theo-
rist sufﬁciently well will thus have an incentive to use this
scheme to exploit CDT agents. (It does not matter whether
the seller subscribes to CDT or EDT.) It should be noted that
even if the buyer uses CDT, his view of the deal matches the
seller’s as soon as the dollar is paid. That is, after observ-
ing his action, he will realize that the box he bought is empty
2This decision problem resembles the widely discussed Death in
Damascus scenario [introduced to the decision theory literature by
Gibbard and Harper, 1981, Section 11] and even more closely the
Frustrater case proposed by Spencer and Wells [2017 ], though these
are not set up to result in an expected ﬁnancial loss.
Copyright c
2020 for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0).
with probability 0:75and thus worth less than a dollar. CDT
knows that it will regret its choice [see Joyce, 2012; Weirich,
1985 for discussions of the phenomenon of anticipated regret
a.k.a. decision instability in CDT].
3 A diachronic Dutch book against causal
decision theory
ADVERSARIAL OFFER results in a loss in expectation for the
causal decision theorist. It is natural to ask whether it is pos-
sible to set up the scenario so that the causal decision theo-
rist ends up with a sure loss; effectively, a Dutch book. Ar-
guably, Dutch books are more convincing than scenarios with
expected losses since the very meaning of “expectations” is
the subject of the debate about EDT and CDT. Of course, if
the seller could perfectly predict the buyer in A DVERSARIAL
OFFER (instead of being right only 75% of the time), then
ADVERSARIAL OFFER would become a Dutch book. But
can we construct a Dutch book without perfect prediction?
We have already observed that in A DVERSARIAL OFFER
the causal decision theorist always regrets his decision after
observing its execution. This suggests the following simple
approach to constructing a Dutch book. After the box is sold,
the seller allows the buyer to reverse his decision for a small
fee (ending up without any box and having lost only the fee).
However, a CDT buyer may then anticipate eventually undo-
ing his choice and therefore not buy a box in the ﬁrst place
[Ahmed, 2014, Section 3.2; though cf. Skyrms, 1993; Ra-
binowicz, 2000].3To get our Dutch book to work, we add
another choice before ADVERSARIAL OFFER .
ADVERSARIAL OFFER WITH OPT-OUT: It is
Monday. The buyer is scheduled to face the A D-
VERSARIAL OFFER on Tuesday. He also knows
that the seller’s prediction was already made on
Sunday.
As a courtesy to her customer, the seller approaches
the buyer on Monday. She offers to not offer the
boxes on Tuesday if the buyer pays her $0:20.
Note that the seller does not attempt to predict whether the
buyer will pay to opt out. Also, we assume that the buyer
cannot, on Monday, commit himself to a course of action to
follow on Tuesday.
It seems that a rational agent should never feel compelled
to accept the Monday offer. After all, doing so loses him
money with certainty, whereas simply refusing both offers
(on Monday and on Tuesday) guarantees that he loses no
money.
CDT, however, recommends opting out on Monday, for the
following reasons. A CDT buyer knows on Monday that if
he does not opt out, he will buy a box on Tuesday (though
he may not yet know which one). Further, he believes that
whatever box he will take on Tuesday will contain $3 with
only 25% probability, thus implying an overall expected pay-
off of 0:25$3$1 =$0:25. This is because, on Monday,
CDT treats the decision on Tuesday in the same way as it
3This, of course, requires that the reversal offer does not come as
a surprise. Throughout, we insist that the buyer knows all the rules
of the game.treats any other random variable in the environment. So the
causal expected utility of not opting out is just what an out-
side observer would expect the payoff of a CDT agent facing
ADVERSARIAL OFFER to be. Because this expected payoff
of$0:25is less than the certain payoff of $0:20that can
be obtained by opting out, CDT recommends opting out.
In fact, for the argument in the previous paragraph to suc-
ceed, it is only necessary that CDT is used on Tuesday; other
decision theories would also recommend accepting the Mon-
day offer, ifthey anticipate that the agent will use CDT on
Tuesday. For instance, if the agent followed EDT on Monday
and CDT on Tuesday (and is aware on Monday that he will
use CDT on Tuesday), then he would still accept the Monday
offer. Similarly, if the seller believes that the buyer will pick
one of the boxes on Tuesday, then she will hope that he re-
jects the Monday offer. Thus, it seems that what creates the
opportunity for a Dutch book is the prospect of buying a box
on Tuesday (as CDT recommends), not the use of CDT on
Monday.
4 Discussion
We differentiate four types of responses to these scenarios
available to supporters of causal decision theory:
1. They could claim that these scenarios are irrelevant for
evaluating decision theories, in the sense that they are
impossible to set up or otherwise out of scope, and there-
fore unpersuasive.
2. They could concede that these scenarios are relevant for
evaluating decision theories, but claim that CDT’s rec-
ommendations in them are acceptable.
3. They could concede that our analysis obliges them to
give up on certain speciﬁc formulations of CDT, but try
to modify CDT to get these scenarios right while main-
taining some of its essence, in particular two-boxing and
the causal dominance principle.
4. They could concede that these scenarios show that the
very core of CDT (two-boxing and the causal dominance
principle) is implausible.
We will discuss these options in turn.
1Surely, if one could show that a CDT agent will or can
never face these scenarios – despite the seller having an obvi-
ous incentive to set them up – that would be the most convinc-
ing defense of CDT. In particular, a causal decision theorist
might claim that sufﬁciently accurate prediction of a CDT
agent is simply impossible.4However, not much accuracy
is required, for the following reasons. The CDT agent will
take one of the two boxes. Even if the seller picks the box to
ﬁll with money uniformly at random, she would therefore be
right half of the time. If she can do any better than that, pre-
dicting correctly with probability 1=2+, then she can extract
money from the CDT agent by putting (instead of $3) some
amount between $2=(12)and $2 in the box predicted not
4For a general discussion of such unpredictability claims in de-
fense of CDT, see Ahmed [2014, Chapter 8 ].
Copyright c
2020 for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0).
to be taken. Thus, the CDT agent needs to be completely un-
predictable in order to avoid being taken advantage of in these
examples.
Most human beings are, generally speaking, at least some-
what predictable in their actions even when such predictabil-
ity can be used against them. For example, in rock-paper-
scissors – which structurally resembles the A DVERSARIAL
OFFER – most people follow exploitable patterns in what
moves they select [see, e.g., Farber, 2015, and references
therein].5Consider such a somewhat predictable person
who aims to be a causal decision theorist. It seems that he
would indeed be vulnerable to the examples discussed ear-
lier. The only defense for the supporter of causal decision
theory would seem to then be that if so, the person in ques-
tion is not truly acting in the way that CDT describes. That
is, acting according to CDT also requires being unpredictable
to the seller, either by succeeding at out-thinking the seller
sufﬁciently often, or by acting sufﬁciently randomly.
Is it reasonable to consider this a requirement of acting ac-
cording to CDT? CDT does not suggest any strict preference
for choosing randomly across options, as opposed to just de-
terministically choosing one of the options that is best accord-
ing to the buyer’s beliefs. Hence, the unpredictability would
have to emerge from the buyer attempting to out-think the
seller. But it does not seem that this is always an attainable
goal. For example, imagine that the buyer is a deterministic
computer program whose source code is known to the seller.
Then regardless of how exactly the agent works, the seller
can predict the buyer’s behavior perfectly [cf. Soares and Fal-
lenstein, 2014, Section 2; Cavalcanti, 2010, Section 5]. We
would thus be forced to conclude that such a program cannot
possibly follow CDT, which to us is an unsatisfactory con-
clusion. Plausibly any other physically realized agent that
chooses deterministically can at least in principle (if not with
current technology) be predicted by creating or emulating an
atom-by-atom copy of that agent [cf. Yudkowsky, 2010, pp.
85ff.].
Even if the supporter of CDT acknowledges that these sce-
narios are possible , he might nevertheless argue that they are
irrelevant , in the sense that the decision theory is not intended
to be used for such scenarios and hence nothing that one could
show about its performance in such a scenario is of signiﬁ-
cance for evaluating the theory. “It is as if one evaluated a
car by testing how it performs underwater.” There is little we
can say about this response. Still, we expect it to be unattrac-
tive to most decision theorists. After all, our scenarios (in
particular the A DVERSARIAL OFFER ) resemble Newcomb’s
problem – the problem that has led to the development of
CDT in the ﬁrst place. Further, if our scenarios were out of
CDT’s scope, then we (and presumably most other decision
theorists) would still be interested in identifying a decision
theory that does make good recommendations for predictable
agents (such as artiﬁcial intelligent agents whose behavior is
5There are multiple rock-paper-scissors bots available online
which attempt to predict their opponent’s future moves based on
past moves (using data from other players). As of July 2019, the
bot at http://www.essentially.net/rsp/ has reportedly played about 2
million rounds and won 57% more often than it lost.determined by a computer program) facing a wide range of
scenarios including the ones given in this paper.
2If our scenarios are within the scope of causal decision
theory, then the supporter of causal decision theory has to
contend with the fact that one can extract expected money
from, and even Dutch-book, CDT agents in them. But he
might question the signiﬁcance of Dutch-book arguments and
other money extraction schemes, either in general or in this
particular context. For some general discussion of whether
(diachronic) Dutch books are conclusive decision-theoretic
arguments, see, e.g., Vineberg [2016 ]or H´ajek [2009 ]. Note,
though, that some of the most inﬂuential arguments in favor
of expected utility maximization (EUM) – of which CDT is
a reﬁnement – are Dutch books. Of course, one may use dif-
ferent arguments to justify EUM. But it would seem odd to
follow Dutch-book arguments to EUM but no further.
Instead of rehashing some of the more generic reasons for
and against the persuasiveness of Dutch books and loss of
money in expectation, we here discuss a response that is spe-
ciﬁc to CDT and A DVERSARIAL OFFER WITH OPT-OUT.6
A causal decision theorist may argue that it is not gener-
ally fair to expect any kind of coherence from CDT’s recom-
mendations when multiple decisions are to be made across
time, due to the different perspectives that the decision maker
adopts (and, arguably, has to adopt) at different points in time.
Consider Newcomb’s problem. Let t0be the time at which
the predictor observes the agent (perhaps using fMRI or the
like) in order to make a prediction. Then, before t0, CDT rec-
ommends committing – and if needed paying money to com-
mit – to one-boxing [cf. Barnes, 1997; Joyce, 1999, pp. 153f.;
Meacham, 2010]. After t0, CDT recommends two-boxing.
However, most decision theorists do not consider this to be a
compelling argument against CDT. The causal decision theo-
rist can easily justify the difference in the decision made by
the fact that, before t0, the commitment decision has a causal
effect on what is in the boxes, and after t0, it does not.
It would be hypocritical for an evidential decision theorist
to disagree, since EDT is dynamically inconsistent in analo-
gous ways. For instance, consider a version of Newcomb’s
problem in which both boxes are transparent [Gibbard and
Harper, 1981, Section 10; also discussed by Gauthier, 1989;
Drescher, 2006, Section 6.2; Arntzenius, 2008, Section 7;
Meacham, 2010, Section 3.2.2]. Let t0
0be the time at which
the EDT agent sees the content of both boxes. Then before t0
0,
EDT recommends committing – and if needed paying money
to commit – to one-boxing. After t0
0, EDT recommends two-
boxing.7The evidential decision theorist can easily justify
this along similar lines: before t0
0, her commitment is evi-
dence about what is in the boxes, and after t0
0it no longer
is.
6For a discussion of similar arguments about other diachronic
Dutch books, see, e.g., Rabinowicz [2008 ].
7Parﬁt’s (1984) hitchhiker [Barnes, 1997 ], XOR Blackmail
[Soares and Levinstein, 2017, Section 2 ]and Yankees vs. Red Sox
[Arntzenius, 2008, pp. 22-23; Ahmed and Price, 2012] similarly ex-
pose dynamic inconsistencies in EDT. Conitzer [2015 ]gives a some-
what different type of scenario – based on the Sleeping Beauty prob-
lem – in which EDT is dynamically inconsistent.
Copyright c
2020 for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0).
Thus, at least some types of dynamic inconsistency do not
constitute strong arguments against a decision theory. How-
ever, in our opinion, the dynamic inconsistency displayed
by CDT in the A DVERSARIAL OFFER WITH OPT-OUTis
much more problematic. For one, it leads to a Dutch book.
Often, the main argument that is given for why a particu-
lar inconsistency is problematic is precisely that it allows
for a Dutch book. Conversely, defenses of dynamic incon-
sistencies [Ahmed, 2014, Section 3.2, for an example in a
Newcomb-like scenario ]often focus on arguing that they do
notallow for Dutch Books.
Further, it seems that some of the reasons for (or defenses
of) dynamic inconsistency in the above decision problems do
not apply to CDT’s dynamic inconsistency in A DVERSARIAL
OFFER WITH OPT-OUT. For CDT in Newcomb’s problem,
there is a particular event at time t0that splits the decision
perspectives: the loss of causal control at t0over the content
of box B. Similarly, for EDT in the Newcomb’s problem with
transparent boxes, that event is the loss of evidential control
[cf. Almond, 2010, Section 4.5] at t0
0over the content of box
B. It is thus easy to argue for defenders of the respective theo-
ries that the perspectives from before and after t0ort0
0should
diverge [Ahmed and Price, 2012, pp. 22-23, Section 4 ]. In
sharp contrast, the A DVERSARIAL OFFER WITH OPT-OUT
lacks any such event between the decision points. The differ-
ence in perspectives for CDT appears to be purely a result of
CDT viewing its current choice differently than it views past
and future decisions.
All that being said, we agree that caution should be taken
when evaluating a decision theory based on scenarios with
multiple decisions across time. In general, more research on
what conclusions can be drawn from such scenarios is needed
[Steele and Stef ´ansson, 2016, Section 6 ]. Nevertheless, we do
not see any clear path by which such research would justify
CDT’s recommendations in the A DVERSARIAL OFFER WITH
OPT-OUT. In any case, even if one is at this point unwilling to
consider scenarios with multiple decision points at all for the
purpose of evaluating decision theories, one would still have
to contend with the simpler A DVERSARIAL OFFER scenario,
in which there is only one decision point.
3If a straightforward interpretation of CDT cannot be de-
fended against our scenarios, one may look to modify it to
avoid expected or sure loss while preserving some of CDT’s
core tenets. In particular, in response to other alleged coun-
terexamples, some authors have tried to modify CDT while
maintaining the causal dominance [Joyce, 1999, Section 5.1 ]
a.k.a. sure thing [Gibbard and Harper, 1981, Section 7 ]prin-
ciple [though see Ahmed, 2012, for an argument against the
motivation behind some of these approaches]. For example,
one may turn to the concept of ratiﬁability. In Newcomb-like
scenarios such as those under discussion here, for any choice
a, we can consider the beliefs about what is in the boxes that
would result from knowing that one will choose a. Then, a
choiceais ratiﬁable if it is an optimal choice – as judged
by CDT – under those beliefs. For example, in Newcomb’s
problem only two-boxing is ratiﬁable, precisely because it is
causally dominant. For an overview of ratiﬁcation and its
relation to CDT, see Weirich [2016, Section 3.6 ]. Unfortu-nately, this concept is of no help in the A DVERSARIAL OF-
FER, because none of the three options (buying B1, buying
B2or declining) is ratiﬁable. For instance, under the beliefs
that would result from knowing that you will take box Bi, it
would be better to buy the other box B3i.
The ratiﬁcationist may respond by claiming that unpre-
dictable randomization should always be possible. If that
were true, then the only ratiﬁable option would be to take
each box with probability 50%, thus gaining money in ex-
pectation. But again, we would like to have a decision the-
ory that works in a broad variety of scenarios, including ones
where the agent expects to be somewhat predictable. Further-
more, even if a true random number generator (TRNG) (e.g.,
one based on nuclear decay) is in fact available, this does not
settle the issue. For example, consider a variant of the A D-
VERSARIAL OFFER in which the seller refrains from putting
money in any box if she predicts the buyer to make different
choices depending on the output of the TRNG. In this A NTI-
RANDOMIZATION ADVERSARIAL OFFER , again no option
is ratiﬁable: under the beliefs that would result from know-
ing that you will make different choices depending on the
TRNG’s output (and therefore choose a box with some posi-
tive probability), you would rather not pick any box. To cir-
cumvent this example, the ratiﬁcationist could argue that the
decision maker should be able to randomize in such a way
thatwhether he is randomizing is unpredictable. However, at
this point, one might just as well assert the impossibility (or
irrelevance) of Newcomb-type scenarios altogether, which we
have addressed in 1.
A different strategy for modifying CDT to avoid the Dutch
book in the A DVERSARIAL OFFER WITH OPT-OUTis the
following. The Dutch book arises from a disagreement be-
tween CDT on Monday and CDT on Tuesday (cf. the discus-
sion under 2). A tempting possibility is to modify CDT so that
it considers all decisions to be made at once. That is, such a
version of CDT – let us refer to it as policy-CDT – prescribes
that one decide on one’s general policy all at once.8In the
ADVERSARIAL OFFER WITH OPT-OUT, there are four possi-
ble policies: opt out, buy B1, buyB2, and buy nothing (where
the last three possibilities include declining the opt-out offer).
When considering these policies, buy nothing dominates opt
out. Hence, policy-CDT will decline the opt-out offer and
thereby avoid the Dutch book. (Note, however, that such a
modiﬁcation of CDT will make no difference to the choices it
prescribes in A DVERSARIAL OFFER , which has only one de-
cision point. Hence, it will still lose money in expectation.)
While this appears to be a promising approach, it is non-
trivial to ﬂesh out, because on other examples it is less clear
what policy-CDT should prescribe. For illustration, consider
the following interpretation of policy-CDT: follow the pol-
icy to which CDT would like to commit ex ante, where “ex
8Policy-CDT resembles Fisher’s [nd]disposition-based decision
theory. Compare Meacham [2010 ]for a discussion of explicit pre-
commitment. Similarly, Gauthier [1989 ]has argued for evaluating
“plans” not decisions in Newcomb-like problems (without basing
this argument on any particular theory like CDT or EDT). A few au-
thors have also proposed policy versions of other, more EDT-like de-
cision theories [Drescher, 2006, Section 6.2; Yudkowsky and Soares,
2018, Section 4].
Copyright c
2020 for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0).
ante” refers to some point in time before the ﬁrst decision of
the scenario. Now, let us consider a version of Newcomb’s
problem which is supplemented by another trivial and un-
related decision – say, whether to eat a peppermint – that
takes place when the agent still has a causal inﬂuence over
the prediction. Then the ex-ante-commitment interpretation
of policy-CDT would recommend one-boxing. To the causal
decision theorist, this may be unacceptable, especially given
that adding the peppermint decision is such a minor modi-
ﬁcation of Newcomb’s problem. Perhaps there is a way to
deﬁne policy-CDT that avoids such dependence on irrelevant
decisions while also prescribing two-boxing, but it is not im-
mediately obvious how to do so.
Many other ways of modifying CDT are worth consider-
ing. For instance, in the A DVERSARIAL OFFER , it may be un-
realistic for the buyer to form a single probability distribution
over box contents. Instead, he may consider multiple different
probability distributions, including one under which box B1
is probably empty and one under which box B2is probably
empty. He could then evaluate each option pessimistically,
i.e., w.r.t. the probability distribution that is worst under that
option. Such a version of CDT would prescribe declining
to buy a box. At the same time, it would recommend two-
boxing in Newcomb’s problem and more generally obey the
causal dominance principle. For a discussion of this maxmin
criterion for choice under multiple probability distributions,
see, e.g., Gilboa and Schmeidler [1989 ]and in particular
game-theoretic interpretations such as that of Gr ¨unwald and
Halpern [2011 ]. A more general discussion of how using sets
of probability distributions (while potentially decision rules
other than the maxmin criterion) is offered by Bradley [2012 ].
In our setting, B1areB2are, roughly, complementary bets in
the causalist’s beliefs. In all worlds in which Biis empty,
B3iis full. As discussed by Bradley, it has been argued that
a rational agent should accept one of a pair of complementary
bets. Indeed, expected utility maximization for a single prob-
ability distribution satisﬁes this complementarity criterion –
to the causalist’s detriment in the Adversarial Offer. Bradley
[2012 ]argues that in general, an agent with imprecise prob-
abilities should not satisfy the complementarity criterion and
that this allows him to avoid Dutch books – though, of course,
he considers Dutch books of a very different type.
4Finally, one may view at least one of the scenarios in
this paper as supporting a persuasive argument against the
very core of CDT. EDT is the obvious alternative. How-
ever, depending on how problematic we ﬁnd EDT’s prescrip-
tions in other cases – such as the Smoking lesion [Ahmed,
2014, Section 4.1–4.3 ]or cases of dynamic inconsistency like
Newomb’s problem with transparent boxes (and the problems
listed in footnote 7) – we may also look to various other
decision theories that have been proposed [Gauthier, 1989;
Spohn, 2012; Poellinger, 2013; Soares and Levinstein, 2017 ].
Acknowledgements
We thank Johannes Treutlein and Jesse Clifton for comments
and discussions.References
[Ahmed and Price, 2012 ]Arif Ahmed and Huw Price.
Arntzenius on ‘why ain’cha rich?’. Erkenntnis , 77(1):15–
30, 7 2012.
[Ahmed, 2012 ]Arif Ahmed. Push the button. Philosophy of
Science , 79(3):386–395, 7 2012.
[Ahmed, 2014 ]Arif Ahmed. Evidence, Decision and
Causality . Cambridge University Press, 2014.
[Almond, 2010 ]Paul Almond. On causation and correlation
part 1: Evidential decision theory is correct, 9 2010.
[Armstrong, 2011 ]Stuart Armstrong. Anthropic decision
theory, Nov 2011.
[Arntzenius, 2008 ]Frank Arntzenius. No regrets, or: Edith
piaf revamps decision theory. Erkenntnis , 68(2):277–297,
2008.
[Aumann et al. , 1997 ]Robert J. Aumann, Sergiu Hart, and
Motty Perry. The absent-minded driver. Games and Eco-
nomic Behavior , 20:102–116, 1997.
[Barnes, 1997 ]R. Eric Barnes. Rationality, dispositions, and
the newcomb paradox. Philosophical Studies: An Inter-
national Journal for Philosophy in the AnalyticTradition ,
88(1):1–28, 10 1997.
[Bostrom, 2010 ]Nick Bostrom. Anthropic Bias: Observa-
tion Selection Effects in Science and Philosophy . Studies
in Philosophy. Routledge, 2010.
[Bradley, 2012 ]Seamus Bradley. Dutch book arguments and
imprecise probabilities. In D. Dieks, W. Gonzalez, S. Hart-
mann, M. St ¨oltzner, and M. Weber, editors, Probabilities,
Laws, and Structures , volume 3 of The Philosophy of Sci-
ence in a European Perspective , pages 3–17. Springer,
Dordrecht, 2012.
[Brams, 1975 ]Steven J. Brams. Newcomb’s problem and
prisoners’ dilemma. The Journal of Conﬂict Resolution ,
19(4):596–612, 12 1975.
[Briggs, 2010 ]Rachael Briggs. Putting a value on beauty.
volume 3 of Oxford Studies in Epistemology , pages 3–34.
Oxford University Press, 2010.
[Cavalcanti, 2010 ]Eric G. Cavalcanti. Causation, decision
theory, and bell’s theorem: A quantum analogue of the
newcomb problem. The British Journal for the Philosophy
of Science , 61(3):569–597, 9 2010.
[Conitzer, 2015 ]Vincent Conitzer. A dutch book against
sleeping beauties who are evidential decision theorists.
Synthese , 192(9):2887–2899, 10 2015.
[Daley and Sadowski, 2017 ]Brendan Daley and Philipp
Sadowski. Magical thinking: A representation result. The-
oretical Economics , 12:909–956, 2017.
[Drescher, 2006 ]Gary L. Drescher. Good and Real – De-
mystifying Paradoxes from Physics to Ethics . MIT Press,
2006.
[Farber, 2015 ]Neil Farber. The surprising psychology of
rock-paper-scissors, 4 2015.
Copyright c
2020 for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0).
[Fisher, nd ]Justin C. Fisher. Disposition-based decision the-
ory. n.d.
[Gauthier, 1989 ]David Gauthier. In the neighbourhood
of the newcomb-predictor (reﬂections on rationality).
InProceedings of the Aristotelian Society, New Series,
1988–1989 , volume 89, pages 179–194. 1989.
[Gibbard and Harper, 1981 ]Allan Gibbard and William L.
Harper. Counterfactuals and two kinds of expected util-
ity. In William L. Harper, Robert Stalnaker, and Glenn
Pearce, editors, Ifs. Conditionals, Belief, Decision, Chance
and Time , volume 15 of The University of Western On-
tario Series in Philosophy of Science. A Series of Books in
Philosophy of Science, Methodology, Epistemology, Logic,
History of Science, and Related Fields , pages 153–190.
Springer, 1981.
[Gilboa and Schmeidler, 1989 ]Itzhak Gilboa and David
Schmeidler. Maxmin expected utility with non-unique
prior. Journal of Mathematical Economics , 18:141–153,
1989.
[Gr¨unwald and Halpern, 2011 ]Peter D. Gr ¨unwald and
Joseph Y . Halpern. Making decisions using sets of
probabilities: Updating, time consistency, and calibration.
Journal of Artiﬁcial Intelligence Research , 42, 2011.
[Hofstadter, 1983 ]Douglas Hofstadter. Dilemmas for super-
rational thinkers, leading up to a luring lottery. Scientiﬁc
American , 248(6), Jun 1983.
[H´ajek, 2009 ]Alan H ´ajek. Dutch book arguments. In The
Handbook of Rational and Social Choice , chapter 7. Ox-
ford University Press, 2009.
[Joyce and Gibbard, 1998 ]James M. Joyce and Allan Gib-
bard. Causal decision theory. In Handbook of Utility The-
ory, Volume 1: Principles. , chapter 13, pages 627–666.
Kluwer, 1998.
[Joyce, 1999 ]James M. Joyce. The Foundations of Causal
Decision Theory . Cambridge Studies in Probability,
Induction, and Decision Theory. Cambridge University
Press, 1999.
[Joyce, 2012 ]James M. Joyce. Regret and instability in
causal decision theory. Synthese , 187:123–145, 2012.
[Lewis, 1979 ]David Lewis. Prisoners’ dilemma is a new-
comb problem. Philosophy & Public Affairs , 8(3):235–
240, 1979.
[Masel, 2007 ]Joanna Masel. A bayesian model of quasi-
magical thinking can explain observed cooperation in the
public good game. Journal of Economic Behavior & Or-
ganization , 64(2):216–231, 10 2007.
[Meacham, 2010 ]Christopher J. G. Meacham. Binding and
its consequences. Philosophical Studies , 149(1):49–71, 5
2010.
[Nozick, 1969 ]Robert Nozick. Newcomb’s problem and
two principles of choice. In Nicholas Rescher et al., ed-
itor, Essays in Honor of Carl G. Hempel , pages 114–146.
Springer, 1969.[Oesterheld, 2019 ]Caspar Oesterheld. Approval-directed
agency and the decision theory of newcomb-like problems.
Synthese , 2019.
[Parﬁt, 1984 ]Derek Parﬁt. Reasons and Persons . Oxford
University Press, 1984.
[Piccione and Rubinstein, 1997 ]Michele Piccione and Ariel
Rubinstein. On the interpretation of decision problems
with imperfect recall. Games and Economic Behavior ,
20:3–24, 1997.
[Poellinger, 2013 ]Roland Poellinger. Unboxing the con-
cepts in newcomb’s paradox: Causation, prediction, de-
cision. 2013.
[Rabinowicz, 2000 ]Wlodek Rabinowicz. Money pump with
foresight. In Michael J. Almeida, editor, Imperceptible
Harms and Beneﬁts , pages 123–154. Springer, 2000.
[Rabinowicz, 2008 ]Wlodek Rabinowicz. Pragmatic Argu-
ments for Rationality Constraints , pages 139–163. Rea-
soning, Rationality and Probability. CSLI Publications,
2008.
[Schwarz, 2015 ]Wolfgang Schwarz. Lost memories and
useless coins: revisiting the absentminded driver. Syn-
these , 192:3011–3036, 2015.
[Shaﬁr and Tversky, 1992 ]Eldar Shaﬁr and Amos Tversky.
Thinking through uncertainty: Nonconsequential reason-
ing and choice. Cognitive Psychology , 24(4):449–474,
1992.
[Skalse, 2018 ]Joar Skalse. A counterexample to perfect de-
cision theories and a possible response. 2018.
[Skyrms, 1993 ]Brian Skyrms. A mistake in dynamic coher-
ence arguments? Philosophy of Science , 60:320–328, 6
1993.
[Soares and Fallenstein, 2014 ]Nate Soares and Benja Fall-
enstein. Toward idealized decision theory. Technical
Report 2014-7, Machine Intelligence Research Institute,
2014.
[Soares and Levinstein, 2017 ]Nate Soares and Benjamin A.
Levinstein. Cheating death in damascus. In Formal Episte-
mology Workshop (FEW) 2017 , University of Washington,
Seattle, USA, 5 2017.
[Spencer and Wells, 2017 ]Jack Spencer and Ian Wells. Why
take both boxes? Philosophy and Phenomenological Re-
search , 2017.
[Spohn, 2012 ]Wolfgang Spohn. Reversing 30 years of dis-
cussion: why causal decision theorists should one-box.
Synthese , 187(1):95–122, 2012.
[Steele and Stef ´ansson, 2016 ]Katie Steele and H. Orri
Stef´ansson. Decision theory. 2016.
[Vineberg, 2016 ]Susan Vineberg. Dutch book arguments.
In Edward N. Zalta, editor, The Stanford Encyclopedia of
Philosophy . Metaphysics Research Lab, Stanford Univer-
sity, spring 2016 edition, 2016.
[Weirich, 1985 ]Paul Weirich. Decision instability. Aus-
tralasian Journal of Philosophy , 63(4):465–472, 1985.
Copyright c
2020 for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0).
[Weirich, 2016 ]Paul Weirich. Causal decision theory. In The
Stanford Encyclopedia of Philosophy . Spring 2016 edition,
2016.
[Yudkowsky and Soares, 2018 ]Eliezer Yudkowsky and Nate
Soares. Functional decision theory: A new theory of in-
strumental rationality, 5 2018.
[Yudkowsky, 2010 ]Eliezer Yudkowsky. Timeless decision
theory, 2010.
Copyright c
2020 for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0).


How valuable is movement growth?

Centre for Effective Altruism – Working Paper , 2015
1 How valuable is movement growth ?
Owen Cotton -Barratt
Executive Summary
Movement growth may be very important for young social movements. It’s obvious
that movement building cannot be always better than direct work, but knowing how
to compare them presents a challenge.
In this article I introduce and explore a model of movement growth which tracks
individuals’ awareness of and inclination towards the movement. I aim to
understand when movement building activities are better or worse than direct work ,
and apply the model to give my views on movement growth in the effective altruist
and related communities.
Part 1: Theory
In the first half of this paper I introduce a model for thinking about movement
growth, and terminology to refer to critical concept s. We model individuals as
having varying levels of awareness about the movement, and varying inclinations
towards it. We assume that these two characteristics can represent the major drivers
of interaction with the movement. We explore the consequences th is
Awareness/Inclination Model (AIM), particularly looking at the long -term
counterfactual effects of direct work compared to ‘publicity’, which aims at increasing
awareness of the movement, and ‘advocacy’, which aims at improving inclinations
towards the movement. This involves analysing different possible long -term
trajectories the movement may be on.
If we accept the model, this has some general implications:
 For early -stage movements, the effects on movement growth are a key
consideration in deciding b etween different activities. For relatively mature
movements, direct work is usually better than movement growth.
 It is more important to focus on increasing awareness than improving
inclination, if:
o the movement has a natural maximum size that we cannot c hange; or
o essentially everyone will join the movement after they know enough
about it; or
o direct work earlier is much more important than direct work later; or
o it is very hard to change inclination relative to awareness .
 Otherwise improving inclination ma y often be better than increasing
awareness (this is sensitive to beliefs about some parameters).
How valuable is movement growth?
2  It is particularly key to avoid being controversial and focus on improving
inclination rather than increasing awareness, if:
o the views of people around them have a significant effect on the
inclinations of people towards the movement; or
o the movement might plateau at a wide range of sizes, depending on
how well -perceived it is; or
o building political consensus will be useful for the direct work.

Part 2: Application
In the second half of the paper, I apply the conceptual tools developed in Part 1 to
answer questions about how to find the best work for the young effective altruism
movement and related areas. The conclusions here are not certain, but represent my
informed best judgement. Some of them are driven purely by qualitative
considerations, and some are b ased in part on numerical estimates.
My conclusions are:
 Getting movement growth right is extremely important for effective altruism.
Which ac tivities to pursue should perhaps be governed even more by their
effects on movement growth than by their direct effects.
 Increasing awareness of the movement is important, but increasing positive
inclination is at least comparably important. Therefore we should genera lly:
o prefer advocacy to publicity ;
o strive to take acts which are seen as good by societal standards as well
as for the movement;
o avoid hostility or needless controversy .
 Direct work has a very important ro le to play in movement building . It is
likely to increase positive inclination, by:
o Demonstrating commitment , and showing that the people engaged in
the movement think the work is valuable ;
o Increas ing the credibility of the area by demonstrating that there is
productive and valuable direct wor k that can be done.
 Within global poverty, work focusing on movement growth may be more
effective than direct work for most people (at the margin today) .
 Within areas that seem promising but do not have an established track
record , direct work aimed at demonstrating that there are credible
interventions may be one of the most effective form s of movement building .

Acknowledgements : Thanks to Tom Ash, Daniel Dewey, Seb Farquhar, Michelle Hutchinson ,
Josh Jacobson, Toby Ord, and Kerry Vaughan for helpful co nversations and comments.
Centre for Effective Altruism – Working Paper , 2015
3 0. Introduction
Suppose you have discovered a great way to help the world. Now you face a trad e-
off: how should you balance pursuing this directly against spreading the word and
convincing others to help pursue it?
The basic pro blem is similar for many possible ways of helping the world. Perhaps
you’ve worked out that washing your hands before surgery helps your patients’
survival rate. Perhaps you’ve realised that treating men and women equally will
help welfare levels and produ ctivity. Perhaps you think that giving money to
charities that have been shown to do a great deal with it is an effective way to help
people. In each case you might have more impact by growing the movement of people
who care about the issue than just doing your part of the direct work1, but it may be
challenging to see how to compare these against each other.
In order to make the best decisions, it’s useful to think about how valuable growing
the movement is compared to the direct work. This is very hard to know for sure,
since we never get to see how all the counterfactuals play out. But I think we can
improve our intuitions by understanding some simple models and getting a clearer
picture of the kind of dynamics that may be at play.
Section 1, the majori ty of the paper, aims to give the reader a set of conceptual tools
for thinking about movement growth. We introduce the Awareness/Inclination
model, and use it to explore the possible life -cycles of movements without
intervention, and the counterfactual ef fects of interventions.
In Section 2 we combine these insights with personal impressions of the effective
altruism movement to provide recommendations.
1. Modelling movement growth
In order to understand the counterfactual impact of a marginal interventio n to foster
movement growth, it’s helpful to first have a picture of what we expect to happen
without that intervention. Generally this will be of the form: (i) the movement grows
as more people become aware of it; (ii) as it grows more direct work is done ; (iii) at
some point the movement size plateaus.
I’m going to introduce a simple model of the space in which this is happening. This
will allow us to analyse the effects of different kinds of intervention.

1 In this paper I count earning to give to direct work as direct work. It is of course indirect in
an important way, but in a direct work / movement growth split, it fits with the direct work.
In one sense, the idea of ‘earning to give’ is one example of a way to help the world that a
person might have discovered.
How valuable is movement growth?
4 1.1 The Awareness/Inclination Model (A IM)
People will know a variety of different things about the movement, and have
different attitudes towards it. As a modelling assumption we will compress these
down to two dimensions: awareness (how much they know about the movement) and
inclination (how favourably disposed they are towards the movement, or how
favourably they would be if they knew more).
Then a person (at a fixed point in time) will be represented by a point in this space .
Some examples are shown in Figure 1:

Figure 1: Individuals in the AIM space
Of course over time people may move around in this space. This will usually be
driven by finding out more about the movement. So when movement occurs, we
should expect it to be primarily in the left -to-right direction of the diagram. As t hey
learn more, people may also become a bit more favourable or hostile to the
movement. It seems to me that significant changes in inclination without significant
increases in awareness are likely to be relatively rare during the growth phase . This
is a s ubstantive assumption which could be worth exploring, but the conclusions of
the model do not require it.

Centre for Effective Altruism – Working Paper , 2015
5 Examples of some trajectories individuals could take through this space are shown in
Figure 2.

Figure 2: Example trajectories as individuals become more aware of the movement
1.2 Life -cycle of a movement
Early on in the life of an idea a large majority of the popul ation will be ignorant of it.
We cannot in general say how favourable they w ill be. Movements where the people
on the left are clustered t owards the top and away from the bottom have more
potential for growth, and less potential to lead to a vocal opposition. It is the people
in the top -right of our diagram who are both knowledgeable and favourably inclined
who will actively pursue and promo te the idea, and may be regarded as the
movement built around that idea. This is illustrated in Figure 3:

How valuable is movement growth?
6
Figure 3: Early in the life of a movement
As the movement starts to grow, some people will become more aware of it, moving
along trajectories to the right and perhaps changing inclination slightly (Figure 4).

Figure 4: Growth of a movement
In our example, the growing awareness resulted in a substantially larger movement,
as well as someone moving into the bottom -right corner of the diagram, becoming
knowledgeable about the movement and also hostile to it. Why should this section
matter? Consider what drives the increases in aware ness and changes in inclination.

Centre for Effective Altruism – Working Paper , 2015
7 At the level of the model it seems that more people being knowledgeable and
favourable to the movement will help to increase the spread of awareness, and that
more of the knowledgeable people being favourable rather than hostile will help to
nudge people’s incl ination in a positive direction. So having a larger , more active
opposition to the mo vement may decrease the frequency of positive inclination -shifts
as people learn more (and increase frequency of negative inclination -shifts). It may
also affect how the movement is able to influence public policy: a small highly
favourable group with no o pposition may be better able to shift policy than a larger
group with a vocal opposition.
What is the lo ng-term picture? There are four stable situations that can come about in
the mid- to long -term as the number of people info rmed about the movement grows :
Scenario A – Saturation : There is a certain amount of work that should usefully be
done on the issue at hand (perhaps each year, or perhaps in total) . When the
movement gets large enough, this work gets done, and there is no need to make
further people aware ; the movement is the correct size and more growth might even
be bad . Existing people drift away from the movement as they see less need to be
involved. e.g. fundraising for specific projects.
Scenario B – Collapse : Something happens to cause negative inclination towards the
movement. This c ould be an inevitable consequence of increased awareness , or could
be caused by some particular event. This affects both existing supporters and people
who do not know about it. A subcategory of collapse is stagnation , where there is no
real opposition but people lose their strong positive inclination, and nobody works
to bring new people into the movement. In either case the negative inclinations stops
the movement from growing, an d it eventually collapses. e.g. Marxism

Figure 5: Scenarios where many people never become aware of the movement

How valuable is movement growth?
8 Scenario C – Controversy : In this scenario almost everyone knows a reasonable
amount about the movement. A fraction of society is favourable enough to be
actively involved, but many people are not in support. Most large movements can
expect to be here. e.g. abortion
Scenario D – Acceptance : It becomes socially normal to support the movement.
Almost everyone knows something about it and agrees.2 e.g. ending the use of lead in
consumer products

Figure 6: Scenarios where virtually everyone becomes aware of the movement
1.3 Effects of interventions
The marginal effect of an intervention may be quite different depending on which
long -term scena rio we are in. For this reason it’s important to have a good
understanding of which are the most likely. To help think about the effects of
interventions, we’ll consider the two intervention archetypes illustrated in Figure 7.
Intervention X – Publicity : This intervention focuses on rai sing awareness of the
movement. It is consistent with the view that all publicity is good publicity. e.g.
performing stunts that are widely reported.

2 The Acceptance scenario has a couple of variations: one where there is a positive feedback
loop which increases inclinations towar ds the movement; a second where people’s natural
inclinations are positive enough that shifting right in the diagram is enough, and the
movement is only limited by awareness. In the very long term Controversy could shift to
Acceptance , for example as in th e abolition of slavery.

Centre for Effective Altruism – Working Paper , 2015
9 Intervention Y – Advocacy : This reaches fewer people than Publicity , but is
concerned chiefly with improving their inclination towards the movement. It is
closer to brand management. e.g. writing a persuasive opinion piece for a newspaper.
Both of these interventions affect the population roughly uniformly; in practice many
interventions may target subgroups.

Figure 7: Two intervention archetypes.
1.4 Shifting timelines
The Saturation and Acceptance scenarios are perhaps the simplest to analyse. In both
cases there is a natural eventual size of the movement, so marginal wor k on
movement growth will not change that size, but change when it is realised. This
suggests that the focus should be on increasing awareness rather than improving
inclination : Publicity has a bigger effect than Advocacy on reaching the end state
quickly. There is still a question of how good increasing awareness is compared to
direct work.
The curve of movement size against time before intervention (black curve in Figure
8) thus has a horizontal asymptote. Suppose that we intervene to make the
movement grow by k people. We will assume for now that our intervention just
move s the entire curve forward in time (red intervention and blue curve in Figure 8).

How valuable is movement growth?
10
Figure 8: Interve ntion to accelerate movement growth
What is the total effect of this? The growth of the movement is advanced in time by a
certain amount a (see Figure 9) . This will produce extra direct work on
corresponding to the area between the blue and black curves .3

Figure 9: Assessing the impact of accelerating movement growth
This area can be shown to be equal to a times the difference between current
movement size and eventual movement size (see Figure 9 – so long as k is small the
area of the extra grey triangle is negligible). If we value direct work equally
regardless of when it happens, this gives a simple way of understanding the value of
acceleration: it is only necessary to estimate the current rate of growth and the
eventual movement size. Then:

3 Note that area in this diagram represents person -years in the movement ; we are assuming
that the direct work done is in proportion to this.

Centre for Effective Altruism – Working Paper , 2015
11 Benefit = (eventual movement size – current movement size) \* advancement period
For example suppose the movement currently has 1 0,000 people in, is growing at 10
people per day, and is projected to reach an eventual size of 100,000 people. Then an
intervention which gains 10 extra new members beyond what would have happened
anywa y for the movement accelerates the process by a day, so long as it also advances
the background awarene ss of the movement at the same rate .4 This means the value of this
advancement is the same as the direct work produced by 90,000 people in the
movement for a day – which is the same as one person for 2 50 years. So the value of
just one person joining can b e estimated as having their future direct work for
around 2 5 years.
Some caveats. A person being worth about 25 years seems reasonable, but this is a bit
of a coincidence of the numbers. The value is quite sensitive to eventual movement
size. If the moveme nt will only reach 20,000 people, each person added only equates
to 2. 5 years of direct work. If it will reach 10 million, each person added (while it’s
currently only 1 0,000 people) produces a whopping 2, 500 years of direct work ,
because they pull forward the future hump of the distribution by a macroscopic
amount . Uncertainty about eventual movement size will complicate this, but much
of the expected value may be driven by scenarios where the movement gets very
large , if these are realistic . In practice interventions won’t advance all parts of the
awareness process equally, so just counting new members is likely to overestimate
the value (though probably by less than an order of magnitude). And i f we value
direct work differently at different ti mes (or di fferently according to how much has
been done) , that would change the analysis, most likely in the direction of making
direct work better relative to movement building.
For these scenarios it seems that movement growth can compare very favourably to
direct work , and that Publicity is often better than Advocacy .5 The major exceptions

4 This corresponds to the assumption that we are simply moving the curve forwards. An
exam ple of an intervention which does not satisfy this assumption might be persuading 10
people who already think of themselves as almost members of Giving What We Can to fill in
the membership form. However doing that together with sufficient outreach to refr esh the
pool of such people probably would satisfy the assumption. This means that outreach which
only shifts people along at one part of the curve should be discounted by up to an order of
magnitude compared to the direct measured shift.
5 However, there is a mechanism which could make Advocacy crucial in these situations too:
relative to Publicity it is probably slower at creating movement members, but it is also much
slower at creating opponents of the movement. If having opponents could significantly slow
movement growth, this indirect effect may mean that in the long term Advocacy speeds
getting the movement to full size more than Publicity .
How valuable is movement growth?
12 are when the movement has already done most of its growth, or if the movement is
simply never expected to get large.
1.5 Shifting the long -term state
The Controversy scenario in some ways admits a similar analysis. Treating the
eventual size of the movement as known and fixed, we can estimate how good
accelerating movement growth is in terms of direct work. But an important
difference is that we may be able to affect the eventua l size of the movement. In
Saturation and Acceptance , movement growth can be very important, and it is
generally worthwhile to speed it up as much as possible by whatever means a re
most effective. In Controversy , long -term stability is achieved when everyo ne knows
about the movement but people have different opinions about it. It may be possible
to change this eventual distribution of opinions, and this could have a large effect.
For example suppose we have a movement of 1 0,000 people growing at 10 people
per day, and p rojected to reach a size of one million people. Suppose there is an
opportunity which would bring in 300 new people – accelerating growth by a month
– but by being controversial, reduce the pool of people interested by 1%. At the
current movem ent size , putting off 1% is just 100 people, so attracting an additional
300 seems like a great deal. But if the change persists and affects the stable level, this
would be reduced by 10,000. Projecting twenty years past the plateau of movement
size, the i ntervention will destroy more than twice as much direct work as it creates
in this model.
Since we hypothesise that a major driver of improving inclination towards the
movement is having lots of people with high inclination and few people who are
opposed, in Controversy it may be more important to focus on improving inclination
towards the movement as a first priority, and raising awareness only as a second
priority. Advocacy is probably a preferable intervention to Publicity .

Centre for Effective Altruism – Working Paper , 2015
13
Figure 10: Distribution of level of inclination in a population of 1 billion people
In Figure 10 we suppose we have a metric for inclination towards the movement
such that the popu lation is normally distributed .6 The horizontal bands show
standard deviations ( z-scores) , and the dots and numbers show how many people lie
in each band, from a population of around a billion people.
Changing the threshold required for eventual participation in the movement can
have a large effect size. For example suppose in one scenario the movement only
catches on in those who are 3.8 standard deviations towards favourable or higher,
whereas in another the movement catches on in those who are 3.7 standard
deviations towards f avourable or higher. Shifting from the first to the second
scenario seems like a small change in required attitude, which might be achievable
by shaping the right positive view of the movement. But its effect is to add 35,000
people to the eventual size of the movement, on a base of 72,000 – almost a 50%
increase! So if how positive p eople will eventually be towards the movement can be
adjusted, that could be crucial (see Figure 1 1).

6 If necessary this might be a synthetic metric , in the same way that IQ is .

How valuable is movement growth?
14
Figure 1 1: Possible eventual movement size s – depending on how extremely favourable
people have to be in order to become involved
Interventions which change the likelihoods of different long -term scenarios may also
be crucial . Changing scenarios is most plausible between Collapse and the other
scenarios. This could have a very large total effect. Avoiding movement collapse may
be difficult to do with po sitive interventions, but it becomes extra important to avoid
events which could catalyse the creation of opponents of the movement and drive
movement collapse.
1.6 Changing the movement

In the above, we’ve imagined that changing the framing around the m ovement can
affect the size it grows to, but that the underlying message and the direct work
associated are essentially the same. It’s also possible that by choosing a slightly
different movement to promote, for example by excising less popular ideas, we c ould
change the growth.

This may be a powerful tool. Because the eventual movement size may be so
sensitive to – among other factors – how appealing the ideas are (as in Figure 11) , it
could be valuable to look for opportuni ties to adjust and improve this . It’s also a tool
that should be used very carefully, however – the value of spreading a different
movement may be substantially different, and it could be too easy to throw the baby
out with the bathwater by removing unpopular components.

Centre for Effective Altruism – Working Paper , 2015
15 1.7 Limitatio ns and extensions

The model we have used necessarily makes a number of simplifications. I think these
have generally been worthwhile, in allowing us to go further and faster through the
analysis without distorting the picture too much. Nonetheless, it’s worth being
aware of possible limitations or shortcomings, and it may be that future work could
explore relaxing these assumptions.

1.7.1 Compressed space of attitudes
Restricting to the 2 -dimensional AIM space meant that we could not represent more
comp lex attitudes, such as distinguishing between someone who has strong positive
affect towards the movement but is not currently taking action on this, and someone
who has much more mixed feelings but is currently pursuing some direct work .
Much of the time I think these two dimensions suffice to capture the important
distinctions, but there may be cases where they fail.

1.7.2 No loss of awareness
We assumed that people tend to move in AIM space only when presented with
something external, and cannot move t o the left. In fact people may drift around
without interacting with others, and in particular may forget and so slowly lose
awareness. It seems likely to me that the pictures presented are still roughly correct
on average for growing movements, but furthe r consideration of this point might
show extra benefits to some interventions.

1.7.3 No different behaviour for distinct subgroups
Perhaps it is easier to affect inclination by large amounts in some groups such as
students. The current model doesn’t make any assumptions about this, but doesn’t
make it easy to model. An extension of the model could perhaps provide easier ways
to think about heterogeneity in the population beyond the position in AIM space.

1.7.4 Growth is primarily endogenous
When we look ed at the marginal effect of a larger movement today, we assumed that
this would accelerate the movement growth process fairly uniformly. This is
reasonable if growth of the movement is driven primarily by people in the
movement. If growth is driven signif icantly by some exogenous factor (for example
uptake of a new technology, reaction to political events, or flow from a related
movement) then the long -term effect of growing the movement today could be much
smaller. Changing the dynamics so that the long -term state alters could still be very
important.

How valuable is movement growth?
16 2. Ap plications
2.1 Types of intervention
Our example interventions Publicity and Advocacy were quite a long way removed
from concrete actions. What might they look like in reality? Since the effects are all
about changing minds (via awareness and inclination), interventions will tend to
take the form of communication – although they could also include taking actions
that can usefully be reported.
Example intervention might include:
 Writing a n introduc tory article in a popular media venue. This would
increase awareness across the population. Depending on style, it might also
increase inclination, or be more controversial and sacrifice inclination for
extra awareness.
 Writing a detailed articulation of an idea important of the movement, to
publish on a website associated with the movement. This would increase
awareness and inclination, primarily in a group near the top -centre of AIM
space. The article you are currently reading is an example of an interven tion
which intends to be of this type.
 Direct work on the problem the movement is aimed at. This will not affect
awareness, but improve inclination as people become more aware.
The last example may be surprising: we started out setting movement building
against direct work, and are now using direct work as an example of movement
building! What’s going on here?
The answer is that while direct work has some movement growth effects , it is usually
not a very effective way of movement building, so there is a meaningful choice
between direct work and (effective) movement growth. However if the ratio of direct
work to movement growth activities is too low , this could hinder the growth o f the
movement, either because it gives a natural line of criticism, or because it is harder to
tell a clear story of what the movement is about. In such cases direct work may
become the most effective form of movement growth at the margin – albeit with a
slant towards direct work that can be communicated clearly or sets a good example.
As a final note, when choosing what to do, it’s important to keep the concept of
comparative advantage in mind. Even if you think a bit more movement growth is
the most effe ctive thing to be doing at the margin, if you have much better
opportunities to pursue direct work, that could be better. Think about the portfolio
of work done by those involved in the movement. Direct work should generally be
done by those best at it, an d movement building by those best at that. The
importance of movement building relative to direct work sets the location of the split
Centre for Effective Altruism – Working Paper , 2015
17 between these activities and shows what people with intermediate skill sets should
be working at .
2.2 Effective altruism
The effective altruism movement is concerned with using evidence and reason to do
the most good it can in the world. It is currently quite small (in the thousands of
people , probably the low thousands ) and growing rapidly (for example membership
of Giving What We Can has roughly doubled each year) .
2.2.1 Decision -relevant parameters
A key decision -relevant parameter is how big the movement will get. It seems that it
would be hard to run out of effective things to do, so Saturation is impossible.
Collapse , Controversy , and Acceptance are all possibilities. Movement collapse is
probably unlikely, but it is very important to avoid.
Knowing how big the movement might get would tell us whether we’re likely to end
up in scenario Controversy or Acceptance , and if Controversy , what the scale of the
eventual plateau might be. I think there is a lot of uncertainty around the eventual
size. It could be valuable to do market research to estimate what the uptake rates
could get to in different slices of society, although this is not straightforward.
If, as seems to me most likely, we are heading towards Controversy , it is also crucial
how much we could hope to affect the eventual size of the movement. Again, this is
hard to estimate, but might be done with market research , seeing how much effect
different framings have on people’s willingness to engage.
2.2.2 Implications
The h igher we think the plateau is, the more we should care about movemen t
growth relative to direct work. The more we think we can affect the level of the
plateau , the more we should prefer quality to speed of growth, focusing mo re on
inclination than awareness. And the higher the discount rate we place on direct
work, the more we should care about direct work compared to movement growth,
and the more ab out spreading awareness than improving inclinations.
My impressions are that it is possible the movement could get very large – millions
of people within twenty to fifty years . Until we have ruled this out, it is important to
keep movement growth a priorit y, and to weight improving inclinations more than
increasing awareness in many trade -offs. Focusing on improving inclinations means
that we should be careful with our framing, trying to explain how we help people to
do things they already wanted to do.
Effective altruism has several facets, as people pursue direct work in multiple areas.
Dealing with developing -world poverty has been perhaps the largest focus area of
the movement. I think that at present movement growth is not significantly limited
How valuable is movement growth?
18 by a lac k of direct work in poverty, and this work is not dramatically more useful
now than in a few years . The implication of this is that it may be better at the margin
if we put a few more resources into movement growth rather than direct work in
poverty. Of co urse this statement has several caveats:
 This is by no means a claim that poverty isn’t a hugely important issue,
and one with great opportunities for direct work available . It may well
be that the big success of effective altruism will be to help eliminat e
poverty. The claim is just that extra direct work today may have less
total impact on poverty than extra movement growth today.
 This doesn’t mean that we should collectively stop doing direct work in
poverty altogether . This is importa nt to continue: because it would be
faithless to abandon the work; becau se it helps attract more people to
such work; and because it helps us learn how to do good such work .
 Some individuals may be best increasing their direct work in poverty,
because they have strong com parative advantage there or because they
could be m ore effective advocates for effectively alleviating poverty if
they could talk about their direct work in poverty.
Does the same argument apply to say that effective altruism should reduce its focus
on di rect work altogether to prefer movement building? Not entirely. I think that the
movement’s growth may be limited by credibility, by ability to assess cause areas or
interventions, or by direct work in other areas such as engagement with policy. In
general the direct work that is done will have an effect on the way that people
perceive the movement, so may affect awareness and will certainly affect inclination.
2.3 Very approximate numerical estimates
Suppose the effective a ltruism movement currently has 3 ,000 people doing an
average of $5,000 direct work per person per year (pre sent movement size equivalent
$15M/yr), and is growing by 10 people per day.
Great uncertainty over eventual movement size. Possibilities, excluding collapse:
 10th percentile: $250 M/yr (50,000 people doing $5,000 per person)
 50th percentile: $3B/yr (1M people doing $3,000 per person)
 90th percentile: $50B / yr (50M people doing $1,000 per person)
Mean approx $10B/yr
Adding a person to the movement ~ bringing this f orward by 0.1 days. Adjust down
to 0.025 days to account for the fact that there are lots of steps in getting people
towards movement.
Centre for Effective Altruism – Working Paper , 2015
19 0.025 days \* $10B/yr = $700,000 worth of direct work (excluding movement building
effects from direct work)
Perhaps adjus t this down a bit for future work holdi ng less value than current work,
and down a bit more for people doing otherwise useful work. Say $3 00k for getting a
person into the movement.
That was for spreading awareness. What about improving inclination? (Numbers for
this are if anything even more crude.)
Suppose increasing inclination of everyone who currently knows something
substantial about the movement ( say 10,000 people , which includes many people not
in the movement ) by 1 σ leads to opinions being 0. 1σ higher when movement reaches
plateau, and that that change lasts 20 years.
Then causing a noticeable (0.2σ, corresponds to ~1.5cm height) shift in one person’s
opinion leads in expectation to 2×10-6 σ shift in population opinion. That leads to an
expect ed ~ 200 people at 90th percentile, and around 25 people in expectation total,
causing about $1 .2M extra direct work. This should be adjusted down for later work
being less valuable than early work – perhaps to $600k .
This is a large effect for improving o pinions. It’s probably most important for
improving the opinions of people near the top and bottom of the inclination
spectrum, as those are the ones who are likely to communicate m ost about the
movement; therefore I suggest increasing this estimate for th ose people as well as for
influential people, and decreasing it for those in the middle.
It might be better to model shifts in opinion as decaying with time and when
spreading to new people. Thi s will probably lead to a lower overall estimate of the
value of changing opinions, since the decay will be biggest in scenarios where the
movement gets very big and inclin ation is particularly important (mock -up in Excel
suggests reduces impact by a factor of 4). Say $150k for a noticeable shift in an opinion .
All o f these numbers are basically pulled out of my head and are neither carefully
modelled nor terribly robust. Nevertheless, they are suggestive, and I’d be mildly
surprised if they were wrong by much over an order of magnitude. It’s important to
remember tha t we excluded the movement growth effects of direct work in order to
make the comparison, so direct work will tend to compare more favourably in
practice than these suggest.
Key empirical questions for better numerical estimates:
1. How big is the movement l ikely to get?
2. How many people know about the movement now?
3. How fast is the movement currently growing?
How valuable is movement growth?
20 4. How much do attitude shifts decay with time?
5. How much are attitude shifts transferred across people?
2.4 Practical upshots
Here I’ll try a little more i nterpretation, to discuss what this means when deciding
between actions . These views are informed by the theory but not entailed by it:
 We should be very careful to avoid actions which could cause negative
inclinations towards the movement for little gain. We should therefore avoid
hostility where possible, and should try to avoid expressing opinions on
controversial topics when we may be perceived as affiliated with the
movement.
 When allocating resources to direct work, we should do so in such a way that
we can demonstrate what we’ve done, and we can learn from the experience,
even if there are associated overheads.
 We should communicate our successes clearly. If a significant amount of the
value of direct work today comes from demonstrating credibility an d
persuading others, it’s important to be able to draw on this easily when
relevant. (Note t his is not the same as trumpeting our successes widely .)
 We should probably avoid moralising where possible , or doing anything else
that might accidentally turn people off. The goal should be to present
ourselves as something society obviously regards as good, so we should
generally conform to social norms. Similarly presenting ourselves as on the
side of the person addressed may help to avoid creating negative in clination
shifts.
 We should try to reach good communicators and thought leaders early and
get them onside. This increases the chance that when someone first hears
about us, it is from a source which is positive, high -status, and eloquent.
2.5 Other areas and movements
How does this analysis translate over to other young movements, or other focus
areas of effective altruism? For such linked movements , it generally seems that
collapse and controversy are the most likely outcomes , that many people are relatively
ignorant of it, and that the community of people who might care about it is much
larger than the community that currently does. So for the same reasons as before it
seems that movement growth is important, and that inclination may be more
impor tant than awareness. In cases where many of the eventual routes to impact are
likely to involve policy, focusing on inclination and avoiding making opponents
could be particularly important.
However, I think there is a significant disanalogy between effect ive work in global
poverty and some more speculative areas such as existential risk reduction. Effective
Centre for Effective Altruism – Working Paper , 2015
21 interventions in global poverty are well -demonstrated, so the growth of attention is
not significantly limited by direct work. In contrast, the growth of attention for
existential risk reduction is substantially limited by a lack of credible direct work in
the area. Therefore, in order to foster movement growth, one of the highest value
activities is to do direct work, and to set an agenda for direct wor k, that clearly helps
with the problem. Choosing work that demonstrates credible ways to help , and
explaining the value of this work, is probably more important than choosing the
direct work with the biggest actual effect in the short term.7 There may also be risks
that need to be addressed in the short term, which could increase the discount rate
on direct work , and so make direct work more valuable relative to movement
building .8

7 See Ord: http://www.fhi.ox.ac.uk/the -timing -of-labour -aimed -at-reducing -existential -risk/
8 See http://www.fhi.ox.ac.uk/Allocati ng-risk-mitigation.pdf