The history and future of AI.

Abstract
--------

The standard model for developing AI systems assumes a fixed, known objective that the AI system is required to optimize through its actions. Systems developed within the standard model have been increasingly successful. I briefly summarize the state of the art and its likely evolution over the next decade. Substantial breakthroughs leading to general-purpose AI are much harder to predict, but they will have an enormous impact on the global economy and on human roles therein. At the same time, I expect that the standard model will become increasingly untenable in real-world applications because of the difficulty of specifying objectives completely and correctly. I propose a new model for AI development in which the machine’s uncertainty about the true objective leads to qualitatively new modes of behaviour that are more robust, controllable, and deferential.

© The Author(s) 2021. Published by Oxford University Press. For permissions please e-mail: journals.permissions@oup.comThis article is published and distributed under the terms of the Oxford University Press, Standard Journals Publication Model (<https://academic.oup.com/journals/pages/open_access/funder_policies/chorus/standard_publication_model>)

JEL
[N10 - General, International, or Comparative](/oxrep/search-results?page=1&tax=JEL/N10)
[O31 - Innovation and Invention: Processes and Incentives](/oxrep/search-results?page=1&tax=JEL/O31)
[O32 - Management of Technological Innovation and R&D](/oxrep/search-results?page=1&tax=JEL/O32)
[O47 - Empirical Studies of Economic Growth; Aggregate Productivity; Cross-Country Output Convergence](/oxrep/search-results?page=1&tax=JEL/O47)

Issue Section:
[Articles](/oxrep/search-results?f_TocHeadingTitle=Articles)

 You do not currently have access to this article.

[Download all slides](/DownloadFile/DownloadImage.aspx?image=&PPTtype=SlideSet&ar=6374673&xsltPath=~/UI/app/XSLT&siteId=5230)


Reward is Enough.

Reward is enough
================

Abstract
--------

In this article we hypothesise that intelligence, and its associated abilities, can be understood as subserving the maximisation of reward. Accordingly, reward is enough to drive behaviour that exhibits abilities studied in natural and artificial intelligence, including knowledge, learning, perception, social intelligence, language, generalisation and imitation. This is in contrast to the view that specialised problem formulations are needed for each ability, based on other signals or objectives. Furthermore, we suggest that agents that learn through trial and error experience to maximise reward could learn behaviour that exhibits most if not all of these abilities, and therefore that powerful reinforcement learning agents could constitute a solution to artificial general intelligence.

Keywords
--------

Artificial intelligenceArtificial general intelligenceReinforcement learningRewardRecommended articles

David Silver
Satinder Singh
Doina Precup
and Richard Sutton


Extending rational models of communication from beliefs to actions.

Extending rational models of communication from beliefs to actions
Theodore R. Sumers1, Robert D. Hawkins2, Mark K. Ho1, Thomas L. Grifﬁths1,2
1Department of Computer Science, Princeton University
2Department of Psychology, Princeton University
fsumers, rdhawkins, mho, tomg g@princeton.edu
Abstract
Speakers communicate to inﬂuence their partner’s beliefs and
shape their actions . Belief- and action-based objectives have
been explored independently in recent computational models,
but it has been challenging to explicitly compare or integrate
them. Indeed, we ﬁnd that they are conﬂated in standard refer-
ential communication tasks. To distinguish these accounts, we
introduce a new paradigm called signaling bandits , generaliz-
ing classic Lewis signaling games to a multi-armed bandit set-
ting where all targets in the context have some relative value.
We develop three speaker models: a belief -oriented speaker
with a purely informative objective; an action -oriented speaker
with an instrumental objective; and a combined speaker which
integrates the two by inducing listener beliefs that generally
lead to desirable actions. We then present a series of sim-
ulations demonstrating that grounding production choices in
future listener actions results in relevance effects and ﬂexible
uses of nonliteral language. More broadly, our ﬁndings suggest
that language games based on richer decision problems are a
promising avenue for insight into rational communication.
Keywords : Communication, rational speech acts, multi-armed
bandits, language games
Introduction
“Language is used for doing things.” (Clark, 1996, p. 3)
Butwhat things? Broadly, accounts of communicative goals
have been formulated in terms of listener beliefs andactions :
“Alan is speaking with the aim of getting Barbara to under-
stand him and to act on that understanding” (Clark, 1996, p.
11). But how do these aims relate? Is language primarily a
tool for informing others, shaping their actions, or some com-
bination of the two?
Classical accounts have emphasized beliefs , framing com-
munication as information transfer between speaker and lis-
tener (Grice, 1975). This is reﬂected in the recent Ratio-
nal Speech Acts (RSA) framework, which typically deﬁnes
a speaker’s utility in terms of epistemic objectives like infor-
mativeness (Frank & Goodman, 2012; Kao, Wu, Bergen, &
Goodman, 2014; Yoon, Tessler, Goodman, & Frank, 2020).
A related perspective from the connectionist literature deﬁnes
meaning in terms of effects on the listener’s latent represen-
tations (Elman, 2009). Yet beliefs themselves are an unsat-
isfying objective, as they are imperceptible and have no real-
world consequences. Grice himself suggested informative-
ness should be generalized to “inﬂuencing or directing the
actions of others” (Grice, 1975, p. 47).
Under an alternative action -oriented view, communication
is an extension of an agent’s basic capability to interfacewith the world, allowing a speaker to act indirectly through
others. This has been explored in game-theoretic pragmat-
ics (Qing & Franke, 2015; Benz & van Rooij, 2007; Franke &
J¨ager, 2016) and emergent communication (Lazaridou & Ba-
roni, 2020). Practical natural language interfaces, including
instruction-following (Tellex et al., 2011) and task-oriented
dialogue systems (Chen, Liu, Yin, & Tang, 2017), typically
learn direct mappings between commands and agent actions.
While such imperative language is effective at inducing im-
mediate actions, it cannot offer a full account of communica-
tion. Humans clearly employ more sophisticated strategies to
“program” others (Lupyan & Bergen, 2016).
To reconcile these perspectives, we propose a uniﬁed com-
putational model that integrates both action-based and belief-
based objectives: communication operates by inﬂuencing in-
termediate beliefs, but its objective is to shape downstream
actions in the world. Under this combined model, speak-
ers reason about the decision problem a listener faces, how
utterances may change their latent beliefs, and ﬁnally how
those beliefs give rise to actions. They then choose utter-
ances to induce belief states that maximize value in expec-
tation over possible actions. Varying the scope of the de-
cision problem allows reasoning at different time horizons,
from “nudges” (Thaler & Sunstein, 2009) that encourage spe-
ciﬁc actions (e.g. getting someone to close a window by
saying “It’s chilly in here”) to intervening on norms more
broadly (Tomasello, 2016).
We ﬁrst extend the Rational Speech Acts (RSA) framework
(Frank & Goodman, 2012) to account for action-based ob-
jectives. We then introduce a new task paradigm, signaling
bandits , which combines traditional signaling games (Lewis,
1969) with multi-armed bandits (Sutton & Barto, 2018). In
Simulation 1, we show that standard reference games are a
special case of the signaling bandit framework and conﬂate
speaker objectives. In Simulation 2, we ﬁnd that action-based
speakers provide decision-relevant information, but employ
hyperbolic and false utterances that improve decisions at the
cost of distorting listener beliefs. Finally, Simulation 3 shows
that when the Combined speaker reasons about a larger set
of possible actions, it becomes more truthful and produces
utterances that improve decision-making over the full distri-
bution of contexts the listener could face. These results sug-
gest that integrated reasoning over listener belief states and
actions creates a pressure to transmit appropriately generaliz-
able information.arXiv:2105.11950v1  [cs.CL]  25 May 2021
Action-Grounded Speaker Models
Suppose a friend is about to step into a busy street. We have
a brief moment to say something– but what should we say?
The basic principle of informativeness suggests that we aim
to reduce their uncertainty about the world as much as possi-
ble. Without a notion of relevance (Sperber & Wilson, 1986),
however, it would be equally informative to mention a pass-
ing cloud or an incoming car. Recent accounts have formal-
ized relevance as a question under discussion (QUD) that col-
lapses the utility of the listener’s beliefs to a coarse-grained
state, such as the current state of the road (e.g. Kao et al.,
2014; Hawkins, Stuhlm ¨uller, Degen, & Goodman, 2015). But
our friend may beneﬁt from a more outcome-grounded notion
of relevance, projecting not to epistemic states but to the deci-
sion problem they face (Benz & van Rooij, 2007): intuitively,
regardless of what they know, we want them to cross safely.
A simple way to ground relevance in the decision problem
is to maximize the likelihood that the listener will take a spe-
ciﬁc action, effectively using that action as the QUD. This
model, which we call the action -oriented speaker, would sug-
gest something like “Cross after that car passes!” However,
rather than choosing a premeditated action from our own per-
spective, we could instead give the listener information that
allows them to act optimally from their perspective. This
combined speaker aims to maximize the full expected util-
ity of the listener’s actions under the induced beliefs. Such a
speaker might say “Look both ways!” which encourages our
friend to check for cars outside of our own ﬁeld of view and
(incidentally) generalizes across intersections.
Belief-oriented speaker . We formalize these models by ﬁrst
introducing the Rational Speech Acts framework (Frank &
Goodman, 2012), which instantiates Gricean maxims (Grice,
1975) as recursive social inference. In this framework, speak-
ers have knowledge of the world state wand choose between
utterances uproportionally to their utility U(u;w), where bS
is a soft-max parameter controlling speaker optimality:
PS(ujw)µexpfbSU(u;w)g (1)
Typically, this utility is deﬁned in terms of the listener’s be-
liefs, their information gain about the state w:
UBelief(ujw) =logPL(wju) (2)
This utility requires the speaker to reason about the listener’s
expected beliefs after hearing the utterance:
PL(wju)µ d JuK(w)P(w) (3)
where dJuK(w)represents the meaning of u, evaluating to one
when utterance uis true of wand zero otherwise. This for-
mulation optimizes for accurate listener beliefs, but lacks a
notion of relevance.
Action-oriented speaker . Our second model extends the ba-
sic RSA framework to reason about actions a listener could
take in the environment. Rather than adding an additional
objective to the epistemic utility, we ground this objectivein a decision problem. Speciﬁcally, we re-formulate the lis-
tener as a reinforcement learning (RL) agent (Sutton & Barto,
2018) with a set of possible actions that may be taken, A. At
each point in time, a subset of those actions are available to
the listener, which we call an action context AA. We as-
sume the listener will choose actions to maximize a reward
function, where the scalar reward value associated with each
action is deﬁned by the world state: R:AW!R. We thus
deﬁne the listener’s policy pLin an action context Ato be
pL(aju;A)µexpfbLRL(a;u)g (4)
where bLis the softmax optimality and RLis the listener’s
expected reward for action a2Aafter hearing utterance u. We
deﬁne expected reward with respect to their posterior beliefs
about the likely state of the world:
RL(a;u) =å
w2WR(a;w)PL(wju) (5)
In this work, we assume speakers are cooperative and have
access to a ground-truth world state w. Action-based ratio-
nal speakers reason about how their utterances will affect lis-
tener actions (Eqs. 3, 4, 5) and communicate to maximize the
listener’s reward. We ﬁrst describe a “pure” action -oriented
speaker, which chooses the highest-reward action a2Aand
optimizes the probability of the listener taking that action:
a,argmax
a2AR(a;w) (6)
UAction(ujA;a) =log[pL(aju;A)] (7)
Critically, rather than aiming for high-performing beliefs in
general, this speaker only considers the listener’s beliefs inso-
far as they are relevant for producing the pre-selected action.
Intuitively, this can be thought of as imperative language.
Combined speaker . Our ﬁnal speaker uniﬁes belief- and
action-oriented objectives by optimizing over both, inducing
beliefs which are likely to maximize rewards in general :
UCombined (ujA;w) =å
a2ApL(aju;A)R(a;w) (8)
Effectively, this utility shifts the locus of decision-making to
the listener: the combined speaker treats them as an indepen-
dent agent and optimizes their beliefs, rather than choosing an
action for them.1We now introduce an experimental setting
to explore these speaker models.
Signaling Bandits
We deﬁne a new language game which enables study of the
speaker models described above. In this setting, speakers can-
not directly signal a unique correct action because all actions
have some relative value. They must instead supply partial
1Note our two action-oriented speakers can be recovered from
a more general utility introducing an additional soft-max over the
reward term R(a;w), yielding the Action utility with b!¥and the
Combined utility with b=1.
information to guide decision making. We ﬁrst review the
structure of Lewis signaling games and note their limitations.
We then describe multi-armed bandits, a setting studied in re-
inforcement learning. Finally, we combine the two to produce
a new game, signaling bandits .
Lewis Signaling Games
Lewis signaling games are two-player collaborative settings
with a speaker and a listener (Lewis, 1969). Following the
notation introduced previously, a signaling game is deﬁned
by a world state w, action context available to the listener,
A, and utterances available to the speaker, U. There is one
action a2Awith a positive reward; other actions have zero
reward. The world state wimplies the correct action a. The
speaker knows wbut the listener does not. During gameplay,
the speaker chooses an utterance u2Uand sends it to the
listener. The listener updates their beliefs, PL(wju), and uses
the posterior to choose an action, pL(aju;A).
Signaling games formalize the coordination problem un-
derlying communication (Krahmer & Van Deemter, 2012;
Frank & Goodman, 2012). However, the interplay of be-
liefs, actions, and rewards is highly constrained. The state
of the world wis synonymous with the correct action a, and
players are indifferent over other actions. It is thus impossi-
ble to discriminate the three speaker objectives deﬁned above
(Eq. 2, 7, 8).2For a richer decision-making setting, we turn
to multi-armed bandits.
Multi-Armed Bandits
A multi-armed bandit is a single-player sequential game. In
each round, the player takes an action and receives a scalar
reward (Sutton & Barto, 2018). Players seek to maximize
their rewards, but are initially ignorant of the reward struc-
ture. Over multiple rounds, they must balance exploration
(choosing a new action to learn its reward) with exploitation
(choosing the most valuable known action). Because payoffs
are scalar, decisions are more nuanced than Lewis games.
Contextual bandits extend this to study learning via ab-
stract information. Actions are now characterized by features,
and rewards are deﬁned with respect to these features. For-
mally, a feature function fdescribes actions: f:A!RK.
Rewards are then deﬁned as a function of these features:
R:f(a)!R. Thus, rather than learn about the reward of
a speciﬁc action, players can learn about the reward of a fea-
turewhich applies to many actions. For example, an animal
might learn that ripe yellow bananas are high-reward, while
rotten brown bananas are low-reward. Associating the payoff
with the color (a feature) rather than the banana (a speciﬁc
action) allows knowledge to transfer to new settings (the next
banana). Contextual bandits have been studied extensively
in reinforcement learning and to a lesser degree for emer-
gent communication (Donaldson, Lachmann, & Bergstrom,
2007). Yet to our knowledge, they have not been used to
2But see Qing and Franke (2015) for evidence in favor of action-
oriented speakers.
1
0
-120-2
3 1 -1
2 0 -2
1 -1 -3
“the blue feature
 is worth -2 points”
“the circle feature
 is worth 2 points”
A
B utterance
 contextstate
worldFigure 1: Signaling bandits combine Lewis signaling games
with multi-armed bandits. (A) The world wis deﬁned by cor-
respondences between features and rewards (table margins),
which combine additively to create possible actions A(table
contents). (B) Two contexts Awith example utterances. The
ﬁrst utterance is true and useful (encouraging the listener to
avoid the blue objects). The second is false (circles are worth
1). It is locally useful (encouraging the listener to choose the
circle) but will lead the listener astray in other contexts.
study communication with an existing language. In the next
section, we introduce a two-player version of this game.
Signaling Bandits
We combine the communication of Lewis games with the re-
ward structure of contextual bandits to create a new game,
signaling bandits . Unlike Lewis games, speakers no longer
communicate concrete information (which action is correct).
Instead, they communicate abstract information (how much
features are worth). We now describe basic gameplay.
As in Lewis games, signaling bandits are two-player games
with a speaker and a listener. Each game is deﬁned by a world
state w, a set of all possible actions Aand a set of speaker ut-
terances U. In each round, the listener faces an action context
AA. However, unlike Lewis games, there is no single “cor-
rect” action. Instead, as in contextual bandits, each action has
a scalar reward deﬁned by the world state w. Here, we assume
features are indicator variables over actions:
f:A!f0;1gK(9)
Rewards Rare linear over these features, parameterized by w:
R(a;w) =w>f(a): (10)
Concretely, this means the world state wis a vector encoding
the reward associated with each feature. A listener with full
world knowledge (every element of w) can calculate exact
rewards using Eq. 10 and thus select the optimal action in
any context. Fig. 1A depicts this visually: wdeﬁnes the value
of individual shapes and colors (table margins), which in turn
determines the reward for each possible action in A(table
contents). The value of an individual feature (one element of
w) thus constitutes partial knowledge about the world.
The speaker helps the listener by providing such partial
knowledge. Formally, Uis a set of tuples of the form h1K;Ri
which specify a given feature and scalar value. As shown
in Fig. 1, these are messages like hBlue, -2iorhCircle, 2i.
Speakers choose utterances and send them to the listener.
The (literal) listener updates their beliefs by setting the corre-
sponding feature to the transmitted scalar value:
pL(wLju) =pL(wLju1K=uR) (11)
and then chooses an action from the available set Aaccording
to their posterior belief over rewards (Eqs. 4 and 5).
Transmitting partial knowledge and constructing different
contexts AAintroduces several important dynamics. First,
it induces relevance effects, since different knowledge will be
useful in different contexts. For example, in Fig. 1B, hGreen,
2iwould improve decision making in both contexts while
hBlue, -2iwould only be relevant for the top one. Second, it
accommodates nonliteral language naturally, as false beliefs
can yield good decisions. In the bottom context of Fig. 1B,
a false messagehCircle, 2imaximizes the probability of the
listener choosing the optimal action. Finally, it allows us to
explore generalization : whether a listener’s beliefs facilitate
good decision making over other action contexts constructed
from the same world. A bias towards communicating gen-
eralized information is implicated in cultural learning (Csibra
& Gergely, 2009; Tessler & Goodman, 2019); thus, modeling
the dynamics of speaker objectives and resulting generaliza-
tion performance is of signiﬁcant theoretical interest.
Signaling bandits creates clean distinctions between lis-
tener beliefs PL(wju), the optimal action a, and the value of
individual actions, R(a;w). This allows for meaningful differ-
ences between speaker models (Eqs. 2, 7, 8). In the following
section, we use simulations to illustrate this. We return to ex-
tensions beyond basic gameplay in the General Discussion.
Simulations
We perform three simulations within the signaling bandits
framework. We ﬁrst describe the general procedure and met-
rics used to measure speaker behaviors. For all simulations,
we set bL=3;bBelief =3;bAction =3, and bCombined =2.
Speaker optimality does not affect the qualitative results; we
analyze optimal speakers ( b!¥) at the end of this sec-
tion. We assume listeners have uniform priors over feature
rewards, and allow speakers to send only a single message.
100
-100100 -100
200 0
0 -200
“blue is 100 points”
“green is 100 points”
“square is 100 points”
“circle is 100 points”utterancesstate
world
context
Figure 2: Simulation 1 constructs a traditional Lewis signal-
ing game within the signaling bandits framework. Note that
two of the possible utterances are false.
Speaker PS(truthful) pL(a)RS(A)RS(A)
Belief 1.00 .500 100 -
Action 1.00 .500 100 -
Combined 1.00 .500 100 -
Table 1: Simulation 1 results. Reference games align all three
speakers’ objectives and so cannot disambiguate them. Gen-
eralization, RS(A), does not apply to this setting.
Procedure . For each simulation, we deﬁne a world state w
and set of allowable utterances U. For an action context A,
we ﬁrst compute each speaker’s distribution over utterances
u2Uas deﬁned by Eqs. 2, 7, and 8. We then compute the
listener’s resulting distribution over actions (Eq. 4).
Evaluation Metrics. We use four metrics to summarize
speaker behavior. First, their probability of choosing a true
utterance, P(truthful). Second, the probability of the listener
choosing the optimal action in A(Eq. 4), which we write as
pL(a)for brevity. Third, the expected reward on the action
context A(Eq. 8), which we write as RS(A). Finally, we want
to know whether speakers are over-optimizing for a particu-
lar context. To evaluate this, we calculate the expected gen-
eralization , which indicates whether the listener’s resulting
belief state yields good performance on other contexts drawn
fromA. Formally, we compute the expected reward of the
speaker’s utterance uacross all possible contexts:
RS(u;w;A) =å
A2[A]3RS(u;w;A)P(A) (12)
where P(A)is the probability of an action context A; here,
we assume a uniform distribution over all contexts of size 3.
Again, we shorten this to RS(A)for clarity. If local perfor-
mance substantially exceeds generalization, RS(A)RS(A),
we say the speaker generalizes poorly: it is optimizing lo-
cal decision-making by providing false or less broadly useful
information.
Simulation 1: Reproducing reference games
Our ﬁrst simulation constructs a Lewis signaling game as a
special case of our more general class of signaling bandits
(Fig. 2). We show that this case cannot distinguish between
our models, motivating Simulations 2 and 3.
Figure 3: Simulation 2 results. Shading indicates probability of speaker choosing that utterance, averaged over all 84 3-action
contexts. X’s indicate true utterances. Left: Belief speakers choose true utterances at random. Center/Right: Action and
Combined speakers focus on decision-relevant features and exaggerate to improve listener decisions.
Speaker PS(truthful) pL(a)RS(A)RS(A)
Belief 1.00 .499 .539 .539
Action .330 .772 1.18 .486
Combined .360 .742 1.28 .522
Table 2: Simulation 2 results. Action and Combined fre-
quently send false messages. They obtain high performance
on the local context, RS(A), but generalize poorly, RS(A).
Setup. To construct a Lewis game with a single target, we
deﬁne wand construct a context Acontaining one action with
arbitrarily high reward (the target) and two with zero reward
(the distractors). We restrict utterances Uto positive mes-
sages (corresponding to possible referential labels).
Results and Discussion. Results are summarized in Table 1.
All three speaker objectives are aligned, and so we ﬁnd that
they behave identically: they choose between the two literally
true messageshGreen, 100iorhCircle, 100i. No model has
any reason to prefer a false utterance, or to prefer one true
utterance over the other.
Simulation 2: Divergent speaker behaviors
We next explore how different speaker models may diverge
for other tasks in our signaling bandit paradigm (Fig. 1).
Setup. We consider the world state depicted in Fig. 1. We
setUto all feature-value tuples and evaluate each speaker’s
behavior across all9
3
=84 possible contexts of 3 actions.
Results and Discussion. We plot each speaker’s probabil-
ity over individual utterances in Fig. 3, and summarize the
results in Table 2. All metrics are averaged across the 84 con-
texts. First, we observe that Belief speakers choose a true
utterance at random, regardless of the action context. This
yields relatively poor performance locally but perfect gener-
alization ( RS(A) =RS(A) = :539). In contrast, both Action
and Combined speakers lie frequently ( PS(truthful) < :5). Ac-
tion speakers tailor their utterances to induce the single opti-
mal action in each set. As a result, they exaggerate whicheverSpeaker PS(truthful) pL(a)RS(A)RS(A)
Belief 1.00 .499 - .539
Action .440 .566 - .748
Combined .534 .627 - .949
Table 3: Simulation 3 results. When communicating about a
larger action context, Action and Combined speakers become
more truthful and generalization improves dramatically.
feature values align with the best action in the immediate con-
text. This strategy succeeds locally: they induce the most-
optimal action a majority of the time ( pL(a) = :772) and
obtain more than twice the reward obtained by the Belief
speaker ( RS(A) =1:18). However, the resulting distortion in
listener beliefs means they generalize poorly ( RS(A) =:486).
Finally, Combined speakers achieve a middle ground. They
obtain the best outcome less frequently than Action speak-
ers (pL(a) =:742), but higher reward locally ( RS(A) =1:28)
and generally ( RS(A) = :522). Sensitivity to the rewards of
all three actions leads them to distort beliefs less than Ac-
tion speakers. We visualize the divergence between Action
and Combined speakers in Fig. 4 in a single action context to
better understand these differences.
Simulation 3: Expanding speaker context
Simulation 2 showed that Action and Combined speakers can
be myopic: they produce messages to induce locally-optimal
actions at the cost of generalization. Simulation 3 explores
how this changes when they optimize over the entire action
space A. We ﬁnd that both Action and Combined speakers
become more truthful and generalize better.
Setup. We use the same world as Simulation 2 (shown in
Fig. 1). We ﬁrst construct a single “global” action context of
all 9 actions: A=A. We compute each speaker’s distribution
over utterances for this 9-action context, then evaluate gener-
alization over 3-action contexts, RS(A). Because there is no
“local” context, we do not compute RS(A).
0
-3
context
1Figure 4: Comparing Action and Combined in one action
context (Simulation 2). Shading indicates difference in ut-
terance probabilities between speakers. The Action speaker
maximizes the listener’s probability of choosing the red cir-
cle:pL(a) = :737;RS(A) = :456. The Combined speaker
achieves higher reward by avoiding the blue square: pL(a) =
:627;RS(A) = :482. Neither sends messages about green,
demonstrating relevance effects.
Results and Discussion. Results are summarized in Table 3.
The Belief speaker is unchanged. Both action-oriented speak-
ers are more truthful and generalize better than Simulation 2.
However, the Action speaker ﬁxates on the single best option
(the green circle). It sends false messages which exagger-
ate its value, e.g.hCircle, 2i, or discourage alternatives, e.g.
hRed, -2i. As a result, it fares poorly whenever a green circle
is not present. This illustrates the brittleness of optimizing to
obtain a speciﬁc action. Because the Combined speaker opti-
mizes in expectation over all actions, it is more likely to send
true messages about extreme values, e.g. hGreen, 2iorhBlue,
-2i. It obtains both higher rewards RS(A)and the optimal
action pL(a)more frequently.
Effects of speaker optimality
While we ﬁxed the speaker optimality parameter bSthrough-
out our simulations, it may interact in important ways with
our model comparison. First, our Belief speaker is insensi-
tive to b: since we assumed that the listener’s prior over w
is uniform, all true utterances are equally valuable. By con-
trast, the Action and Combined speakers are sensitive to b
in different ways, since they take the soft-max over differ-
ent quantities (Action over log-probabilities and Combined
over expected utilities). At any given b, Combined domi-
nated Action on all metrics, so we tuned bto equate them
(bAction =3,bCombined =2). At the same time, it is informa-
tive to explore their asymptotic behavior as b!¥, which is
summarized in Table 4. In Simulation 2, we ﬁnd that Action
and Combined still send false messages and generalize poorly
in the limit. Yet when multiple messages are equally likely toContext Speaker PS(truthful) pL(a)RS(A)RS(A)
Both Belief 1.00 .499 .539 .539
Local Action .458 .942 1.63 .690
Local Combined .488 .942 1.67 .710
Global Action .500 .624 - .958
Global Combined 1.00 .731 - 1.28
Table 4: Speaker behavior as b!¥.Local context corre-
sponds to Simulation 2, and Global context to Simulation 3.
induce an optimal action, the Combined speaker converges
to preferring truthful ones while the Action speaker is indif-
ferent. This difference results in dramatically better perfor-
mance in Simulation 3: the Combined speaker converges on
producing the utterance hGreen, 2i, while the optimal Action
speaker remains ambivalent between hGreen, 2iandhCircle,
2i. In sum, the Combined speaker’s sensitivity to underlying
reward structure leads it to consistently produce more truthful
and generalizable utterances across contexts and in the limit
of optimality.
General Discussion
Humans communicate to inﬂuence one anothers’ beliefs and
actions. Here, we explored different ways speakers can
reason about these objectives. We introduced two action-
oriented speaker models which optimize for a downstream
decision problem , grounding relevance in the listener’s actual
decision context (Roberts, 2012). Critically, we proposed that
rational “Combined” speakers should consider both beliefs
and actions: they should communicate to induce belief states
that are likely to produce high-value actions. To distinguish
speaker models, we introduced a new communication game,
signaling bandits. Signaling bandits generalizes Lewis sig-
naling games to multi-armed bandits, formalizing communi-
cation in richer decision settings. Simulations show that the
Combined speaker prefers generalizable information that is
likely to produce high-value actions across a distribution of
possible future contexts. This ﬁnding raises intriguing con-
nections to belief-oriented accounts of generics (e.g.“Birds
ﬂy”; Tessler & Goodman, 2019) as well as biases towards
generalizable examples in non-linguistic pedagogy (Csibra &
Gergely, 2009; Tomasello, 2016). We are thus optimistic that
such speaker models may provide a bridge from communica-
tive principles to social learning more broadly.
This work represents a small step towards a deeper explo-
ration of action-grounded models of rational communication.
First, human experiments are needed to validate our simu-
lations. Second, we considered only literal listeners in col-
laborative settings. Pragmatic listeners may reason about a
speaker’s objectives and knowledge, as well as the action con-
text the speaker considered (Goodman & Stuhlm ¨uller, 2013).
Finally, we explored only single-round gameplay; iterated
games would allow for richer interactions. Speakers could
observe listener actions and infer their beliefs via inverse re-
inforcement learning (Ng & Russell, 2000). A single message
followed by learner actions in multiple contexts would force
speakers to optimize for a distribution over contexts. This
would make generalization an objective rather than an inci-
dental effect, as in optimal reward design (Singh, Lewis, &
Barto, 2009). Listeners could learn both socially (via speaker
messages) and individually (via their own actions). We hope
we have successfully signaled the high value of research in
this paradigm!
Acknowledgements
We thank our anonymous reviewers for their thoughtful feed-
back. This work was supported by NSF grant #1545126 and
John Templeton Foundation grant #61454 to TLG and NSF
grant #1911835 to RDH.
Code for simulations available at:
https://github.com/tsumers/
signaling-bandits
References
Benz, A., & van Rooij, R. (2007). Optimal assertions, and
what they implicate. A uniform game theoretic approach.
Topoi ,26(1), 63–78.
Chen, H., Liu, X., Yin, D., & Tang, J. (2017). A survey
on dialogue systems: Recent advances and new frontiers.
ACM SIGKDD Explorations Newsletter ,19(2), 25–35.
Clark, H. H. (1996). Using Language . Cambridge University
Press.
Csibra, G., & Gergely, G. (2009). Natural pedagogy. Trends
in Cognitive Sciences ,13(4), 148–153.
Donaldson, M. C., Lachmann, M., & Bergstrom, C. T.
(2007). The evolution of functionally referential mean-
ing in a structured world. Journal of Theoretical Biology ,
246(2), 225–233.
Elman, J. L. (2009). On the meaning of words and dinosaur
bones: Lexical knowledge without a lexicon. Cognitive
Science ,33(4), 547-582.
Frank, M. C., & Goodman, N. D. (2012). Predicting prag-
matic reasoning in language games. Science ,336(6084),
998–998.
Franke, M., & J ¨ager, G. (2016). Probabilistic pragmatics,
or why Bayes’ rule is probably important for pragmatics.
Zeitschrift f ¨ur Sprachwissenschaft ,35(1), 3 - 44.
Goodman, N. D., & Stuhlm ¨uller, A. (2013). Knowledge
and implicature: Modeling language understanding as so-
cial cognition. Topics in Cognitive Science ,5(1), 173–184.
Grice, H. P. (1975). Logic and conversation. In Syntax and
Semantics: Vol. 3: Speech Acts (p. 41-58). New York:
Academic Press.
Hawkins, R. D., Stuhlm ¨uller, A., Degen, J., & Goodman,
N. D. (2015). Why do you ask? Good questions provoke
informative answers. In Proceedings of the 39th Annual
Conference of the Cognitive Science Society.Kao, J. T., Wu, J. Y ., Bergen, L., & Goodman, N. D. (2014).
Nonliteral understanding of number words. Proceedings of
the National Academy of Sciences ,111(33), 12002–12007.
Krahmer, E., & Van Deemter, K. (2012). Computational gen-
eration of referring expressions: A survey. Computational
Linguistics ,38(1), 173–218.
Lazaridou, A., & Baroni, M. (2020). Emergent multi-agent
communication in the deep learning era. arXiv preprint
arXiv:2006.02419 .
Lewis, D. (1969). Convention: A Philosophical Study . John
Wiley & Sons.
Lupyan, G., & Bergen, B. (2016). How language programs
the mind. Topics in Cognitive Science ,8(2), 408-424.
Ng, A. Y ., & Russell, S. J. (2000). Algorithms for inverse
reinforcement learning. In Proceedings of the 17th Inter-
national Conference on Machine Learning.
Qing, C., & Franke, M. (2015). Variations on a Bayesian
theme: Comparing Bayesian models of referential reason-
ing. In Bayesian Natural Language Semantics and Prag-
matics (pp. 201–220). Springer.
Roberts, C. (2012). Information structure: Towards an inte-
grated formal theory of pragmatics. Semantics and Prag-
matics ,5, 6–1.
Singh, S., Lewis, R. L., & Barto, A. G. (2009). Where do
rewards come from? In Proceedings of the 31st Annual
Conference of the Cognitive Science Society.
Sperber, D., & Wilson, D. (1986). Relevance: Communi-
cation and cognition (V ol. 142). Harvard University Press
Cambridge, MA.
Sutton, R. S., & Barto, A. G. (2018). Reinforcement Learn-
ing: An Introduction . MIT press.
Tellex, S., Kollar, T., Dickerson, S., Walter, M. R., Banerjee,
A. G., Teller, S., & Roy, N. (2011). Understanding natural
language commands for robotic navigation and mobile ma-
nipulation. In Proceedings of the 25th AAAI Conference on
Artiﬁcial Intelligence.
Tessler, M. H., & Goodman, N. D. (2019). The language of
generalization. Psychological Review ,126(3), 395.
Thaler, R. H., & Sunstein, C. R. (2009). Nudge: Improving
Decisions about Health, Wealth, and Happiness . Penguin.
Tomasello, M. (2016). Cultural learning redux. Child Devel-
opment ,87(3), 643–653.
Yoon, E. J., Tessler, M. H., Goodman, N. D., & Frank,
M. C. (2020). Polite speech emerges from competing so-
cial goals. Open Mind ,4, 71–87.

Theodore R
Sumers
Robert D
Hawkins
Mark K
Ho
Thomas L
Griffiths


Approximate Causal Abstractions.

Approximate Causal Abstraction
Sander Beckers
Dept. of Philosophy and Religious Studies
Utrecht University
srekcebrednas@gmail.comFrederick Eberhardt
Humanities and Social Sciences
California Instituite of Technology
fde@caltech.eduJoseph Y. Halpern
Dept. of Computer Science
Cornell University
halpern@cs.cornell.edu
Abstract
Scientiﬁc models describe natural phenomena
at different levels of abstraction. Abstract de-
scriptions can provide the basis for interven-
tions on the system and explanation of ob-
served phenomena at a level of granularity that
is coarser than the most fundamental account
of the system. Beckers and Halpern (2019),
building on work of Rubenstein et al. (2017),
developed an account of abstraction for causal
models that is exact. Here we extend this
account to the more realistic case where an
abstract causal model offers only an approx-
imation of the underlying system. We show
how the resulting account handles the discrep-
ancy that can arise between low- and high-
level causal models of the same system, and
in the process provide an account of how one
causal model approximates another, a topic of
independent interest. Finally, we extend the
account of approximate abstractions to prob-
abilistic causal models, indicating how and
where uncertainty can enter into an approxi-
mate abstraction.
1 INTRODUCTION
Scientiﬁc models aim to provide a description of real-
ity that offers both an explanation of observed phenom-
ena and a basis for intervening on and manipulating the
system to bring about desired outcomes. Both of these
aims lead to a consideration of models that represent the
causal relations governing the system. They also imply
the need for scientiﬁc models that describe the system
at a granularity or level of description appropriate for
the user and suitable for interventions that are feasible.
Such more abstract causal models do not capture all the
Figure 1: Climate example adapted from Chalupka et
al. (2016), in which a high-level causal model for the
phenomenon of El Ni ˜no is constructed from low-level
(high-dimensional) wind ~Wand sea surface temperature
~Tmeasurements. ~Uis an unmeasured confounder and
is the mapping between the models. See text for details.
detailed interactions that occur at the most fundamental
level of the system, nor do they, in general, represent out-
comes of the system completely accurately at the abstract
level. Nevertheless, such abstract causal models can (at
least) approximately explain the phenomena, and can be
informative about how the system will respond to inter-
ventions that are speciﬁed only at the abstract level.
This paper provides a formal account of such approxi-
mate abstractions for causal models that builds on the
deﬁnition of an abstraction provided by Beckers and
Halpern (2019) (see Section 2), which in turn built on
the work of Rubenstein et al. (2017). That notion of ab-
straction implicitly assumed an underlying causal system
that permitted an exact description of the system at the
abstract level. Here we weaken that assumption to han-
dle what we take to be the more realistic case, namely,
that abstract causal models will capture the underlying
system in only an approximate way.
As a simpliﬁed working example to illustrate our points
we use the case of the wind and sea surface temper-
ature patterns over the equatorial Paciﬁc that give rise
to the high-level climate phenomena of El Ni ˜no and La
Ni˜na, as described by Chalupka et al. (2016). They con-
sidered the question of how the El Ni ˜no climate phe-
nomenon related to the underlying wind and sea surface
temperature patterns that constitute it (see Fig. 1). At the
low level they considered two high-dimensional vector-
valued variables representing the wind speeds ~Wand
the sea surface temperatures ~T, respectively, on a grid
of geographical locations in the equatorial Paciﬁc. They
assumed (with some justiﬁcation from climate science)
that wind speed ~Wis a cause of sea surface temperature
~T, that is,~T=fL(~W;~U)for some high-dimensional
functionfL(:)and exogenous causes ~U. They allowed
the possibility that ~Umay be a confounder of ~Wand
~T, so that there might be an additional causal relation
~W=gL(~U). Leaving details about feedback and tem-
poral delay aside, they were interested in whether the
same system could be described at a higher level, using
a low-dimensional structural equation E=fH(C;~U),
where there is a surjective mapping fromR(~T;~W),
the set of possible values of ~Tand~W, toR(E;C). In
the language of this paper, they were searching for an
abstract causal description of the system. They required
that the high-level model retain a causal interpretation,
in the sense that if one intervened on C, there would still
be a well-deﬁned causal effect on E, no matter how the
intervention on Cwas interpreted as an intervention on
the underlying set of variables ~W.
Chalupka et al. (2016) were able to learn such a high-
level model, and one of the states of E(the high-level
description of the sea surface temperatures) indeed cor-
responded to what would commonly be described as an
El Ni ˜no occurring, conventionally deﬁned by an average
temperature deviation in a rectangular region of the Pa-
ciﬁc. However, the high-level description was not per-
fect: it provided an informative causal description of the
underlying systems and allowed for predictions that ap-
proximated the actual outcomes. Here we make precise
the nature of such an approximation between a high-level
and low-level causal model of the same system. In the
process, we deﬁne what it means for one causal model to
approximate another.
Although our running example is a vastly simpliﬁed cli-
mate model, the challenge of approximately modeling
phenomena at a more abstract level is part of almost ev-
eryscientiﬁc model. For example, it was Robert Boyle’s
great insight that, despite its inaccuracies for real gases in
practice, the ideal gas law still provides an approximate
abstract description of the behavior of the molecules of a
gas in a container that is extraordinarily useful for un-
derstanding and manipulating real systems. Approxi-mate abstractions can take a variety of forms in scientiﬁc
practice, ranging from idealizations and discretizations
to other forms of simpliﬁcation and dimension reduction
(as in the climate example). Our account captures these
in a uniﬁed formal framework.
The main contribution of this paper is to present a frame-
work that offers a foundation for analyzing abstraction
and approximation in causal models. We provide what
we believe are sensible deﬁnitions of approximation and
approximate abstraction, and a conceptual discussion of
these notions. In addition, we provide some technical re-
sults regarding the difﬁculty of determining whether an
approximate abstraction can be viewed as the composi-
tion of an approximation and an exact abstraction.
2 PRELIMINARIES
Since we are interested in scientiﬁc models that support
explanations of phenomena and can inform interventions
on a system, we start by deﬁning a deterministic causal
model with a set of possible interventions. We use ex-
ogenous and endogenous variables to distinguish those
inﬂuences that are external to the system and those that
are internal. The deﬁnitions follow the framework devel-
oped by Halpern (2016).
Deﬁnition 2.1 : A signatureSis a tuple (U;V;R), where
Uis a set of exogenous variables,Vis a set of endoge-
nous variables, andR, a function that associates with ev-
ery variable Y2U[V a nonempty setR(Y)of possible
values forY(i.e., the set of values over which Yranges ).
If~X= (X1;:::;Xn),R(~X)denotes the crossproduct
R(X1)R (Xn).
For simplicity in this paper, we assume that signatures
are ﬁnite, that is,UandVare ﬁnite, and the range of
each variable Y2U[V is ﬁnite.
Deﬁnition 2.2 : Abasic causal model Mis a pair (S;F),
whereSis a signature andFdeﬁnes a function that as-
sociates with each endogenous variable Xastructural
equationFXgiving the value of Xin terms of the values
of other endogenous and exogenous variables. Formally,
the equation FXmapsR(U[VfXg)toR(X), so
FXdetermines the value of X, given the values of all the
other variables inU[V .
Note that there are no functions associated with exoge-
nous variables, since their values are determined outside
the model. We call a setting ~ uof values of exogenous
variables a context .1
1We remark that the notion of context used here, which goes
back to (Halpern and Pearl 2005), is similar to that of Boutilier
The value of Xmay not depend on the values of all other
variables.Ydepends onXin context~ uif there is some
setting of the endogenous variables other than XandY
such that if the exogenous variables have value ~ u, then
varying the value of Xin that context results in a varia-
tion in the value of Y; that is, there is a setting ~ zof the
endogenous variables other than XandYand valuesx
andx0ofXsuch thatFY(x;~ z;~ u )6=FY(x0;~ z;~ u).
In this paper, we restrict attention to recursive (or
acyclic ) models, that is, models where there is a partial
orderon variables such that if Ydepends onX, then
XY.2In a recursive model, given a context ~ u, the
values of all the remaining variables are determined (we
can just solve for the value of the endogenous variables
in the order given by ) We often write the equation for
an endogenous variable as X=f(~Y); this denotes that
the value of Xdepends only on the values of the vari-
ables in~Y, and the connection is given by f. Our climate
example is recursive, since ~T=fL(~W;~U).
Anintervention has the form ~X ~ x, where~Xis a set
of endogenous variables. Intuitively, this means that the
values of the variables in ~Xare set to~ x. The structural
equations deﬁne what happens in the presence of inter-
ventions. Setting the value of some variables ~Xto~ x
in a causal model M= (S;F)results in a new causal
model, denoted M~X ~ x, which is identical to M, except
thatFis replaced byF~X ~ x: for each variable Y =2~X,
F~X ~ x
Y =FY, while for each X0in~X, the equation FX0
for is replaced by X0=x0(wherex0is the value in ~ x
corresponding to ~ x).
Halpern and Pearl (2005) and Halpern (2016) implic-
itly assumed that all interventions can be performed in
a model. For reasons that will become clear when deﬁn-
ing abstraction, we follow Rubenstein et al. (2017) and
Beckers and Halpern (2019) in adding the notion of “al-
lowed interventions” to a causal model. This allows us to
capture situations where not all interventions are of inter-
est to the modeler and/or some interventions may not be
feasible. We can then deﬁne a causal model Mas a tuple
(S;F;I), where (S;F)is a basic causal model and I
is a set of allowed interventions . We sometimes write a
causal model M= (S;F;I)as(M0;I), whereM0is
the basic causal model (S;F), if we want to emphasize
the role of the allowed interventions.
Given a signature S= (U;V;R), aprimitive event
is a formula of the form X=x, forX2 V and
et al. (1996), in that both are assignments of values to variables.
However, here it is used in particular to denote an assignment
of values to all the exogenous variables.
2Halpern (2016) calls this strongly recursive , in order to dis-
tinguish it from models in which the partial order depends on
the context. This distinction has no impact on our results.x2 R(X). A causal formula (over S)is one of the
form [Y1 y1;:::;Yk yk]', where'is a Boolean
combination of primitive events, Y1;:::;Ykare distinct
variables inV, andyi2R(Yi). Such a formula is ab-
breviated as [~Y ~ y]'. The special case where k= 0
is abbreviated as '. Intuitively, [Y1 y1;:::;Yk
yk]'says that'would hold if Yiwere set to yi, for
i= 1;:::;k .
A causal formula  is true or false in a causal model,
given a context. As usual, we write (M;~ u)j= if the
causal formula  is true in causal model Mgiven context
~ u. Thej=relation is deﬁned inductively. (M;~ u)j=X=
xif the variable Xhas valuexin the unique (since we
are dealing with recursive models) solution to the equa-
tions inMin context~ u(i.e., the unique vector of val-
ues that simultaneously satisﬁes all equations in Mwith
the variables inUset to~ u). The truth of conjunctions
and negations is deﬁned in the standard way. Finally,
(M;~ u)j= [~Y ~ y]'if(M~Y=~ y;~ u)j='.
To simplify notation, we sometimes write M(~ u)to de-
note the unique element of R(V)such that (M;~ u)j=
V=~ v. Similarly, given an intervention ~Y ~ y,
M(~ u;~Y ~ y)denotes the unique element of R(V)such
that(M;~ u)j= [~Y ~ y](V=~ v).
These deﬁnitions allow us to describe the climate system
both in terms of a low-level causal model MLand a high-
level model MH:
ML= ((UL;VL;RL);FL;IL)
= ((f~Ug;f~W;~Tg;RL);fgL;fLg;IL)and
MH= ((UH;VH;RH);FH;IH)
= ((f~Ug;fC;Eg;RH);fgH;fHg;IH):
Chalupka et al. (2016) treated ~Uas not only exogenous,
but also unobserved, and therefore made no claim about
its dimensionality in the high-level model. We do not ex-
plicitly spell outRLandRHhere; the description of the
climate model indicates that RL(C)RL(E)is much
smaller thanRL(~W)RL(~T): many different low-level
(vector-valued) states may correspond to one high-level
low-dimensional state. Finally, for our climate example
we have not yet said anything about interventions, so IL
andIHare currently placeholders.
2.1 Abstraction
Beckers and Halpern (2019) gave a sequence of suc-
cessively more restrictive deﬁnitions of abstraction for
causal model. The ﬁrst and least restrictive deﬁnition
is the notion of exact transformation due to Rubenstein
et al. (2017). Examples given by Beckers and Halpern
show that the notion of exact transformation is arguably
too ﬂexible. Thus, in this paper the notion of abstrac-
tion we consider is that of -abstractions , introduced by
Beckers and Halpern, which can be viewed as a restric-
tion of exact transformations3and avoids some of their
problems. However, nothing hinges on this choice: all
deﬁnitions can just as well be interpreted using the less
restrictive notions of abstractions.
The key to deﬁning all the notions of abstraction
from a low-level to a high-level causal model consid-
ered by Beckers and Halpern is the abstraction func-
tion:RL(VL)!RH(VH)that maps endogenous
states ofMLto endogenous states of MH. This
is a generalization of the surjective mapping from
R(~T;~W)toR(E;C)discussed in the introduction.
In the formal deﬁnition, we need two additional func-
tions:U:RL(UL)!RH(UH), which maps exoge-
nous states of MLto exogenous states of MH, and
!:IL!IH, which maps low-level interventions to
high-level interventions. Beckers and Halpern (2019)
show that, given their deﬁnition of abstraction, Uand
!can be derived from . We brieﬂy review the rele-
vant deﬁnitions here; we refer the reader to their paper
for more details and motivation.
Deﬁnition 2.3: Given a setVof endogenous variables,
~XV, and~ x2R(~X), let
Rst(V;~ x) =f~ v2R(V) :~ xis the restriction of ~ vto~Xg:
Given:RL(VL)!RH(VH), deﬁne!(~X ~ x) =
~Y ~ yif~YVH,~ y2RH(~Y), and(Rst(VL;~ x)) =
Rst(VH;~ y)(where, as usual, given TRL(VL), deﬁne
(T) =f(~ vL) :~ vL2Tg). It is easy to see that, given
~Xand~ x, there can be at most one such ~Yand~ y. If
such a~Yand~ ydo not exist, we take !(~X ~ x)to be
undeﬁned. LetI
Lbe the set of interventions for which
!is deﬁned, and letI
H=!(I
L).
Note that if is surjective, then it easily follows that
!(;) =;, and for all ~ vL2RL(VL),!(VL ~ vL) =
VH (~ vL).
With this deﬁnition, the need for the intervention sets
ILandIHbecomes clear: in general, not all low-level
interventions will neatly map to a high-level interven-
tion, since the abstraction function may aggregate vari-
ables together; some low-level interventions will consti-
tute only a partial intervention on a high-level variable.
3Exact-transformations relate probabilistic causal mod-
els, while-abstractions relate (deterministic) causal models.
Beckers and Halpern (2019) show that we can compare the
two by proving that every -abstraction is what they call a uni-
form-transformation : speciﬁcally, if MHis a-abstraction
ofML, then for every probability PrL, there exists a probabil-
ityPrHsuch that (ML;PrL)is an exact-transformation of
(MH;PrH).The “allowed intervention” sets ensure that the set of in-
terventions can be suitably restricted to retain only those
that can actually be abstracted. Similarly, there may be
cases where the high-level model does not support all
interventions because they may not be well-deﬁned. For
example, what does it mean in the ideal gas law to change
temperature, while keeping pressure and volume con-
stant? It is not even clear that such an intervention is
meaningful.
Of course, a minimal requirement for any causal model
to be a-abstraction of some other model is that the sig-
natures of both models need to be compatible with .
Beckers and Halpern (2019) add two further minimal re-
quirements. We capture all of them by requiring the two
causal models to be -consistent:
Deﬁnition 2.4 : If:RL(VL)! RH(VH), then
(MH;IH)and(ML;IL)are-consistent ifis surjec-
tive,IH=!(IL), andjRL(UL)jjRH(UH)j.
Deﬁnition 2.5 : (MH;IH)is a-abstraction of
(ML;IL)if(MH;IH)and(ML;IL)are-consistent
and there exists a surjective Usuch that for all ~ uL2
RL(UL)and~X ~ x2IL,(ML(~ uL;~X ~ x)) =
MH(U(~ uL);!(~X ~ x)):
Abstraction means that for each possible low-level
context-intervention pair, the two ways of moving up “di-
agonally” to a high-level endogenous state always lead to
the same result. The ﬁrst way is to start by applying ML
to get a low-level state, and then moving up to a high-
level state by applying , whereas the second way is to
ﬁrst move to a high-level context and intervention (by ap-
plyingUand!), and then to obtain a high-level state
by applying MH.
A common and useful form of abstraction occurs when
the low-level variables are clustered, so that the clusters
form the high-level variables. Roughly speaking, the in-
tuition is that in the high-level model, one variable cap-
tures the effect of a number of variables in the low-level
model. This makes sense only if the low-level variables
that are being clustered together “work the same way” as
far as the allowed interventions go. The following deﬁ-
nition makes this special case of abstraction precise.
Deﬁnition 2.6 : IfVH=fY1;:::;Yng, then:
RL(VL)! RH(VH)isconstructive if there exists a
partitionP=f~Z1;:::;~Zn+1gofVL, where~Z1;:::;~Zn
are nonempty, and mappings i:R(~Zi)! R (Yi)
fori= 1;:::;n such that= (1;:::;n); that is,
(~ vL) =1(~ z1):::n(~ zn), where~ ziis the projection
of~ vLonto the variables in ~Zi, andis the concatena-
tion operator on sequences. If MHis a-abstraction of
Figure 2: The climate model exempliﬁes constructive ab-
straction. Left:andUmap low-level variables to their
high-level counterparts. Right:!maps low-level in-
terventions to the high-level intervention. Several differ-
ent low-level interventions on ~Wmay correspond to the
same intervention on C.
MLthen we say it is constructive ifis constructive and
IL=I
L.
In this deﬁnition, we can think of each ~Zias describ-
ing a set of microvariables that are mapped to a single
macrovariable Yi. The variables in ~Zn+1(which might
be empty) are ones that are marginalized away.
The climate example almost exactly ﬁts the notion of
constructive abstraction: the variables in the high-level
model,CandE, each correspond to a vector-valued low-
level variable, ~Wand~T, but~Wand~Tcould have been
replaced by disjoint sets of variables. Consequently,
maps states from the same low-level variable to the same
high-level variable (see Fig. 2, left). Although interven-
tions are practically not feasible in the climate case, hy-
pothetically they are perfectly well-deﬁned: an interven-
tion onCcan be instantiated at the low level by several
different interventions on ~W(see Fig. 2, right). Finally,
given an intervention on ~W, we have that
ML(~ u;~W ~ w)j= (~T=~t)iff
MH(U(~ u);!(~W ~ w))j= (E=(~t)):
The correspondence between MLandMHisexact . In
fact, the high-level model MHconstructed by Chalupka
et al. (2016) did not satisfy this biconditional precisely,
but had to approximate it. We maintain that, in general,
high-level models in science are only approximate ab-
stractions .
3 APPROXIMATE ABSTRACTION
In order to deﬁne what it means for one causal model to
be an approximation of another, we need a way of mea-
suring the “distance” between causal models. We take a
distance function to simply be a function that associates
with a pair (M1;M2)a distance, that is, a non-negative
real number. We show how various distance functions oncausal models can be deﬁned, starting from a metricdV
on the state spaceR(V)of a causal model. (Recall that a
metric on a space Xis a function d:XX!IR+such
that (a)d(x;x0) = 0 iffx=x0, (b)d(x;y) =d(y;x),
and (c)d(x;y) +d(y;z)d(x;z).) Such a metric dV
is typically straightforward to deﬁne. Given two states
s1ands2, we can compare the value of each endogenous
variableXins1ands2. The difference in the values
determines the distance between s1ands2.
The choice of distance function is application-dependent.
Different researchers looking at the same data may be
interested in different aspects of the data. For ex-
ample, suppose that the model is deﬁned in terms
of 5 variables, X1;:::;X 5.X3might be gen-
der andX4might be height. Suppose that we
restrict to distance functions that takes the dis-
tance between (x1;:::;x 5)and(x0
1;:::;x0
5)to be of
the formp
w1(x1x0
1)2++w5(x5x0
5)2, where
w1;:::;w 5are weights that represent the importance (to
the researcher) of each of these ﬁve features. One re-
searcher might not be interested in gender (so doesn’t
care if her predictions about gender are incorrect), and
thus might take w3= 0; another researcher might care
about gender and not about height, so she might take
w3= 1 andw4= 0. While, as we shall see, the choice
of distance function makes a crucial difference in eval-
uating the “goodness” of an approximate abstraction, in
light of the above, we leave the choice of distance func-
tion unspeciﬁed.
In the remainder of the paper, we assume that the state
spaceR(V)of endogenous variables for each causal
model comes with a metric dV. We provide a number of
ways of lifting the metric dVon states to a distance func-
tiondon models, and then use the distance function to
deﬁne both approximation and approximate abstraction.
Our intuition for the distance function is based on how
causal models are typically used. Speciﬁcally, we are in-
terested in how two models compare with regard to the
predictions they make about the effects of an interven-
tion. Our intuition is similar in spirit to that behind the
notion of structural intervention distance considered by
Peters and B ¨uhlmann (2015), although the technical deﬁ-
nitions are quite different. (We discuss the exact relation-
ship between our approach and theirs in the next section,
in the context of probabilistic models.)
We start with the simplest setting where this intuition can
be made precise, one where the models M1andM2differ
only with regards to their equations. We say that two
models are similar in this case. If two models are similar,
then, among other things, we can assume that they have
the same metric dV. In this setting, we can compare the
effect of each allowed intervention ~X ~ xin the two
models. That is, for each context ~ u, we can compare the
statesM1(~ u;~X ~ x)andM2(~ u;~X ~ x)that arise after
performing the intervention ~X ~ xin context~ uin each
model. We get the desired distance function by taking
the worst-case distance between all such states.
Deﬁnition 3.1: Deﬁne a distance function dmax on pairs
of similar models by taking
dmax(M1;M2) =
max~X ~ x2I,~ u2R(U)fdV(M1(~ u;~X ~ x);M2(~ u;~X ~ x))g:
The causal model M1is admax-approximation ofM2
ifdmax(M1;M2).
Thus,M1is admax-approximation of M2if the pre-
dictions ofM1are always within of the predictions of
M2.
We apply similar ideas to deﬁning approximate abstrac-
tion. But now we no longer have a distance function de-
ﬁned on causal models with the same signature. Rather,
the distance function dis deﬁned on pairs (ML;MH)
consisting of a low-level and high-level causal model
(which, in general, have different signatures), related by
a a surjective mapping . The idea behind dis that we
start with a low-level intervention ~X ~ x, consider its
effects inML, lift this up to MHusing, and compare
this to the effects of !(~X ~ x)inMH.
Deﬁnition 3.2: Fix a surjective map :R(VL)!
R(VH). Deﬁne the distance function don pairs of-
consistent models (ML;MH)by taking
d(ML;MH) = minUsurjective max~X ~ x2IL,~ uL2RL(UL)
(dVH((ML(~ uL;~X ~ x));MH(U(~ uL);!(~X ~ x)))):
MHis a-approximate abstraction ofMLif
d(ML;MH).
We take the minimum over all functions Ubecause the
function that lifts the low-level contexts up to the high-
level contexts does not play a major role. We thus simply
focus on the best choice of U.
To get an intuition for an approximate abstraction, con-
sider the climate example again. For a low-level inter-
vention~W ~ wand a low-level context ~ u, there are two
ways of lifting their effect to MH(see Fig. 3). The ﬁrst is
to start by applying MLto the context-intervention pair
to determine a low-level state ~t, and then apply to ob-
tain the high-level state E=e=(~t). (Recall that ML
can be viewed as a function from context-intervention
pairs to states inRL(VL).) The second is to ﬁrst lift
the intervention ~W ~ wtoIH, that is, an intervention
onC, and the context to UHby applying !andU.
Figure 3: Approximate abstraction in the climate exam-
ple. We measure (for the worst-case choice of low-level
intervention ~W ~ wand context ~ u) the distance be-
tween the values of the variable Eobtained by (a) ﬁrst
applyingMLto~ uand~W ~ wand then abstracting by
applyingvs. (b) abstracting the intervention and con-
text (by applying !andU, respectively) and then ap-
plyingMH.
Then we apply MH(~ uH;C c), which again gives a
high-level endogenous state e0. We are identifying the
degree to which MHapproximates MLwith the worst-
case distance between the two ways of lifting the context-
intervention pairs, for an optimal choice of U.
The following straightforward results show that our no-
tion of approximate abstraction is a sensible generaliza-
tion of both the notion of an exact abstraction and the
notion of approximation between similar models.
Proposition 3.3: MHis a-0-approximate abstraction
ofMLiffMHis a-abstraction of ML.
Proposition 3.4: IfM1andM2are similar, then M2
is aId--approximate abstraction of M1(where Idis
the identity function on R(V)) iffM2is admax--
approximation of M1.
4 APPROXIMATE ABSTRACTION FOR
PROBABILISTIC CAUSAL MODELS
Aprobabilistic causal model M= (S;F;I;Pr)is just a
causal model together with a probability Pron contexts
~ u. In this section, we assume that all causal models are
probabilistic, and extend the notion of approximation to
probabilistic causal models. We again start by consid-
ering the simplest setting, where we have probabilistic
models that differ only in their equations. We again call
such models similar . Now we have several reasonable
distance functions.
Deﬁnition 4.1: Deﬁne a distance function dmax on pairs
of similar probabilistic causal models by taking
dmax(M1;M2) = max~X ~ x2I P
~ u2R(U)Pr(~ u)dV(M1(~ u;~X ~ x);M2(~ u;~X ~ x))
:
The probabilistic causal model M1is admax-approxi-
mation ofM2ifdmax(M1;M2).
Here we have just replaced the max over contexts in Def-
inition 3.1 by an expectation over contexts.
But we may not always just be interested in the expected
distance. We may, for example, be more concerned with
the likelihood of serious prediction differences, and not
be too concerned about small differences. This leads to
the following deﬁnition.
Deﬁnition 4.2: Deﬁne a distance function don pairs of
similar probabilistic causal models by taking
d(M1;M2) = max~X ~ x2I
Pr(f~ u:dV(M1(~ u;~X ~ x);M2(~ u;~X ~ x))g:
The probabilistic causal model M1is ad-approxima-
tionofM2ifd(M1;M2).
We can now extend these ideas to approximate abstrac-
tion. We ﬁrst extend the deﬁnition of -abstraction to the
probabilistic setting.
Note that we can view Pras a probability measure on
R(V), by taking Pr(~ v) =f~ u:M(~ u) =~ vg. An inter-
vention~X ~ xalso induces a probability Pr~X ~ xon
R(V)in the obvious way:
Pr~X ~ x(~ v) = Pr(f~ u:M(~ u;~X ~ x) =~ vg):
In the deterministic notion of abstraction, we require that
the two high-level states obtained by the two different
ways of lifting the effects of a low-level intervention to
the high level be equal. In the probabilistic notion, we
require that the two different probability distributions ob-
tained by the two ways of lifting an intervention be equal.
Deﬁnition 4.3: MHis a-abstraction ofMLifMHand
MLare-consistent and for all interventions ~X ~ x2
IL, we have that (Pr~X ~ x
L) = Pr!(~X ~ x)
H:
We can now extend our deﬁnitions to the approximate
scenario just as we did for deterministic causal models.
Deﬁnition 4.4: Fix a surjective map :R(VL)!
R(VH). Deﬁne the distance function don pairs of-
consistent probabilistic causal models by taking
d(ML;MH) = minfU:U(PrL)=Pr Hgmax~X ~ x2IL
(P
~ uL2RL(UL)PrL(~ uL)
dVH((ML(~ uL;~X ~ x));MH(U(~ uL);!(~X ~ x)))):
MHis a-approximate abstraction ofMLif
d(ML;MH).For the climate example this deﬁnition implies the fol-
lowing: Suppose we introduce probabilities by specify-
ing distributions over the contexts ~ uLand~ uH. The re-
sulting probabilistic causal model (MH;PrH)is a-
approximate abstraction of (ML;PrL)if the expectation
(in terms of PrL) of the difference between the states e
ande0of the high-level temperature variable Eis less or
equal to, whereeande0are determined exactly in ac-
cordance with the two pathways in Fig. 3, selecting the
worst-case intervention ~W ~ wand the best case U.
Analogously to the deterministic case, we have the fol-
lowing straightforward results.
Proposition 4.5: MHis a-0-approximate abstraction
ofMLiffMHis a-abstraction of ML.
Proposition 4.6: IfM1andM2are similar, then M2is
aId--approximate abstraction of M1iffM2is admax-
-approximation of M1.
In Deﬁnition 4.4 we consider the worst-case low-level
intervention to deﬁne the distance. In many cases, how-
ever, the whole point of an abstraction is to be able to
exclude rare low-level boundary cases (e.g., when the
ideal gas law is taken to refer only to equilibrium states).
Moreover, often the actual manipulations that we can
perform are known to us only at the high-level, because
the low-level implementation of the intervention is unob-
servable to us. For example, in setting the room temper-
ature to 70F we do not generally consider the instantia-
tion of that intervention which superheats one corner of
the room and freezes the rest such that the mean kinetic
energy works out just right. As Spirtes and Scheines
(2004) show, in the absence of any further information,
such ambiguous manipulations can be quite problematic.
Fortunately, often our knowledge of the mechanism of
how a high-level intervention is implemented does give
us signiﬁcant probabilistic information. For example, we
might know that the heater has a fan that circulates the
air, most likely resulting in relatively uniform distribu-
tions of the kinetic energies of the particles.
We capture this information using what we call an inter-
vention distribution.
Deﬁnition 4.7 : Given a surjective map
:R(VL)!R (VH), an intervention distribu-
tion PrIis a distribution on IL IHsuch that
PrI(~X ~ xj~Y ~ y)>0iff!(~X ~ x) =~Y ~ y.
We think of PrI(~X ~ xj~Y ~ y)as telling us how
likely the high-level intervention ~Y ~ yis to have been
implemented by the low-level intervention ~X ~ x.
Deﬁnition 4.8: Given a surjective map :R(VL)!
Figure 4: Probabilistic approximate abstraction for the
climate example: An intervention on the high-level wind
variableCresults, given the distribution over contexts ~ u
(omitted for clarity), in the red distribution over the high-
level temperature variable E. The intervention C c
can be instantiated in the low-level wind map ~Win many
ways, according to the intervention distribution PrI.
Mapping the (manipulated) distribution over ~Wusing
MLresults in a distribution over the low-level temper-
ature map~Tthat combines uncertainty from PrIand
PrU. Abstracting this distribution using toEresults in
the black distribution over E.MHis a probabilistic --
approximate abstraction of MLif the distance between
the expectations of these distributions is less than .
R(VH)and an intervention-distribution PrI, the dis-
tance function dPrIon pairs of-consistent probabilistic
causal models is deﬁned by taking
dPrI(ML;MH) = minfU:U(PrL)=PrHgmax~Y ~ y2IH
(P
~ uL2RL(UL)PrL(~ uL)P
~X ~ x2ILPrI(~X ~ xj~Y ~ y)
dVH((ML(~ uL;~X ~ x));MH(U(~ uL);!(~X ~ x)))):
Intuitively, for each high-level intervention ~Y ~ y, we
take the expected distance between the two ways of lift-
ing a low-level intervention to ~Y ~ y. In computing
the expectation, there are two sources of uncertainty: the
likelihood of a given context (this is determined by PrL)
and the likelihood on each low-level intervention that
maps to~Y ~ y(this is given by PrI( j~Y ~ y)).
Fig. 4 illustrates the point for the climate example.
We can also deﬁne d
andd;PrI
in a manner completely
analogous to Deﬁnition 4.2. We omit these deﬁnitions
for reasons of space, but the details should be clear.
It is worth comparing our approach to other approaches
to determining the distance between causal models. The
more standard way to compare two causal models is to
compare their causal graphs. The causal graph is a di-
rected acyclic graph (dag) that has nodes labeled by vari-
ables; (the node labeled) Xis an ancestor of (the node
labeled)YiffXY. Dags have been compared usingwhat is called the structural Hamming distance (SHD)
(Acid and de Campos 2003), where the SHD between
GandH(which are assumed to have an identical set of
nodes) is the number of pairs of nodes (i;j)on which
GandHdiffer regarding the edge between iandj(ei-
ther because one of one of them has an edge and the
other does not, or the edges are oriented in different di-
rections). As Peters and B ¨uhlmann (2015) observe, the
SHD misses out on some important information in causal
networks. In particular, it does not really compare the ef-
fect of interventions. They want a notion of distance that
takes this into account, as do we. However, they take into
account the effect of interventions in a way much closer
in spirit to SHD. Roughly speaking, in our language,
given two similar causal models M1andM2, they count
the number of pairs (X;Y )of endogenous variables such
that intervening on Xleads to a different distribution
overYinM1andM2. More formally, let PrM;X x
Y
denote the marginal of PrM;X xon the variable Yin
modelM. (Since we want to compare probability distri-
butions in two different models, we add the model to the
superscript.) The SID between similar models M1and
M2is the number of pairs (X;Y )such that there exists
anx2R(X)such that PrM1;X x
Y6= PrM2;X x
Y .
Although SID does compare the predictions that two
models make, it differs from our distance functions in
several important respects. First, it compares just the
effect of interventions on single variables, whereas we
allow arbitrary interventions. We believe that it is impor-
tant to consider arbitrary interventions, since sometimes
variables act together, and it takes intervening on more
than one variable to distinguish two models. Second,
we are interested in how far apart two distributions are,
not just the fact that they are different. Finally, we want
a deﬁnition that applies to models that are not similar,
since this is what we need for approximate abstraction.
5 COMPOSING ABSTRACTION AND
APPROXIMATION
It can be useful to understand an approximate abstrac-
tion as the result of composing an approximation and an
abstraction, in some order. For example, we can explain
the ideal gas law in terms of thinking of frictionless elas-
tic collisions between particles (this is an approximation
to the truth) and then abstracting by replacing the kinetic
energy of the particles by their temperature (a measure
of average kinetic energy). Here we examine the extent
to which an approximate abstraction can be viewed this
way. We start with two easy results showing that if we
compose an approximation and an abstraction in some
order, then we do get an approximate abstraction.
Proposition 5.1 : If(MH;IH)is a-abstraction of
(ML;IL)and(M0
H;IH)is admax--approximation of
(MH;IH)then(M0
H;IH)is a-approximate abstrac-
tion of (ML;IL).4
In Proposition 5.1 we considered an abstraction followed
by an approximation. Things change if we do things in
the opposite order; that is, if we consider an approxima-
tion followed by an abstraction. For suppose that M0
Lis a
dmax-approximation of MLandMHis a-abstraction
ofML. Now it is not in general the case that MHis ad-
-approximate abstraction of M0
L. The problem is that
when assessing how good an abstraction MHis ofM0
L,
we are comparing two high-level states (using dVH). But
when comparing MLtoM0
L, we usedVL. In general,
dVL(~ vL;~ v0
L)anddVH((~ vL);(~ v0
L))may be unrelated.
Proposition 5.2 : If(MH;IH)is a-abstraction of
(ML;IL)and(ML;IL)is admax-approximation of
(M0
L;IL), then (MH;IH)is a-kapproximate ab-
straction of (M0
L;I0
L), wherekis
max~X ~ x2IL;~ uL2RL(UL)
dVH((M0
L(~ uL;~X ~ x));(ML(~ uL;~X ~ x)))
dVL(M0
L(~ uL;~X ~ x);ML(~ uL;~X ~ x)):
While composing an approximation and an abstraction
gives us an approximate abstraction, the full paper (2019)
contains two examples that show that we cannot in gen-
eral decompose an approximate abstraction into an ab-
straction composed with an approximation or an approx-
imation composed with an abstraction.
As the following result shows, we can test whether an
approximate abstraction can be viewed as the result of
composing an abstraction and an approximation.
Theorem 5.3: If(MH;IH)is a--approximate ab-
straction of (ML;IL)then the problem of determin-
ing whether there exists a model (M0
H;IH)(resp.,
(M0
L;IL)) that is similar to (MH;IH)and is an abstrac-
tion of (ML;IL)(resp., (ML;IL)) is in nondeterministic
time polynomial in jILjjRL(UL)j.
There is one important special case where we are guar-
anteed to be able to ﬁnd an appropriate intermediate low-
level model, and to do so in polynomial time: if the map-
pingis constructive.
Theorem 5.4: If:RL(VL)!RH(VH)is construc-
tive, the causal models (ML;IL)and(MH;IH)are-
consistent, and (MH;IH)is a--approximate abstrac-
tion of (ML;IL), then we can ﬁnd a model (M0
L;IL)
4Proofs of all technical results can be found in the full paper
(Beckers, Eberhardt, and Halpern 2019).that is similar to (ML;IL), and such that (MH;IH)
is a-abstraction of (M0
L;IL)in time polynomial in
jILjjR0
L(UL)j.
The full paper contains an example showing that being
constructive does not similarly guarantee the existence of
M0
Hin the ﬁrst half of Theorem 5.3.
6 DISCUSSION AND CONCLUSIONS
By deﬁning notions of abstraction, approximation, and
approximate abstraction, we have presented a framework
that relates causal models that describe the same sys-
tem at (possibly) different levels of granularity. While
coarser models offer a degree of simpliﬁcation by omit-
ting details, they also in general entail a loss in accu-
racy with respect to the fundamental description. Our
framework shows how to quantify this loss in accuracy
by deﬁning a distance metric that captures the degree to
which a more abstract causal model approximates a more
detailed causal model. High- and low-level causal mod-
els of the same system can vary on almost any dimen-
sion. They need not share the same equations, the same
variables, or the same interventions. They may involve
entirely distinct state spaces. Inevitably, then, there is
some degree of choice as to what one deems relevant to
the approximation.
Starting with deterministic causal models, we provided a
general method for quantifying the “goodness” of an ap-
proximate abstraction. As an interesting special case, our
approach allows for the comparison of causal models that
operate at the same level of detail. We then extended to
probabilistic causal models, and considered several dif-
ferent choices for quantifying the distance between mod-
els. Finally, we considered the extent to which we could
decompose an approximate abstraction into an abstrac-
tion and an approximation.
Given the ubiquitous use of causal models in the social
and natural sciences that are known not to capture all the
causally relevant details, the framework we presented of-
fers a principled way to assess the trade-off between ab-
straction and accuracy.
Acknowledgements
Beckers was supported by the grant ERC-2013- CoG
project REINS 616512. Eberhardt was supported in
part by NSF grants 1564330 and BCS-1845958, and
HRGC grant 13601017. Halpern was supported in part
by NSF grants IIS-1703846 and IIS-1718108, ARO grant
W911NF-17-1-0592, and a grant from the Open Philan-
thropy project. We thank the UAI reviewers for many
useful comments.
References
Acid, S. and L. M. de Campos (2003). Searching for
Bayesian network structures in the space of re-
stricted acyclic partially directed graphs. Journal
of A.I. Research 18 , 445–490.
Beckers, S., F. Eberhardt, and J. Y . Halpern
(2019). Approximate causal models. Available at
arxiv.org/abs/1906.11583.
Beckers, S. and J. Y . Halpern (2019). Abstract-
ing causal models. In Proc. Thirty-Third
AAAI Conference on Artiﬁcial Intelligence
(AAAI-19) . The full version is available at
arxiv.org/abs/1812.03789.
Boutilier, C., N. Friedman, M. Goldszmidt, and
D. Koller (1996). Context-speciﬁc independence
in Bayesian networks. In Proc. Twelfth Confer-
ence on Uncertainty in Artiﬁcial Intelligence (UAI
’96), pp. 115–123.
Chalupka, K., T. Bischoff, P. Perona, and F. Eberhardt
(2016). Unsupervised discovery of El Ni ˜no using
causal feature learning on microlevel climate data.
InProc. 32nd Conference on Uncertainty in Arti-
ﬁcial Intelligence (UAI 2016) , pp. 72–81.
Halpern, J. Y . (2016). Actual Causality . Cambridge,
MA: MIT Press.
Halpern, J. Y . and J. Pearl (2005). Causes and ex-
planations: a structural-model approach. Part I:
Causes. British Journal for Philosophy of Sci-
ence 56 (4), 843–887.
Peters, J. and P. B ¨uhlmann (2015). Structural in-
tervention distance (SID) for evbaluating causal
graphs. Neural Computation 27 (3), 725–747.
Rubenstein, P. K., S. Weichwald, S. Bongers, J. M.
Mooij, D. Janzing, M. Grosse-Wentrup, and
B. Sch ¨olkopf (2017). Causal consistency of struc-
tural equation models. In Proc. 33rd Confer-
ence on Uncertainty in Artiﬁcial Intelligence (UAI
2017) .
Spirtes, P. and R. Scheines (2004). Causal inference
of ambiguous manipulations. Philosophy of Sci-
ence 71 , 833–845.

Sander Beckers
Frederick Eberhardt
Joseph Y Halpern


A Note on the Existence of Ratifiable Acts.

A Note on the Existence of Ratiable Acts
Joseph Y. Halpern
Cornell University
Computer Science Department
Ithaca, NY 14853
halpern@cs.cornell.edu
http://www.cs.cornell.edu/home/halpern
August 15, 2018
Abstract
Sucient conditions are given under which ratiable acts exist.
1 Introduction
Jerey (1983) suggested that you should choose an act that would be best for \the person you
expect to be when you will have chosen". He called an act that satised this property ratiable .
Whether one should always choose ratiable acts is a matter of ongoing debate. Skyrms (1990b)
argues that there are situations where ratiable acts are appropriate and other situations where
they are not, whereas Harper (1986) seems to suggest that they are always appropriate. Joyce
(2012) gives a more recent perspective.
Although I personally do not believe that ratiable acts are always appropriate, I do not
enter into this discussion here. Rather, I examine more closely the question of when ratiable
acts exist. As Rabinowicz (1985) shows by example, in arguably natural problems, there can
exist several ratiable acts, a unique ratiable act, or no ratiable \pure" acts (i.e., ones
where there is no randomization). Harper (1986) points out that existence of ratiable act will
require randomization. This should not be surprising. Ratiability has always been understood
as an equilibrium notion. It is well known that Nash equilibrium does not always exist in
pure strategies; to show that a Nash equilibrium always exists, Nash (1951) had to use mixed
strategies (where players can randomize over pure strategies). However, as Richter (1984) shows,
when there is a cost to randomizing, ratiable acts may not exist.1
While Harper (1986) is willing to ignore settings where there is no ratiable act, saying
\I regard cases where no act is ratiable as genuinely pathological and have no qualms about
allowing that causal utility theory makes no recommendations in them", it does not seem to
me that charging for randomization is pathological. There is a cognitive cost to randomizing.
Work supported in part by NSF under grants 1703846 and 1718108 and a grant from the Open Philanthropy
Foundation.
1This is also a problem for Nash equilibrium; see Section 3.
1
In any case, there still remains the question of when ratiable acts do exist. Skyrms (1990a,
1990b) suggests conditions under which ratiable acts exist, but does not prove any theorem.
Indeed, he says \The question of the existence of ratiable acts . . . is a delicate one which calls
for careful examination" (Skyrms 1990b).
Skyrms' conditions for the existence of ratiable acts are essentially that we allow random-
ization and that there is no cost for randomizing. While the intuition behind these assumptions,
and the need for them, is clear, making them precise is not trivial. Moreover, it turns out that
these assumptions do not quite suce for ratiable acts to exist. Here I show that once we
formalize these assumptions appropriately then, together with an additional assumption about
how conditioning works, they suce to show that ratiable acts exist. Not surprisingly, once
we have the appropriate conditions, the proof of the existence of ratiable acts is quite similar
to that of the proof of the existence of Nash equilibrium. In the next section, I provide the
details.
2 The existence of ratiable acts
Before proving the existence of ratiable acts, I rst have to dene the notion. Jerey (1983)
does not provide a formal denition; I use Harper's (1984) formalization, also used by Skyrms
(1990a, 1990b) and later authors.
I follow the standard Savage (1954) approach to decision theory. Specically, I assume that
a state space Sand a setOof outcomes are given. An actis a function from states to outcomes.
LetAdenote the set of acts. For simplicity in this paper, I assume that S,O, and therefore
Aare nite and nonempty (although for Savage, it is important that Scan be innite; indeed,
his postulates force Sto be innite).
To make sense of ratiable acts, it has typically been assumed that there is a probability
Pr onSAand a utility function umapping outcomes to utilities (real numbers).2Given an
acta, letuabe the function on states dened by setting ua(s) =u(a(s)).
Denition 2.1: An act aisratiable if, for all acts a02A,
EPrja(ua)EPrja(ua0); (1)
whereEPrja(u) denotes the expected utility of a function uon states with respect to the prob-
ability Prja:
EPrja(u) =X
s2S(Prja)(s)u(s):
I remark that Prjais sometimes denoted Pr( ja), and (Prja)(s) is typically written Pr( sja);
I use these notations interchangeably. Intuitively, Denition 2.1 says that ais ratiable if the
2Actually we do not quite need a probability Pr on SA. Rather, for each act a2A, we must have
a conditional probability Pr aonS. Intuitively, Pr a(s) = Pr( sja), the probability of sgiven that act ais
performed. To my mind, having such a family of conditional probabilities is more reasonable than having a
probability on SA. To be more consistent with the literature, I assume a probability Pr on SAhere,
although the reader can easily check that all that is ever used are the conditional probability Pr( ja). I also
implicitly assume through that Pr( a) (or, more precisely, Pr( Sfag)) is positive, so that conditioning on ais
well dened.
2
agent would be at least as happy with aas with any other act a0conditional on ahaving been
performed.
Next, I make precise the notion that there is \no cost for randomizing". Given A(which
a view as a set of pure|deterministic|acts), let Abe the convex closure ofA, so that if
A=fa1;:::; ang, then Aconsists of all acts of the form 1a1++nan, where1++n= 1
andi0 fori= 1;:::;n . The act 1a1++nanis interpreted as \perform a1with
probability 1and:::and perform anwith probability n". Similarly, let Obe the convex
closure ofO. Extenduto a function u:O!IRin the obvious way, by taking
u(1o1++non) =1u(o1) ++nu(on): (2)
The assumption that there is no cost for randomization is captured by assuming that
(1a1++nan)(s) =1a1(s) ++nan(s) : (3)
the outcome of performing the act 1a1++nanin statesis, with probability 1, the outcome
of performing act a1in statesand . . . and with probability n, the outcome of performing an
in states. That is, performing a convex combination of acts in state sleads to the obvious
convex combination of outcomes in O; there are no \untoward" outcomes. By way of contrast,
if there were a penalty for randomizing, both a1(s) and a2(s) might give the outcome $1,000,
although (:5a1+:5a2)(s) gives an outcome of $0. Of course, this is inconsistent with (3). In
combination with (2), (3) ensures that the utility of performing the act 1a1++nanis the
appropriate convex combination of the utilities of performing a1;:::; an.
Assumption (3) does not suce to show the existence of ratiable acts. To explain why, I rst
introduce the additional assumption that is needed, which involves conditioning on randomized
acts. More precisely, we need to extend Pr to a probability PronSA. Again, as I pointed
out above, all we really need to know is Pr( ja) for acts a2A. Supposed that we know
Pr(ja) and Pr(ja0) for a;a02A. What should Pr(ja+ (1)a0) be? One approach to
dening this is to use Jerey's rule and take
Pr(ja+ (1)a0) =Pr(ja) + (1) Pr(ja0): (4)
This seems reasonable: with probability ,awill be played, in which case Pr( ja) describes the
conditional probability; and with probability (1 ),a0will be played, in which case Pr( ja0)
describes the conditional probability. While this is reasonable and what I will assume to get
the result, note that it is a nontrivial assumption.
To understand the issues involved, consider a variant of Newcomb's problem: There are two
boxes, a red box and a blue box, and an extremely accurate predictor has put $1,000 in one
of them and nothing in the other. The agent must choose one of these boxes; that is, we take
A=fr;bg. We can take Oto consist of two outcomes: getting $1,000 and getting nothing. If
the predictor predicts that the agent will choose the red box, then he puts the $1,000 in the
blue box; if he predicts that the agent will choose the blue box, he puts the $1,000 in the red
box. Thus, conditional on choosing the red box, the agent would prefer to choose the blue box;
conditional on choosing the blue box, the agent would prefer to choose the red box. So neither
rnorbis ratiable.
Now suppose that we extend AtoA. To make this concrete, suppose that we allow a
randomizing device; the agent can set the device to a number between 0 and 1, and will
3
then play the act r+ (1)b. Assuming (3) and (4), a straightforward argument shows that
choosing= 1=2 gives the unique ratiable act. However, suppose that we change the game so
that if the predictor predicts that the agent randomizes, that is, if the predictor predicts that
the agent will set strictly between 0 and 1, then the predictor will put $1,000 in the red box.
Let the state space Sconsist of two states; in s1, the predictor puts $1,000 in the red box,
and ins2, the predictor puts $1,000 in the blue box. We can assume that (3) holds: the outcome
of randomizing over acts is the appropriate convex combinations of outcomes. On the other
hand, (4) does not hold; randomizing is strong evidence for state s1.
Assuming (3) and (4) allows us to prove the existence of a ratiable act in A; that is, there
are ratiable acts once we allow randomization.
Theorem 2.2: If (3) and (4) hold, then there is always a ratiable act in A
Proof: The argument proceeds very much like the argument for the existence of a Nash
equilibrium, using Kakutani's (1941) xed-point theorem. I rst state Kakutani's theorem, and
then explain all the terms in the statement of the theorem by showing how they hold in the
setting of interest.
Kakutani's Fixed-Point Theorem: IfXis a non-empty, compact, and convex subset of
IRn, the function f:X!2Xhas a closed graph, and f(x) is non-empty and convex for all
x2X, thenfhas a xed point, that is, there exists some x2Xsuch thatx2f(x).
We take the Xin Kakutani's theorem to be A. IfjAj=n, then we can identify the element
1a1++nan2Awith the tuple ( 1;:::; n)2IRn. With this identication, Ais a
closed subset of IRn(\closed" just means that if x1;x2;:::is a sequence of element in Athat
converges to y, theny2A); by construction it is convex (if x;y2A, then so is x+ (1)y
for2[0;1]); it is nonempty, since it includes AandAis non-empty; and it is clearly a
bounded subset of IRn(since all components in a tuple in Aare between 0 and 1). Since A
is closed and bounded, by standard results, it is compact.
Let the function fin the theorem be the best-response function. More precisely, given
x2A, letgx:AA!IRbe dened by taking
g(x;y) =X
s2SPr(sjx)u(y(s));
that is,g(x;y) describes how the agent feels about ygiven that she actually played x. Assump-
tions (3) and (4), together with the denition of u, guarantee that gis a continuous function.
Now dene
f(x) =fy:8z2A(g(x;y)g(x;z))g:
That is,y2f(x) if the agent feels that playing ywould have been one of her best choices,
if she actually plays x. Note that xis ratiable precisely if x2f(x). The fact that f(x)
is a convex set follows from our assumption on the utility function: if y;y02f(x), then
g(x;y) =g(x;y0) =g(x;y + (1)y0), soy+ (1)y02f(x) for all2[0;1]. Since A
compact and gis continuous, g(x;) takes on a maximum in A, sof(x) is nonempty. The
functionfhas a closed graph; that is, whenever xn!x,yn!y, andyn2f(xn), then
y2f(x). Thatfhas a closed graph follows easily from the fact that gis continuous. Since the
4
assumptions of Kakutani's xed-point theorem hold, fhas a xed point. As I observed above,
this xed point is a ratiable act.3This completes the proof.
3 Discussion
I have provided sucient conditions for the existence of ratiable acts. The key conditions are
the assumption that there is no cost for randomizing, which is captured by requiring that the
outcome of performing a convex combination of deterministic acts is the convex combination
of the outcomes of the individual deterministic acts, and an assumption that can be viewed as
saying that randomizing has no evidential value, which is captured by requiring that condition-
ing on a convex combination of deterministic acts is equivalent to the convex combination of
conditioning on the individual deterministic acts.
Interestingly, the fact that charging for randomization can aect the existence of equilibrium
has also been noted in a game-theoretic context. Halpern and Pass (2015) consider a setting
where, associated with each action in a normal-form game, there may be a cost. They show that,
in this setting, the analogue of Nash equilibrium may not exist. Interestingly, their example
involves a cost for randomization. Consider rst the standard rock-paper-scissors game where,
as usual, rock beats scissors, scissors beats paper, and paper beats rock. If a player's choice
beats the other player's choice, then he gets a payo (utility) of 1 and the other player gets
1; if both players make the same choice, they both get 0. As is well known, this game has a
unique Nash equilibrium, where players randomize, choosing each action with probability 1 =3.
Now suppose that we modify the game slightly, and charge for randomizing. That is, if
a player randomizes, no matter how he randomizes, his payos are decreased by  >0 (so,
for example, he gets 1 if his choice beats the other player's choice and if there is a
draw). The intuition here is that randomization is cognitively expensive. In any case, with this
utility function, it is easy to see that there is no Nash equilibrium, no matter how small is
(as long as it is positive). For suppose that we have a Nash equilibrium ( a1;a2). Ifa1involves
randomization, then player 1 can do better by deterministically playing the best response to
whichever action player 2 puts the highest probability on. (If player 2 puts equal probability
on several actions, player 1 chooses one of them and plays a best response to that.) This
deterministic choice gets at least as high a payo as a1ignoring the cost of randomizing,
and thus must be strictly better than a1when the cost of randomization is taken into account.
Thus, player 1 does not randomize in equilibrium. A similar argument shows that player 2 does
not randomize in equilibrium either. But it is clear that there is no equilibrium for this game
where both players use deterministic strategies.
Halpern and Pass also show that if randomization is free (and a few other technical con-
ditions hold), then the analogue of Nash equilibrium does exist in their framework. Very
roughly speaking, \randomization is free" means that the cost of a randomized act of the form
1a1++nanis the sum of 1times the cost of a1and . . . and ntimes the cost of an. The
similarity to (3) should be clear. Note that this condition is violated in the rock-paper-scissors
example.
3Exactly the same arguments show that, under the assumptions above, Skyrms' (1990a) deliberation process,
which is intended to model which strategy to play in a game or which act to choose in a decision problem, also
converges.
5
But what should we do if this condition does not hold? Charging extra for randomiza-
tion, as in the rock-paper-scissors example, does not seem so unreasonable, given the cognitive
costs of randomizing. Interestingly, as observed by Halpern and Pass (2015), there are rock-
paper-scissors tournaments (indeed, even a rock-paper-scissors world championship), and books
written on rock-paper-scissors strategies (Walker and Walker 2004). Championship players are
clearly not randomizing uniformly (they could not hope to get a higher payo than an opponent
by doing this). This leaves open the question of what an appropriate solution concept would
be in such a situation. This question also applies in the case of ratiability, but is perhaps less
serious in that context. I agree with Joyce (2012) that the agent should choose the best act
according to the prescriptions of causal decision theory. If this results in some regret after the
fact, then so be it|it was still the right choice. But for those who believe that ratiability is
a normative requirement, this is an important question that must be addressed.
References
Halpern, J. Y. and R. Pass (2015). Algorithmic rationality: Game theory with costly com-
putation. Journal of Economic Theory 156 , 246{268.
Harper, W. (1984). Ratiability and causal decision theory: comments on Eells and Seiden-
feld. In PSA: Proceedings of the Biennial Meeting of the Philosophy of Science Association,
Vol. 2 , pp. 213{228.
Harper, W. (1986). Mixed strategies and ratiability in causal decision theory. Erkennt-
nis 24 (1), 25{36.
Jerey, R. C. (1983). The Logic of Decision . Chicago: University of Chicago Press. second
edition.
Joyce, J. M. (2012). Regret and instability in causal decision theory. Synthese 187 (1), 123{
145.
Kakutani, S. (1941). A generalization of Brouwer's xed point theorem. Duke Mathematical
Journal 8 (3), 457{459.
Nash, J. (1951). Non-cooperative games. Annals of Mathematics 54 , 286{295.
Rabinowicz, W. (1985). Raticationism without ratication: Jerey meets Savage. Theory
and Decision 19 (2), 171{200.
Richter, R. (1984). Rationality revisited. Australasian Journal of Philosophy 62 (4), 392{403.
Savage, L. J. (1954). Foundations of Statistics . New York: Wiley.
Skyrms, B. (1990a). The Dynamics of Rational Deliberation . Cambridge, MA: Harvard Uni-
versity Press.
Skyrms, B. (1990b). Ratiability and the logic of decision. Midwest Studies in Philosophy XV ,
44{56.
Walker, G. and D. Walker (2004). The Ocial Rock Paper Scissors Strategy Guide . New
York: Simon & Schuster, Inc.
6