Selecting the Partial State Abstractions of MDPs: A Metareasoning Approach with Deep Reinforcement Learning.

Selecting the Partial State Abstractions of MDPs:
A Metareasoning Approach with Deep Reinforcement Learning
Samer B. Nashed1∗, Justin Svegliato2∗, Abhinav Bhatia1, Stuart Russell2, Shlomo Zilberstein1
Abstract — Markov decision processes (MDPs) are a common
general-purpose model used in robotics for representing se-
quential decision-making problems. Given the complexity of
robotics applications, a popular approach for approximately
solving MDPs relies on state aggregation to reduce the size of
the state space but at the expense of policy fidelity—offering
a trade-off between policy quality and computation time.
Naturally, this poses a challenging metareasoning problem:
how can an autonomous system dynamically select different
state abstractions that optimize this trade-off as it operates
online? In this paper, we formalize this metareasoning problem
with a notion of time-dependent utility and solve it using deep
reinforcement learning. To do this, we develop several general,
cheap heuristics that summarize the reward structure and
transition topology of the MDP at hand to serve as effective
features. Empirically, we demonstrate that our metareasoning
approach outperforms several baseline approaches and a strong
heuristic approach on a standard benchmark domain.
I. I NTRODUCTION
MDPs are a common general-purpose model used in
robotics for representing sequential decision-making prob-
lems [23]. However, the complexity of solving MDPs scales
poorly with the number of features reasoned about in the
environment, limiting their applicability. To address this
limitation, a range of approximate solvers for MDPs have
been proposed that seek to trade a small reduction in policy
quality for a large reduction in computation time.
A particularly effective approximate solver for MDPs,
recently proposed by Nashed et al. [17], solves a sequence
of partially abstract MDPs in order to solve an MDP. In
a partially abstract MDP, some states are considered at
maximum fidelity while other states are considered at lower
fidelity using an abstract representation. This can greatly
reduce the size of the state space while still resulting in a near
optimal policy by using a detailed representation for states
where it is most necessary. Still, for a partially abstract MDP
to be effective, it requires a suitable abstraction function that
maps a state in an MDP to an abstract state in an abstract
MDP. Since there has been substantial work on generating
abstraction functions for planners, ranging from symbolic
planners [6], [27], [13] to stochastic planners [1], [7], [26],
this paper assumes that a suitable abstraction function already
exists via either learning or careful expert design.
Given a specific abstraction function, a partially abstract
MDP uses an expansion strategy to determine which states
This work was supported by NSF grants IIS-1813490 and IIS-1954782.
∗Both authors contributed equally.
1University of Massachusetts, Amherst, MA, USA. Emails: {snashed,
abhinavbhati, shlomo }@cs.umass.edu
2University of California, Berkeley, CA, USA. Emails: {jsvegliato,
russell }@berkeley.edu
Fig. 1. Two partially abstract MDPs that were constructed using different
expansion strategies: a cheap expansion strategy that often results in lower
policy quality ( top) and an expensive expansion strategy that often results in
higher policy quality ( bottom ). Small circles are ground states, large circles
are abstract states, and arrows are transitions between states. The red ground
state is the current state, the green ground states have high reward, and the
green abstract states contain a ground state that has high reward.
of the MDP to reason about at maximum fidelity and
which states of the MDP to reason about at lower fidelity.
Concretely, this means that the expansion strategy selects
the abstract states to expand in the partially abstract MDP.
Ideally, any approach to selecting an expansion strategy
(illustrated in Fig. 1) should optimize a formal notion of
time-dependent utility by managing the trade-off between
policy quality and computation time given the domain of
operation and the available computational resources. Most
importantly, such an approach should generalize to any MDP
and require little knowledge in the details of that MDP.
In this paper, we therefore (1) offer a metareasoning
approach to selecting different expansion strategies that
optimizes a formal notion of time-dependent utility and (2)
express it as deep reinforcement learning problem. Moreover,
we (3) propose several general, cheap heuristics that summa-
rize the reward structure and transition topology of the MDP
at hand to generate effective features for deep reinforcement
learning. Empirically, we demonstrate that our metareasoning
approach outperforms several baseline approaches and a
strong heuristic approach on a standard benchmark domain.
II. R ELATED WORK
There are many approaches to approximately solving
MDPs, including performing dynamic programming, using
partial policies, and employing state abstractions. See [17]
for a thorough discussion of how these approaches relate
to partially abstract MDPs. Similarly, reinforcement learning
with time-dependent utility, introduced by Horvitz [10], has
been used in a variety of metareasoning problems [24], [22],
[4]. Although our approach uses reinforcement learning and
time-dependent utility similar to this work, we focus on a
different problem—that of learning how to select different
expansion strategies dynamically online during operation.
Online planning and learning over different abstractions
in general is a common problem across many areas of
artificial intelligence and encompasses several related sub-
problems. These include handling the non-Markovian nature
of state and action abstractions [2], learning context-specific
independences present in certain tasks [5], and learning
temporal abstractions in the form of progressively more
abstract skill controllers [14]. However, here, we restrict our
attention to state abstractions in the form of state aggregation
where multiple states in a larger (ground) problem form a
single state in a smaller (abstract) problem.
Online selection of state abstractions has been studied in
the context of both reinforcement learning and planning.
In reinforcement learning, abstractions are generally used if
the state space is large and training data is sparse, which
leads to poor experiential coverage. Methods include learning
the best state abstraction from a set of state abstractions
via hypothesis testing [12] and dynamically selecting state
abstractions of increasing granularities based on confidence
intervals of Q-values [25]. In planning, similar techniques
have been to applied to sample-based tree search algorithms.
For example, the PARSS algorithm is an algorithm that
adjusts state abstractions during tree search by starting with
coarse state abstractions and refining them given the variance
of the Q-values over actions at a specific abstract state [11].
An extensive body of research has investigated reasoning
over state abstractions during planning. Early work proposed
a hierarchy of state abstractions, represented as factored
semi-MDPs, that may have multiple intermediate state ab-
stractions that can be swapped in and out depending on
the environment [21]. Later work proposed algorithms for
dynamically eliminating state factors in states that were
estimated to not impact the policy by comparing two partially
abstract policies made with different state abstractions [3].
Finally, there have been specific applications, such as multi-
agent planning, where specialized partition schemes have
been introduced and adapted to an online setting [16].
This paper proposes a metareasoning framework that takes
advantage of powerful deep reinforcement learning methods
to learn a policy for selecting different expansion strategies.
Following work on SSPs, we use general, cheap heuristic
features that avoid relying on the specifics of an MDP. Most
importantly, we define this as a metareasoning problem that
optimizes a formal notion of time-dependent utility.III. B ACKGROUND
In this section, we review the formal definitions of a
ground MDP, an abstract MDP, and a partially abstract MDP.
a) Ground MDPs: A ground MDP is a tuple M=
⟨S, A, T, R ⟩. The space of states is S. The space of actions
isA. The transition function T:S×A×S→[0,1]
represents the probability of reaching a state s′∈Safter
performing an action a∈Ain a state s∈S. The reward
function R:S×A→Rrepresents the immediate reward
of performing an action a∈Ain a state s∈S. A
solution is a policy π:S→Aindicating that an action
π(s)∈Ashould be performed in a state s∈S. A policy
πinduces a value function Vπ:S→Rrepresenting
the expected discounted cumulative reward Vπ(s)∈Rfor
each state s∈Sgiven a discount factor 0≤γ < 1.
An optimal policy π∗maximizes the expected discounted
cumulative reward for each state s∈Sgiven the equation
V∗(s) = max a∈A
R(s, a) +γP
s′∈ST(s, a, s′)V∗(s′)
.
b) Abstract MDPs: Specifying an abstract MDP ¯Mof
a ground MDP Mrequires two functions [15]. First, an
abstraction function ϕ:S→¯Smaps a ground state s∈S
to an abstract state ¯s∈¯S. Second, an inverse abstraction
function ϕ−1:¯S→ P(S)maps an abstract state ¯s∈¯Sto
a set of ground states S⊆ P(S), where P(S)is the power
set of S. The condition ϕ(s) = ¯s⇔s∈ϕ−1(¯s)must hold
for each ground state s∈Sand abstract state ¯s∈¯S.
An abstract MDP is a tuple ¯M=⟨¯S, A, ¯T,¯R⟩[15].
The space of abstract states is ¯S={ϕ(s)|s∈
S}such that a set of ground states Sis abstracted by
an abstraction function ϕ. The space of ground actions
isA. The abstract transition function is ¯T(¯s, a,¯s′) =P
s∈ϕ−1(¯s)ψ(s)P
s′∈ϕ−1(¯s′)T(s, a, s′). The abstract reward
function is ¯R(¯s, a) =P
s∈ϕ−1(¯s)ψ(s)R(s, a). Note that a
weighting function ψ:S→[0,1]represents the probability
of being in a ground state s∈Sin an abstract state ϕ(s)∈¯S.
c) Partially Abstract MDPs: A partially abstract MDP
˜Mcombines a ground MDP Mand an abstract MDP ¯M
as a tuple ˜M=⟨˜S, A, ˜T,˜R⟩[17]. The space of partially
abstract states is ˜S=α∪βwith a set of ground states
α={ϕ−1(¯s)|¯s∈Γ}and a set of abstract states β={¯S\Γ}
such that a set of expanded abstract states Γ⊆¯Sis expanded
by an inverse abstraction function ϕ−1. The space of ground
actions is A. The partially abstract transition function ˜T:
˜S×A×˜S→[0,1]is composed of the ground/abstract
transition functions Tand¯T:
˜T(˜s, a,˜s′) =

T(˜s, a,˜s′) if˜s∈α,˜s′∈αP
s′∈ϕ−1(˜s′)T(˜s, a, s′) if˜s∈α,˜s′∈βP
s∈ϕ−1(˜s)ψ(s)T(s, a,˜s′)if˜s∈β,˜s′∈α
¯T(˜s, a,˜s′) if˜s∈β,˜s′∈β
The partially abstract reward function ˜R:˜S×A→Ris
composed of the ground/abstract reward functions Rand¯R:
˜R(˜s, a) =(
R(˜s, a)if˜s∈α
¯R(˜s, a)if˜s∈β
IV. S ELECTING PARTIAL STATE ABSTRACTIONS
The problem of selecting an expansion strategy to de-
termine the abstract states to expand in a partially abstract
MDP involves managing the trade-off between policy quality
and computation time. We frame this as a metareasoning
problem, the main advantage being that it expresses this
trade-off in terms of time-dependent utility, providing deep
reinforcement learning with an appropriate objective. Meth-
ods for similar metareasoning problems typically use heuris-
tics based on statistical measures to manage this trade-off.
Here, we introduce the first approach that selects expansion
strategies for partially abstract MDPs decision-theoretically.
A. Metareasoning for Partial State Abstractions
We begin by introducing the metareasoning problem for
partial state abstractions. This problem requires a time-
dependent utility that represents the utility of a policy in
terms of its quality and computation time. Intuitively, a policy
of a specific quality computed in a second has higher utility
than a policy of the same quality computed in an hour. A
time-dependent utility is therefore expressed as the difference
between an intrinsic value that reflects the utility of a policy
given its quality (but not computation time) and a time cost
that reflects the utility of a policy given its computation time
(but not quality) [10]. We define this function below.
Definition 1. Given a policy of quality q∈Φand compu-
tation time t∈Ψ, atime-dependent utility U: Φ×Ψ→R
can be expressed as the difference between two functions
U(q, t) =UI(q)−UC(t)where UI: Φ→R+is the intrinsic
value andUC: Ψ→R+is the time cost .
Given this time-dependent utility, the one-step metareason-
ing problem for partial state abstractions is the problem of
selecting the abstract states to expand in a given partially
abstract MDP. Naturally, a solution to this problem must
optimize time-dependent utility: we must select the abstract
states to expand in the partially abstract MDP that balances
the quality and computation time of its resulting policy.
Formally, given a set of abstract states Γi∈ P(¯S)to expand
in a partially abstract MDP ˜Miand its resulting policy πiof
policy quality q(πi)and computation time t(πi), this one-
step metareasoning problem is as follows.
arg max
Γi∈P(¯S)U(q(πi), t(πi))
This can be challenging to solve given substantial uncertainty
over the policy πiresulting from a partially abstract MDP
˜Mithat expands the abstract states Γi∈ P(¯S).
In real-time settings, an autonomous system often lazily
plans and acts online. Hence, during operation, we assume
that the autonomous system is either (1) executing an old
local policy πwhen it encounters a visited current state sor
(2) solving for a new local policy π′when it encounters an
unvisited current state s′. We can therefore view the union
of each local policy πias a joint global policy πΥas in
our recent work [17] that grows in quality and computation
time with each local policy πi. Intuitively, this presents asequential metareasoning problem for selecting the abstract
states to expand in a sequence of partially abstract MDPs
where the resulting local policies πiof each partially abstract
MDP ˜Mitogether compose a joint global policy πΥthat
must optimize time-dependent utility. Formally, given the
abstract states Υ = [Γ 1, . . . , Γh]expanded in a sequence
of partially abstract MDPs [˜M1, . . . , ˜Mh]over the unvisited
states{s1, . . . s h} ∈Shand the joint global policy πΥof
quality q(πΥ)and computation time t(πΥ), this sequential
metareasoning problem is as follows.
arg max
ΥU(q(πΥ), t(πΥ))
In practice, it is often beneficial to approximate this
sequential metareasoning problem as a sequence of inde-
pendent one-step metareasoning problems as follows.
arg max
Γ1∈P(¯S)U(q(π1), t(π1)) +···+ arg max
Γh∈P(¯S)U(q(πh), t(πh))
B. Reinforcement Learning for Partial State Abstractions
We now cast the sequential metareasoning problem for
partial state abstractions as an MDP. Each time an unvisited
state si∈Sis encountered, the MDP must select the
abstract states Γito expand in the partially abstract MDP ˜Mi.
Intuitively, the states include the quality and computation
time of the current joint global policy along with the reward
structure and transition topology of the ground MDP and
abstract MDP while the actions include expansion strategies
that select the abstract states to expand in the partially
abstract MDP. We define this metareasoning problem below.
Definition 2. The sequential metareasoning problem for
partial state abstractions is a tuple ⟨Φ,Ψ, F,ˆS,ˆA,ˆT,ˆR⟩
given a ground MDP Mand an abstract MDP ¯M:
•Φ ={q0, q1, . . . , q NΦ}is a set of qualities.
•Ψ ={t0, t1, . . . , t NΨ}is a set of computation times.
•F=F0×F1× ··· × FNFis a set of features that
summarize the reward structure and transition topology
of the ground MDP Mand abstract MDP ¯M.
•ˆS= Φ×Ψ×Fis a set of states of computation: each
state s∈ˆSreflects the current joint global policy πΥ
of quality q(πΥ)∈Φand computation time t(πΥ)∈Ψ.
•ˆAis a set of actions of computation: the set of expansion
strategies that each select different abstract states Γito
expand in a partially abstract MDP ˜Mi.
•ˆT:ˆS×ˆA×ˆS→[0,1]is an unknown transition function
that represents the probability of reaching state s′=
(q′, t′, f′)∈ˆSafter performing action a∈ˆAin state
s= (q, t, f )∈ˆS.
•ˆR:ˆS×ˆA×ˆS→Ris a reward function that represents
the expected immediate reward, ˆR(s, a, s′) =U(q′, t′)−
U(q, t), of reaching state s′= (q′, t′, f′)∈ˆSafter
performing action a∈ˆAin state s= (q, t, f )∈ˆS.
Note that the reward function is consistent with the
objective of optimizing the time-dependent utility: execut-
ing a sequence of expansion strategies until a joint global
policy πΥof quality q(πΥ)∈Φand computation time
t(πΥ)∈Ψemits a cumulative reward equal to the time-
dependent utility U(q(πΥ), t(πΥ)). This is a form of reward
shaping —equivalent to emitting a reward of U(q, t)once at
the end of an episode in terms of the objective—that guides
reinforcement learning with a reward at each time step [18].
We use deep reinforcement learning to learn an optimal
metareasoning policy by performing a series of simulations
that each use an expansion strategy to select the abstract
states to expand in a sequence of partially abstract MDPs.
Here, an agent learns a policy as a neural network by
performing actions and observing rewards in the environ-
ment, making it a good fit for metareasoning for three rea-
sons. First, by balancing exploitation and exploration, it can
learn how to select an expansion strategy given the reward
structure and transition topology of the ground MDP and
abstract MDP. Next, by ignoring large unreachable regions
of the state space, it can reduce the overhead of learning
which expansion strategy to select. Finally, by using a neural
network that extracts the relationship between large input
and output spaces, it can encode the effects of an expansion
strategy on the resulting policy of a partially abstract MDP
in a way that generalizes to novel states of computation.
C. Calculating Time-Dependent Utility
Typically, in metareasoning, a solution quality qis defined
as the approximation ratio, q=c∗
c, where c∗is the cost of
the optimal solution and cis the cost of the given solution.
However, since computing the cost of an optimal solution to
a complex problem is often infeasible, a solution quality can
be estimated as the approximation ratio, q=¯c∗
c, where ¯c∗is
a lower bound on the cost of the optimal solution and cis
the cost of the given solution. Generally, a solution quality
q= 0 means no solution was computed while a solution
quality q= 1 means an optimal solution was computed.
We need a specific definition of solution quality in the
context of MDPs. Here, the quality q(π)of a policy πis
defined as the approximation ratio,
q(π) =Vπ
V∗=P
s∈Sd(s)Vπ(s)P
s∈Sd(s)V∗(s),
where Vπis the value function of the policy πandV∗is the
value function of the optimal policy π∗, given a probability
d(s)of starting in a state s∈S. However, since computing
the value of an optimal policy of a complex MDP is often
infeasible, the optimal value function V∗must be replaced
with an upper bound on the value function ¯V∗.
Given the quality q(πΥ)and computation time t(πΥ)of
the current joint global policy πΥ, we can define the time-
dependent utility U(q(πΥ), t(πΥ))using an intrinsic value
UI(q(πΥ))and a time cost UC(t(πΥ)). First, given a tunable
parameter α, we model the intrinsic value as UI(q(πΥ)) =
αq(πΥ). Second, given a tunable parameter β, we model the
time cost as UC(t(πΥ)) =P
i∈h[eβt(πi)−1]such that πiis
the local policy solved for the unvisited states {s1, . . . s h} ∈
Sh. The rates αandβare typically given in the problem
depending on the value and urgency for a policy [8].
Given this time-dependent utility, it is possible to ex-
press the reward function of the metareasoning problem.Formally, given the current state of computation s=
(q(πΥ), t(πΥ),·)∈ˆSand the successor state of computation
s′= (q(π′
Υ), t(π′
Υ),·)∈ˆSthat reflect the current joint global
policy πΥand successor joint global policy π′
Υalong with
an expansion strategy a∈ˆAused to solve for a new local
policy πthat improves the joint global policy πΥ, we can
express the reward function in the following way.
ˆR(s, a, s′) =U(q(π′
Υ), t(π′
Υ))−U(q(πΥ), t(πΥ))
=α[q(π′
Υ)−q(πΥ)]−eβt(π)
V. R EPRESENTING THE STATE OF COMPUTATION
In this section, we introduce 6 features that compose the
state of computation in the sequential metareasoning problem
for partial state abstractions. These features can easily be
computed for a ground MDP Mand abstract MDP ¯Mand
reflect their reward structure ortransition topology .
A. Reward Structure
We define 3 features below describing the availability of
immediate reward around the current ground/abstract state.
1)Reward Frequency :The feature f1is the number of
positive reward ground states reachable within hactions of
the current ground state normalized by the total number of
reachable ground states.
2)Reward Proximity :The feature f2is the minimum
number of actions required to reach the nearest positive
reward ground state from the current ground state normalized
by the diameter diam( M) of the ground MDP M.
3)Reward Information :A main weakness of state ab-
stractions is that they induce artificial information boundaries
within the state space. For example, when a set of ground
states is compressed into an abstract state, the abstract MDP
loses information about any ground state that has successor
ground states in other abstract states. This is because succes-
sor ground states may be aggregated with other ground states
that are not reachable in a single action, which is detrimental
when a ground state with high reward successor ground states
can no longer be distinguished from a ground state without
high reward successor ground states. Therefore, the feature
f3is1/(1 +|diam(¯s)−δ|), where diam( ¯s) is the diameter
of the graph of the ground states in the current abstract state
¯s∈¯Sandδis the distance to the nearest high reward ground
state such that this value approaches 1or0as these ground
states move toward or away from this boundary.
B. Transition Topology
We define 3 features below describing the local transition
topology surrounding the current ground/abstract state.
1)Transition Entropy :The feature f4is the entropy of
the abstract successor state distribution at the current abstract
state assuming that actions are selected randomly. This is a
rough measure of the probability that actions performed at
the current abstract state will transition to different abstract
states that may be worth reasoning over more closely. A
higher entropy at the current abstract state indicates a higher
probability of transitioning to different abstract states.
Fig. 2. An example of (k, h)-reachability where k= 3 andh≤4. The
green region is allstates reachable from a given ground state swithin k
actions. The yellow /redregions are the states from which the set of important
ground states SGis still reachable/unreachable within hactions.
2)State Visitation :The feature f5is the expected dis-
counted number of times that the current abstract state will be
visited given an abstract start state distribution and an optimal
abstract policy. We compute this for all abstract states ¯s′∈¯S
by performing dynamic programming with the equation
λ(¯s′) =¯d(¯s′) +γX
¯s∈¯S¯T(¯s,¯π∗(¯s),¯s′)λ(¯s),
where ¯π∗(¯s)is the optimal abstract policy and ¯d(¯s′)is the
abstract start state distribution.
3)Important Ground State Reachability :The feature f6
is a novel measure of reachability from a given ground state
to a set of nearby important ground states.
Definition 3. A set of important ground states SGis(k,h)-
reachable from a given ground state sif, after performing
any arbitrary sequence of actions a1, . . . , a k, there is at least
one important ground state sg∈SGthat is still reachable
within hactions given a probability ϵ >0.
Similar to recent work on SSPs [20], [19], this measure
establishes an envelope (illustrated in Fig. 2) of ground states
in which the probability of reaching a set of important ground
states SGfrom a given ground state sis always greater than
zero. In general, while certain MDPs may have transition
topologies that permit calculating (k, h)-reachability exactly,
it usually must be estimated. To do this, we provide a
constant time estimation procedure in Algorithm 1. Here,
the accuracy of the estimate improves with the number of
samples parameterized by nandm. We choose kto be
proportional to the diameter of the current abstract state as
this is the maximum number of actions that can performed
by the local policy solved for the current abstract state.
VI. E XPERIMENTS
We now evaluate the proposed approach (D QN) against a
set of baseline approaches on a standard benchmark domain.
a) Hypothesis: Any approach to the metareasoning
problem for partial state abstractions should try to optimize
time-dependent utility by selecting the abstract states to
expand in a sequence of partially abstract MDPs. Ideally,
the approach should identify two cases. First, there are
cases in which cheap and expensive expansion strategiesAlgorithm 1: ESTIMATE (k, h)-REACHABILITY
1:Input: An MDP M, a ground state s, a set of important ground
states SG, and the parameters k,h,n, and m
2:Output: The probability that the set of important ground states SG
are(k, h)-reachable from the ground state s
3:Sk← ∅
4:fori∈ {1, . . . , n }do
5: s′←s
6: forj∈ {1, . . . , k }do
7: s′←SIMULATE RANDOM ACTION (M, s′)
8: Sk←Sk∪ {s′}
9:σ←0,ρ← ∅
10:forsk∈Skdo
11: fori∈ {1, . . . , m }do
12: s′←sk
13: forj∈ {1, . . . , h }do
14: s′←SIMULATE RANDOM ACTION (M, s′)
15: ifs′∈SGor∃p∈ρsuch that s′∈pthen
16: σ←σ+ 1
17: ρ←ρ∪PATH(sk, s′)
18: break
19:return σ/n
result in roughly equal policy quality , reducing computation
time at negligible sacrifice to policy quality. Second, there
are cases in which expensive expansion strategies result in
much higher policy quality than a cheap expansion strategy,
boosting policy quality at marginal amortized computation
time. Our hypothesis is that the proposed approach will
learn to exploit these two cases and hence optimize time-
dependent utility beyond the baseline approaches.
b) Experimental Setup: All approaches were evaluated
on100random simulations. For each simulation, we record
three metrics: the values for the policy quality ,computation
time, and time-dependent utility of the final policy. The
proposed approach was trained on1000 random simulations
using deep Q-learning with standard settings. The neural
network has two hidden layers of 64and32nodes with ReLU
activation and a linear output layer of 3nodes. The step
size is 0.0001 . The exploration strategy is ϵ-greedy action
selection with an exploration probability ϵthat is annealed
from 1to0.1over 1000 episodes. The experience buffer
capacity is ∞. The number of steps is 20000 . The buffer ini-
tialization period is 200. The target network update interval
is1000 . The minibatch size is 64. All simulations for training
and evaluation were generated using different randomization
seeds to measure generalizability to unfamiliar simulations.
c) Standard Benchmark Domain: We consider the
Earth observation domain proposed in early work on ground
MDPs [9] and recently modified in recent work on partially
abstract MDPs [17]. In this domain, a satellite orbiting Earth
indefinitely must take photos of points of interest Pwith
weather levels Wthat change stochastically. The satellite
starts at longitude x∈Xwith its camera focused at latitude
y∈Y. Given the rates ∆Yand∆X, the satellite can
then either do N OOPERATION , shift its camera N ORTH to
latitude (y+ ∆ Y)∈Y, shift its camera S OUTH to latitude
(y−∆Y)∈Y, or take an I MAGE of a point of interest at
latitude y∈Yand longitude x∈X. Concurrent to each
action, the satellite orbits from east to west described by
Naive Greedy Proactive Hybrid DQN
Approach0.00.51.0Policy Quality
Computation Time [seconds]
Time-Dependent UtilityFig. 3. Left: The policy quality of the final policy relative to the optimal policy over all evaluation simulations for each approach. Center: The frequency
of computation times of the final policy over all evaluation simulations for each approach. Right: The distribution of time-dependent utilities of the final
policy over all evaluation simulations for each approach. Putting these figures together, we observe that the proposed approach optimizes time-dependent
utility more effectively than the baseline approaches by learning how to manage the trade-off between policy quality and computation time.
Fig. 4. An example policy for the proposed approach that selects expansion
strategies. There are eight abstract states that each contain 2×4ground states
where hatch marks denote a point of interest. Each band within an abstract
state represents a specific expansion strategy: blue for N AIVE ,orange for
GREEDY , and green for P ROACTIVE such that darker shading denotes
higher probability. This policy shows how the proposed approach exploits
reward structure and transition topology of the earth observation MDP to
dynamically select the expansion strategy optimizing time-dependent utility.
longitude ((x+ ∆ X) mod |X|)∈Xwhere the modulo
operator creates periodic boundary conditions to represent
continuous orbits around earth. Most importantly, given the
IMAGE action, the satellite earns a reward proportional to
image quality such that image quality is a function of the
weather w∈W. The formal definitions of the ground,
abstract, and partially abstract MDPs are in recent work [17].
d) Baseline Approaches: We consider pure and hybrid
approaches that expand the current abstract state anda set of
informative abstract states . The N AIVE approach expands no
informative abstract states. The G REEDY approach expands
informative abstract states that contain a point of interest
within 1abstract state of the current abstract state. The
PROACTIVE approach expands informative abstract states
that are reachable from the current abstract state to any
abstract state that contains a point of interest within 2abstract
states of the current abstract state. The H YBRID approach
uses either the N AIVE , GREEDY , or P ROACTIVE approach
depending on the (k, h)-reachability of the current ground
state and the occupancy frequencies of the abstract MDP.
e) Experimental Results: Fig. 3 shows that the pro-
posed approach optimizes time-dependent utility beyond the
baseline approaches. First, N AIVE , GREEDY , and P ROAC -
TIVE exhibit poor time-dependent utility ( 29.6,36.6,32.2).
This is because they lead to either high policy quality in too
much computation time or low policy quality. Next, H YBRID
exhibits better time-dependent utility ( 40.7) by heuristically
reasoning over expansion strategies via careful expert design.
Finally, D QNexhibits the best time-dependent utility ( 44.7)
by performing explicit optimization using deep reinforce-
ment learning. Overall, the proposed approach decision-
theoretically selects expansion strategies (like in Fig. 4)
based on whether investing computation time would result
in a worthwhile improvement in policy quality.VII. C ONCLUSION
This paper introduces the metareasoning problem of se-
lecting different expansion strategies and solves it using deep
reinforcement learning with several general, cheap heuristics
that reflect the MDP at hand. Empirically, we show that
our metareasoning approach outperforms several baseline
approaches and a strong heuristic approach on a standard
benchmark domain. In future work, we will explore the
generalizability of this work to MDPs of varied topologies.
REFERENCES
[1] D. Abel, D. Arumugam, L. Lehnert, and M. Littman. State abstractions for
lifelong reinforcement learning. In ICML , 2018.
[2] A. Bai, S. Srivastava, and S. J. Russell. Markovian state and action abstractions
for MDPs via hierarchical MCTS. In IJCAI , 2016.
[3] J. Baum, A. E. Nicholson, and T. I. Dix. Proximity-based non-uniform
abstractions for approximate planning. JAIR , 43, 2012.
[4] A. Bhatia, J. Svegliato, S. B. Nashed, and S. Zilberstein. Tuning the hyperparam-
eters of anytime planning: A metareasoning approach with deep reinforcement
learning. In ICAPS , 2022.
[5] R. Chitnis, T. Silver, B. Kim, et al. CAMPS: Learning context-specific
abstractions for efficient planning. arXiv:2007.13202 , 2020.
[6] N. S. Flann. Learning appropriate abstractions for planning in formation
problems. In 6th ICML , 1989.
[7] X. Fu, G. Yang, P. Agrawal, and T. Jaakkola. Learning task informed abstractions.
In38th ICML , 2021.
[8] E. A. Hansen and S. Zilberstein. Monitoring and control of anytime algorithms.
AIJ, 126(1-2):139–157, 2001.
[9] A. Hertle, C. Dornhege, T. Keller, R. Mattm ¨uller, et al. An experimental
comparison of classical, fond and probabilistic planning. In KI, 2014.
[10] E. Horvitz and G. Rutledge. Time-dependent utility and action under uncertainty.
In7th UAI , 1991.
[11] J. Hostetler, A. Fern, and T. Dietterich. Sample-based tree search with fixed and
adaptive state abstractions. JAIR , 60, 2017.
[12] N. Jiang, A. Kulesza, and S. Singh. Abstraction selection in model-based
reinforcement learning. In ICML , 2015.
[13] C. A. Knoblock. Generating abstractions for planning. AIJ, 68(2), 1994.
[14] G. Konidaris, S. Kuindersma, R. Grupen, and A. Barto. Robot learning from
demonstration by constructing skill trees. IJR, 31(3), 2012.
[15] L. Li, T. J. Walsh, and M. L. Littman. Towards a unified theory of state abstraction
for MDPs. In ISAIM , 2006.
[16] A. Ma, M. Ouimet, and J. Cort ´es. Dynamic domain reduction for multi-agent
planning. In MRS , 2017.
[17] S. B. Nashed, J. Svegliato, M. Brucato, C. Basich, R. Grupen, and S. Zilberstein.
Solving Markov decision processes with partial state abstractions. In ICRA , 2021.
[18] A. Y . Ng, D. Harada, and S. J. Russell. Policy invariance under reward
transformations. In ICML , 1999.
[19] L. Pineda and S. Zilberstein. Soft labeling in stochastic shortest path problems.
In18th AAMAS , 2019.
[20] L. E. Pineda, K. H. Wray, and S. Zilberstein. Fast SSP solvers using short-sighted
labeling. In 31st AAAI , 2017.
[21] K. Steinkraus and L. P. Kaelbling. Combining dynamic abstractions in large
MDPs. Technical report, MIT, 2004.
[22] J. Svegliato, P. Sharma, and S. Zilberstein. A model-free approach to meta-level
control of anytime algorithms. In ICRA , 2020.
[23] J. Svegliato, K. H. Wray, S. J. Witwicki, J. Biswas, and S. Zilberstein. Belief
space metareasoning for exception recovery. In IROS , 2019.
[24] J. Svegliato, K. H. Wray, and S. Zilberstein. Meta-level control of anytime
algorithms with online performance prediction. In 27th IJCAI , 2018.
[25] M. Tamassia, F. Zambetta, W. L. Raffe, F. Mueller, and X. Li. Dynamic choice
of state abstraction in Q-learning. In ECAI , 2016.
[26] M. Tomar, A. Zhang, R. Calandra, M. E. Taylor, and J. Pineau. Model-invariant
state abstractions for model-based RL. arXiv:2102.09850 , 2021.
[27] A. Unruh and P. S. Rosenbloom. Abstraction in problem solving and learning.
In11th IJCAI , 1989.

Samer B
Nashed*
Justin Svegliato*
Abhinav Bhatia
Shlomo Zilberstein
Stuart Russell


Human biases limit cumulative innovation.

royalsocietypublishing.org/journal/rspb
Research
Cite this article: Thompson B, Griffiths TL.
2021 Human biases limit cumulativeinnovation. Proc. R. Soc. B 288: 20202752.
https://doi.org/10.1098/rspb.2020.2752
Received: 3 November 2020
Accepted: 8 February 2021
Subject Category:
Behaviour
Subject Areas:
cognition, behaviour
Keywords:
cultural evolution, innovation, functionlearning, inductive bias, Bayesian, optimization
Author for correspondence:
Bill Thompsone-mail: wdt@princeton.eduHuman biases limit cumulative innovation
Bill Thompson and Thomas L. Griffiths
Departments of Psychology and Computer Science, Princeton University, Princeton, NJ 08544, USA
BT, 0000-0001-7348-3531
Is technological advancement constrained by biases in human cognition?
People in all societies build on discoveries inherited from previous gener-ations, leading to cumulative innovation. However, biases in human
learning and memory may influence the process of knowledge transmission,
potentially limiting this process. Here, we show that cumulative innovationin a continuous optimization problem is systematically constrained by
human biases. In a large ( n= 1250) behavioural study using a transmission
chain design, participants searched for virtual technologies in one of fourenvironments after inheriting a solution from previous generations. Partici-
pants converged on worse solutions in environments misaligned with
their biases. These results substantiate a mathematical model of cumulativeinnovation in Bayesian agents, highlighting formal relationships between
cultural evolution and distributed stochastic optimization. Our findings pro-
vide experimental evidence that human biases can limit the advancement ofknowledge in a controlled laboratory setting, reinforcing concerns about bias
in creative, scientific and educational contexts.
1. Introduction
All human societies depend on technology [1] —tools, concepts, skills and subsis-
tence techniques that help people solve physical and computational problems.One of the most distinctive aspects of human technology is that many innovations
we rely on every day would be ‘unlikely or impossible for us to have achieved by
ourselves ’[2]. From kayaks to cryptographic hash functions, efficient solutions to
environmental challenges depend on cumulative innovation :‘in most times and
places, individuals don ’t invent tools; tools evolve gradually ’[3] as innovations
are transmitted from one generation to the next. This process of transmissionand accumulation of knowledge is called cumulative cultural evolution (CCE) [4].
Here, we examine whether biases in human learning and memory can influ-
ence the outcome of cumulative innovation in a large behavioural experiment.Mathematical models suggest that efficient accumulation of knowledge depends
on high-fidelity transmission [5] of successful innovations [6] to new individuals
who themselves may innovate further [7] —the so-called ratchet effect [8]. Aspects
of this process can be studied experimentally. Behavioural studies (reviewed in [9])
show that cumulative innovation under controlled laboratory conditions allows
experimental micro-societies to discover increasingly functional solutions to pro-blem-solving and optimization tasks in both physical [10 –14] and virtual [15 –18]
settings (e.g. paper aeroplane [10] or virtual arrowhead [16] design tasks).
Performance depends on factors such as population structure [11,19 –21] and
transmission dynamics [22], as predicted by models [17,23,24].
However, theories of cultural evolution [6,25] also suggest that the outcome
of cumulative innovation may be constrained by a more fundamental problem.Imagine trying to construct a canoe after travelling in one built by somebody
else. In many contexts, the transmission of knowledge from person to person
is an imperfect process relying on inference, estimation, and reconstruction ofideas [25]. This introduces a problem of induction —the need to generalize
beyond what is observable —that can only be solved by inductive bias [26].
Bias is a requirement for inference whenever aspects of a learning problem arenot fully determined by the data observed. For instance, the structure and par-
ameters of tools, artefacts or subsistence techniques cannot be directly
© 2021 The Author(s) Published by the Royal Society. All rights reserved.
 Downloaded from https://royalsocietypublishing.org/ on 07 April 2021

communicated from one mind to another: instead, these
design features must often be inferred from limited experi-
ence, incomplete description, noisy perceptual information,
or potentially inaccurate or partial memories.
Inductive biases can also play an important role in instances
of ambiguous or conflicting social testimony. For example,
Henrich et al. [27] showed that human biases in category-
based induction may help to explain observed patterns of
culturally evolved adaptive food taboos in a Fijian population.
However, in specific cases where cultural information is indeci-sive, biases in categorization and similarity-based inferences
can lead to non-adaptive taboos [27]. Cases like this exemplify
the potential for inductive biases to influence decision-makingand learning in functional settings when information trans-
mission is limited. While teaching [28 –31] and other
pragmatic mechanisms for cooperative inference [32,33] canimprove information transmission, inferential bottlenecks [3]
create the opportunity for biases in human cognition to repeat-
edly influence the outcome of cultural transmission, raising thequestion of their long-term consequences.
Cognitive approaches to cultural evolution [25,34,35] have
emphasized this possibility: ‘Human cognitive abilities act
[…] as a filter on the representations capable or likely to be
widely distributed in human populations ’[25]. As a result
of the need for inductive generalization, artefact designs,tool-use techniques, behavioural repertoires, or technical con-
cepts which people find difficult to remember or understand
are systematically less likely to survive the process of culturaltransmission. Instead, the process may favour decisions, con-
cepts, or design choices that people find intuitive. This
perspective is supported by formal models [34 –36]. These
dynamics may arise as a consequence of inductive biases
that are themselves the result of development and learning
[37], or from innate or adaptive features of human cognition[38]. Behavioural studies show that human biases can influ-
ence the transmission of conventional behaviours such as
artificial languages [39], signalling systems [40], stereotypes[41], motor programs [42] and musical rhythms [43]. Cultural
transmission by serial reproduction (SR) can even be viewed
abstractly as an efficient algorithm for revealing cognitivebiases [44].
However, the impact of human biases has been difficult to
assess in functional settings. Understanding these effects posesa challenge for theories of technology [3,45,46] and for exper-
imental studies of cumulative innovation in structured
problem-solving settings, which can be difficult to quantify
in cognitive terms [4,46 –49]. Studies of innovation in simpler
optimization paradigms offer the potential for more continuity.
People are excellent at solving optimization problems as indi-
viduals [50], networked groups [51] and cultural populations[24,52], often outperforming advanced algorithms [50,53].
Studies have focused on how social learning strategies
(reviewed in [54]) influence the potential for accumulation inmulti-modal, continuous optimization landscapes [22], and
on causal understanding [24] for example. However, the poten-
tial for human inductive biases to influence these dynamicsremains unknown, because studies often implement direct
transfer of solutions between individuals. Without a quanti-
tative theory relating psychological biases to inference andcumulative innovation in a specific experimental setting,
these basic predictions have been difficult to test [4,55].
To help assess the impact of cognitive biases on CCE, we
developed a probabilistic cognitive model of learning and
innovation in chains of Bayesian learners. The model pro-
vides insight into functional and cognitive perspectives byrevealing a formal connection between CCE and distributed
optimization. Our analysis predicts that the biases of learners
should limit cumulative innovation in continuous optimiz-ation problems. To test this prediction, we conducted a
large behavioural experiment in which transmission chains
of participants attempted to solve a multidimensional con-tinuous optimization problem. Previous studies focused on
complex functional landscapes [22,52] and learning from
multiple models [16]. By contrast, our study used a trans-mission chain design (figure 1), and we constructed a
simple hill-shaped landscape whose optimum should be
easy to discover (figure 2). These simplifications allowed usto examine the biases of our participants that emerge directly
as a consequence of the process of knowledge transmission.
Our key manipulation was to transpose the abstract
optimization problem into four virtual environments, each
defined by a different optimal solution to the same under-
lying problem. In this setting, the relevant biases arereflected in the kinds of solutions that people are likely to
consider spontaneously or intuitively, before observing how
other people solve the task or engaging in within-taskenvironment 2
optimum:environment 1
optimum:environment 4
optimum:environment 3
optimum:
n = 250 n = 250 n = 250 n = 250chain seed chain seed chain seed chain seed
Figure 1. Transmission chain design. Participants were organized into transmission chains of 10 generations . A participant at generation tinherited the arrowhead
designed by the participant at generation t−1 of their chain. The principle experimental manipulation was the design of the highest-scoring virtual arrowhead
(illustrated above), which varied between treatments (environments E1 –E4). Each treatment was replicated in 25 independent chains. (Online version in colour.)royalsocietypublishing.org/journal/rspb Proc. R. Soc. B 288: 202027522
 Downloaded from https://royalsocietypublishing.org/ on 07 April 2021

learning. We estimated our participants ’biases in the task
using a SR paradigm, allowing us to independently measurethe consistency between these biases and each of the optimal
solutions in the four virtual environments. We then examined
whether information accumulation was equally effective inall four environments, or whether information accumulation
was constrained in environments less consistent with our
participants ’biases.
2. Model
One way to connect models of cultural transmission with cog-
nitive theories of intelligence is to analyse populations of
Bayesian learners [56]. Some of the earliest models of cultural
transmission were based on this approach [6]. Bayesian modelsnaturally express the cognitive structures important to theories
of cultural transmission, such as technical knowledge [57],
cognitive attractors [45], latent solutions [58], learning biases[36], adaptive instincts [38,59] and content biases [6]. These
inductive biases are expressed by a prior distribution over a
design space of possible solutions to a problem. Intuitiveor learnable solutions have high prior probability. Coun-
terintuitive solutions have lower prior probability. A prior
distribution which assigns differential probabilities to solutionsencodes inductive bias. For example, a simplicity bias [60]
assigns more complex solutions lower probability. More struc-
tured or domain-specific priors can result from conceptualor semantic knowledge, such as intuitive physics [61], or
inductive biases for particular structural forms [62].
Bayesian models have been instrumental in understanding
how people ’s inductive biases can shape SR [63] and iterated
learning [56]. These models have focused on the emergence
of conventional behaviours [36,64,65], particularly humanlanguage [36,39,56,66]. We used this probabilistic paradigm
to analyse CCE in populations of individuals attempting to
solve an optimization problem and transmitting their solutionsto new learners. Following influential models of CCE
[6,7,34,67], we analysed transmission of a real-valued solution
θ. This allowed us to view CCE in terms of two components:inductive biases, expressed as ap r i o r distribution p(θ); and the
underlying optimization problem, expressed as a fitness land-scape or objective function f(θ). We examined the dynamics
of cultural transmission under the assumption that the relevant
prior distribution is Gaussian with mean μand variance δ
2,t h a t
perception of transmitted designs is subject to Gaussian noise
(with variance σ2) and that innovation can be expressed as
incremental steps along the gradient ruf(u) of the fitness land-
scape (see Methods).
(a) Predictions
The mathematical framework that descends from combiningthese cognitive and functional assumptions has a surpris-ingly familiar structure. A transmission chain of individuals
solving a continuous optimization problem through social
learning by Bayesian inference and individual learningthrough bounded innovation has dynamics:
utþ1¼ut/C0aruf(ut)þl(m/C0ut)þetþ1,( 2 :1)
where etþ1is zero-mean Gaussian noise, αis an innovation
rate parameter, and λ=σ2/(σ2+δ2). Equation (2.1) describes
a form of stochastic gradient descent —an optimization
algorithm at the heart of modern artificial intelligence andmachine learning [68]. This connection provides a formal
basis for the view that CCE can be interpreted as a form of
distributed computation [69], while also offering insightinto the relationship between function and cognition —
in equation (2.1), human inductive biases act as a form of
regularization . The balance of transmission fidelity and
bias strength plays the role of a regularization parameter,
determining the impact of human biases on the process
(see Methods).
We calculated the equilibrium behaviour of this process in
hill-climbing problems (see Methods), and found that it con-
verges towards a compromise between the mean μof the
prior distribution p(θ) and the optimum θ* of the objective
function f(θ). This compromise can be written as
E
t!1[ut]¼mþb(u/C3/C0m): (2:2)score landscapescore
(calories)
base width
(pixels)base width
(pixels)length
(pixels)
length
(pixels)stimulus design space
900
800
700
600
500
400
1401201008060 6080100120140(a)( b)
Figure 2. Optimization problem. ( a) Illustration of the design space of possible arrowheads implied by the ability to manipulate base width and length (between 50
and 150 pixels). Arrowheads are regularly sampled points for illustration purposes in a continuous underlying space of possible designs. ( b) The objective function
(score landscape) f(θ). Yellow denotes higher scores, blue denotes lower scores. The score landscape was a simple, smooth, hill-shaped function (quadratic surface)
with maximum at θ* (here θ*bw= 115, θ*l= 115, illustrating environment E4). Across environments, the position of the optimum design ( θ*) varied but the
shape of the landscape remained constant and as displayed here. (Online version in colour.)royalsocietypublishing.org/journal/rspb Proc. R. Soc. B 288: 202027523
 Downloaded from https://royalsocietypublishing.org/ on 07 April 2021

Here, 0 ≤β≤1 interpolates between processes dominated
by function, converging towards optimal solutions ( β→1),
and processes dominated by inductive biases ( β→0)
converging towards intuitive solutions closer to μ. This
interpolation parameter is determined by a principled
relationship between bias strength, transmission fidelity, the
rate of innovation, and the steepness of the fitness function(see Methods).
The general result in equation (2.2) shows that previous
analyses of CCE [34,67] and SR [70] can be recapitulated asspecial cases. Equation (2.2) predicts that participants transmit-
ting solutions to a continuous optimization problem will
discover a compromise between the mean μof the prior p(θ)
and the optimum design θ*. Task success should be higher in
environments more consistent with participants ’biases.
3. Results
To test these predictions, we conducted a pre-registered
(https://osf.io/hx9jd/) behavioural study using a transmissionchain design (see figure 1 and Methods). Participants completed
an optimization task after briefly observing the solution pro-
duced by the previous participant and reconstructing it frommemory. We adapted an influential paradigm developed by
Mesoudi and colleagues [16,17,22,71,72]. Participants designed
virtual arrowheads (figure 2) to earn as many calories of food
as possible on a virtual hunt by modifying continuous proper-
ties of its design (the width of the arrowhead ’s base and its
overall length, see figure 2 a). Different combinations of arrow-
head features (base width = θ
bw,l e n g t h= θl)l e a dt od i f f e r e n t
scores determined by the landscape f(θ)i nf i g u r e2 .N o t et h a t
our goal is not to assess human biases in general for this task,which are likely to vary across populations [27]. Our aim is
to estimate the biases of this specific population and predict
their impact.
(a) Estimated biases
Participants ( n= 250, 25 chains of 10 generations) assigned to
the SR treatment completed a simplified version of the task(reconstruction of an inherited arrowhead design after brief
observation, but no individual learning). Previous research
[73] shows that SR results in samples from the prior p(θ). We
used this connection to estimate people ’s biases (see Methods)
about the stimulus design space (figure 2). The sample-mean
arrowhead design can be taken as an estimate ( ^
m) of the
mean μof the prior distribution —the prototype arrowhead
design. Figure 3 illustrates the estimated prototype. In this
figure, faint depictions of all 250 arrowheads produced by par-ticipants in the SR treatment are overlaid uniformly. The
position of this prototype in the design space is also shown
in figure 3 (inset). The prototype design is close to the centreof the design space, but is statistically significantly longer
(M= 107.5, Z= 20 187, p< 0.001, Wilcoxon signed-rank test)
and wider ( M= 103.2, Z= 16 874, p= 0.025) than would be
expected under random sampling.
(b) Task success
Participants ( n= 1000) assigned to optimization treatments
(E1–E4) first reconstructed the inherited design before complet-
ing four individual learning trials (practice hunts )i nw h i c h
they tested nearby variations on their arrowhead and gainedperformance feedback. All four environments were deter-
mined by the landscape f(θ) shown in figure 2. However, we
manipulated the location of the optimum arrowhead design
(peak of the hill). Optimum designs are shown in figure 1.
We calculated the consistency of each environment with
human biases by measuring the absolute design space distancebetween the optimum design ( θ*) and the prototype ( ^
m).
Larger distance means the optimum is less consistent with
human biases. Environments are ranked in order of least tomost consistent: E1 is least consistent ( j
u/C3
bw/C0^mbwj¼28:2,
ju/C3
l/C0^mlj¼32:5); E2 ( ju/C3
bw/C0^mbwj¼11:75,ju/C3
l/C0^mlj¼32:5)
and E3 ( ju/C3
bw/C0^mbwj¼28:22,ju/C3
l/C0^mlj¼7:5) are more
consistent, E4 is most consistent ( ju/C3
bw/C0^mbwj¼11:75,
ju/C3
l/C0^mlj¼7:5).
Figure 4 shows task success (produced arrowhead scores)
across treatments. Participants assigned to E4 achieved the
highest scores ( M= 950, s.d.=66), followed by E3 ( M= 934,
s.d. = 78) and E2 ( M= 883, s.d.=111). Participants assigned
to the E1 treatment had least success ( M= 856, s.d. = 151).
A pre-registered one-way analysis of variance showed that
the effect of environment on task success was significant(F(3, 996) = 41.66, p< 0.0001). However, a Levene test for
homogeneity of variance showed that the assumption of
equal variances was not met. A non-parametric alternative(the Kruskal –Wallis test by rank) confirmed that the effect
of environment on task success was significant ( H(3) =
113.4, p< 0.001). Pairwise comparisons using Wilcoxon ’s
rank-sum test with Bonferroni adjustments showed that
participants in the E4 treatment achieved significantly
higher task success than participants in treatments E3(p= 0.006), E2 ( p< 0.0001) and E1 ( p< 0.0001). Participants
in the E3 treatment achieved significantly higher task
success than participants in treatments E2 ( p< 0.001) and
E1 ( p< 0.0001). There was no statistically significant differ-
ence in task success between participants in the E1 and E2
treatments ( p= 1).
Figure 3. Experimental results (bias estimate). Visualization of the sample-
based estimate of the mean ^mof the prior distribution p(θ) implicitly
assumed by participants. Plot shows arrowheads produced by all n= 250
participants in the serial reproduction treatment faintly overlaid, illustrating
^m. Inset shows prior mean arrowhead ( ^m, black) positioned in design
space relative to optimum arrowheads in optimization treatments E1 –E4.
(Online version in colour.)royalsocietypublishing.org/journal/rspb Proc. R. Soc. B 288: 202027524
 Downloaded from https://royalsocietypublishing.org/ on 07 April 2021

(c) Arrowheads discovered
The main prediction of our formal analysis is that people will
converge on a linear compromise between the prototype ^m
and the optimum θ*. We performed a pre-registered
regression analysis to test this prediction. We analysed base
width and length separately. First, we performed a regressionanalysis with arrowhead base width as dependent variable
and the difference between the prototype and the optimum
(^
mbw/C0u/C3
bw) as predictor. Our formal hypothesis was sup-
ported. Accounting for the mean of the prior ( β= 0.98, p<
0.0001) and the difference between the prior and optimum
(β= 0.33, p< 0.0001) accounted for 97% of the variance in
arrowhead base width (adjusted R2= 0.966). Second, we per-
formed the same analysis with arrowhead length as the
dependent variable. Accounting for the mean of the prior(β= 1., p< 0.0001) and the difference between prior and opti-
mum ( β= 0.34, p< 0.0001) accounted for 97% of the variance
in arrowhead length (adjusted R
2= 0.968).
(d) Exploratory analyses
We first established that optimization chains had converged
to an equilibrium state. Regression analyses of the optimiz-
ation treatments with environment, generation, and chainid as predictors showed generation was not a statistically
significant predictor of task success ( β= 1.43, p= 0.45), arrow-
head base width ( β=−0.51, p= 0.12), or length ( β= 0.16,
p= 0.65) after generation three, indicating convergence
(figure 5 b). We then examined the contributions of bias and
individual learning to task success. Figure 5 shows expectedtask success under a random walk over the fitness landscape
(dot-dash lines). We quantified this expectation by numerical
integration over the fitness landscape within the range of thedesign space, which provides an approximation to the aver-
age task success that would be achieved by selecting an
arrowhead design uniformly at random under the constraintsof the task in each environment without transmission or inno-
vation. This baseline helps to contextualize task success rates
achieved through transmission and innovation.As an additional reference point, we also examined task
success rates under a model that includes innovation but not
biased learning. Our mathematical analysis showed that
unbiased learning via Bayesian inference ( δ
2→∞) in the con-
text of gradient-based innovation predicts discovery of the
optimal solution (see equation (2.2) and Methods). Simulation
of a simpler model of innovation supports this prediction.We simulated the experimental conditions (25 chains of 10
generations per treatment) under a model that assumes indi-
viduals sample four arrowhead designs uniformly at randomwithin the bounded region of design space surrounding the
design they inherited, and transmit the highest scoring
sampled design. This process leads to convergence towardsthe optimum design in all treatments, and therefore does not
provide an alternative explanation of our findings (Kruskal –
Wallis test by rank showed no significant differences in tasksuccess between environments after generation 5 at α=0 . 0 1
in 912 of 1000 simulations of the experiment).
Figure 5 shows scores that would have been awarded to
arrowheads designed by participants in the SR treatment
(pale box plots). These hypothetical scores quantify the distri-
bution of task success we would expect if participants reliedexclusively on their biases as a source of information influen-
cing their arrowhead designs. The comparison is useful
because it establishes a baseline score distribution in theabsence of cultural accumulation of information about the
score landscape f. Examining arrowheads after generation
three, we found that the hypothetical scores of participantsin the SR treatment were significantly different from baselines
in E4 ( Z= 14 719, p< 0.001, Wilcoxon signed-rank test), E3
(Z= 13 219, p< 0.001) and E2 ( Z= 10 577, p< 0.001), but not
in E1 ( Z= 8810, p= 0.098). Participants ’biases helped in
some but not all environments. However, arrowheads
designed by participants in the optimization treatmentsachieved significantly greater task success than arrowheads
designed in the SR treatment in all environments (Mann –
Whitney one-sided tests, E1: W= 20 255.5, p< 0.001;
E2: W= 19 985.5, p< 0.001; E3: W= 22 830, p< 0.001; E4:
W= 21 035.5, p< 0.001; two-sided tests concurred). Individual
learning contributed to task success in all environments.
4. Discussion
Is technological advancement potentially constrained byhuman biases? This question is central to theories of human
technological culture, but has been difficult to answer
empirically. Functional perspectives highlight the capacity todiscover efficient solutions to diverse environmental problems
[1,5,6,16]. Cognitive perspectives highlight the transforma-
tive impact of human cognition on cultural transmission[25,34,55,74]. The integration of these perspectives into formal
cognitive models and behavioural experiments is a challenge
for modern approaches to human culture [4,46,48].
We developed a probabilistic model of CCE to quantify
these effects. Our analysis combined a Bayesian formulation
of cultural transmission with an optimization-based model ofinnovation through individual learning. The resulting math-
ematical framework predicts a trade-off between inductive
biases and objective function in cumulative innovation.
Our analysis highlights formal connections between
cultural evolution, stochastic optimization algorithms, and
Bayesian inference, extending known connections betweentask success
across conditions
500 600 700 800 900 1000
arrowhead score (calories)E3
E2
E1E4
Figure 4. Experimental results (task success). Task success across optimiz-
ation treatments. Triangles show mean score for individual chains. Faint
dots show individual participant scores. Greater distance between the opti-mum and the prior mean resulted in worse performance. (Online version
in colour.)royalsocietypublishing.org/journal/rspb Proc. R. Soc. B 288: 202027525
 Downloaded from https://royalsocietypublishing.org/ on 07 April 2021

evolution, learning, and optimization [75 –81] to cultural
transmission. An optimization objective is an assumptionof the model; however, the fact that the population dynamics
has the form of a well-defined algorithm computing sol-
utions to this optimization problem is not an assumption,but a finding that derives from the combination of simpler
assumptions about bounded individual cognition. This con-
nection to stochastic optimization algorithms establishes therelevance of a large mathematical literature on the properties
of these processes that may help illuminate aspects of
human technological cultures in the same way that evol-utionary theory has offered quantitative insights and
predictions. Here, for example, we found that human induc-
tive biases can act as a form of regularization. Regularizationis known to play an important role in the generalization pro-
file of stochastic optimization processes [82], raising the
question of whether analogous functional consequences ofhuman learning biases may also be relevant to cultural
accumulation, for instance.
More generally, our analysis illustrates a way of viewing
cultural processes directly in terms of the computations
performed by individuals and by the population, creating
continuity between models of cognitive and culturalprocesses. Computational approaches to cognition view
learning in terms of the information processing carried outby individuals —the computations people perform and their
dependencies on data and bias. Our results extend this per-
spective to the cultural process as a whole, by formalizingthe computation performed by the population . This offers a
quantitative formulation of the idea that populations can
implement distributed computations over generations [69],and that cultural processes can lead to products that go
beyond the information processing limits of individuals
[1]—a core principle of CCE that has been notoriously
difficult to quantify [4].
Our study contributes to an increasingly detailed empirical
picture of how human cognition can influence cultural evol-ution [14,31,42,54], by quantifying biases that are introduced
simply as a consequence of the process of cultural transmission.
Our model makes predictions about the impact of inductivebiases in general on the outcome of cultural transmission.
To test these predictions, we selected a specific population of
participants, measured their biases on a widely used task,and showed that the results aligned with our model predic-
tions. Participants performed worse at solving a continuous
optimization problem when the solution was less consistenttask success over generations(a)( b)( c)
 task successscore
generation
in chainrepro.
chainsopt.
chains****
****
****
****
 6007008009001000
6007008009001000
6007008009001000
6007008009001000score score scorearrowheads
random
walkmaximum
scoreoptimum discovered
orrvvdd dd eeoovvisisveveveveveveveveisisvvvvvvovovvvvvvvvvvveeerredeeredd cceeeereerrerrrreer
mimcoco
iimd
ttiiiiiiiiic ee sccscccssssss
mmeded
tmimc d oooo
discovered
optimum
optimum discoveredoptimum discovered
environment E4environment E3environment E2environment E1
11 0 23456789
Figure 5. Experimental results (task success baselines and comparisons). Task success overall ( a) and over generations ( b). Solid box plots show arrowhead score
within-treatment distributions. Faded box plots show scores that would have been awarded to arrowheads produced by participants in the serial repro duction
treatment if they had been evaluated in against the scoring function. Triangles ( a) show mean score. Dot-dashed lines show score expected under a random
walk over the scoring function. Dotted lines show maximum score. ( c) Optimum arrowhead and the mean design of produced arrowheads positioned within
the stimulus design space. Faint dots show individual arrowhead designs in stimulus space. (Online version in colour.)royalsocietypublishing.org/journal/rspb Proc. R. Soc. B 288: 202027526
 Downloaded from https://royalsocietypublishing.org/ on 07 April 2021

with the relevant prior distribution, confirming mathematical
predictions and substantiating the assumptions of influential
theories [3,25].
The current study focuses on a gradient optimization
setting and is limited to transmission-chain population struc-
tures. These are significant simplifications. While these
simplifications allowed us to identify a basic property of cumu-lative innovation, examining the impacts of inductive bias in
more structured tasks and richer population dynamics is an
important goal for future research. Extending our gradient-based model of innovation to more sophisticated algorithms
that better approximate human search is also a priority. A
more complete understanding requires analysis of importantcases such as preferential selection among multiple social
models [6], heterogeneous or dynamic priors, and varying
environmental objectives. Whether these factors counteract oramplify the impacts of inductive bias remains unknown.
Another important assumption of our analyses is that the
prototype arrowhead design measured using SR is a usefulapproximation to inductive biases that are at least partly
shared among participants. SR has desirable properties as an
estimation procedure: it provides a mathematically motivated,data-efficient approach to estimation of a prior distribution
(see Methods and e.g. [83]) from participant responses in a
social task, and previous research has shown that this approachaccords with alternative methods [44,83]. However, this
method and our model rely on the assumption that participant ’s
inferences are approximately Bayesian with respect to a priordistribution that is shared, which is a clear simplification. Our
model and experimental results indicate that quantifying a
shared prototype design by estimating the mean of a priordistribution through SR provides an empirically useful approxi-
mation. However, the estimated prior prototype design should
be interpreted as a useful summary statistic of a potentiallycomplex distribution, rather than a platonic ideal design.
Future research might profitably explore this simplification.
Our findings provide experimental evidence that human
inductive biases can impact the discoveries people are able to
make as a population in a controlled laboratory setting, as pre-
dicted by theories of cultural evolution [3,6,25,34,55]. Sometheories have emphasized how technical knowledge [57] and
other adaptive human priors [25,38] are well aligned with
environmental problems. Other theories have emphasizedthe many problems people face with highly counterintuitive
solutions [1,6,16]. Our framework is applicable to both
contexts. Our results suggest that there are significant connec-tions between evolutionary theories of cumulative culture and
computational theories of inductive inference. Understanding
these connections may help to integrate diverse approachesto the evolution of human technology and clarify conditions
that facilitate effective accumulation from a cognitive perspec-
tive. These results reinforce the importance of mechanismsthat help to alleviate bias in creative, educational and
scientific contexts.
5. Methods
(a) Model
We made the simplifying assumption that dimensions θiof a multi-
dimensional parametrization Θcan be treated independently.
Under a Gaussian prior distribution p(ui)¼N(ui;mi,d2
i) and Gaus-
sian noise-corrupted observation of ut
i(with variance s2
i), thesampling distribution p(^uijui,mi,si,di) for the design induced ( ^ui)
is the posterior distribution, which is Gaussian with expectation
E[^ui]¼s2
i
simiþd2
i
siui,( 5 :1)
where si¼s2
iþd2
i. We quantified bounded innovation by assum-
ing that people experiment with nearby variations on an inherited
solution and use performance feedback to attempt a (guided) inno-vation. A generic way to capture the outcome of this process isgradient-based learning [84,85], casting innovation as a step along
the gradient (slope) of fdenoted by r
uf(ui). The rate of innovation
(the magnitude of the step α> 0) is a model parameter. The chain
dynamics are given by equation (2.1). The expected change pergeneration is
E[
utþ1
i/C0ut
i]¼s2
i
simiþd2
i
siuti/C0aruf(ut
i)/C0ut
i
¼s2
i
si(mi/C0ut
i)/C0aruf(ut
i):(5:2)
which implies no further accumulation in expectation if ruf(ui)
reaches the threshold
r/C3
i¼s2
i
simi/C0ut
i
a: (5:3)
Iff(θi) is a quadratic surface with optimum θi*then f(θi) has gradi-
entsruf(ui)¼a(ui/C0u/C3
i) for some constant a. Solving for r/C3
ileads
to equation (2.2) with
bi¼aas
aasþs2(5:4)
(b) Prior estimation
Mathematical results show that cultural transmission by SR in a
transmission chain can be analysed as a Markov chain in whichthe representation (the arrowhead design θ
tin the current con-
text) induced by an individual at generation tdepends only on
the representation induced by the individual at generation t−1
[56]. If the inferences made by the individuals in this chain canbe approximately characterized as Bayesian inference under aprior distribution p(θ), then it can be shown that (under relatively
general technical conditions: see [56]) the chain converges to astationary equilibrium distribution in which the probabilitythat any individual induces a design θis the same as the prob-
ability of the design under the prior distribution p(θ). This
result indicates that the arrowheads produced by individuals inSR chains can be seen as samples from the prior distribution.
One benefit of this finding is that it justifies analysing the
designs produced along the chain, rather than for examplebeing restricted to analysing the designs produced at the finalgeneration of the process, providing a more stable estimate ofthe properties of the prior distribution (such as its expectation,which we analysed as ^
mi¼(1=n)Pn
k¼1ui, where kranges over
alln= 250 participants assigned to the SR treatment). Each of
the 25 SR chains was initialized at one of five arrowhead designs(assigned using block randomization) chosen to offer anunbiased coverage of the design space (designs occupied fourcorners and the centre of a square positioned at the centre ofdesign space; these designs are pictured in figure 1). Estimatingprior distributions by SR has been evaluated by prior researchand shown to have desirable characteristics and to accord withalternative methods in the domains of function learning [44]and category learning [73], for example.
(c) Participants
Participants were recruited on Amazon ’s Mechanical Turk over
two sessions (US only, approval rating ≥95) and compensatedroyalsocietypublishing.org/journal/rspb Proc. R. Soc. B 288: 202027527
 Downloaded from https://royalsocietypublishing.org/ on 07 April 2021

1 USD for participation plus performance-dependent compen-
sation of up to 0.50 USD. Most participants completed in lessthan 5 min. Participants who failed attention checks (constructedfrom pilot data: minimum completion time; minimum reproduc-tion accuracy; at least two unique arrowheads evaluated onlearning trials) were excluded and a replacement was immedi-ately re-recruited. No demographic data (e.g. age, genderidentity) were collected.
(d) Stimuli
Scores were determined by a quadratic function f(θi) = (1/
2)a(θi−θi*)2+c, where θi*is the optimum value for feature θi.
We required a parametrization which did not include negativescores and had an optimum (maximum) score in a semanticallyreasonable range. To meet these requirements, we set α=−30
and c= 10 000 and divided the result by 100. The maximum
available score was 1000 calories. To award a score, we computedf(θ
i) for both arrowhead dimensions and awarded the mean.
(e) Procedure
Participants were assigned to one of five treatments (E1 - E4, SR)
using block randomization. The experiment had three trial types:an inheritance trial (IT); testing trials (TT × 4) and a productiontrial (PT). In IT, participants observed an arrowhead producedby the participant at the previous generation (or one of fiveseeds at generation 1). The inherited design was presented inthe centre of the screen on a white background for 3000 ms, pre-ceded by a fixation cross (1000 ms). Participants were instructedto reconstruct the design as accurately as possible using HTMLrange-sliders which dynamically redrew the arrowhead inthe centre of the screen. Range-slider initial position wasrandomized. Participants could not proceed until both sliders
had been moved. The arrowhead appeared upon modificationof the sliders. Participants completed one task-familiarizationtrial (recreating a briefly observed random arrowhead design)before the inheritance trial.
During TT participants tested variations on their reconstruc-
tion of the inherited design. Range-sliders allowed testing ofdesigns within 30 units (pixels) either side of the participant ’s
reconstruction. Participants could click a ‘Shoot ’button at any
time to test the current design. Clicking the shoot button trig-gered an animation in which the current design flew off screenright. Score was presented immediately after test, concludingTT. The participant ’s initial reconstruction and its score, along
with any tested designs and their scores, were presentedthroughout at the top of the screen. In PT, participants wereinstructed that the AD produced on this trial will determine abonus payment, at a rate of 0.10 USD per 200 calories. Previousdesigns and their scores remained visible. Score feedback waspresented after PT, concluding the experiment.
Ethics. This research was approved by the Committee for Protection of
Human Subjects at University of California, Berkeley.
Data accessibility. Experimental data, reproducible analyses, experiment
implementation codebase and pre-registration are available at
https://osf.io/hx9jd/.
Competing interests. We declare we have no competing interests.
Funding. This work was supported in part by NSF grant no. 1456709
and DARPA Cooperative agreement no. D17AC00004.
Acknowledgements. We thank Matthew Hardy for feedback. A prelimi-
nary version of this research was presented at the 41st AnnualMeeting of the Cognitive Science Society and appears in its proceed-
ings [86]. All experimental results presented here are new. We are
grateful to the attendees of this meeting for valuable feedback.
References
1. Henrich J 2015 The secret of our success: how culture
is driving human evolution, domesticating ourspecies, and making us smarter . Princeton, NJ:
Princeton University Press.
2. Caldwell CA, Atkinson M, Renner E. 2016
Experimental approaches to studying cumulative
cultural evolution. Curr. Dir. Psychol. Sci. 25,
191–195. (doi:10.1177/0963721416641049)
3. Boyd R, Richerson PJ, Henrich J. 2013 The cultural
evolution of technology. In Cultural evolution (eds
PJ Richerson, MH Christiansen), pp. 119 –142.
New York, NY: The MIT Press.
4. Mesoudi A, Thornton A. 2018 What is cumulative
cultural evolution? Proc. Biol. Sci. 285, 20180712.
5. Lewis HM, Laland KN. 2012 Transmission fidelity is
the key to the build-up of cumulative culture. Phil.
Trans. R. Soc. B 367, 2171 –2180. (doi:10.1098/rstb.
2012.0119)
6. Boyd R, Richerson PJ 1985 Culture and the
evolutionary process . Chicago, IL: University of
Chicago Press.
7. Henrich J. 2004 Demography and cultural evolution:
how adaptive cultural processes can producemaladaptive losses —the Tasmanian case. Am.
Antiquity 69, 197–214. (doi:10.2307/4128416)
8. Tomasello M, Kruger AC, Ratner HH. 1993 Cultural
learning. Behav. Brain Sci. 16, 495–511. (doi:10.
1017/S0140525X0003123X)9. Caldwell CA, Atkinson M, Renner E. 2016
Experimental approaches to studyingcumulative cultural evolution. Curr. Dir.
Psychol. Sci. 25, 191–195. (doi:10.1177/
0963721416641049)
10. Caldwell CA, Millen AE. 2008 Experimental
models for testing hypotheses about cumulative
cultural evolution. Evol. Human Behav.
29, 165–171. (doi:10.1016/j.evolhumbehav.2007.
12.001)
11. Fay N, De Kleine N, Walker B, Caldwell CA. 2019
Increasing population size can inhibit cumulative
cultural evolution. Proc. Natl Acad. Sci. USA 116,
6726–6731. (doi:10.1073/pnas.1811413116)
12. Muthukrishna M, Shulman BW, Vasilescu V,
Henrich J. 2014 Sociality influences cultural
complexity. Proc. R. Soc. B 281, 20132511. (doi:10.
1098/rspb.2013.2511)
13. Scanlon LA, Lobb A, Tehrani JJ, Kendal JR. 2019
Unknotting the interactive effects of learningprocesses on cultural evolutionary dynamics. Evol.
Hum. Sci. 1, e17. (doi:10.1017/ehs.2019.17)
14. Derex M, Bonnefon JF, Boyd R, Mesoudi A. 2019
Causal understanding is not necessary for the
improvement of culturally evolving technology. Nat.
Hum. Behav. 3, 446–452.
15. Derex M, Beugin MP, Godelle B, Raymond M. 2013
Experimental evidence for the influence of groupsize on cultural complexity. Nature 503, 389–391.
(doi:10.1038/nature12774)
16. Mesoudi A. 2008 An experimental simulation of the
‘copy-successful-individuals ’cultural learning
strategy: adaptive landscapes, producer –scrounger
dynamics, and informational access costs. Evol.
Hum. Behav. 29, 350–363. (doi:10.1016/j.
evolhumbehav.2008.04.005)
17. Mesoudi A, O ’Brien MJ. 2008 The cultural transmission
of great basin projectile-point technology I: an
experimental simulation. Am. Antiquity 73,3–28.
(doi:10.1017/S0002731600041263)
18. Zwirner E, Thornton A. 2015 Cognitive requirements
of cumulative culture: teaching is useful but notessential. Sci. Rep. 5, 16781. (doi:10.1038/srep16781)
19. Derex M, Boyd R. 2015 The foundations of the
human cultural niche. Nat. Commun. 6, 8398.
(doi:10.1038/ncomms9398)
20. Derex M, Boyd R. 2016 Partial connectivity increases
cultural accumulation within groups. Proc. Natl
Acad. Sci. USA 113, 2982 –2987. (doi:10.1073/pnas.
1518798113)
21. Mason W, Watts DJ. 2012 Collaborative learning in
networks. Proc. Natl Acad. Sci. USA 109, 764–769.
(doi:10.1073/pnas.1110069108)
22. Acerbi A, Tennie C, Mesoudi A. 2016 Social learning
solves the problem of narrow-peaked search
landscapes: experimental evidence in humans.royalsocietypublishing.org/journal/rspb Proc. R. Soc. B 288: 202027528
 Downloaded from https://royalsocietypublishing.org/ on 07 April 2021

R. Soc. Open Sci. 3, 160215. (doi:10.1098/rsos.
160215)
23. Creanza N, Feldman MW. 2014 Complexity in
models of cultural niche construction with selectionand homophily. Proc. Natl Acad. Sci. USA 111
(Suppl. 3), 10 830 –10 837. (doi:10.1073/pnas.
1400824111)
24. Derex M, Perreault C, Boyd R. 2018 Divide and
conquer: intermediate levels of population
fragmentation maximize cultural accumulation. Phil.
Trans. R. Soc. B 373, 20170062. (doi:10.1098/rstb.
2017.0062)
25. Sperber D, Claidière N. 2008 Defining and
explaining culture (comments on Richerson and
Boyd, Not by genes alone). Biol. Phil. 23, 283–292.
(doi:10.1007/s10539-005-9012-8)
26. Vapnik VN. 1999 An overview of statistical learning
theory. IEEE Trans. Neural Netw .10, 988–999.
(doi:10.1109/72.788640)
27. Henrich J, Heine SJ, Norenzayan A. 2010 The
weirdest people in the world? Behav. Brain Sci. 33,
61–83. (doi:10.1017/S0140525X0999152X)
28. Caldwell CA, Renner E, Atkinson M. 2017 Human
teaching and cumulative cultural evolution. Rev.
Phil. Psychol. 9,1–20.
29. Castro L, Toro MA. 2014 Cumulative cultural
evolution: the role of teaching. J. Theor. Biol. 347,
74–83. (doi:10.1016/j.jtbi.2014.01.006)
30. Ho MK, MacGlashan J, Littman ML, Cushman F.
2017 Social is special: a normative framework for
teaching with and learning from evaluativefeedback. Cognition 167,9 1–106. (doi:10.1016/j.
cognition.2017.03.006)
31. Morgan TJH et al. 2015 Experimental evidence for
the co-evolution of hominin tool-making teaching
and language. Nat. Commun. 6, 6029. (doi:10.1038/
ncomms7029)
32. Yang SCH, Yu Y, Givchi A, Wang P, Vong WK, Shafto
P. 2017 Optimal cooperative inference. See http:/ /
arxiv.org/abs/1705.08971.
33. Wang J, Wang P, Shafto P. 2020 Sequential cooperative
Bayesian inference. See http:/ /arxiv.org/abs/2002.05706.
34. Claidière N, Sperber D. 2007 The role of attraction in
cultural evolution. J. Cogn. Cult. 7,8 9–111. (doi:10.
1163/156853707X171829)
35. Acerbi A, Charbonneau M, Miton H, Scott-Phillips T.
2019 Cultural stability without copying. See https://
osf.io/vjcq3.
36. Kirby S, Dowman M, Griffiths TL, Kirby S,
Griffiths TL, Dowman M. 2007 Innateness and
culture in the evolution of language. Proc. Natl
Acad. Sci. USA
104, 5241 –5245. (doi:10.1073/pnas.
0608222104)
37. Kemp C, Perfors A, Tenenbaum JB. 2007 Learning
overhypotheses with hierarchical Bayesian models.Dev. Sci. 10, 307–321. (doi:10.1111/j.1467-7687.
2007.00585.x)
38. Boyer P. 1998 Cognitive tracks of cultural
inheritance: how evolved intuitive ontology governs
cultural transmission. Am. Anthropol. 100, 876–889.
(doi:10.1525/aa.1998.100.4.876)
39. Kirby S, Tamariz M, Cornish H, Smith K. 2015
Compression and communication in the culturalevolution of linguistic structure. Cognition 141,
87–102. (doi:10.1016/j.cognition.2015.03.016)
40. Verhoef T, Kirby S, de Boer B. 2014 Emergence of
combinatorial structure and economy through iteratedlearning with continuous acoustic signals. J. Phon. 43,
57–68. (doi:10.1016/j.wocn.2014.02.005)
41. Martin D, Hutchison J, Slessor G, Urquhart J,
Cunningham SJ, Smith K. 2014 The spontaneous
formation of stereotypes via cumulative cultural
evolution. Psychol. Sci. 25, 1777 –1786. (doi:10.
1177/0956797614541129)
42. Saldana C, Fagot J, Kirby S, Smith K, Claidière N.
2019 High-fidelity copying is not necessarily the keyto cumulative cultural evolution: a study in monkeys
and children. Proc. R. Soc. B 286, 20190729.
(doi:10.1098/rspb.2019.0729)
43. Ravignani A, Delgado T, Kirby S. 2017 Musical
evolution in the lab exhibits rhythmic universals.
Nat. Hum. Behav. 1,1–7. (doi:10.1038/s41562-
016-0007)
44. Kalish ML, Griffiths TL, Lewandowsky S.
2007 Iterated learning: intergenerationalknowledge transmission reveals inductive biases.
Psychon. Bull. Rev. 14, 288–294. (doi:10.3758/
BF03194066)
45. Claidière N, Scott-Phillips TC, Sperber D. 2014
How Darwinian is cultural evolution? Phil.
Trans. R. Soc. B 369, 20130368. (doi:10.1098/rstb.
2013.0368)
46. Nettle D. 2020 Selection, adaptation, inheritance
and design in human culture: the view from thePrice equation. Phil. Trans. R. Soc. B 375, 20190358.
(doi:10.1098/rstb.2019.0358)
47. Dietrich A. 2004 The cognitive neuroscience of
creativity. Psychon. Bull. Rev. 11, 1011 –1026.
(doi:10.3758/BF03196731)
48. Heyes C. 2018 Enquire within: cultural evolution and
cognitive science. Phil. Trans. R. Soc. B 373,
20170051. (doi:10.1098/rstb.2017.0051)
49. Fogarty L, Creanza N, Feldman MW. 2015 Cultural
evolutionary perspectives on creativity and human
innovation.
Trends Ecol. Evol .30, 736–754. (doi:10.
1016/j.tree.2015.10.004)
50. Borji A, Itti L. 2013 Bayesian optimization explains
human active search. See https://dl.acm.org/doi/10.
5555/2999611.2999618.
51. Mason W A, Jones A, Goldstone RL. 2008
Propagation of innovations in networked groups.
J. Exp. Psychol.: General 137, 422–433. (doi:10.
1037/a0012798)
52. Yahosseini KS, Moussaïd M. 2020 Comparing
groups of independent solvers and transmissionchains as methods for collective problem-solving.
Sci. Rep. 10,1–9. (doi:10.1038/s41598-020-
59946-9)
53. Wu Y, Ren M, Liao R, Grosse R. 2018 Understanding
short-horizon bias in stochastic meta-optimization.
See http:/ /arxiv.org/abs/1803.02021.
54. Kendal RL, Boogert NJ, Rendell L, Laland KN,
Webster M, Jones PL. 2018 Social learning
strategies: bridge-building between fields. Trends
Cogn. Sci. 22, 651–665. (doi:10.1016/j.tics.2018.
04.003)55. Miton H, Charbonneau M. 2018 Cumulative culture
in the laboratory: methodological and theoretical
challenges. Proc. R. Soc. B 285, 20180677. (doi:10.
1098/rspb.2018.0677)
56. Griffiths TL, Kalish ML. 2007 Language evolution by
iterated learning with Bayesian agents. Cogn. Sci.
31, 441–480. (doi:10.1080/15326900701326576)
57. Osiurak F, Reynaud E. 2019 The elephant in the
room: what matters cognitively in cumulative
technological culture. Behav. Brain Sci .43, e156.
(doi:10.1017/S0140525X19003236)
58. Tennie C, Call J, Tomasello M. 2009 Ratcheting up
the ratchet: on the evolution of cumulative culture.Phil. Trans. R. Soc. B 364, 2405 –2415. (doi:10.1098/
rstb.2009.0052)
59. Sperber D 1996 Explaining culture: a naturalistic
approach . Oxford, UK: Blackwell.
60. Chater N, Vitányi P, Vitanyi P. 2003 Simplicity: a
unifying principle in cognitive science. Trends Cogn.
Sci.7,1 9–22. (doi:10.1016/S1364-6613(02)00005-0)
61. Smith KA, Battaglia PW, Vul E. 2018 Different
physical intuitions exist between tasks, notdomains. Comput. Brain Behav. 1, 101–118. (doi:10.
1007/s42113-018-0007-3)
62. Kemp C, Tenenbaum JB. 2008 The discovery of
structural form. Proc. Natl Acad. Sci. USA 105,
10 687 –10 692. (doi:10.1073/pnas.0802631105)
63. Canini KR, Griffiths TL, Vanpaemel W, Kalish ML.
2014 Revealing human inductive biases for category
learning by simulating cultural transmission.
Psychon. Bull. Rev. 21
, 785–793. (doi:10.3758/
s13423-013-0556-3)
64. Navarro DJ, Perfors A, Kary A, Brown SD, Donkin C. 2018
When extremists win: cultural transmission via iteratedlearning when populations are heterogeneous. Cogn.
Sci.42,2 1 0 8 –2149. (doi:10.1111/cogs.12667)
65. Perfors A, Navarro DJ. 2014 Language evolution can
be shaped by the structure of the world. Cogn. Sci.
38, 775–793. (doi:10.1111/cogs.12102)
66. Thompson B, Kirby S, Smith K. 2016 Culture
shapes the evolution of cognition. Proc. Natl Acad.
Sci. USA 113, 4530 –4535. (doi:10.1073/pnas.
1523631113)
67. Henrich J, Boyd R. 2002 On modeling cognition and
culture: why cultural evolution does not require
replication of representations. J. Cogn. Cult. 2,
87–112. (doi:10.1163/156853702320281836)
68. Kingma DP, Ba JL. 2015 Adam: a method for
stochastic optimization. In 3rd Int. Conf. on Learning
Representations, ICLR 2015: conference track
proceedings . See http:/ /arxiv.org/abs/1412.6980v9.
69. Smaldino PE, Richerson PJ. 2013 Human cumulative
cultural evolution as a form of distributed
computation. In Handbook of human computation
(ed. P Michelucci), pp. 979 –992. New York, NY:
Springer.
70. Xu J, Griffiths TL. 2010 A rational analysis of the
effects of memory biases on serial reproduction.Cogn. Psychol. 60, 107–126. (doi:10.1016/j.
cogpsych.2009.09.002)
71. Mesoudi A. 2011 An experimental comparison of
human social learning strategies: payoff-biased
social learning is adaptive but underused. Evol.royalsocietypublishing.org/journal/rspb Proc. R. Soc. B 288: 202027529
 Downloaded from https://royalsocietypublishing.org/ on 07 April 2021

Hum. Behav. 32, 334–342. (doi:10.1016/j.
evolhumbehav.2010.12.001)
72. Mesoudi A, Chang L, Murray K, Lu HJ. 2015 Higher
frequency of social learning in China than in theWest shows cultural variation in the dynamics of
cultural evolution. Proc. R. Soc. B 282, 20142209.
(doi:10.1098/rspb.2014.2209)
73. Griffiths TL, Christian BR, Kalish ML. 2008 Using
category structures to test iterated learning as a
method for identifying inductive biases. Cogn. Sci.
32,6 8–107. (doi:10.1080/03640210701801974)
74. Morin O. 2016 Reasons to be fussy about cultural
evolution. Biol. Phil. 31, 447–458. (doi:10.1007/
s10539-016-9516-4)
75. Watson RA, Szathmáry E. 2016 How can evolution
learn? Trends Ecol. Evol. 31, 147–157. (doi:10.1016/
j.tree.2015.11.009)
76. Kouvaris K, Clune J, Kounios L, Brede M, Watson RA.
2017 How evolution learns to generalise: using theprinciples of learning theory to understand theevolution of developmental organisation. PLOS
Comput. Biol. 13, e1005358. (doi:10.1371/journal.pcbi.
1005358)
77. Mandt S, Hoffman MD, Blei DM. 2017 Stochastic
gradient descent as approximate Bayesian inference.
See http:/ /arxiv.org/abs/1704.04289.
78. Harper M. 2009 The replicator equation as an
inference dynamic. See http:/ /arxiv.org/abs/0911.
1763.
79. Suchow JW, Bourgin DD, Griffiths TL. 2017
Evolution in mind: evolutionary dynamics,
cognitive processes, and Bayesian inference.
Trends Cogn. Sci. 21, 522–530. (doi:10.1016/j.tics.
2017.04.005)
80. Maclaurin D, Duvenaud D, Adams RP. 2015 Early
stopping is nonparametric variational inference.See http:/ /arxiv.org/abs/1504.01344.
81. Aitchison L. 2018 A unified theory of adaptive
stochastic gradient descent as Bayesian filtering.See http:/ /arxiv.org/abs/1807.07540.82. Krogh A, Hertz JA. 1992 A simple weight decay can
improve generalization. See https://papers.nips.cc/
paper/1991/file/8eefcfdf5990e441f0fb6f3fad709e21-
Paper.pdf.
83. Griffiths TL, Kemp C, Tenenbaum JB. 2008
Bayesian models of cognition. In Cambridge
handbook of computational psychology (ed. R Sun),
pp. 59 –100. Cambridge, UK: Cambridge University
Press.
84. Jordan C, Jordán K. 1965 Calculus of finite
differences , vol. 33. Providence, RI: American
Mathematical Society.
85. Boyd S, Boyd SP, Vandenberghe L. 2004 Convex
optimization . Cambridge, UK: Cambridge University
Press.
86. Thompson B, Griffiths TL. 2019 Inductive biases
constrain cumulative cultural evolution. In Proc.
annual meeting of the Cognitive Science Society , vol.
41. See https://cogsci.mindmodeling.org/2019/
papers/0205/index.html.royalsocietypublishing.org/journal/rspb Proc. R. Soc. B 288: 2020275210
 Downloaded from https://royalsocietypublishing.org/ on 07 April 2021