Sparse Skill Coding: Learning Behavioral Hierarchies with Sparse Codes.

Under review as a conference paper at ICLR 2020
SPARSE SKILL CODING : LEARNING BEHAVIORAL
HIERARCHIES WITH SPARSE CODES
Anonymous authors
Paper under double-blind review
ABSTRACT
Many approaches to hierarchical reinforcement learning aim to identify sub-goal
structure in tasks. We consider an alternative perspective based on identifying
behavioral ‘motifs’—repeated action sequences that can be compressed to yield
a compact code of action trajectories. We present a method for iteratively com-
pressing action trajectories to learn nested behavioral hierarchies of arbitrary depth,
with actions of arbitrary length. The learned temporally extended actions provide
new action primitives that can participate in deeper hierarchies as the agent learns.
We demonstrate the relevance of this approach for tasks with non-trivial hierar-
chical structure and show that the approach can be used to accelerate learning in
recursively more complex tasks through transfer.
1 I NTRODUCTION
Despite the many successes of deep reinforcement learning (RL) in recent years (Mnih et al., 2015;
Schulman et al., 2017; Silver et al., 2016; Levine et al., 2016), long-term credit assignment and
search complexity remain fundamental challenges. One of the primary strategies for managing this
complexity has been to incorporate hierarchical, temporally-extended actions. Hierarchies hand-
designed using domain knowledge can provide substantial training beneﬁts (Sutton et al., 1999;
Barto & Mahadevan, 2003). However, a major challenge in hierarchical reinforcement learning is
to develop general methods for discovering useful hierarchical representations without relying on
domain expertise.
Many objectives for the hierarchy learning problem have been proposed, with notable focus on facili-
tating transfer to downstream tasks and facilitating efﬁcient exploration of the state space (Eysenbach
et al., 2018; Frans et al., 2017; Solway et al., 2014). We pose the hierarchy learning problem as
follows: given a distribution of tasks, what determines the optimal set of representations for action
sequences? We approach this question by considering the problem faced by human decision-makers.
Humans are fundamentally resource-constrained. Energy is limited, computation is expensive, and
solutions to problems must be computed in real-time. Across cortical areas, a common strategy for
dealing with these constraints is to reduce computational complexity by storing representations that
efﬁciently encode the statistics of the domain. This idea originated as the efﬁcient coding hypothe-
sis (Barlow, 1961), and has been empirically corroborated in sensory and motor systems (Barlow,
1961; Olshausen & Field, 1996; Hrom ´adka et al., 2008; Poo & Isaacson, 2009; Vinje & Gallant,
2000).
We extend the efﬁcient coding hypothesis to the problem of representation learning for planning,
and propose a method for discovering temporally extended actions by learning an efﬁcient code
of the behavior required by a task distribution. This is a novel formulation of the classic notion
of “chunking” from cognitive psychology (Chase & Simon, 1973; Simon, 1991), which motivated
early work on hierarchical reinforcement learning (Korf, 1985; Stolle & Precup, 2002), and aligns
with empirical neuroscience results suggesting that organisms represent their motor output in terms
of a sparse efﬁcient code of high-level “motor primitives” (Flash & Hochner, 2005). An efﬁcient
code for a sequence of actions compresses the sequence into a minimum-length description that
factorizes the input distribution (Cover & Thomas, 2012). The beneﬁt of using such a code in the
context of decision making is that it provides the building blocks for solving related problems using a
minimal set of decision points, delineated by a minimal set of skills that capture the statistics of the
behavior required by the task distribution. As a consequence, this approach subsumes several distinct
1
Under review as a conference paper at ICLR 2020
objectives for hierarchy learning proposed in the literature; an efﬁcient code of behavior required by a
problem space reduces the number of decision points required (Harb et al., 2017), facilitates transfer
to tasks drawn from the same distribution (Solway et al., 2014), facilitates efﬁcient exploration of the
state space [cite], and decomposes a task into a natural set of sub-tasks (Bacon et al., 2017; Fox et al.,
2017).
The problem of ﬁnding a compact code for sequential data with long-range dependencies and nested
hierarchical structure is equivalent to the problem of ﬁnding a minimum-length program that can
generate the data—that is, ﬁnding a program with minimum Kolmogorov complexity (Kolmogorov,
1965). The Kolmogorov complexity of a sequence is not ﬁnitely computable and can thus only
be approximated. Drawing inspiration from this idea, we propose a relatively simple approach
for approximating minimum-description-length codes of sequences of actions through iterative
convolutional sparse coding and compression, with structure similar to classic string compression
methods such as the Nevill-Manning algorithm (Nevill-Manning & Witten, 1997). With this method,
we are able to extract compact, hierarchically nested representations of action trajectories, with
temporally extended actions of arbitrary lengths. We incorporate this method into the RL problem by
equipping the agent with the capacity to compress its behavior and augment its action space with the
learned representations after each task it faces.
2 P RELIMINARIES
Reinforcement learning: We consider a taskas a ﬁnite horizon Markov decision process, consisting
of a set of states S, a set of actions A, a transition function p(St+1=s0jSt=s;At=a)that deﬁnes
how actions move an agent between states, and a reward function r(St=s;At=a;St+1=st+1)
that deﬁnes the reward the agent receives by taking action ain statesand ending up in state st+1.
The objective is to ﬁnd the policy :S!Athe expected cumulative reward
E"TX
t=1r(St=s;At=a;St+1=st+1)#
:
Sparse coding: Sparse coding (Olshausen & Field, 1996) is an unsupervised algorithm for learning
a dictionary that reconstructs a signal using a minimal set of non-zero coefﬁcients on the dictionary.
The sparse coding model assumes that a signal Xis generated as a linear combination of ﬁlters W
with coefﬁcients splus additive Gaussian noise :
xi=nX
j=1Wjsi;j+ (1)
The objective is to reconstruct the input with minimal distortion while using the minimal number of
non-zero coefﬁcients. The ﬁlters and coefﬁcients are jointly optimized:
min
w;smX
i=1jjxinX
j=1Wjsi;jjj2
2+X
i;jjjsi;jjj1 (2)
with thel1norm imposing a penalty on the number on non-zero activations, and modulating the
trade-off between the accuracy and sparsity of the representations. To learn sparse codes for time
series data, one can augment the basic sparse coding model by replacing scalar-valued coefﬁcients
with vector-valued coefﬁcients and matrix multiplication by convolution, which allows basis functions
to appear at all possible shifts in the signal:
min
w;smX
i=1jjxinX
j=1Wjsi;jjj2
2+X
i;jjjsi;jjj1 (3)
We use this convolutional formulation of the sparse coding model to encode trajectories of actions
generated by an RL agent. We note the similarity between the sparsity constraint and the Minimum
Description Length (MDL) principle, which states that the best model ^M2M is that which can
describe a data sample xcompletely using the fewest number of bits,
2
Under review as a conference paper at ICLR 2020
^M= arg min
M2^ML(x;M )
whereL(x;M )is the codelength assignment function deﬁning the theoretical code length required to
describe (x;M )uniquely. Underlying the MDL is the idea that a model that is able to (losslessly)
compress data must do so by capturing its structure and regularities. We use sparse coding to
approximate this objective.
3 S PARSE SKILL CODING
We propose a method, sparse skill coding (SSC), for discovering hierarchically nested codes for
action sequences using a variant of convolutional sparse coding. Given a trajectory 2Ztconsisting
ofttimesteps of ndiscrete actions, we wish to ﬁnd a minimal set of multi-step actions that encodes
this trajectory. We represent trajectories as binary matrices Tnt2[0;1], where actions are one-hot
encoded.
The standard sparse coding model learns a single layer code and requires ﬁxing the size of the
dictionary elements (the length of the actions) in advance. We propose an alternative method that can
discover potentially hierarchically-nested dictionary elements of arbitrary length with an iterative
coding and compression process.
At all stages, the size of the dictionary elements is set to 2-timesteps. A dictionary and sparse code is
found for the batch of trajectories, by minimizing equation 2. The dictionary element awhich has the
highest explained variance is then selected and assigned an integer code n+ 1. The dimension of the
matrixTis increased to Tn+1t. All 2-step time windows that yielded an active coefﬁcient on this
dictionary element are then replaced with a 1-step one hot vector encoding the dictionary element’s
integer code n+ 1. The length of the trajectory is thus decreased by the number of occurrences of that
dictionary element w. We denote this compression procedure with the function (T;a). This process
is repeated for the new matrix Tn+1tw, forkiterations. In this manner, dictionary elements can
be discovered that contain previously compressed sequences.
Algorithm 1: Sparse Skill Coding
Input: Batch ofmtrajectories encoded as binary matrices Tnt2[0;1]
Output: Dictionary of Khigh-level actions DK
1fork= 1toKdo
2 minw;sPm
i=1jjTiPn
j=1Wjsi;jjj2
2+P
i;jjjsi;jjj1
3s!= arg maxPm
is
4Dk=W!
5Tk= (T;Dk)
6endfor
7returnDK
The result of this process is a set of (potentially nested) high-level actions that capture the statistical
structure of trajectories generated on a task. An agent’s action space can then be augmented to include
these high-level actions, which can facilitate transfer to tasks drawn from the same distribution.
4 R ELATED WORK
Early work in hierarchical reinforcement learning demonstrated that well-designed sub-goals or
high-level actions can signiﬁcantly speed the discovery of shortest-path solutions (Sutton et al., 1999;
Barto & Mahadevan, 2003; Dayan & Hinton, 1993) and facilitate transfer to related tasks (Konidaris
& Barto, 2007). Later work demonstrated the advantages of incorporating pre-deﬁned sub-goals into
deep reinforcement learning (Kulkarni et al., 2016) or pre-learned skills (Tessler et al., 2017), but left
open the question of how to discover these sub-goals or skills automatically.
Recent work have attempted to discover these temporally extended actions by optimizing for reusable
behaviors shared across tasks (Frans et al., 2017), maximizing diversity in exploration (Florensa et al.,
3
Under review as a conference paper at ICLR 2020
2017; Eysenbach et al., 2018; Gregor et al., 2016; Achiam et al., 2018), or by ﬁnding bottlenecks
in demonstrations (Kipf et al., 2018; Co-Reyes et al., 2018), after which these temporally extended
actions are combined with a high-level policy to learn on downstream tasks.
However, in contexts in which bottleneck states are less apparent, approaches for end-to-end learning
of temporally extended actions and policies, such as options (Bacon et al., 2017; Harb et al., 2017)
frequently degenerate to learning either single-step options or only a single option for the entire
trajectory. On the other hand, approaches that mitigate this degeneracy by ﬁxing the horizon length
of each sub-policy (Nachum et al., 2018; Frans et al., 2017). Furthermore, while in theory methods
such as options (Sutton et al., 1999) or hierarchies of abstract machines (Parr & Russell, 1998) could
learn nested behavior, in practice because the number of contexts grows exponentially with depth,
most approaches focus on learning two-level hierarchies, with the exception of (Fox et al., 2017)
which proposes a method for learning deeper nested hierarchies, but with a ﬁxed number of options
available at each depth.
Nested structure is characteristic of problems in natural language processing (Socher et al., 2011)
or program induction (Parisotto et al., 2016), but approaches in these ﬁelds usually have access to
additional top-down supervision on tree structure. Our method discovers variable-length temporally
extended actions in a bottom-up fashion from demonstration , and we show that our method is
able to nest temporally extended actions and transfer to recursively structured environments where
bottlenecks are not that clearly apparent.
5 E XPERIMENTS AND RESULTS
In our experiments, we ask the following questions:
Can sparse skill coding learn temporally extended actions that reﬂect the nested hierarchy of
a task?
Do the temporally extended actions learned from sparse skill coding better capture behav-
ioral motifs than hierarchical RL methods that learn to identify sub-goals?
Can an agent transfer these temporally extended actions to learn more quickly on a series of
recursively more complex environments?
We ﬁnd that in contrast to those learned in subgoal-based hierarchical approaches, the temporally
extended actions learned from sparse skill coding reﬂect commonly repeated patterns of behavior that
can be used to build a nested hierarchy, and such a nested hierarchy enables the agent to continually
transfer to recursively more complex environments.
To evaluate our approach, we consider the Lightbot domain (explained in further detail in Sec-
tion 5.1.1) and the classic four rooms domain. We compare with an option-critic baseline (Bacon
et al., 2017) trained with proximal policy optimization (Schulman et al., 2017).
5.1 E XPERIMENT 1: L EARNING SPARSE SKILLS FROM DEMONSTRATION
To understand the properties of representations learned with this method, we ﬁrst present a qualitative
analysis of the representations learned by sparse skill coding performed on trajectories generated by
an expert policy, and contrast these learned representations with those learned via option-critic (Bacon
et al., 2017) on the same task.
5.1.1 T ASKS
We compare the abstractions learned by sparse skill coding and option-critic on a task that highlights
the relevance of identifying behavioral “motifs” over subgoal states.
Lightbot: The Lightbot domain (Figure 1) is adapted from a game developed to teach children how
to program. For each level in the game, there exists a compact, hierarchical program that generates
the solution. In the original game, the objective is to ﬁnd the shortest program that solves the level.
Whereas Sanborn et al. (2018) used the Lightbot domain to study hierarchical learning in humans,
we adapt the Lightbot game as a novel domain for hierarchical RL methods: the agent begins in a
random location and direction in the room and must navigate the room to turn on all of the lights
4
Under review as a conference paper at ICLR 2020
(blue tiles) using ﬁve basic actions: walk ,jump ,right ,left , andlight (which turns on the
light). This domain presents a challenging sparse reward task: the agent receives a positive reward of
only if it successfully turns off all lights.
Figure 1: The Lightbot domain.
5.1.2 S UB-GOALS VS . M OTIFS
The repeated patterns in the solutions for Lightbot puzzles serve to test whether methods that discover
nested hierarchical structure, such as ours, are able to learn re-usable temporally extended-actions
that better reﬂect the structure of the environment than methods that chain together subtrajectories
between sub-goals. Figure 2 visualizes the action sequences generated while optimizing a policy with
proximal policy optimization (PPO) (Schulman et al., 2017) in the Lightbot and Four Rooms domain.
In environments with nested hierarchical structure, such as the Lightbot domain, compressible
sequential structure emerges in the agent’s action sequences. This structure can be compactly encoded
with a short, hierarchical code. In domains more conducive to sub-goal approaches, such as Four
Rooms, behavioral motifs are less apparent; solutions chain together sequences of repeated actions
(e.g. [ right ,right ,right ], [down ,down ,down ]). Such sequences could be compressed with
a run-length encoding scheme, but lack the nested structure that requires hierarchical compression
schemes.
Figure 2: Hierarchical structure in the trajectories of a PPO agent in the Lightbot domain.
Figure 3: Convergence of action trajectories in the four rooms domain. Converged trajectories do not
contain the hierarchically nested structure present in the Lightbot domain.
5.1.3 R ESULTS
An expert policy was obtained with PPO on the Lightbot puzzle in Figure 4 under a shaped reward
structure, where +10 reward was received for every light turned on, and -1 for every other action.
The policy was trained to convergence with a learning rate of 105for 10,000 episodes, with a
gradient update every 100 episodes and a maximum episode length of 100 timesteps. A batch of
5
Under review as a conference paper at ICLR 2020
1,000 trajectories was generated from the converged policy and encoded with sparse skill coding
for 8 iterations, yielding a set of 8 nested hierarchical actions. The action space of a new agent was
augmented with these 8 actions and its policy was trained to convergence.
We compare the skills learned with sparse skill coding to options learned with option-critic (Bacon
et al., 2017) trained with PPO. The same hyperparameters were used for both algorithms, with the
addition of a deliberation cost (Harb et al., 2017) of 0:05for option-critic.
Figure 4 shows the normalized cumulative terminations per state for each option and skill learned
by these two methods in the Lightbot domain. Options learned with option-critic show some
specialization, but are highly redundant and fail to capture the nested structure inherent in the task.
Sparse skill coding learns separable skills that reﬂect the structure of the environment.
Figure 4: Normalized cumulative terminations per state for options learned via option-critic (left) and
skills learned via sparse skill coding (right).
5.2 E XPERIMENT 2: L EARNING SKILLS FOR CONTINUAL TRANSFER
A motivation for learning high-level temporally-extended actions in the ﬁrst place is that it reduces
thecognitive cost of choosing a series of actions to the cognitive cost of choosing only one action.
Therefore, the potential beneﬁt of discovering behavioral motifs as high-level actions is that such
high-level actions not only could be re-used in various related contexts, but could serve as primitives
for building even higher-level motifs for even more complex domains. The decision to add a high-
level action to the agent’s repertoire of skills pays upfront the cognitive cost of taking that particular
series of primitive actions, such that the agent need not pay such a cost when invoking the high-level
action for future learning.
We are interested in understanding the implications that the iterative encapsulation of higher and
higher-level actions have as the agent faces a task more complex than tasks it has trained on previously.
Environments that exhibit a recursive or fractal structure, such as the Tower of Hanoi, offer a natural
suite of tasks that grow rapidly in complexity from the perspective taking primitive actions, but whose
solutions are straightforward if sub-solutions to easier problems may be re-used. Many real-world
problems have many nested layers of complexity and such fractal environments boils such nested
structure into its purest form, allowing us to take a ﬁrst step towards understanding how an intelligent
agent may re-use primitive sub-solutions to enable learning on more complex versions of problems it
has encountered.
In the quantitative results that follow, we are not as interested in the asymptotic performance of SSC
compared to standard approaches for continual transfer as much as we are in the speed at which SSC
adapts as well as the compositional structure of the trajectories that SSC learns.
5.2.1 T ASKS
We consider two domains, the Tower of Hanoi puzzle (Figure 5) and Fractal Lightbot (Figure 6).
These domains are organized into two levels each as follows:
Level Tower of Hanoi Fractal Lightbot
0 2 disks 1 cross
1 3 disks 2 crosses
6
Under review as a conference paper at ICLR 2020
For level 1, we initialize an SSC agent with weights from a PPO agent trained on level 0 and with
an augmented action space created from encoding trajectories from the PPO agent trained on level
0. We also compare with PPO and option-critic agents that were (1) trained from scratch and (2)
transferred from the previous level 0.
Tower of Hanoi: The Tower of Hanoi is a classic puzzle that has been extensively studied in cognitive
psychology and planning (Anderson, 1990). In the Tower of Hanoi, the player must move a stack
ofndisks from one peg to another by moving each disk one at a time, with the restriction that the
player cannot place a larger disk on top of a smaller one. One of the notable properties of this task is
that the graph of its state space is a fractal resembling the Sierpinski triangle. Due to its cyclic nature,
the optimal solution to the task is a recursive algorithm, which requires 2n1moves. We note that
the recursive structure of this tasks can be exploited by an agent transferring its learning across tasks
of increasing complexity, as the solutions to the nstep problem are contained within the solutions to
then+1 step problem.
We model each Tower of Hanoi puzzle as a sparse reward reinforcement learning problem in which
the agent receives a reward of 0 from the environment for every action taken and a reward of 10 for
successfully transferring the tower of disks. In addition, the agent incurs a cognitive cost of -1 for
every action taken. On each episode, the tower of disks is initialized on a random peg.
Figure 5: (Left) The Tower of Hanoi. (Right) State space for the three disk problem.
Fractal Lightbot: Fractal Lightbot is an adaptation of the Lightbot puzzles built on top of the
Minigrid environment (Chevalier-Boisvert et al., 2018), which permits the use of images as state
representations. The dimensions of the observations are ﬁxed as the complexity of the puzzles
increases; the agent’s observations are 9x9 images showing overhead views of the portion of the
environment that is directly in front of the agent, which changes as the agent moves around. Unlike
the original Lightbot game, we removed the possibility of tiles at multiple heights, and the agent
is able to turn on the light when it is in front of rather than on top of the light tile so that it is not
occluded.
We also model each Fractal Lightbot puzzle as a sparse reward reinforcement learning problem in
which the agent receives a reward of 0 from the environment for every action taken and a reward of
100 for successfully transferring the tower of disks. In addition, the agent incurs a cognitive cost of -1
for every action taken. On each episode, the agent is initialized in a random location and direction.
Figure 6: Lightbot puzzles that grow in a fractal manner. left: level 0, right : level 1
7
Under review as a conference paper at ICLR 2020
Figure 7: Transfer learning in the Tower of Hanoi. Each line averages over 3 different random seeds.
Error bars show 95% conﬁdence intervals.
Figure 8: Transfer learning in Fractal Lightbot. Each line averages over 3 different random seeds.
Error bars show 95% conﬁdence intervals.
5.2.2 R ESULTS
Figure 7 compares SSC with our baselines on transferring from level 0 (2 disks) to level 1 (3 disks)
for Tower of Hanoi, and Figure 8 compares SSC with our baselines on transferring from level 0 (1
cross) to level 1 (2 crosses) for Fractal Lightbot. We observe that SSC performs much better than
option critic. SSC transfer slightly slower than PPO, possibly because exploring with long high-level
actions potentially is more costly.
6 D ISCUSSION
Our goal is (1) to understand how to design an algorithm that can discover nested behavioral
hierarchies of arbitrary depth, with actions of arbitrary length and (2) understand how reducing the
cognitive cost of choosing actions with high-level actions affect transfer. Sparse skill coding is our
method for studying these questions. As our method is a bottom-up method for learning nested
hierarchies of temporally extended actions, we are able to avoid the computational complexity that
make learning nested hierarchies of more than two levels difﬁcult. We have also shown the distinction
between hierarchies characterized by subgoals and hierarchies characterized by recurring motifs. We
hope this paper motivates future work in learning nested behavioral hierarchies.
REFERENCES
Joshua Achiam, Harrison Edwards, Dario Amodei, and Pieter Abbeel. Variational option discovery
algorithms. arXiv preprint arXiv:1807.10299 , 2018.
8
Under review as a conference paper at ICLR 2020
John Robert Anderson. The adaptive character of thought . Psychology Press, 1990.
Pierre-Luc Bacon, Jean Harb, and Doina Precup. The option-critic architecture. In Association for
the Advancement of Artiﬁcial Intelligence , pp. 1726–1734, 2017.
Horace Barlow. Possible principles underlying the transformations of sensory messages. In Rosenblith
W (ed.), Sensory Communication , chapter 13, pp. 217–234. MIT press, 1961.
Andrew G Barto and Sridhar Mahadevan. Recent advances in hierarchical reinforcement learning.
Discrete Event Dynamic Systems , 13(4):341–379, 2003.
William G Chase and Herbert A Simon. Perception in chess. Cognitive psychology , 4(1):55–81,
1973.
Maxime Chevalier-Boisvert, Lucas Willems, and Suman Pal. Minimalistic gridworld environment
for openai gym. https://github.com/maximecb/gym-minigrid , 2018.
John D Co-Reyes, YuXuan Liu, Abhishek Gupta, Benjamin Eysenbach, Pieter Abbeel, and Sergey
Levine. Self-consistent trajectory autoencoder: Hierarchical reinforcement learning with trajectory
embeddings. arXiv preprint arXiv:1806.02813 , 2018.
Thomas M Cover and Joy A Thomas. Elements of information theory . John Wiley & Sons, 2012.
Peter Dayan and Geoffrey E Hinton. Feudal reinforcement learning. In Advances in neural information
processing systems , pp. 271–278, 1993.
Benjamin Eysenbach, Abhishek Gupta, Julian Ibarz, and Sergey Levine. Diversity is all you need:
Learning skills without a reward function. arXiv preprint arXiv:1802.06070 , 2018.
Tamar Flash and Binyamin Hochner. Motor primitives in vertebrates and invertebrates. Current
opinion in neurobiology , 15(6):660–666, 2005.
Carlos Florensa, Yan Duan, and Pieter Abbeel. Stochastic neural networks for hierarchical reinforce-
ment learning. arXiv preprint arXiv:1704.03012 , 2017.
Roy Fox, Sanjay Krishnan, Ion Stoica, and Ken Goldberg. Multi-level discovery of deep options.
arXiv preprint arXiv:1703.08294 , 2017.
Kevin Frans, Jonathan Ho, Xi Chen, Pieter Abbeel, and John Schulman. Meta learning shared
hierarchies. arXiv preprint arXiv:1710.09767 , 2017.
Karol Gregor, Danilo Jimenez Rezende, and Daan Wierstra. Variational intrinsic control. arXiv
preprint arXiv:1611.07507 , 2016.
Jean Harb, Pierre-Luc Bacon, Martin Klissarov, and Doina Precup. When waiting is not an option:
Learning options with a deliberation cost. arXiv preprint arXiv:1709.04571 , 2017.
Tom ´aˇs Hrom ´adka, Michael R DeWeese, and Anthony M Zador. Sparse representation of sounds in
the unanesthetized auditory cortex. PLoS biology , 6(1):e16, 2008.
Thomas Kipf, Yujia Li, Hanjun Dai, Vinicius Zambaldi, Edward Grefenstette, Pushmeet Kohli, and
Peter Battaglia. Compositional imitation learning: Explaining and executing one task at a time.
arXiv preprint arXiv:1812.01483 , 2018.
Andrei N Kolmogorov. Three approaches to the quantitative deﬁnition oﬁnformation’. Problems of
information transmission , 1(1):1–7, 1965.
George Konidaris and Andrew G Barto. Building portable options: Skill transfer in reinforcement
learning. In International Joint Conference on Artiﬁcial Intelligence , volume 7, pp. 895–900, 2007.
Richard E Korf. Macro-operators: A weak method for learning. Artiﬁcial intelligence , 26(1):35–77,
1985.
Tejas D Kulkarni, Karthik Narasimhan, Ardavan Saeedi, and Josh Tenenbaum. Hierarchical deep
reinforcement learning: Integrating temporal abstraction and intrinsic motivation. In Advances in
neural information processing systems , pp. 3675–3683, 2016.
9
Under review as a conference paper at ICLR 2020
Sergey Levine, Chelsea Finn, Trevor Darrell, and Pieter Abbeel. End-to-end training of deep
visuomotor policies. The Journal of Machine Learning Research , 17(1):1334–1373, 2016.
V olodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare,
Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level control
through deep reinforcement learning. Nature , 518(7540):529, 2015.
Oﬁr Nachum, Shixiang Shane Gu, Honglak Lee, and Sergey Levine. Data-efﬁcient hierarchical
reinforcement learning. In Advances in Neural Information Processing Systems , pp. 3303–3313,
2018.
Craig G Nevill-Manning and Ian H Witten. Identifying hierarchical structure in sequences: A
linear-time algorithm. Journal of Artiﬁcial Intelligence Research , 7:67–82, 1997.
Bruno A Olshausen and David J Field. Emergence of simple-cell receptive ﬁeld properties by learning
a sparse code for natural images. Nature , 381(6583):607, 1996.
Emilio Parisotto, Abdel-rahman Mohamed, Rishabh Singh, Lihong Li, Dengyong Zhou, and Pushmeet
Kohli. Neuro-symbolic program synthesis. arXiv preprint arXiv:1611.01855 , 2016.
Ronald Parr and Stuart J Russell. Reinforcement learning with hierarchies of machines. In Advances
in neural information processing systems , pp. 1043–1049, 1998.
Cindy Poo and Jeffry S Isaacson. Odor representations in olfactory cortex:sparse coding, global
inhibition, and oscillations. Neuron , 62(6):850–861, 2009.
Sophia Sanborn, David Bourgin, Michael Chang, and Thomas Grifﬁths. Representational efﬁciency
outweighs action efﬁciency in human program induction. In Proceedings of the Annual Meeting of
the Cognitive Science Society , 2018.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy
optimization algorithms. arXiv preprint arXiv:1707.06347 , 2017.
David Silver, Aja Huang, Chris J Maddison, Arthur Guez, Laurent Sifre, George Van Den Driessche,
Julian Schrittwieser, Ioannis Antonoglou, Veda Panneershelvam, Marc Lanctot, et al. Mastering
the game of go with deep neural networks and tree search. nature , 529(7587):484, 2016.
Herbert A Simon. The architecture of complexity. In Facets of systems science , pp. 457–476. Springer,
1991.
Richard Socher, Cliff C Lin, Chris Manning, and Andrew Y Ng. Parsing natural scenes and natural
language with recursive neural networks. In Proceedings of the 28th international conference on
machine learning (ICML-11) , pp. 129–136, 2011.
Alec Solway, Carlos Diuk, Natalia C ´ordova, Debbie Yee, Andrew G Barto, Yael Niv, and Matthew M
Botvinick. Optimal behavioral hierarchy. PLoS computational biology , 10(8):e1003779, 2014.
Martin Stolle and Doina Precup. Learning options in reinforcement learning. In International
Symposium on abstraction, reformulation, and approximation , pp. 212–223. Springer, 2002.
Richard S Sutton, Doina Precup, and Satinder Singh. Between MDPs and semi-MDPs: A framework
for temporal abstraction in reinforcement learning. Artiﬁcial intelligence , 112(1-2):181–211, 1999.
Chen Tessler, Shahar Givony, Tom Zahavy, Daniel J Mankowitz, and Shie Mannor. A deep hierar-
chical approach to lifelong learning in minecraft. In Thirty-First AAAI Conference on Artiﬁcial
Intelligence , 2017.
William E Vinje and Jack L Gallant. Sparse coding and decorrelation in primary visual cortex during
natural vision. Science , 287(5456):1273–1276, 2000.
10
Under review as a conference paper at ICLR 2020
A D ETAILS FOR EXPERIMENT 2
A.1 A GENT DETAILS
Tower of Hanoi: Observations for an m-disk, 3-peg Hanoi task are represented with m3vectors
encoding the location of each disk. Each action is parameterized as source peg, target peg
which automatically moves the topmost disk on the source peg on top of the topmost disk of
thetarget peg. Observations are encoded with a 3-layer fully-connected network with a hidden
dimension of 256 units and ReLU activations at each layer. The last layer produces the action
distribution.
Fractal Lightbot: Observations for the Fractal Lightbot task are encoded with a 3-layer CNN with
hidden dimensions of 16, 32, and 64 to encode the image observations, with kernels of size (2;2),
stride of 1, and 2 fully-connected output layers of 256 dimensions, with ReLU activations at every
layer. The last layer produces the action distribution.
A.2 T RAINING DETAILS
For level 1, we initialize an SSC agent with weights from a PPO agent trained on level 0 and with an
augmented action space created from encoding trajectories from the PPO agent trained on level 0.
We compare with the following baselines:
A PPO agent trained on level 1 from scratch.
A PPO agent trained on level 1 with weights initialized from training a PPO agent on level
0.
An option-critic agent trained on level 1 from scratch.
An option-critic agent trained on level 1 with weights initialized from training an option-
critic agent on level 0.
SSC is trained using PPO. Because the environments are all sparse reward environments, we collect
the minimum amount of whole episodes whose aggregate number of transitions is greater than or
equal to 4096 before doing every gradient update. For PPO we use a clip ratio of 0.1 and a weight
decay penalty of 1e-5. Option critic was initialized with 4 options.
The hyperparameters for each agent to converge were found using an informal search:
For Fractal Lightbot, we used a learning rate decay of 0.99 every 100 updates. For Tower of
Hanoi, we used a learning rate decay of 0.95 every 100 updates.
We did not set a ﬁxed horizon for the episodes and trained all models for 1,000,000
transitions. For training PPO and option critic from scratch on level 1 in Fractal Lightbot,
we trained for 3,000,000 transitions because this was the amount needed for the agents to
converge.
When transferring from level 0 to level 1, we found that initializing the agents from the last
checkpoint from level 0 had a difﬁcult time exploring the new level because (1) the environ-
ment has sparse rewards and (2) the weights were optimized for level 0 only. Therefore, we
initialize all agents from a checkpoint saved one-third the way through training before the
return curve plateaus. For every task this point occurred after the agent has trained on about
4000 episodes, so we used the checkpoint at 4000 episodes as a standard.
11

Sophia Sanborn
Michael Chang
Sergey Levine
Thomas Griffiths


What the Baldwin Effect affects depends on the nature of plasticity.

What the Baldwin Effect affects depends on the nature of plasticity
===================================================================

Abstract
--------

In a process known as the Baldwin Effect, developmental plasticity, such as learning, has been argued to accelerate the biological evolution of high-fitness traits, including language and complex intelligence. Here we investigate the evolutionary consequences of developmental plasticity by asking which aspects of a plastic trait are the focus of genetic change. The aspects we consider are: (*i*) dependencies between elements of a trait, (*ii*) the importance of each element to fitness, and (*iii*) the difficulty of acquiring each element through plasticity. We also explore (*iv*) how cultural inheritance changes the relationship between plasticity and genetic change. We find that evolution by natural selection preferentially fixes elements that are depended upon by others, important to fitness, or difficult to acquire through plasticity, but that cultural inheritance can suppress and even reverse genetic change. We replicate some of these effects in experimental evolutionary simulations with human learners. We conclude that what the Baldwin Effect affects depends upon the mechanism of plasticity, which for [behavior](/topics/neuroscience/behavior-neuroscience "Learn more about behavior from ScienceDirect's AI-generated Topic Pages") and cognition includes the [psychology of learning](/topics/social-sciences/psychology-of-learning "Learn more about psychology of learning from ScienceDirect's AI-generated Topic Pages").

Introduction
------------

The interaction between developmental plasticity, expressed through processes such as learning, and biological evolution has received considerable attention over the past few decades (Gabora, 2008; Scheiner, 1993; Via et al., 1995). However there remains considerable debate over whether plasticity is a driving force in evolution, taking the lead in adaptation with genetic change following afterwards (Laland, Wray, & Hoekstra, 2014; Pigliucci, Murren, & Schlichting, 2006; West-Eberhard, 2003). One way in which plasticity and evolution may interact is the Baldwin Effect (Baldwin, 1896; Weber & Depew, 2003), which proposes that evolution proceeds by selection favoring genetic variants that support adaptive traits that arise via plasticity. Over time, accumulated genetic change allows traits to be reliably acquired by all members of the species and may reduce their plasticity. This can be contrasted with an aplastic model in which novel traits arise purely through genetic mutation.

Baldwin's theory, originally called “organic selection”, has had variable success since its proposal by several figures in the late 19th century (Baldwin, 1896; Lloyd Morgan, 1896; Morgan & Harris, 2015; Osborne, 1896; Richards, 1987; Weber & Depew, 2003). Initially regarded as an important part of the evolutionary process, a scandal concerning Baldwin's personal life, the development of mathematical evolutionary theory, and an increasing segregation between developmental and evolutionary biology all contributed to Baldwin's theory falling out of favor in the early 20th century (Richards, 1987). Simpson briefly mentioned it shortly after the formation of the Modern Synthesis, coining the term “Baldwin Effect” (Simpson, 1953), but it remained on the periphery of evolutionary thinking.

Despite this mixed history, and although the Baldwin Effect remains contested (Richards, 1987; Weber & Depew, 2003), it has made a modest resurgence in the last few decades. In particular, it has been invoked in several cases that are a challenge for aplastic models of evolution, such as when fitness landscapes are heavily skewed with only a small number of genotypes associated with high fitness (Hinton & Nowlan, 1987). This *needle*-*in*-*a*-*haystack* problem is hard for selection to solve without plasticity because the multitude of low fitness genotypes create a flat fitness landscape. In a demonstration that has come to be associated with the Baldwin Effect, Hinton and Nowlan (1987) conducted a series of simulations in which they introduced plasticity by including an allele that did not specify the phenotype, but instead prompted the organism to developmentally explore different phenotypic options. The existence of this allele not only accelerated the rate at which beneficial phenotypes were discovered, but also accelerated the evolution of the beneficial genotype because the plastic alleles created a fitness gradient that natural selection could follow.

Other work has built on these results, showing the strengths and limitations of the Baldwin Effect. For instance, while the ability of plasticity to accelerate the *appearance* of beneficial phenotypes has received support (Ancel, 1999; Fontanari & Santos, 2017; Santos, Szathmáry, & Fontanari, 2015), theory suggests it ultimately slows their genetic fixation because once traits can be reliably acquired through plasticity there is little selective advantage in further genetic change (Ancel, 1999, Ancel, 2000; Fontanari & Santos, 2017). Moreover, once a beneficial phenotype has appeared, assuming it has a sufficient fitness benefit, it can spread as rapidly in a non-plastic sexually reproducing population as in a plastic population (Santos et al., 2015). Other potential limitations include the observation that plastic traits may need to be widespread before selection can overcome genetic drift and increase aplastic genetic variants that support them (Chater, Reali, & Christiansen, 2009), and that the hypothesized reduction in plasticity following plastic adaptation may be sufficiently slow that plasticity is minimally reduced before the environment is likely to change again (Scheiner, Barfield, & Holt, 2017). Nonetheless, many models of evolution support the general plausibility of a process where (*i*) adaptation is initially plastic, (*ii*) plasticity increases in response to environmental novelties, and (*iii*) environmental stability leads to the replacement of plasticity with fixed genetic influence (Ancel, 1999; Lande, 2009; Scheiner et al., 2017).

Many animal behaviors are potential candidates for such an evolutionary process. For example, archer fish propel jets of water from their mouth to capture flying insects (Schuster, 2007), shaping their mouth to focus the jets on prey at different distances (Gerullis & Schuster, 2014). Plasticity is clearly relevant to this behavior as young fish need to learn the technique by watching others (Schuster, Wöhl, Griebsch, & Klostermeier, 2006). While it is possible that this behavior evolved without plasticity, this would have required the existence of a mutation that directly caused (at least a primordial form of) the behavior. Alternatively, ancestral archer fish may have learned to target insects above the water's surface and subsequent genetic change reinforced and supported this behavior. Such a genetic response can be seen in their eyes which have evolved to accurately perceive airborne insects from underwater (Temple, Hart, Marshall, & Collin, 2010), a trait that presumably followed, not preceded, the behavior.

Other traits for which plasticity has been argued to be important are those involving coordination between individuals, most prominently language (Deacon, 1997; Pinker & Bloom, 1990). Aplastic accounts must suppose that language arose through genetic change. However, this implies that it first appeared in a single individual. As the fitness benefit of language requires more than one individual to possess it, this would hinder its spread. Plasticity has been used in two ways to solve this problem. In the more limited case, language may have first arisen via a genetic mutation, but was nonetheless able to spread between individuals due to plasticity thereby bringing fitness benefits to its users (Pinker & Bloom, 1990). Assuming the mutation enhances language acquisition, there would then be a selective pressure favoring its spread. Other theories go further, suggesting that language both arose and spread via plasticity and that genetic change was entirely in response, favoring genetic variants that supported language acquisition (i.e., the Baldwin Effect) (Deacon, 1997).

These arguments apply to all cases where phenotypes need to be coordinated across individuals, including animal communication systems, such as birdsong. While it is possible that song could have evolved without plasticity (for instance, through sensory exploitation), consistent with the Baldwin Effect (Ancel, 1999, Ancel, 2000), song development is often plastic (e.g. Thorpe, 1961) and species-typical song can emerge with remarkably limited stimuli (Feher, Suzuki, Okanoya, Ljubicic, & Tchernichovski, 2014).

Another means by which plasticity has been suggested to drive evolutionary change, this time in the case of human intelligence, is as an “evolutionary crane” (Dennett, 2003). This account supposes that our ancestors used their (more limited) cognition to develop behaviors that increased their fitness and that were sufficiently hard to acquire that selection favored genetic variants supporting their acquisition. Rather than being trait specific, genetic change increased our general cognitive capacities, which led to the discovery of even more complex behaviors and thereby redoubled selection on cognition. Thus, our evolution featured a coevolution of cognition and its products that resulted in huge changes to our cognitive abilities. In support of this argument, theoretical work has found that this process will occur provided a series of increasingly complex and successful possible behaviors exists, and that it can produce evolutionary dynamics similar to the hominin archaeological record (Morgan, 2016).

While this pervious work has identified cases in which plasticity may have played an important role, it remains unclear precisely how we might expect plasticity to affect genetic change. Which genes will be targeted? To what extent will behaviors become fixed? Here, we address these questions with theoretical analyses addressing how the effect of developmental plasticity on genetic change is modulated by (*i*) how traits interact with each other to produce fitness benefits, (*ii*) how important traits are to fitness, and (*iii*) how readily traits are acquired through plasticity. We then extend these analyses to include cultural inheritance, allowing the prevalence of the traits in the population to affect the probability they are acquired through plasticity. Finally, we test the predictions resulting from our theoretical analyses in an experimental simulation with a population of human learners. Our results present a clearer picture of what the Baldwin Effect affects: traits that are difficult to acquire, important to fitness, or that are required by other traits are likely to come under increasing genetic influence. These results have significant implications for understanding the potential role of the Baldwin Effect in explaining human traits such as language and higher-level cognition.

Section snippets
----------------

The simulation framework
------------------------

Based on Hinton and Nowlan (1987), we consider a population of 1000 asexual, haploid organisms whose fitness is determined by the acquisition of *n* traits. The acquisition of the *t*th trait by the *i*th individual is affected by a corresponding genetic locus, *G**i*,*t* . Accordingly, each organism has *n* genetic loci. Each locus contains one of two possible alleles: *fixed* and *plastic*. A fixed allele means that the organism is guaranteed to acquire the corresponding trait, while a plastic allele means

Experiment
----------

The above simulations rely on simplifying assumptions about the nature of learning and the traits in question. This means there is a need to verify whether theoretical results hold with real learning capacities which may depart from these assumptions. For instance, errors may be biased, or individuals may vary in their ability to learn or in their motivations. Therefore, to empirically test some of our theoretical findings, we conducted an “experimental evolutionary simulation” with human

Discussion
----------

Developmental plasticity has been argued to be an important part of the evolutionary process explaining how selection (i) can solve a *needle*-*in*-*a*-*haystack* problem (Hinton & Nowlan, 1987), (ii) can favor traits that rely on coordination across multiple individuals, specifically language (Pinker & Bloom, 1990), and (iii) can favor complex cognition (Deacon, 1997; Dennett, 2003; Morgan, 2016). Here, we built on these results using theoretical and experimental simulations to explore the impact of

Author contributions
--------------------

TJHM and TLG conceived of the project. TJHM created the models. TJHM and JWS created and ran the experiments. TJHM analyzed the data. All authors wrote the manuscript.

Acknowledgements
----------------

A preliminary version of this work was presented at the Annual Conference of the Cognitive Science Society and appears in the proceedings (Morgan & Griffiths, 2015). This work was funded by NSF grants 1456709 and 1408652.

References (53)
---------------

* L.W. Ancel### [A quantitative model of the Simpson-Baldwin effect](/science/article/pii/S0022519398908335)

### Journal of Theoretical Biology

(1999)
* L.W. Ancel### [Undermining the Baldwin expediting effect: Does phenotypic plasticity accelerate evolution?](/science/article/pii/S0040580900914849)

### Theoretical Population Biology

(2000)
* K. Aoki *et al.*### [Evolution of learning strategies in temporally and spatially variable environments: A review of theory](/science/article/pii/S0040580913001081)

### Theoretical Population Biology

(2014)
* T.J. Dewitt *et al.*### [Costs and limits of phenotypic plasticity](/science/article/pii/S0169534797012743)

### Trends in Ecology & Evolution

(1998)
* M. Ehn *et al.*### [Adaptive strategies for cumulative cultural learning](/science/article/pii/S0022519312000756)

### Journal of Theoretical Biology

(2012)
* L. Gabora### [The cultural evolution of socially situated cognition](/science/article/pii/S1389041707000368)

### Cognitive Systems Research

(2008)
* P. Gerullis *et al.*### [Archerfish actively control the hydrodynamics of their jets](/science/article/pii/S0960982214009221)

### Current Biology

(2014)
* S.J. Lycett *et al.*### [A 3D morphometric analysis of surface geometry in Levallois cores: Patterns of stability and variability across regions and their implications](/science/article/pii/S030544031200492X)

### Journal of Archaeological Science

(2013)
* H.J. Muller### [The relation of recombination to mutational advance](/science/article/pii/0027510764900478)

### Mutation Research, Fundamental and Molecular Mechanisms of Mutagenesis

(1964)
* M. Santos *et al.*### [Phenotypic plasticity, the Baldwin effect, and the speeding up of evolution: The computational roots of an illusion](/science/article/pii/S0022519315000715)

### Journal of Theoretical Biology

(2015)

- S. Schuster *et al.*### [Animal cognition: How archer fish learn to down rapidly moving targets](/science/article/pii/S096098220601013X)

### Current Biology

(2006)
- S. Via *et al.*### [Adaptive phenotypic plasticity: Consensus and controversy](/science/article/pii/S0169534700890618)

### Trends in Ecology & Evolution

(1995)
- D.S. Adler *et al.*### Early Levallois technology and the Lower to Middle Paleolithic transition in the Southern Caucasus

### Science

(2014)
- A.V. Badyaev### Evolutionary significance of phenotypic accommodation in novel environments: An empirical test of the Baldwin effect

### Philosophical Transactions of the Royal Society, B: Biological Sciences

(2009)
- J. Baldwin### A new factor in evolution

### The American Naturalist

(1896)
- R. Boyd *et al.*### The cultural niche: Why social learning is essential for human adaptation

### Proceedings of the National Academy of Sciences

(2011)
- N. Chater *et al.*### Restrictions on biological adaptation in language evolution

### Proceedings of the National Academy of Sciences

(2009)
- N. Chomsky### On cognitive structures and their development: A reply to Piaget
- T.W. Deacon### The symbolic species: The co-evolution of language and the brain

(1997)
- D. Dennett### The Baldwin effect: A crane, not a skyhook
- J.W. Earl *et al.*### Mystery of the poisoned expedition

### Nature

(1994)
- O. Feher *et al.*### Birds tutored with their own developing song produce wildtype-like song as adults
- J. Felsenstein### The evolutionary advantage of recombination

### Genetics

(1974)
- J.F. Fontanari *et al.*### The revival of the Baldwin effect

### European Physical Journal B

(2017)
- M. Hébert *et al.*### Inexperienced preys know when to flee or to freeze in front of a threat

### PNAS

(2019)
- C. Heyes### Cognitive gadgets

(2018)
View more referencesCited by (4)
------------

* ### [Sinking In: The Peripheral Baldwinisation of Human Cognition](/science/article/pii/S136466132030214X)

2020, Trends in Cognitive SciencesCitation Excerpt :However, very little of this information from cognitive science has penetrated research on Baldwinisation. Modelling suggests that the probability of Baldwinisation depends crucially on 'details of developmental processes, such as the psychological mechanisms of learning' [17], but most applications of the Baldwin hypotheses continue to assert that behaviour X has been genetically assimilated without asking what type of learning drove the assimilation of X, or about the neurocognitive processes that were targets of selection. In short, the Baldwin effect is a promising hypothesis rather than an empirical effect, and, although Baldwin was a psychologist, cognitive science is conspicuously absent from research done in his name.

Show abstractThe Baldwin effect is a hypothetical process in which a learned response to environmental change evolves a genetic basis. Modelling has shown that the Baldwin effect offers a plausible and elegant explanation for the emergence of complex behavioural traits, but there is little direct empirical evidence for its occurrence. We highlight experimental evidence of the Baldwin effect and argue that it acts preferentially on peripheral rather than on central cognitive processes. Careful scrutiny of research on taste-aversion and fear learning, language, and imitation indicates that their efficiency depends on adaptively specialised input and output processes: analogues of scanner and printer interfaces that feed information to core inference processes and structure their behavioural expression.
* ### [Phenotypic plasticity and archetype: a response to common objections to the biological theory of archetype and instinct](https://doi.org/10.1111/1468-5922.12883)

2023, Journal of Analytical Psychology
* ### [Rethinking Norm Psychology](https://doi.org/10.1177/17456916221112075)

2023, Perspectives on Psychological Science
* ### [Connections between Older Greek Adults’ Implicit Attributes and Their Perceptions of Online Technologies](https://doi.org/10.3390/technologies8040057)

2020, Technologies
Recommended articles (6)
------------------------

* Research article### [Challenges to both reliability and validity of masculinity-preference measures in menstrual-cycle-effects research](/science/article/pii/S0010027720300202)

Cognition, Volume 197, 2020, Article 104201Show abstractAlthough it remains contentious, women's changeable attraction to masculine faces has been used to inform evolutionary ideas about human mating strategies. Typical experiments in this area use two-alternative-forced-choice (2afc) over a few pairs of similar images. The reliability of these measures is analysed suggesting that many studies have too few trials to be reliable. In the current experiment, fertility shifts in preferences for masculinised faces (and Africanised faces) were explored using both attractiveness ratings and a 2afc method over 80 pairs. The 2afc method showed a fertility shift in preferences whereas attractiveness ratings did not show a shift. Further, it was demonstrated how the size of the preferences shown in the 2afc tasks correlated with general face-matching performance. It is concluded that fertility is associated with improved face-processing accuracy and hence 2afc designs have poor validity as measures of masculinity preference. These issues of validity and reliability may have contributed to the contentious nature of fertility effects on preferences. Further, validity and reliability need to be considered in any study where a change in preference is identified using a comparative-preference task.
* Research article### [On the limits of evidence accumulation of the preconscious percept](/science/article/pii/S0010027719302537)

Cognition, Volume 195, 2020, Article 104080Show abstractWhen a stimulus is repeated the memory representation for that stimulus is strengthened and performance in memory tests increases. To what extent this effect requires that each exposure elicits a fully-fledged conscious percept? In two Rapid Serial Visual Presentation (RSVP) experiments we explored whether the memory representations for words would accumulate evidence trough repeated exposure when none of those presentations induced a conscious percept. Participants were instructed to identify repeated items inserted in different RSVP streams and we isolated the first instance that an item was consciously perceived as repeating. The results showed that the probability of detecting a repetition for the first time was constant across repetitions. This effect signals a limit to the evidence accumulation process through repeated exposure. We discuss whether conscious perception modulates the decay of memory representations with below-threshold items resulting in extremely fleeting memory representations.
* Research article### [Exploring the influence of ownership history on object valuation in typical development and autism](/science/article/pii/S0010027720300068)

Cognition, Volume 197, 2020, Article 104187Show abstractItems with special histories (e.g. celebrity owners) or qualities (e.g. limited editions) are more valuable than similar “inauthentic” items. Typically developing (TD) children privilege authenticity and are particularly influenced by who objects belong to. Here, we explore why children and adults over-value items with special ownership histories and examine how autism spectrum disorder (ASD) affects object valuation. In Studies 1 and 2, TD children perceived items belonging to famous owners (with “good” or “bad” reputations) to be more valuable than similar items belonging to non-famous owners. However, they ascribed significantly higher values to items belonging to famous heroes than infamous villains when compared. Children with ASD did not over-value objects with special ownership histories, but their valuations were moderated by qualities unrelated to ownership (e.g. rarity). In Study 3, adults with ASD assigned high values to authentic items with special ownership histories but were more likely to keep inauthentic objects than neurotypical adults. Our findings show that association with a famous owner is sufficient to increase an item's value for TD children and adults (with and without ASD). The degree of added value may be determined by the famous owner's character for TD children, but not adults. By contrast, children with ASD value objects via a different strategy that prioritizes material qualities over ownership history. However, the awareness of authenticity displayed by adults with ASD suggests that the emergence of ownership history as an important influence on object evaluation may be developmentally delayed in ASD, rather than completely absent.
* Research article### [The effect of contextual plausibility on word skipping during reading](/science/article/pii/S0010027720300032)

Cognition, Volume 197, 2020, Article 104184Show abstractRecent eye-movement evidence suggests readers are more likely to skip a high-frequency word than a low-frequency word independently of the semantic or syntactic acceptability of the word in the sentence. This has been interpreted as strong support for a serial processing mechanism in which the decision to skip a word is based on the completion of a preliminary stage of lexical processing prior to any assessment of contextual fit. The present large-scale study was designed to reconcile these findings with the plausibility preview effect: higher skipping and reduced first-pass reading times for words that are previewed by contextually plausible, compared to implausible, sentence continuations that are unrelated to the target word. Participants' eye movements were recorded as they read sentences containing a short (3–4 letters) or long (6 letters) target word. The boundary paradigm was used to present parafoveal previews which were either higher or lower frequency than the target, and either plausible or implausible in the sentence context. The results revealed strong, independent effects of all three factors on target skipping and early measures of target fixation duration, while frequency and plausibility interacted on later measures of target fixation duration. Simulations using the *E*-Z Reader model of eye-movement control in reading demonstrated that plausibility effects on skipping are potentially consistent with the assumption that higher-level contextual information only affects post-lexical integration processes. However, no current model of eye movements in reading provides an explicit account of the information or processes that allow readers to rapidly detect an integration failure.
* Research article### [Temporal and spatial ensemble statistics are formed by distinct mechanisms](/science/article/pii/S0010027719303026)

Cognition, Volume 195, 2020, Article 104128
* Research article### [Object features reinstated from episodic memory guide attentional selection](/science/article/pii/S0010027719303324)

Cognition, Volume 197, 2020, Article 104158Show abstractWhen observers search for an object in the environment, they compare the incoming sensory information to the attentional template, a representation of the target in visual working memory (VWM). Previous studies have shown that visual search is more efficient when the attentional template is precise. We pursued the hypothesis that the attentional template in VWM is automatically complemented by features from long-term memory, possibly to increase its precision. At the beginning of the experiment, observers learned associations between shape and color. Then, we tested whether selecting one of these shapes was influenced by the previously associated color. To this end, we ran a saccadic selection task consisting of a memory and choice display. In the memory display, the target shape was presented at central fixation and participants were instructed to foveate this shape in the subsequent choice display. In the choice display, the target shape appeared together with a distractor shape at eccentric positions. Importantly, the target shape was colorless (gray) in the memory display so that only shape, but not color was loaded into VWM. However, saccades went more frequently to the target shape when it was shown in the learned color than when this color was shown in the distractor. Thus, the color of the target shape was reinstated from episodic memory to complement the attentional template in VWM.
[View full text](/science/article/pii/S0010027719303397)

Thomas J
H
Morgan
Jordan W
Suchow
Thomas L
Griffiths