Enhancing metacognitive reinforcement learning using reward structures and feedback.

Enhancing metacognitive reinforcement learning
using reward structures and feedback
Paul M. Krueger1(pmk@berkeley.edu)
Falk Lieder1(falk.lieder@berkeley.edu)
Thomas L. Grifﬁths (tom grifﬁths@berkeley.edu)
Department of Psychology, University of California Berkeley, Berkeley, CA 94720 USA
1These authors contributed equally.
Abstract
How do we learn to think better, and what can we do to pro-
mote such metacognitive learning? Here, we propose that cog-
nitive growth proceeds through metacognitive reinforcement
learning. We apply this theory to model how people learn how
far to plan ahead and test its predictions about the speed of
metacognitive learning in two experiments. In the ﬁrst experi-
ment, we ﬁnd that our model can discern a reward structure that
promotes metacognitive reinforcement learning from one that
hinders it. In the second experiment, we show that our model
can be used to design a feedback mechanism that enhances
metacognitive reinforcement learning in an environment that
hinders learning. Our results suggest that modeling metacog-
nitive learning is a promising step towards promoting cognitive
growth.
Keywords: Decision-Making; Planning; Metacognitive Rein-
forcement Learning; Cognitive Training
Introduction
One of the most remarkable aspects of the human mind is its
ability to improve itself based on experience. Such learning
occurs in a range of domains, from simple stimulus-response
mappings, motor skills, and perceptual abilities, to problem
solving, cognitive control, and learning itself (C. S. Green &
Bavelier, 2008; Bavelier, Green, Pouget, & Schrater, 2012).
Demonstrations of cognitive and brain plasticity have in-
spired cognitive training programs. The success of cognitive
training has been mixed and the underlying learning mecha-
nisms are not well understood (Owen et al., 2010; Anguera
et al., 2013; Morrison & Chein, 2011). Feedback is an im-
portant component of many effective cognitive training pro-
grams, but it remains unclear what makes some feedback
structures more effective than others, and there is no prin-
cipled method for designing optimal feedback structures.
To address these problems, we model cognitive plasticity
as metacognitive reinforcement learning. This perspective al-
lows us to translate methods for accelerating reinforcement
learning in robots (Ng, Harada, & Russell, 1999) into feed-
back structures for cognitive training in humans.
Here, we evaluate this approach in the domain of planning.
As a ﬁrst step, we developed a metacognitive reinforcement
learning model of how people learn how many steps to plan
ahead in sequential decision problems, and we test its predic-
tions empirically. The results of our ﬁrst experiment suggest
that our model can discern which reward structures are more
conducive to metacognitive learning. In our second experi-
ment, we ﬁnd that feedback structures designed based on our
model can accelerate learning to plan.We start by introducing the theory of reinforcement learn-
ing that our approach is based upon. The following two sec-
tions apply this theory to model the problem of deciding how
to decide and the process by which people learn to do so.
We then use this theory to motivate a novel computational
method for designing feedback structures that promote cog-
nitive plasticity and experimentally test the predictions of our
theory. We close with a discussion of the implications of our
results for cognitive training.
Planning and reinforcement learning
A sequential decision problem can be modeled as a Markov
decision process (MDP)
M= (S;A;T;g;r;P0); (1)
where Sis the set of states, Ais the set of actions, T(s;a;s0)
is the probability that the agent will transition from state sto
state s0if it takes action a, 0g1 is the discount factor
on future rewards, r(s;a;s0)is the reward generated by this
transition, and P0is the probability distribution of the initial
state S0(Sutton & Barto, 1998). A policy p:S7!Aspeci-
ﬁes which action to take in each of the states. The expected
sum of discounted rewards that a policy pwill generate in the
MDP Mstarting from a state sis known as its value function
Vp
M(s) =E"
¥
å
t=0gtr(St;p(St);St+1)#
: (2)
The optimal policy p?
Mmaximizes the expected sum of dis-
counted rewards, that is
p?
M=argmax
pE"
¥
å
t=0gtr(St;p(St);St+1)#
: (3)
Solving large planning problems is often intractable be-
cause the number of possible action sequences grows expo-
nentially with the number of steps one plans ahead. When
the state space Sis discrete and relatively small, dynamic
programming can be used to ﬁnd optimal plans in polyno-
mial time (Littman, Dean, & Kaelbling, 1995). But the high-
dimensional, continuous state spaces people have to plan with
in real life are too large for these methods. Instead, peo-
ple seem to rely on approximate planning strategies (Huys
et al., 2015) and often decide primarily based on immediate
and proximal outcomes while neglecting the long-term con-
sequences of their actions (Myerson & Green, 1995). Despite
its fallibility, looking only a few steps ahead can drastically
simplify the planning problem, and this may often be a ne-
cessity for bounded agents with imperfect knowledge of the
environment (Jiang, Kulesza, Singh, & Lewis, 2015). Since
cutting corners in the decision process is both necessary and
problematic, good decision-making requires knowing when
that is admissible and when it is not. Knowing how much to
plan is therefore an important metacognitive skill to learn.
Previous work suggests that this metacognitive skill can
be learned through trial and error (Lieder & Grifﬁths, 2015).
Learning through trial and error can be understood in terms of
reinforcement learning (Sutton & Barto, 1998). While certain
reinforcement learning algorithms can, in principle, learn to
solve arbitrarily complex problems, reinforcement learning
can also be very slow—especially when rewards are sparse
and the optimal policy is far from the learner’s initial strategy.
A common approach to remedy this problem is to give the
algorithm pseudo-rewards for actions that do not achieve the
goal but lead in the right direction (Ng et al., 1999).While
previous work has developed this idea to accelerate learning
a direct mapping from states to actions, we will leverage it to
accelerate learning to plan.
Deciding how to decide
People can use many different decision strategies. This poses
the problem of deciding how to decide (Boureau, Sokol-
Hessner, & Daw, 2015). Previous research on meta-decision-
making has focused on the arbitration between habits ver-
sus planning (Keramati, Dezfouli, & Piray, 2011; Dolan
& Dayan, 2013). While this is an important meta-control
problem, it is only one part of the puzzle because people
are equipped with more than one goal-directed decision-
mechanism. Hence, when the model-based system is in
charge, it has to be determined how many steps it should plan
ahead. Ideally, the chosen planning horizon should achieve
the optimal tradeoff between expected decision quality ver-
sus decision time (Vul, Goodman, Grifﬁths, & Tenenbaum,
2014) and mental effort (Shenhav et al., 2017).
Here, we make the simplifying assumption that people al-
ways choose the action that maximizes their sum of expected
rewards over the next hsteps, for some value of hthat differs
across decisions. A planning horizon of h=1 entails looking
only at the immediate outcome of each action (myopic one-
step planning) whereas a planning horizon larger than one
entails solving a sequential decision problem to form a multi-
step plan. Under this assumption, the meta-decision problem
is to select a planning horizon hfrom a set H=f1;2;;g,
execute the plan, select a new planning horizon, and so on.
More formally, this problem can be formalized as a meta-
level MDP (Hay, Russell, Tolpin, & Shimony, 2012). In our
task, the meta-level MDP is
Mmeta= (Smeta;H;Tmeta;rmeta); (4)
where the meta-level state m2Smeta=f0;1;2;3;4gencodesthe number of remaining moves, and the meta-level action
h2H=f1;2;3;4gis the planning horizon used to make a
decision. The meta-level reward function rmetaintegrates the
cost of planning with the return of the resulting action:
rmeta(mk;hk) =cost(hk)+h
å
t=1r(st;plan(k;hk)
t); (5)
where plan(k;h)
tis the tthaction of the plan formed by looking
hsteps ahead in the meta-level state mk. The meta-decision-
maker receives this reward after the plan has been executed in
its entirety. If the meta-decision-maker selects short planning
horizons there can be multiple plan-act-reward-learn cycles
within a single trial. The cost of planning cost (hk)is deter-
mined by the branching factor bof the decision tree according
to
cost(hk) =lbhkhk; (6)
where bhkis the number of plans, hkis the number of steps
per plan, and lis the cost per planning step. *
Metacognitive reinforcement learning
Solving the problem of deciding how to decide optimally is
computationally intractable but the optimal solution can be
approximated through learning (Russell & Wefald, 1991).
We propose that people use reinforcement learning (Sutton
& Barto, 1998) to approximate the optimal solution to the
meta-decision problem formulated in Equation 4.
Model
Our model of metacognitive reinforcement learning builds on
the semi-gradient SARSA algorithm (Sutton & Barto, 1998)
that was develop to approximately solve MDPs with large or
continuous state spaces. Speciﬁcally, we assume that people
learn a linear approximation to the meta-level Q-function
Qmeta(mk;hk)7
å
j=1wjfj(mk;hk); (7)
whose features fcomprise one indicator variable for each pos-
sible planning horizon h(f1=1(h=1);;f4=1(h=4)),
one indicator variable for whether or not the agent planned
alllsteps until the end of the task ( f5=1(h=l)), the num-
ber of steps that were left unplanned ( f6=maxf0;lhg),
and the number of steps the agent planned too far ( f7=
maxf0;hlg). The semi-gradient SARSA algorithm learns
the weights of these features by gradient descent. To bring
it closer to human performance, our model replaces its gra-
dient descent updates by Bayesian learning. Concretely, the
weights ware learned by Bayesian linear regression of the
bootstrap estimate ˆQ(mk;hk)of the meta-level value function
onto the features f. The bootstrap estimator
ˆQ(mk;hk) =rmeta(mk;hk)+hµt;f(m0;h0)i (8)
*This equation assumes a constant branching factor and an upper
bound on the complexity of planning. People’s planning time likely
increases less than exponentially fast with the planning horizon but
our approximation may be sufﬁcient for small problems.
is the sum of the immediate meta-level reward and the pre-
dicted value of the next meta-level state m0. The predicted
value of m0is the scalar product of the the posterior mean µt
of the weights wgiven the observations from the ﬁrst tac-
tions (where t=åk
n=1hn) and the features f(m0;c0)ofm0and
the planning horizon h0that will be selected in that state.
We assume that the prior on the feature weights reﬂects that
it is beneﬁcial to plan until the end ( P(f5) =N(µ=1;s=
0:1)), although planning is costly ( P(f1) =P(f2) =P(f3) =
P(f4) =N(µ=1;s=0:1)), and that planning too much is
more costly than planning too little ( P(f7) =N(µ=1;s=
0:1)andP(f6) =N(µ=0;s=0:1)).
Given the posterior on the feature weights w, the planning
horizon his selected by Thompson sampling. Speciﬁcally,
to make the kthmeta-decision, a weight vector ˜ wis sampled
from the posterior distribution of the weights given the series
of meta-level states, selected planning horizons, and resulting
value estimates experienced so far. That is,
˜wkP(wjEk); (9)
where the set Ek=fe1;;ekgcontains the meta-decision-
maker’s experience from the ﬁrst kmeta-decisions; to be
precise, each meta-level experience ej2Ekis a tuple
mj;hj;ˆQ(mj;cj;µj)
containing a meta-level state, the com-
putation selected in it, and the bootstrap estimates of its Q-
value. The sampled weight vector ˜ wis then used to predict
the Q-values of each possible planning horizon h2Hac-
cording to Equation 7. Finally, the planning horizon with the
highest predicted Q-value is used for decision-making.
By proposing metacognitive reinforcement learning as a
mechanism of cognitive plasticity, our model suggests that re-
ward and feedback are critical for cognitive growth. Concep-
tualizing metacognitive reinforcement learning as a regres-
sion problem suggests that learning how to best think about
a problem should require less practice the stronger the cor-
relation between the features f(m;c)(i.e., the predictors) and
the resulting reward net the cost of thinking (i.e., the crite-
rion; Green, 1991). Here, we apply our model to predict
how quickly people can learn that more planning leads to
better results from the reward structure of the practice prob-
lems. According to the model, learning should be fastest
when the reward increases deterministically with the plan-
ning horizon both within and across problems. By contrast,
learning should be slower when this relationship is degraded
by additional variability in the rewards that is unrelated to
planning. The following experiments test this prediction and
illustrate the model’s utility for designing feedback structures
that promote metacognitive learning.
Experiment 1: Reward structures can help or
hinder learning to plan
Methods
We recruited 304 adult participants from Amazon Mechan-
ical Turk. The task took about 25 minutes, and participants
were paid $2.50 plus a performance-dependent bonus of up to
Figure 1: Screenshot of a problem from Experiment 1.
$2.00. Participants played a series of ﬂight planning games.
The environment consisted of six different cities, each con-
nected to two other cities (Figure 1). Participants began each
trial at a given city, and were tasked with planning a speciﬁed
number of ﬂights. Each ﬂight was associated with a known
gain or loss of money, displayed onscreen. Thus, the partic-
ipants’ task was to plan a route that would maximize their
earnings or minimize their losses, based on the number of
planning steps required for that game.
The experiment comprised thirteen trials total: a sequence
of three practice problems which required planning 2, 3, and
3 steps ahead, respectively, followed by ten 4-step prob-
lems, with a break after trial eight. The order of the two
3-step problems was randomized, and the order of the ten
4-step problems was randomized across the last ten trials of
the experiment. Participants were assigned randomly to one
of two conditions: environments with reward structures de-
signed to promote learning (“diagnostic rewards”), or envi-
ronments with reward structures designed to hinder learning
(“non-diagnostic rewards”).
The problems of the diagnostic rewards condition were au-
tomatically generated to exhibit four characteristics:
1. For each l-step problem, planning h<lsteps ahead gener-
ateslhsuboptimal moves. In other words, each myopic
planner makes the maximum possible number of mistakes.
2. When the number of moves is l, then planning lsteps ahead
yields a positive return, but planning h<lsteps ahead
yields a negative return.
3. The return increases monotonically with the planning hori-
zon from 1 to the total number of moves.
4. Each starting position occurs at least once.
The reward structures used for the non-diagnostic rewards
condition were created by shifting the diagnostic reward
structures so as to degrade the correlation between planning
horizon and reward. Concretely, for half of the problems all
rewards were shifted down such that no amount of planning
could achieve a return better than $10. Since the original
problems were such that the 1-step planner always performed
worst, the shift wasr1+X
lwhere r1is the return of the 1-step
planner, lis the number of steps in the planning problem,
andXis a random number between 10 and 20 that differed
across problems ( XUniform ([10;20])). For the other half
of the problems, all rewards were shifted up by r1+X
lsuch
that all planners achieve a return of at least +$10. These re-
ward structures make it extremely difﬁcult for metacognitive
reinforcement learning to discover that planning is valuable,
because the random shifts greatly diminish the correlation be-
tween planning horizon and reward.
Results
Both model simulations and human behavior demonstrated
enhanced learning in environments with diagnostic rewards.
Figure 2 shows the mean performance of the metacognitive
reinforcement learning model, and the mean performance of
human participants. Here, performance is measured as rela-
tive reward
Rrel= (RRmin)=(RmaxRmin); (10)
where Ris the total reward received during the trial, and Rmin
andRmaxare the highest and lowest possible total reward on
that trial, respectively.
To measure the effects of condition and trial number
on performance in human participants, we ran a repeated-
measures ANOV A. This revealed a signiﬁcant effect of both
trial number ( F(9;2989) =3:44;p<0:001) and condition
(F(9;3029) =15:26;p<0:0001), such that participants im-
proved over time, and participants with diagnostic feedback
performed better than those without. To measure learning
in each group, we ran a simple linear regression of the rel-
ative reward on the trial number. This revealed a signiﬁcant
regression equation for participants who received diagnostic
rewards ( F(2;302) =11:28;p<0:01), with an R2of 0:59,
but not for participants who received non-diagnostic rewards
(F(2;302) =3:51;p>0:05), with an R2of 0:31, suggesting
that improvement in performance occurred with diagnostic
rewards, but not without.
To analyze the frequency with which participants chose
the optimal route, we performed a multinomial logistic re-
gression of whether or not each participant chose the optimal
route on trial number and group. This revealed signiﬁcant
effects of trial number ( p<106) and group ( p<0:0001).
In addition, we found that participants interacting with a di-
agnostic reward structure learned to plan signiﬁcantly further
ahead than participants interacting with the non-diagnostic
reward structure. When there were four steps left, the aver-
age planning horizon was 2 :96 with diagnostic rewards com-
pared to 2 :65 with non-diagnostic rewards ( t(596) =2:94,
Figure 2: Model predictions and human performance in Ex-
periment 1. Error bars indicate the standard error of the mean.
Model predictions were averaged over 500 simulations.
p<0:01). When the rewards were diagnostic of good plan-
ning, participants’ choices in the ﬁrst step of the 4-step prob-
lems accorded 10 :3% more frequently with 4-step planning
(t(302) =3:57;p<0:001). For 3 remaining steps there
was a signiﬁcant increase in choices according with opti-
mal 1-step ( p<0:01), 2-step ( p<0:01) and 4-step plan-
ning ( p<0:01). For 2 remaining steps, there was a signif-
icant increase in choices according with optimal 1-step plan-
ning ( p<0:0001) without a decrease in agreement with other
planning horizons. Finally, on the last move participants’
choices in the environment with diagnostic rewards corre-
sponded 5 :8% more frequently with optimal 1-step planning
(t(302) =3:71;p<0:001), and signiﬁcantly less frequently
with 2-step and 3-step planning ( p<0:01 and p<0:001). In
summary, diagnostic rewards led to better agreement between
the planning horizon and the number of remaining steps.
Experiment 2: Using feedback to promote
learning to plan
When one has control over the reward structure of an environ-
ment, creating rewards tailored to faster learning may be fea-
sible. However, often environmental rewards are ﬁxed. In Ex-
periment 2, we tested whether providing feedback may be an
effective alternative approach to accelerating learning. When
participants do not plan enough to ﬁnd the optimal route, this
could be because the time cost of planning an optimal route
outweighs its beneﬁts. To change that, we provided feedback
in the form of timeout penalties for short-sighted decisions.
Methods
We recruited 324 adult participants on Amazon Mechani-
cal Turk. The task took about 30 minutes, and participants
were paid $3 :00 plus a performance-dependent bonus of up
to $2 :00. Participants played twenty trials of the ﬂight plan-
ning game described above. These trials were divided into a
training block and a testing block. The training block con-
sisted of six trials requiring 2-step planning, followed by ten
trials requiring 3-step planning. The testing block consisted
of four additional 3-step trials. The order of the 2-step tri-
als and the order of the 3-step trials were randomized across
subjects. Participants were randomly assigned to either the
feedback condition or the control condition.
In the training block, participants in the feedback condition
were told their apparent planning horizon at the end of every
trial and penalized with a timeout that reﬂected the amount
of planning they had eschewed. Concretely, we set the dura-
tions of the timeouts such that the cost of short-sighted de-
cisions was proportional to the amount of necessary planning
the participant had eschewed. Speciﬁcally, the forgone cost of
planning was estimated by cost =2lˆh, where lis the number
of moves for that trial, ˆhis the participant’s apparent plan-
ning horizon, and 2 is the branching factor since each step
entailed a binary decision. The participant’s planning hori-
zon was estimated by the number of consecutive moves con-
sistent with the optimal policy, beginning with the last move,
followed by the second-to-last, etc. At the end of each trial of
the ﬁrst block, participants in the feedback group were penal-
ized with a timeout delay for sub-optimal routes. The delay
was calculated as 7 (cost1)seconds. During this period,
participants were unable to proceed to the next trial. If partic-
ipants performed the optimal route, they were able to proceed
immediately to the next trial.
The control group received no feedback and had to wait
a ﬁxed amount of time at the end of every trial in block 1,
regardless of their performance. This ﬁxed period was set to
8 seconds, to match the mean timeout period for participants
in the feedback group (7.9 seconds). Neither group received
feedback or delays in the test block.
The planning problems presented in this experiment were
created in two steps. In the ﬁrst step, we created 2- and 3-
step problems with maximally diagnostic reward structures
(according to the criteria used in Experiment 1) subject to
the constraint that the ﬁrst move with the highest immediate
reward was optimal for exactly half of those problems. In the
second step, we modiﬁed these problems so as to deteriorate
the correlation between planning horizon and reward using
the same method we employed to create the non-diagnostic
reward structures used in Experiment 1.
Model Predictions
We applied the metacognitive reinforcement learning model
described above to the problem of learning how many steps
one should plan ahead. We simulated a run of the experi-
ment described above with 1000 participants in each condi-
tion. The simulations predicted a gradual increase in the rel-
ative return from the ﬁrst 3-step problem to the last one (see
Figure 3). With feedback, the relative return increased faster
and reached a higher level than without feedback.
Results
To quantify the effects of condition and trial number on per-
formance (measured as relative reward), we ran a mixed-
Figure 3: Results of Experiment 2. The metacognitive RL
model predicts that feedback accelerate learning to plan. Hu-
man behavior shows a similar pattern of results.
design repeated-measures ANOV A on participant perfor-
mance during the 3-step trials. This revealed a signiﬁcant
effect of feedback ( F(9;4521) =8:54;p<0:01) and trial
number ( F(9;4521) =1:85;p<0:05) on relative reward. To
measure learning in each group, we performed a simple lin-
ear regression of relative reward on trial number for the 3-
step trials in the training block (i.e., when participants in the
feedback group received feedback). This revealed a signiﬁ-
cant regression equation for the feedback group ( F(2;322) =
5:28;p=0:05), with an R2of 0:40 but not for the control
group ( F(2;322) =1:57;p>0:05), with an R2of 0:16. This
suggests that participants who received feedback improved
during the training block but the control group did not.
Feedback increased the model’s average performance in
both the training block and the transfer block. We next
tested whether the enhanced learning of the feedback group
during training resulted in better performance in the trans-
fer block (trials 17-20) where they no longer received any
feedback. A two-sample t-test revealed that the feedback
group’s advantage in the testing block was nearly signiﬁcant
(t(1294) =1:53;p=0:063). Figure 3 compares our partici-
pants’ performance to the model predictions.
As predicted by our model, a multinomial logistic regres-
sion of whether or not each participant chose the optimal
route on trial number and feedback, revealed signiﬁcant ef-
fects of trial number ( p<0:0001) and feedback ( p<0:01).
Feedback appeared to increase people’s planning horizons:
when there were two remaining moves, the choices of the
feedback group accorded 4% less often with myopic choice
(t(1398) =2:17;p<0:05), 7% more often with optimal 2-
step planning ( t(1398) =3:44;p<0:001), and 4% more of-
ten with optimal 3-step planning ( t(1398) =2:43;p<0:05).
Discussion
In this article, we have introduced a computational model
of how people learn to decide better. Its central idea is that
learning how to think can be understood as metacognitive re-
inforcement learning. Our model extends previous research
on strategy selection learning (Lieder et al., 2014; Lieder &
Grifﬁths, 2015) by capturing that choosing cognitive oper-
ations is a sequential decision problem with potentially de-
layed rewards rather than a one-shot decision. The new model
correctly predicted the effects of reward structure and feed-
back on learning to plan: Experiment 1 suggested that our
model captures the effect of reward structures on the speed of
metacognitive learning. We then applied our theory to design
feedback for people’s performance in environments whose
reward structure is not diagnostic of good planning. Experi-
ment 2 conﬁrmed the model’s prediction that this intervention
would be effective.
Our results suggest two pragmatic approaches to promot-
ing cognitive growth: ﬁrst, designing reward structures that
are diagnostic of the quality of reasoning, planning, and
decision-making; second, providing feedback on the process
by which a decision was made. In Experiment 2 we followed
the latter approach by designing feedback based on the cost
of planning; but other types of feedback may also be useful.
If cognitive plasticity is based on model-free reinforcement
learning as assumed by our theory, then its speed should crit-
ically depend on how well the feedback people receive upon
performing cognitive operations reﬂects their value. There-
fore, feedback structures that align immediate feedback with
long-term value should be maximally effective at promot-
ing cognitive plasticity and learning to make better decisions.
Future experiments should test this hypothesis by designing
feedback structures using the optimal gamiﬁcation method in-
troduced by Lieder and Grifﬁths (2016). Feedback designed
using optimal gamiﬁcation could be especially beneﬁcial be-
cause the underlying method of reward shaping is designed
to accelerate model-free reinforcement learning (Ng et al.,
1999). Critically, to promote learning how to decide, people
should decide without any assistance and only receive feed-
back after their choice.
We hope that our theory of metacognitive reinforcement
learning will be a step towards establishing a scientiﬁc foun-
dation for designing feedback for cognitive training and other
interventions for promoting cognitive growth. Future work
will evaluate alternative forms of feedback, address the prob-
lem of transfer and retention, and design more effective train-
ing paradigms using tasks that are maximally diagnostic of
how people think and decide.
Acknowledgments. This work was supported by grant number
ONR MURI N00014-13-1-0341. The authors thank Thomas Hills,
Peter Dayan, Rika Antonova, Silvia Bunge, Stuart Russell, Amitai
Shenhav, and Sebastian Musslick for feedback and discussions.
References
Anguera, J. A., Boccanfuso, J., Rintoul, J. L., Al-Hashimi, O.,
Faraji, F., Janowich, J., . . . others (2013). Video game training
enhances cognitive control in older adults. Nature ,501(7465), 97–
101.Bavelier, D., Green, C. S., Pouget, A., & Schrater, P. (2012). Brain
plasticity through the life span: learning to learn and action video
games. Annual review of neuroscience ,35, 391–416.
Boureau, Y.-L., Sokol-Hessner, P., & Daw, N. D. (2015). Deciding
how to decide: self-control and meta-decision making. Trends in
cognitive sciences ,19(11), 700–710.
Dolan, R. J., & Dayan, P. (2013). Goals and habits in the brain.
Neuron ,80(2), 312–325.
Green, C. S., & Bavelier, D. (2008). Exercising your brain: a review
of human brain plasticity and training-induced learning. Psychol-
ogy and aging ,23(4), 692.
Green, S. B. (1991). How many subjects does it take to do a regres-
sion analysis. Multivariate behavioral research ,26(3), 499–510.
Hay, N., Russell, S., Tolpin, D., & Shimony, S. (2012). Select-
ing computations: Theory and applications. In N. de Freitas &
K. Murphy (Eds.), Uncertainty in artiﬁcial intelligence: Proceed-
ings of the twenty-eighth conference. P.O. Box 866 Corvallis, Ore-
gon 97339 USA: AUAI Press.
Huys, Q. J. M., Lally, N., Faulkner, P., Eshel, N., Seifritz, E., Ger-
shman, S. J., . . . Roiser, J. P. (2015). Interplay of approximate
planning strategies. Proceedings of the National Academy of Sci-
ences ,112(10), 3098–3103.
Jiang, N., Kulesza, A., Singh, S., & Lewis, R. (2015). The depen-
dence of effective planning horizon on model accuracy. In Proceed-
ings of the 2015 international conference on autonomous agents
and multiagent systems (pp. 1181–1189).
Keramati, M., Dezfouli, A., & Piray, P. (2011). Speed/accuracy
trade-off between the habitual and the goal-directed processes.
PLoS Comput Biol ,7(5), e1002055.
Lieder, F., & Grifﬁths, T. L. (2015). When to use which heuristic: A
rational solution to the strategy selection problem. In Proceedings
of the 37th annual conference of the cognitive science society.
Lieder, F., & Grifﬁths, T. L. (2016). Helping people make better
decisions using optimal gamiﬁcation. In Proc. 38th annu. conf.
cogn. sci. soc., philadelphia (pp. 2075–80).
Lieder, F., Plunkett, D., Hamrick, J. B., Russell, S. J., Hay, N., &
Grifﬁths, T. (2014). Algorithm selection by rational metareason-
ing as a model of human strategy selection. In Z. Ghahramani,
M. Welling, C. Cortes, N. Lawrence, & K. Weinberger (Eds.), Ad-
vances in neural information processing systems 27 (pp. 2870–
2878). Curran Associates, Inc.
Littman, M. L., Dean, T. L., & Kaelbling, L. P. (1995). On the
complexity of solving markov decision problems. In Proceedings
of the eleventh conference on uncertainty in artiﬁcial intelligence
(pp. 394–402).
Morrison, A. B., & Chein, J. M. (2011). Does working memory
training work? the promise and challenges of enhancing cogni-
tion by training working memory. Psychonomic bulletin & review ,
18(1), 46–60.
Myerson, J., & Green, L. (1995). Discounting of delayed rewards:
Models of individual choice. Journal of the experimental analysis
of behavior ,64(3), 263–276.
Ng, A. Y ., Harada, D., & Russell, S. (1999). Policy invariance under
reward transformations: Theory and application to reward shaping.
In I. Bratko & S. Dzeroski (Eds.), Proceedings of the 16th Annual
International Conference on Machine Learning (pp. 278–287). San
Francisco: Morgan Kaufmann.
Owen, A. M., Hampshire, A., Grahn, J. A., Stenton, R., Dajani, S.,
Burns, A. S., . . . Ballard, C. G. (2010). Putting brain training to
the test. Nature ,465(7299), 775–778.
Russell, S., & Wefald, E. (1991). Principles of metareasoning. Ar-
tiﬁcial Intelligence ,49(1-3), 361–395.
Shenhav, A., Musslick, S., Lieder, F., Kool, W., Grifﬁths, T., Cohen,
J., & Botvinick, M. (2017). Toward a rational and mechanistic
account of mental effort. Annual Review of Neuroscience ,40.
Sutton, R. S., & Barto, A. G. (1998). Reinforcement learning: An
introduction . Cambridge, MA, USA: MIT press.
Vul, E., Goodman, N., Grifﬁths, T. L., & Tenenbaum, J. B. (2014).
One and done? optimal decisions from very few samples. Cognitive
science ,38(4), 599–637.


If it’s important, then I’m curious: Increasing perceived usefulness stimulates curiosity.

Cognition 226 (2022) 105193
Available online 10 June 2022
0010-0277/© 2022 The Authors. Published by Elsevier B.V. This is an open access article under the CC BY license ( http://creativecommons.org/licenses/by/4.0/ ).If it’s important, then I’m curious: Increasing perceived usefulness
stimulates curiosity
Rachit Dubeya,*, Thomas L. Griffithsb, Tania Lombrozoc
aDepartment of Computer Science, Princeton University, Princeton, USA
bDepartments of Psychology and Computer Science, Princeton University, Princeton, USA
cDepartment of Psychology, Princeton University, Princeton, USA
ARTICLE INFO
Keywords:
Curiosity
Intervention
Education
Science communication
Utility-value intervention
Social curiosity ABSTRACT
Curiosity is considered essential for learning and sustained engagement, yet stimulating curiosity in educational
contexts remains a challenge. Can people ’s curiosity about a scientific topic be stimulated by providing evidence
that knowledge about the topic has potential value to society? Here, we show that increasing perceptions of
‘social usefulness ’ regarding a scientific topic also increases curiosity and subsequent information search. Our
results also show that simply presenting interesting facts is not enough to influence curiosity, and that people are
more likely to be curious about a scientific topic if they perceive it to be useful personally and socially. Given the
link between curiosity and learning, these results have important implications for science communication and
education more broadly.
1.Introduction
Consider the following scenario. A new species of fruit fly has been
discovered in Malaysia – remarkably, the fruit flies can jump up to 5 feet
high in the air. Biologists believe that these fruit flies can improve our
understanding of insect locomotion. Now imagine another scenario.
Another species of fruit fly has been discovered in Indonesia – remark -
ably, the fruit flies share 95% of their DNA with humans. Biologists
believe that these fruit flies could help us understanding the origins of
cancer in humans. Which species of fruit fly are you more curious to
learn about?
Curiosity has long been recognized as an important impetus to sci-
entific discovery and to everyday information-seeking (Berlyne, 1950;
Simon, 2001; Kidd & Hayden, 2015 ). A dominant viewpoint is that cu-
riosity is non-instrumental . Correspondingly, most prior work has focused
on perceived uncertainty and/or novelty as drivers of curiosity, where
curiosity is pursued for its own sake (Schmidhuber, 1991; Berlyne, 1960;
Oudeyer et al., 2007; Kobayashi et al., 2019 ). Yet the introductory
example suggests that expected practical usefulness, a canonically instrumental consideration, may also direct our curiosity towards some
stimuli over others.1
The role of instrumental factors, such as perceived usefulness, has
been recognized in recent theoretical work on curiosity (Dubey and
Griffiths, 2020 ; Golman, Loewenstein, Molnar, & Saccardo, 2020 ; Liquin
and Lombrozo, 2020b ).2 For instance, one theoretical account (Dubey &
Griffiths, 2020 ) links curiosity to ‘value of knowledge ’, which is a
function of people ’s current understanding of a topic and the perceived
usefulness of that topic, and posits that curiosity is piqued whenever one
perceives an opportunity to increase the value of one’s knowledge (i.e.,
topics that either increase understanding or perceived usefulness or
both). While prior research has explored ways to stimulate curiosity by
motivating reduction of uncertainty (Gentry et al., 2002; Law et al.,
2016; Clark et al., 2019 ), this account additionally suggests that it
should also be possible to direct curiosity towards scientific topics –
topics that may seem initially unimportant – if people come to appre -
ciate their practical usefulness.
Prompted by this potential link between curiosity and usefulness, in
this work, we consider stimulating curiosity by changing the perceived
*Corresponding author.
E-mail address: rdubey@princeton.edu (R. Dubey).
1 In the introductory example, both of the newly discovered species of fruit flies are similar in terms of novelty as well as their potential to reduce uncertainty, but
the second species of fruit flies would presumably stimulate greater curiosity, and we suggest this is at least in part due to its greater usefulness.
2 We additionally note that although Loewenstein ’s influential theory of curiosity does not explicitly consider usefulness in its formal account, it does hypothesize
that people will be more curious about topics that are important to them (Loewenstein, 1994 ). Recent formulations of the theory have formally studied how curiosity
is driven by information that people perceive as being important to them (Golman & Loewenstein, 2018; Golman et al., 2020 ).
Contents lists available at ScienceDirect
Cognition
u{�~zkw! s{yo| kro>! ÐÐÐ1ow �o�to~1m{y2w {mk�o2m{rzt�!
https://doi.org/10.1016/j.cognition.2022.105193
Received 22 September 2020; Received in revised form 24 May 2022; Accepted 30 May 2022
Cognition 226 (2022) 105193
2usefulness of a scientific topic. Given that scientific topics often include
basic research with highly indirect effects for any given individual, a
focus on immediate practical usefulness might have limited applica -
bility. Instead, we here focus on social usefulness – that is, on how
knowledge of a scientific topic might benefit others and society more
broadly. This has the advantage of targeting a wider range of science,
and additionally allows us to explore a relatively understudied aspect of
curiosity: its social role (Sinha et al., 2017; Dubey et al., 2021 ). Given
the importance of curiosity for fostering learning (Kang et al., 2009;
Gruber et al., 2014; Marvin & Shohamy, 2016; Gruber & Ranganath,
2019; Wade & Kidd, 2019; Gross et al., 2020 ) and potentially even
reducing political polarization (Kahan et al., 2017; Erceg et al., 2020 ),
successfully stimulating curiosity could have tremendous theoretical
and practical value. Theoretically, finding a causal connection between
perceived usefulness and curiosity would lend support to accounts of
curiosity that feature usefulness, and go beyond prior demonstrations of
a correlation between the two. Practically, it would offer a method for
educators and science communicators to motivate learning about
science.
2.Experiment 1: confirming the correlation between curiosity
and usefulness
In Experiment 1, we aimed to confirm the existence of a correlation
between curiosity and perceived social usefulness for a broad range of
scientific questions. Prior to collecting the data, we pre-registered the
data collection protocol, stimuli, and analysis plan (https://osf.
io/3pnkv ).
2.1. Participants
We recruited 300 participants from Prolific. They earned $1.15 for
participating in a study that took approximately 5-6 minutes to com-
plete. We removed participants who failed a simple attention check.
Four participants were thus excluded, but their inclusion does not affect
the significance of our findings.
2.2. Stimuli
The stimuli used in the experiment were 100 diverse scientific
questions sampled from Reddit ’s Explain Like I’m Five subreddit. Out of
these 100 questions, 50 questions were the highest-rated questions of all
time (i.e., the questions that were the most popular and received the
highest amount of engagement; number of up-votes was greater than
1000 for these questions). The other 50 questions were moderately
popular questions, as reflected in up-votes between 200 and 600, and
were taken directly from a prior study on curiosity (Dubey et al., 2021 ).
We chose this diverse set of questions in order to elicit a wide range of
curiosity in our participants. Note that we did not include questions that
received fewer than 200 up-votes, as many of these questions contained
grammatical errors or were poorly written.
2.3. Procedure
At the start of the experiment, each participant was assigned 10
questions randomly sampled from our 100-question database. For each
question, participants were asked to rate their curiosity (‘How curious
are you to know the answer to this question? ’) on a scale from 1 (‘not at
all curious ’) to 7 (‘very curious ’). This was the key variable of interest.
Further, we asked participants to provide their ratings for two di-
mensions of usefulness: personal usefulness (‘To what extent would
knowing the answer to this question benefit you personally? ’) and social
usefulness (‘To what extent would knowing the answer to this question
benefit others and society in general? ’). Both were measured on a scale
from 1 (‘not beneficial at all’) to 7 (‘very beneficial ’).
Lastly, we asked participants to rate their confidence in knowing the answer to the question, on a scale of 1–7 (‘How confident are you that
you know the correct answer to this question? ’). We added this question
to ensure that any effect of personal or social usefulness on curiosity is
not reducible to confidence. After participants provided the ratings for
their assigned 10 questions, the experiment was finished and they were
debriefed and paid for their participation.
2.4. Results
To test the effect of confidence, personal usefulness, and social use-
fulness of curiosity, we ran a mixed-effects linear model with confi -
dence, personal usefulness, and social usefulness as fixed effects, and
question ID and subject ID as random effects. As shown in Table 1, both
personal and social usefulness were positive and significant predictors of
curiosity, even controlling for confidence.
2.5. Discussion
Experiment 1 confirmed the existence of a correlational link between
perceived usefulness and curiosity on a broad range of everyday scien -
tific questions. We also found that both personal and social usefulness
were significant predictors of curiosity.
These results are consistent with accounts of curiosity that incorpo -
rate a role for instrumental value. However, they go beyond most prior
work in identifying a role for social usefulness that appears to be distinct
from personal usefulness. Why might social usefulness be associated with
curiosity? One possibility is that socially useful information is valued in
its own right, and so prompts curiosity for this reason. Another possi -
bility, however, is that our question about social usefulness captured
indirect forms of personal usefulness – for instance, the expectation that
something might be of personal use in the future (because it is of use to
others more broadly), or that one might gain social capital by knowing
information that is useful to others. Rather than disentangling these
possibilities here, our subsequent studies probe the power of social
usefulness: can perceptions of social usefulness be leveraged to promote
curiosity about science?
3.Experiment 2: does usefulness influence curiosity?
Having established a correlation between usefulness and curiosity,
we next investigated whether curiosity about a scientific topic can be
influenced by manipulating its perceived social usefulness, and whether
this affects subsequent information search.
3.1. Participants
We recruited 240 participants from Amazon Mechanical Turk
(AMT). They earned $1.00 for participating in a study that took
approximately 7-8 minutes to complete. For both Experiments 2 and 3,
Table 1
Experiment 1: results from the mixed linear effect model.
Coef. Std. Error z PF|z| [0.025 0.975]
Intercept 2.37 0.13 18.08 0.00 2.11 2.63
Confidence 0.11 0.02 5.54 0.00 0.07 0.14
Personal usefulness 0.16 0.02 7.98 0.00 0.12 0.19
Social usefulness 0.16 0.02 8.20 0.00 0.12 0.19
Question ID 0.13 0.02
Subject ID 1.29 0.09      R. Dubey et al.
Cognition 226 (2022) 105193
3sample sizes were determined prior to data collection.3
3.2. Stimuli
Two short articles describing the biology of fruit flies and two short
articles describing the biology of rats were used. For each of the two
topics (i.e., fruit fly and rat), one article was ‘high-use ’ and the other was
‘low-use ’. The high-use article emphasized how research about that
animal could be highly beneficial to medicine, while the low-use article
raised questions about whether research concerning that animal could
generate any medical benefits. Thus, compared to the low-use article,
the high-use article was designed to primarily increase perceptions of
social usefulness (although it might also indirectly increase perceptions
of personal usefulness). For both Experiment 2 and 3, all articles were
matched in terms of length and, as much as possible, for general content
and style. (See Supplementary Materials for complete stimuli.)
3.3. Procedure
3.3.1. Phase 1
At the beginning of Phase 1, participants were presented with one of
the two scientific topics, either ‘biology of fruit flies’ or ‘biology of rats’
(counter-balanced). Participants then responded to the following on a
scale from 1-7:
1. Usefulness: “To what extent would knowing about this phenomenon
be useful to you in the future? ” 1 (Not useful at all) to 7 (Very useful)
2. Understanding: “Please rate how well you feel you understand this
phenomenon. ” 1 (Very vaguely) to 7 (Very thoroughly)
3.Curiosity: “Please rate your curiosity in knowing about this phe-
nomenon. ” 1 (Not curious at all) to 7 (Very curious)
The first question was included to verify the successful manipulation
of usefulness, and the second to ensure that the effect of usefulness on
curiosity couldn ’t be reduced to perceived understanding. Curiosity
rating was the key variable of interest in Phase 1. After providing their
ratings, participants were presented with the assigned article for that
topic and were instructed to read it as carefully as possible. The article
advocated for either high or low-use (counter-balanced). After reading
the article, participants were asked to re-rate usefulness, understanding,
and curiosity about the initial topic. This procedure was then repeated
for the other topic, which was always paired with the other level of use
(for example, someone who received the high use article for rats would
then receive the low use article for fruit flies; also refer to Fig. 1). 3.3.2. Phase 2
In the second phase, participants were instructed that they would be
presented with some facts about the two topics (four for each topic, eight
in total), but that they could only select five of those facts. The eight fact
choices were then presented (e.g., “Rat Fact #3201 ”), and participants
indicated their five choices. The corresponding facts were shown to
participants after they indicated their choices.
3.4. Results
3.4.1. Phase 1
We evaluated each of our dependent variables in a mixed ANOVA
with time point (2: pre vs. post ratings) as a within-subjects factor,
article usefulness (2: low vs. high) as a within-subjects factor, and high
use topic (2: rat, fruit fly) as a between-subjects factor.
We first confirmed that our manipulation of usefulness successfully
affected perceived usefulness ratings. Indeed, there was an expected
interaction between time and usefulness, F(1, 239)47.69,
MSE35.3, pD0.001, such that the high-use article led to an increase
in usefulness ratings of 1.16 from pre to post, whereas the low-use article
led to an increase of 0.40 (Fig. 2a). We next investigated the change in
understanding ratings and found that there was an interaction between
time and understanding, F(1239)29.1, MSE16.8, pD0.001, such
that the mean understanding rating increased by 0.63 for the low-use
stimuli and by 1.16 for the high-use stimuli (Fig. 2b).
Finally, we considered our primary measure of interest, curiosity,
and found that there was an expected interaction between time and
usefulness, F(1239)32.69, MSE21.6, pD0.001, with the increase
in curiosity being greater for the high-use stimuli (1.04) compared to the
low-use stimuli (0.44), t(478)4.71, pD0.001 (Fig. 3a). Note that no
other effects were significant for any of the above ANOVAs.
Fig. 1.Design of Experiment 2. The experiment was divided into two phases. In Phase 1, participants were presented with one of the two topics (fruit flies or rats)
and asked to provide ratings for curiosity, understanding, and usefulness. They then read an article about that topic and once again rated curiosity, understanding,
and usefulness. This procedure was repeated for the second topic. In Phase 2, participants had the choice to reveal five out of eight facts presented to them (four facts
from each topic). The chosen facts were then presented one by one. Note that instructions were provided before each phase.
Fig. 2.Effect of usefulness manipulation on perceived usefulness and under -
standing (Experiment 2). (a) Change in usefulness ratings for participants who
received the low-use and high-use stimuli before and after they read the cor-
responding articles. (b) Change in understanding ratings for participants before
and after they read the corresponding articles. 3Based on pilot data, we aimed to recruit at least 60 participants per con-
dition, which required 240 participants in Experiment 2, given two conditions
with counterbalanced order. R. Dubey et al.
Cognition 226 (2022) 105193
4We next considered whether understanding or perceived usefulness
mediated the effect of our usefulness manipulation on curiosity. We
found partial mediation for both; however, the effect of perceived use-
fulness in predicting curiosity survived the inclusion of understanding as
a predictor (refer to Supplemental Materials for analyses).
3.4.2. Phase 2
We next investigated whether participants were more likely to reveal
facts about the high-use stimuli compared to the low-use stimuli. Par-
ticipants indeed revealed more facts about the high-use stimuli (3 vs. 2,
Fig. 3b). A one-sample t-tests revealed that the mean number of facts
revealed about the high-use stimulus was significantly higher than the
chance value of 2.5 whether the high-use topic was fruit flies, t(244)
4.06, pD0.001, or rats, t(232)6.79, pD0.001.
3.5. Discussion
Experiment 2 found that participants became more curious about
stimuli after reading information that suggested the topic was of high
(vs. low) use (Phase 1), and that they were more likely to reveal addi-
tional information about the high-use topic (Phase 2). This suggests that
manipulating perceived usefulness, in this case the social usefulness of
biological information via medical applications, led to an increase in
curiosity about the corresponding scientific topic. However, Experiment
2 also has some potential concerns. For one, we found that the effect of
our manipulation on curiosity was partially mediated by understanding.
Thus, it could be that changes in understanding (and not perceived
usefulness) partially drove the effects of our manipulation on curiosity.
Another concern is that even though our goal was to lower the perceived
usefulness for the low-use stimuli, it could be that our manipulation
instead lowered their perceived informativeness (the low-use stimuli
contained the line: “some of these pet projects, they really don’t make a
whole lot of sense and sometimes these dollars go to projects that have
little or nothing to do with the public good ” and people could have
inferred there is nothing to be learned by doing research on these
topics). We address these concerns in Experiment 3.
4.Experiment 3: what are the more effective usefulness cues?
Experiment 3 had three aims. First, we aimed to test the influence of
perceived usefulness on curiosity while controlling for understanding.
Second, we aimed to study the influence of usefulness on curiosity while
controlling for informativeness. Last, we aimed to isolate effects of
personal and/or social usefulness from more generic claims of
importance. 4.1. Participants
We recruited 203 participants from AMT (n67, 72, and 64 for
INTEREST, ECOLOGY, and MEDICAL respectively). They earned $0.35
for participating in a study that took approximately 2-3 minutes to
complete.
4.2. Stimuli
Three short articles describing the biology of fruit flies were used. In
the INTEREST condition, the article presented interesting facts about
fruit fly reproduction . In the ECOLOGY condition, the article illustrated
how fruit flies are useful to the environment . In the MEDICAL condition,
the article provided evidence that fruit flies are useful to medical research .
(See Supplementary Materials for complete stimuli.)
4.3. Procedure
Participants were randomly assigned to read a short article about the
‘biology of fruit flies’ (INTEREST, ECOLOGY, or MEDICAL) and pro-
vided ratings before and after they read the article. Participants were
first presented with the scientific topic and were asked to rate their
understanding, perceived usefulness, and curiosity as in Experiment 2.
After providing these ratings, participants were presented with the
assigned article and then asked to re-rate understanding, perceived
usefulness, and curiosity about that topic. Participants were also asked
to respond to the following on a scale of 1–7: “Please rate how surprising
you found the previously shown information on fruit flies to be”, from 1
(Not surprising at all) to 7 (Very surprising). This was added as a control
to ensure that any potential increase in curiosity was not caused simply
by surprise.
We hypothesized that participants ’ increase in understanding would
be similar across the three conditions, but that perceived usefulness
would not be. Further, we also hypothesized that the INTEREST condi -
tion, despite providing interesting information, would not be deemed as
useful and would also result in a smaller increase in curiosity (which
would address the second potential confound from Exp 2). The contrast
between the ECOLOGY and MEDICAL conditions also allows us to test
whether curiosity is especially sensitive to evidence for potential social
(or even personal) usefulness: while ECOLOGY emphasized importance
and value to other species, only MEDICAL emphasized potential use-
fulness for individual humans and society.
4.4. Results
We first investigated how participants ’ understanding changed after
they read the corresponding articles across the three conditions. Par-
ticipants ’ understanding ratings increased significantly after they read
the article for all three conditions, t(142)2.34, pD0.05, for IN-
TEREST, t(126)3.57, pD0.001, for ECOLOGY, and t(132)3.88,
pD0.001, for MEDICAL (Fig. 4a). A one-way ANOVA revealed that
these three groups were not significantly different from each other, F
(2200)2.64, MSE5.5, p0.07, indicating that the increase in un-
derstanding was similar across all conditions, which successfully elim-
inates the potential confounds from Experiment 2.
We next evaluated how perceived usefulness changed across condi -
tions. Although participants ’ usefulness ratings increased numerically
for all three conditions (Fig. 4b), this increase was not significant for
INTEREST, t(142)1.52, p0.13, although it was for both ECOL -
OGY, t(126)3.31, pD0.01, and MEDICAL, t(132)5.01, pD0.01.
This suggests that simply presenting interesting facts about a topic was
not enough to influence perceived usefulness. Further, a one-way
ANOVA revealed a significant effect of condition, F(2200)9.25,
Fig. 3.Usefulness influences curiosity (Experiment 2). (a) Mean increase in
participants ’ curiosity about a topic after reading a ‘low- ’ or ‘high-use ’ article.
(b) Mean number of facts chosen after reading a ‘low- ’ or ‘high-use ’ article
about that topic. R. Dubey et al.
Cognition 226 (2022) 105193
5MSE19.8, pD0.001, with MEDICAL significantly higher than
ECOLOGY, t(129)2.1, pD0.05, and ECOLOGY significantly higher
than INTEREST, t(134)2.2, pD0.05.
We next analyzed how much surprise each article evoked (Fig. 4c)
and found that there was a significant difference for the surprise ratings
across the three conditions, F(2200)5.12, MSE16.7, pD0.01.
Specifically, MEDICAL was significantly more surprising than ECOL -
OGY, t(129)3.19, pD0.01, and was significantly more surprising
than INTEREST, t(137)2.24, pD0.05, but ECOLOGY and INTEREST
did not differ, t(134)0.92, p0.36.
We next evaluated the change in participants ’ curiosity ratings.
Although participants ’ curiosity ratings increased for all three condi -
tions, that increase was not significant for INTEREST, t(142)1.45,
p0.15, but it was significant for ECOLOGY, t(126)2.13, pD0.05,
and for MEDICAL, t(132)3.68, pD0.01. A one-way ANOVA
confirmed a significant effect of condition, F(2200)5.14, MSE9.1,
pD0.01. Follow-up t-tests showed that MEDICAL was significantly
different from ECOLOGY, t(129)2.13, pD0.05, but ECOLOGY was
not significantly different from INTEREST, t(134)0.89, p0.38.
These results suggest that if people perceive stimuli to be less useful,
then they are less likely to become curious about them.
As in Experiment 2, we tested whether the effect of our usefulness
manipulation on curiosity was mediated by perceived usefulness and
found that the effect of the usefulness manipulation on curiosity was
fully mediated by perceived usefulness, whereas understanding and
surprise only partially mediated this effect (refer to Supplemental Ma-
terials) Fig. 5. 4.5. Discussion
The findings from Experiment 3 reveal that not all cues to value are
equal: participants were more likely to become curious about a scientific
topic if they learned it might be more useful for humans and society
versus the environment and other species (MEDICAL vs. ECOLOGY).
Further, Experiment 3 succeeded in identifying an effect of usefulness
that could not be explained by differences in understanding or surprise.
Our results suggest that simply presenting interesting facts that do not
offer evidence of usefulness is not enough to induce curiosity (INTER -
EST), even if those facts boost understanding and induce surprise.
5.General discussion
Across three studies, we show that the perceived usefulness of in-
formation predicts (Exp 1) and influences (Exp 2-3) both self-reported
curiosity (Exp 1-3) and subsequent information search (Experiment 2).
Moreover, the effects of perceived usefulness on curiosity cannot be
reduced to perceived understanding or surprise (Exp 3).
5.1. Implications
The idea that curiosity is influenced by both instrumental and non-
instrumental considerations is not new (Rossing & Long, 1981 ). Re-
searchers have long recognized a role for both of these factors in influ-
encing curiosity, but many have deliberately focused on the latter, in
part because information search can be driven by factors other than
curiosity, making non-instrumental information search an especially
diagnostic marker of curiosity (Eliaz & Schotter, 2007; Kidd & Hayden,
2015 ). However, in recent years, the field has largely come to adopt the
assumption that curiosity is, by definition, the urge to acquire non-
instrumental information (Cervera, Wang, & Hayden, 2020 ; FitzGibbon,
Lau, & Murayama, 2020 ; Gottlieb & Oudeyer, 2018 ; Hsee & Ruan, 2016 ;
Kobayashi et al., 2019 ; Oudeyer, Baranes, & Kaplan, 2013 ).4 By
demonstrating a link between usefulness and curiosity, our work chal-
lenges this prevalent assumption and also provides support to recent
accounts that explicitly link curiosity to utility (Abir et al., 2022 ; Dubey
and Griffiths, 2020 ; Golman et al., 2020 ; Liquin and Lombrozo, 2020b ).
That said, our findings are consistent with the idea that curiosity is
typically experienced as an intrinsic and non-instrumental drive (see also
Liquin & Lombrozo, 2020a; Gottlieb et al., 2020; Sharot & Sunstein,
2020 ).
We also make a contribution in revealing a particularly powerful
instrumental consideration: the perceived usefulness of scientific infor-
mation. Our studies suggest that perceived usefulness has both personal
and social dimensions (Exp 1), and that it goes beyond generic claims of
value or importance (Exp 3). Further, the connection between curiosity
and information-seeking (demonstrated in Exp 2) makes interventions
Fig. 4.Effect of usefulness manipulation on understanding, usefulness, and surprise (Experiment 3). (a) Mean change in understanding ratings. (b) Mean change in
usefulness ratings. (c) Mean surprise ratings of participants in each condition.
Fig. 5.People ’s curiosity is higher when they perceive something to be more
useful (Experiment 3). Mean change in curiosity ratings for the three different
conditions. Participants ’ curiosity increased the most in MEDICAL, in which
they read an article that provided evidence that fruit flies are highly beneficial
to medicine. 4 Although not all such work makes this assumption (e.g. Loewenstein, 1994;
van Lieshout et al., 2018, 2020 ) R. Dubey et al.
Cognition 226 (2022) 105193
6on curiosity especially relevant for educational contexts. Indeed, the
importance of perceived utility (a construct closely related to usefulness)
has been documented within the education literature, with studies
showing that students ’ perceived ‘utility value ’ i.e., how valuable they
think a task would be, influences motivation as well as the allocation of
study time (Atkinson, 1964; Wigfield & Eccles, 2000; Eccles & Wigfield,
1995; Hulleman et al., 2008, 2010; Harackiewicz et al., 2012; Brown
et al., 2015; Dunlosky & Ariel, 2011; Ariel et al., 2009 ). While curiosity
and motivation are typically considered distinct, and potentially involve
different computational and neural mechanisms, our findings contribute
to a body of work that suggests both are influenced by instrumental
considerations, raising new questions about how curiosity might shape
motivation. Finally, our focus on social usefulness opens up new ques-
tions about when and why different forms of usefulness might matter.
When it comes to motivating interest in basic research, history, or
distant parts of the world, the case for immediate personal usefulness
may be hard to make, but focusing on social usefulness may nonetheless
be an effective route to manipulating perceived usefulness.
5.2. Limitations
Our work has several limitations, which also present opportunities
for future work. For one, the usefulness cues provided in our experi -
mental stimuli were relatively explicit and strong. This could limit the
generalizability of our findings to more naturalistic contexts where
usefulness is not always so clearly identified, but rather must be inferred
from social and contextual cues (see also Dubey et al., 2021 ). Relatedly,
the different articles in Experiment 3 were quite different from each
other (despite being similar in length and writing style); future work
would benefit from using a broader range of stimuli, including materials
with more direct relevance to formal and informal educational settings.
Another limitation of our work is that we did not explicitly differentiate
personal and social usefulness in our interventions (Experiments 2 and
3), and so we do not know which kind of usefulness is most effective in
stimulating curiosity. For example, is students ’ curiosity for scientific
topics more likely to be increased by demonstrating that a scientific
topic could provide personal benefit (e.g., knowing algebra might help
them get better jobs), or by demonstrating that knowing more about a
scientific topic could help solve societal issues? Notably, our in-
terventions in Experiments 2-3 involved information of personal and
social value in the sense that it had personal and social implications, but
the information was not actually actionable for individual participants;
it may be that actionable information has even larger effects on
curiosity.
Another limitation of our work is that we do not differentiate be-
tween curiosity and information-search beyond including measures of
both. While we find corresponding effects of usefulness on both mea-
sures in Experiment 2, it is possible for someone to be curious but still
avoid obtaining information, or to seek information despite lacking
curiosity. Despite this limitation, we make some progress in improving
our understanding of curiosity by showing that like information-seeking,
which has been shown to be influenced by both instrumental and non-
instrumental cues (Sharot & Sunstein, 2020; Kelly et al., 2021 ), curios -
ity can also be driven by instrumental considerations. That said, several
key theoretical questions about curiosity and usefulness remain. Studies
have shown that people can become curious about irrelevant and
sometimes even potentially harmful stimuli (Hsee & Ruan, 2016; Cab-
rero et al., 2019; FitzGibbon et al., 2019; Lau et al., 2020 ), and that
people are sometimes averse to information, even when that informa -
tion is potentially useful to them (Sweeny et al., 2010; Golman et al.,
2017; Charpentier et al., 2018 ). Understanding how curiosity interacts
with usefulness in these contexts is an important future direction.
5.3. Concluding remarks
Despite the above open questions, our work shows that self-reported curiosity and information-seeking behavior can be influenced by
perceived usefulness, and it points to effective strategies for stimulating
curiosity about science. The results from Experiment 3 suggest that
simply presenting information that is interesting or surprising is insuf-
ficient to drive curiosity. Instead, a more effective way to stimulate
curiosity is to present information in a way that allows people to more
clearly appreciate its personal and social usefulness. These results will be
of importance to education researchers and science communicators
concerned with piquing curiosity for public health or general under -
standing of science. Perhaps fruit flies will never be welcome in our
homes, but by learning about their usefulness, information about them
could be more welcome in our minds.
Appendix A.Supplementary data
Supplementary data to this article can be found online at 10.1016/j.
cognition.2022.105193 .
References
Abir, Y., Marvin, C. B., van Geen, C., Leshkowitz, M., Hassin, R. R., & Shohamy, D.
(2022). An energizing role for motivation in information-seeking during the early
phase of the covid-19 pandemic. Nature Communications, 13, 1–10.
Ariel, R., Dunlosky, J., & Bailey, H. (2009). Agenda-based regulation of study-time
allocation: When agendas override item-based monitoring. Journal of Experimental
Psychology: General, 138, 432.
Atkinson, J. W. (1964). An introduction to motivation. Princeton, N.J.: Van Nostrand .
Berlyne, D. E. (1950). Novelty and curiosity as determinants of exploratory behaviour
British. Journal of Psychology. General Section, 41, 68–80.
Berlyne, D. E. (1960). Mcgraw-hill series in psychology. Conflict, arousal, and curiosity .
Brown, E. R., Smith, J. L., Thoman, D. B., Allen, J. M., & Muragishi, G. (2015). From
bench to bedside: A communal utility value intervention to enhance students ’
biomedical science motivation. Journal of Educational Psychology, 107, 1116 .
Cabrero, J. M. R., Zhu, J. Q., & Ludvig, E. A. (2019). Costly curiosity: People pay a price
to resolve an uncertain gamble early. Behavioural processes, 160, 20–25.
Cervera, R. L., Wang, M. Z., & Hayden, B. Y. (2020). Systems neuroscience of curiosity.
Current Opinion in Behavioral Sciences, 35, 48–55.
Charpentier, C. J., Bromberg-Martin, E. S., & Sharot, T. (2018). Valuation of knowledge
and ignorance in mesolimbic reward circuitry. Proceedings of the National Academy of
Sciences, 115, E7255 –E7264 .
Clark, S., Harbaugh, A. G., & Seider, S. (2019). Fostering adolescent curiosity through a
question brainstorming intervention. Journal of adolescence, 75, 98–112.
Dubey, R., & Griffiths, T. L. (2020). Reconciling novelty and complexity through a
rational analysis of curiosity. Psychological Review, 127, 455–476.
Dubey, R., Mehta, H., & Lombrozo, T. (2021). Curiosity is contagious: A social influence
intervention to induce curiosity. Cognitive science, 45, e12937 .
Dunlosky, J., & Ariel, R. (2011). Self-regulated learning and the allocation of study time.
Psychology of learning and motivation. volume, 54, 103–140.
Eccles, J. S., & Wigfield, A. (1995). In the mind of the actor: The structure of adolescents ’
achievement task values and expectancy-related beliefs. Personality and Social
Psychology Bulletin, 21, 215–225.
Eliaz, K., & Schotter, A. (2007). Experimental testing of intrinsic preferences for
noninstrumental information. American Economic Review, 97, 166–169.
Erceg, N., Ru¯zoj¯ci˘c, M., & Galic, Z. (2020). Misbehaving in the corona crisis: The role of
anxiety and unfounded beliefs . PsyArXiv Preprint .
FitzGibbon, L., Komiya, A., & Murayama, K. (2019). The lure of counterfactual curiosity:
People incur a cost to experience regret . OSF Preprints .
FitzGibbon, L., Lau, J. K. L., & Murayama, K. (2020). The seductive lure of curiosity:
Information as a motivationally salient reward. Current Opinion in Behavioral
Sciences, 35, 21–27.
Gentry, J. W., Burns, A. C., Dickinson, J. R., Putrevu, S., Chun, S., Hongyan, Y.,
Williams, L. A., Bare, T., & Gentry, R. A. (2002). Managing the curiosity gap does
matter: What do we need to do about it?. Developments in Business Simulation and
Experiential Learning. In Proceedings of the Annual ABSEL conference .
Golman, R., Hagmann, D., & Loewenstein, G. (2017). Information avoidance. Journal of
Economic Literature, 55, 96–135.
Golman, R., & Loewenstein, G. (2018). Information gaps: A theory of preferences
regarding the presence and absence of information. Decision, 5, 143.
Golman, R., Loewenstein, G., Molnar, A., & Saccardo, S. (2020). The Demand for, and
Avoidance of, Information . Catonsville, MD: Management Science Advance online
publication.
Gottlieb, J., Cohanpour, M., Li, Y., Singletary, N., & Zabeh, E. (2020). Curiosity,
information demand and attentional priority. Current Opinion in Behavioral Sciences,
35, 83–91.
Gottlieb, J., & Oudeyer, P. Y. (2018). Towards a neuroscience of active sampling and
curiosity. Nature Reviews Neuroscience, 19, 758–770.
Gross, M. E., Zedelius, C. M., & Schooler, J. W. (2020). Cultivating an understanding of
curiosity as a seed for creativity. Current Opinion in Behavioral Sciences, 35, 77–82. R. Dubey et al.
Cognition 226 (2022) 105193
7Gruber, M. J., Gelman, B. D., & Ranganath, C. (2014). States of curiosity modulate
hippocampus-dependent learning via the dopaminergic circuit. Neuron, 84,
486–496.
Gruber, M. J., & Ranganath, C. (2019). How curiosity enhances hippocampus-dependent
memory: The prediction, appraisal, curiosity, and exploration (pace) framework.
Trends in cognitive sciences, 23, 1014 –1025 .
Harackiewicz, J. M., Rozek, C. S., Hulleman, C. S., & Hyde, J. S. (2012). Helping parents
to motivate adolescents in mathematics and science: An experimental test of a
utility-value intervention. Psychological Science, 23, 899–906.
Hsee, C. K., & Ruan, B. (2016). The pandora effect: The power and peril of curiosity.
Psychological science, 27, 659–666.
Hulleman, C. S., Durik, A. M., Schweigert, S. B., & Harackiewicz, J. M. (2008). Task
values, achievement goals, and interest: An integrative analysis. Journal of
Educational Psychology, 100, 398–416.
Hulleman, C. S., Godes, O., Hendricks, B. L., & Harackiewicz, J. M. (2010). Enhancing
interest and performance with a utility value intervention. Journal of Educational
Psychology, 102, 880–895.
Kahan, D. M., Landrum, A., Carpenter, K., Helft, L., & Hall Jamieson, K. (2017). Science
curiosity and political information processing. Political Psychology, 38, 179–199.
Kang, M. J., Hsu, M., Krajbich, I. M., Loewenstein, G., McClure, S. M., Wang, J. T. Y., &
Camerer, C. F. (2009). The wick in the candle of learning: Epistemic curiosity
activates reward circuitry and enhances memory. Psychological Science, 20, 963–973.
Kelly, C., Sharot, T., et al. (2021). Individual differences in information-seeking. Nature
communications, 12, 1–13.
Kidd, C., & Hayden, B. Y. (2015). The psychology and neuroscience of curiosity. Neuron,
88, 449–460.
Kobayashi, K., Ravaioli, S., Baran ˇes, A., Woodford, M., & Gottlieb, J. (2019). Diverse
motives for human curiosity. Nature human behaviour, 3, 587–595.
Lau, J. K. L., Ozono, H., Kuratomi, K., Komiya, A., & Murayama, K. (2020). Shared
striatal activity in decisions to satisfy curiosity and hunger at the risk of electric
shocks. Nature Human Behaviour, 4, 531–543.
Law, E., Yin, M., Goh, J., Chen, K., Terry, M. A., & Gajos, K. Z. (2016). Curiosity killed the
cat, but makes crowdwork better. In Proceedings of the 2016 CHI Conference on
Human Factors in Computing Systems (pp. 4098 –4110) .
van Lieshout, L., Vandenbroucke, A. R., Müller, N. C., Cools, R., & de Lange, F. P. (2018).
Induction and relief of curiosity elicit parietal and frontal activity. Journal of
Neuroscience, 38, 2579 –2588 . van Lieshout, L. L., de Lange, F. P., & Cools, R. (2020). Why so curious?. quantifying
mechanisms of information seeking. Current Opinion in Behavioral Sciences, 35,
112–117.
Liquin, E. G., & Lombrozo, T. (2020). Explanation-seeking curiosity in childhood. Current
Opinion in Behavioral Sciences, 35, 14–20.
Liquin, E. G., & Lombrozo, T. (2020). A functional approach to explanation-seeking
curiosity. Cognitive Psychology, 119, 101276 .
Loewenstein, G. (1994). The psychology of curiosity: A review and reinterpretation.
Psychological Bulletin, 116, 75–98.
Marvin, C. B., & Shohamy, D. (2016). Curiosity and reward: Valence predicts choice and
information prediction errors enhance learning. Journal of Experimental Psychology:
General, 145, 266–272.
Oudeyer, P. Y., Baranes, A., & Kaplan, F. (2013). Intrinsically motivated learning of real-
world sensorimotor skills with developmental constraints. Intrinsically motivated
learning in natural and artificial systems (pp. 303–365). Berlin, Heidelberg: Springer .
Oudeyer, P. Y., Kaplan, F., & Hafner, V. V. (2007). Intrinsic motivation systems for
autonomous mental development. IEEE Transactions on Evolutionary Computation, 11,
265–286.
Rossing, B. E., & Long, H. B. (1981). Contributions of curiosity and relevance to adult
learning motivation. Adult Education, 32, 25–36.
Schmidhuber, J. (1991). Curious model-building control systems. In Proceedings of 1991
IEEE International Joint Conference on Neural Networks (pp. 1458 –1463) .
Sharot, T., & Sunstein, C. R. (2020). How people decide what they want to know. Nature
Human Behaviour , 1–6.
Simon, H. A. (2001). Seek and ye shall find”: How curiosity engenders discovery Designing for
science: Implications from everyday, classroom, and professional settings (pp. 5–20).
Sinha, T., Bai, Z., & Cassell, J. (2017). A new theoretical framework for curiosity for
learning in social contexts. European conference on technology enhanced learning (pp.
254–269). Springer .
Sweeny, K., Melnyk, D., Miller, W., & Shepperd, J. A. (2010). Information avoidance:
Who, what, when, and why. Review of General Psychology, 14, 340–353.
Wade, S., & Kidd, C. (2019). The role of prior knowledge and curiosity in learning.
Psychonomic bulletin & review, 26, 1377 –1387 .
Wigfield, A., & Eccles, J. S. (2000). Expectancy-value theory of achievement motivation.
Contemporary Educational Psychology, 25, 68–81. R. Dubey et al.