The Structure of Goal Systems Predicts Human Performance.

The Structure of Goal Systems Predicts Human Performance
David D. Bourgin1a(ddbourgin@berkeley.edu)
Falk Lieder1a(falk.lieder@berkeley.edu)
Daniel Reichman1a(daniel.reichman@gmail.com)
Nimrod Talmon1b(nimrodtalmon77@gmail.com)
Thomas L. Grifﬁthsa(tom grifﬁths@berkeley.edu)
aDepartment of Psychology
University of California Berkeley
Berkeley, CA 94720 USA
bDepartment of Computer Science and Applied Mathematics
Weizmann Institute of Science
Rehovot, Israel
1These authors contributed equally.
Abstract
Most psychological theories attribute people’s failure to
achieve their goals exclusively to insufﬁcient motivation or
lack of skill. Here, we offer a complementary explanation that
emphasizes the inherent complexity of the computational prob-
lems that arise from the structure of people’s goal systems.
Concretely, we hypothesize that people’s capacity to achieve
their goals can be predicted from combinatorial parameters of
the structure of the network connecting their goals to the means
available to pursue them. To test this hypothesis, we expressed
the relationship between goals and means as a bipartite graph
where edges between means and goals indicate which means
can be used to achieve which goals. This allowed us to map
two computational challenges that arise in goal achievement
onto two classic NP-hard problems: Set Cover and Maximum
Coverage. The connection between goal pursuit and NP-hard
problems led us to predict that people should perform bet-
ter with goal systems that are tree-like. Three behavioral ex-
periments conﬁrmed this prediction. Our results imply that
network parameters that are instrumental to algorithm design
could also be useful for understanding when and why people
struggle in their goal pursuits.
Keywords: decision-making; goals; rational analysis; graph
theory; computational complexity
Introduction
The ability to set and achieve high-level goals, such as creat-
ing a CogSci paper, is a critical feature of human intelligence
and a key challenge for artiﬁcial intelligence systems (Newell
& Simon, 1972). Critically, everyday problem solving re-
quires people to juggle multiple goals in parallel (Atkinson &
Birch, 1970; Miller, Galanter, & Pribram, 1960). Concretely,
when people are given ten minutes to list their current pursuits
they will report about 15 goals on average and each of those
goals typically entails multiple subgoals at several levels of
abstraction (Little & Gee, 2007).
It is generally agreed that there are many situations in
which people fail to act on their goals (Baumeister, Heather-
ton, & Tice, 1994). The predominant explanations of such
failures are lack of motivation, lack of planning (Gollwitzer,
1999), failure to delay gratiﬁcation (Mischel, Shoda, & Ro-
driguez, 1989), or the depletion of the capacity for self-
control (Muraven & Slessareva, 2003). Here, we explore an
alternative explanation for people’s failure to achieve theirgoals: the inherent complexity of the underlying computa-
tional problem.
The relationship between means and goals can be for-
malized in a bipartite graph whose vertices are divided into
a set of means M=fm1;;mkgand a set of goals G=
fg1;;glg. In this graph, a mean mis connected to a goal g
if and only if selecting mwill achieve the goal g. Such net-
works are called goal systems in the psychological literature
(Kruglanski et al., 2002). For instance, the vertices at the top
of the goal system illustrated in Figure 1 might correspond to
your goals to become a proliﬁc scientist ( g1), be a wonder-
ful partner ( g2), become a great parent ( g3), get physically ﬁt
(g4), and enjoy life to the fullest ( g5).
Finding the best conﬁguration of means for achieving a set
of goals can involve considerable computational challenges.
It has been suggested that ﬁndings from theoretical computer
science can shed light on how people cope with hard com-
putational problems (Van Rooij, 2008). For example, Van
Rooij (2008) advocated applying the theory of ﬁxed param-
eter tractability to study how people cope with hard compu-
tational problems. Yet, while previous research on problem
solving has investigated which strategies people use to solve
NP-hard problems (MacGregor & Ormerod, 1996; MacGre-
gor, Ormerod, & Chronicle, 2000), this literature has focused
on the Traveling Salesman problem and other problems that
are structurally distinct from those that arise in goal pursuit.1
Here, we will ﬁll this gap by analyzing goal achievement
through the lens of computational complexity theory.
In theoretical computer science, it is well known that the
performance of many combinatorial optimization algorithms
critically depends on certain graph-theoretic properties of the
networks they are applied to (Kleinberg & Tardos, 2006). For
instance, a well-documented phenomenon in algorithm de-
sign, artiﬁcial intelligence, and operational research is that
NP-hard optimization problems often become easier on trees
and tree-like graphs. Indeed, when restricted to trees, many
1One exception is the work of Carruthers, Masson, and Stege
(2012) which found that the planarity of graphs has no effect on
human performance in the Vertex Cover problem.
Figure 1: Example of goal system: means appear at the bot-
tom marked by m1:::m4. Goals appear on top marked by
g1:::g5. A goal can be attained if at least one mean con-
nected to it is chosen. The set fm1;m3gis a minimal set of
means covering all goals
NP-hard problems can be solved by efﬁcient polynomial-time
algorithms, such as divide-and-conquer methods and greedy
algorithms. Here, we show that human performance on two
goal management tasks is also well predicted by graph the-
oretic measures of the tree-likeness of the underlying goal
system. Our results offer a fresh computational perspective
on why people fail to achieve their goals. Our experimental
results align well with theoretical knowledge from computer
science and highlight that ﬁndings from computational com-
plexity are relevant to cognitive psychology.
The plan for the paper is as follows. We start by formal-
izing two common challenges of goal achievement in terms
of two classic NP-hard problems: Set Cover and Maximum
Coverage. Next we derive our theoretical prediction by argu-
ing that people’s performance on these problems should in-
crease with graph theoretic metrics of how tree-like the goal
system is. We then test this prediction in three behavioral ex-
periments and conclude with a summary of our ﬁndings and
directions for future work.
Formal analysis and predictions
Here we formalize, as well-deﬁned NP-hard problems, two
computational challenges that arise in means selection prob-
lems where one seeks to choose a set of means that are instru-
mental to the ends one is trying to achieve.
The ﬁrst problem we consider is trying to achieve as many
goals as possible with a ﬁxed budget that limits the number of
activities that one can perform. We formalize this challenge
in terms of the Maximum Coverage problem (MC). In it, we
are given a bipartite graph H= (C;D;F)where C;Dare the
sides of the bipartition, and Fis the set of edges connecting
vertices in Cto vertices in D. We are also given a nonnegative
integer kjCj. We seek to ﬁnd a set C0C, of cardinality
at most k, maximizing the number of vertices in Dcovered
by vertices in C0(a vertex b2Dis covered by a vertex a2C
if(a;b)2F). As a goal system, the set Ccorresponds to
means, the set Dcorresponds to goals, and Frepresents to
interconnections between goal and means. Observe that we
assume that once a goal is covered by a single mean then it
will be achieved. This assumption is made in order to sim-
plify the experimental task, allowing for a simple and cleandescription.
The second problem that we study is trying to achieve a
given set of goals as efﬁciently as possible by selecting the
minimal number of means that will accomplish all goals. We
formalize this challenge in terms of the Set Cover problem
(SC). In the Set Cover problem, we are given a bipartite graph
G= (A;B;E), where A;Bare the sides of the bipartition and
Eis the set of edges connecting vertices in Ato vertices in B.
As a goal system, the set Acorresponds to means, the set B
corresponds to goals, and the set Ecorresponds to intercon-
nections between goals and means. In the SC problem, our
goal is to cover all vertices in Bby a set A0Aof minimal
cardinality. Both MC and SC are NP-hard not only to solve
exactly but also to approximate (Feige, 1998).
These two computational problems (MC and SC) capture
essential aspects of means selection problems that have been
studied in the psychological literature.
For example, it is generally agreed that people try to se-
lect means in order to maximize the number of attained goals
(Kruglanski et al., 2002; Zhang, Fishbach, & Kruglanski,
2007). Furthermore, representing goals as graphs and assum-
ing interconnectedness between goals appear either explicitly
or implicitly in several papers (Kruglanski et al., 2002; Tha-
gard & Millgram, 1995).
We shall use the following two performance measures in
quantifying how well people do in MC and SC. Our ﬁrst mea-
sure simply quantiﬁes the ability to ﬁnd the optimal solution.
This is a binary measure that equals 1 if the person ﬁnds the
optimal solution and 0 otherwise. Given an algorithm Afor
MC or SC, the second measure, referred to as the solution
quality , ranges between 0 (worst) and 1 (best). For the MC
task, the solution quality is the number of goals achieved by A
divided by maximum number of goals that could be achieved
with kmeans. For the SC task, the solution quality is the
minimum number of means that achieves all goals divided by
the number of means selected by A. Solution quality (which
is also referred to as the approximation ratio ofA) is widely
used in quantifying the quality of approximation algorithms
for NP-hard problems (Vazirani, 2013).
The hardness of MC and SC gives a ﬁrst indication for
why people might ﬁnd it difﬁcult to juggle multiple goals
at the same time. Yet, not every instance of these problems
is equally difﬁcult. Next, we introduce graph-theoretic mea-
sures that might be useful for distinguishing harder instances
from easier ones.
Features and predictions
A tree is a connected graph without cycles. Many NP-hard
problems on graphs with small treewidth (Robertson & Sey-
mour, 1986, see below), allow exact or approximate algo-
rithms which are signiﬁcantly better than what is known to
be achievable on worst-case instances.
The idea that tree-like graphs might be easier for people
to deal with guided our search for features that quantify how
similar a given network is to a tree. Four such features are
presented below.
Treewidth . Treewidth (tw) is a combinatorial parameter
that is associated with a graph.2Low treewidth implies
that the nodes and edges of the network can be arranged in
a way that resembles a tree (e.g., Kloks, 1994; Robertson
& Seymour, 1986). For a precise deﬁnition of treewidth
see Kloks (1994).
Combinatorial expansion . Given a graph G= (V;E)and a
nonempty subset SV, let¶(S)be the set of edges cross-
ing the cut (S;VnS)and let N(S)be the set of all vertices in
VnShaving a neighbor in S. The vertex-expansion of Gis
deﬁned as min SV;0<jSjjVj=2jN(S)j
jSj. The edge-expansion of
Gis min SV;0<jSjjVj=2j¶(S)j
jSj. Trees of bounded degree have
expansion O(1=jVj), hence large expansion suggests that
the graph is dissimilar to a tree. We computed both vertex
and edge expansion by solving an integer linear programs
(ILPs) using IBM’s CPLEX.
Spectral expansion . The adjacency matrices of the graphs
we consider are symmetric, hence have nreal eigenvalues
l1l2:::ln. The classical (discrete) Cheeger’s in-
equality (e.g., Alon and Milman, 1985) implies that the
larger dl2is, the larger the edge expansion of the graph.3
MC and SC instances with low treewidth ware known
to have exact algorithms that run in time O(2wn)(Alber &
Niedermeier, 2002), hence instances with low treewidth are
likely to be easier to deal with algorithmically. Current al-
gorithms that compute tree-decompositions are quite com-
plicated, therefore it seems unlikely that people would use
them to solve SC and MC problems. Nevertheless, treewidth
might affect the performance of people’s heuristics: the sim-
ilarity between low treewidth graphs and trees might make
the kinds of algorithms that people might use, such as greedy
and divide-and-conquer methods, much more effective. Con-
versely, as worst-case instances of MC and SC are hard
even to approximate, and as hard instances often have large
treewidth and expansion (Clementi & Trevisan, 1999), it is
likely that it will be hard not only to solve exactly, but also to
ﬁnd approximate solutions for instances of large treewidth.
Similar reasoning applies to our expansion measures. We
therefore hypothesize that treewidth, vertex-expansion, edge-
expansion, and the spectral gap of Gare negatively correlated
with the quality of people’s solution to SC and MC problems
and frequency with which they ﬁnd an optimal solution.
Additional predictors
A feedback vertex set (FVS) is a subset of vertices whose
removal from a given graph results in a forest. The size
of a minimal feedback vertex set is an alternative measure
to the similarity of a graph to a tree. Hence we used this
2Here we will only consider undirected graphs.
3We included the spectral gap as it admits an efﬁcient polynomial
algorithm as opposed to the other three predictors which are NP-
hard. As such, it may prove as a practical predictor of performance.feature as well. We calculated FVS using the implementa-
tion based on (Iwata, 2016; Wahlstr ¨om, 2014) available at
https://github.com/wata-orz/fvs.
Previous empirical hardness models have found additional
features of graphs to be useful: the diameter, average eccen-
tricity, and average path length (Leyton-Brown, Nudelman, &
Shoham, 2009). We thus included these features. The diam-
eter of a graph is the longest distance between two vertices
in the graph (where the distance, dist(u;v), is the number of
edges in a shortest path connecting uandv; all graphs con-
sidered are connected). The eccentricity e(v)of a vertex vin
an undirected graph G= (V;E)is the maximal distance of a
vertex in Vnvfrom v. The average eccentricity (AvgEcc) is
1
nåv2Ve(v)(nis the number of vertices). The average path
length (AvgPath) is2
n(n1)ådist(u;v)where summation is
taken over all pairs of distinct vertices.
Behavioral experiments
To test how our predictors relate to human performance in
MC and SC, we conducted three crowdsourced behavioral
experiments. We used a between-subjects design where each
participant was assigned randomly to one of twenty graphs
with treewidths varying from 4 to 13. In the ﬁrst experiment,
participants were asked to solve the SC problem, and in the
second experiment participants were asked to solve the MC
problem. In each case, the problem was graphically repre-
sented as a bipartite graph with 48 vertices. The 24 vertices
at the bottom represent the available means (activities A-Z)
and 24 vertices at the top represent the goals. Each edge from
a means vertex to a goal vertex implies that completing that
activity is sufﬁcient to achieve the goal. In the SC task, par-
ticipants were asked to select a minimal number of activities
to achieve all of the goals. In the MC task, participants were
asked to choose ﬁve activities that achieve as many goals as
possible. The third and ﬁnal experiment asked participants to
solve a SC problem where goals are given semantic content
and real values, and a different visual display is used to elimi-
nate possible visualization effects. In each experiment we re-
stricted our analyses to goal systems in which every goal and
mean vertex had exactly 4 neighbors (i.e., each graph was
4-regular). This restriction meant that each graph required
the same amount of memory to enable processing, ensuring
that any difference between conditions cannot be explained
by working memory limitations.
Experiment 1: Human performance on Set Cover
Methods We recruited 655 participants on Amazon Me-
chanical Turk. Participants were paid $1 :25 and could earn
a performance-dependent bonus of up to $2. Each participant
was randomly assigned to one of 20 conditions that differed
only in the graph structure of the SC problem participants
were asked to solve. After consenting to participate, partici-
pants read a cover story about a person trying to choose which
set of activities (e.g., volunteer to improve the company’s
website and work out at the gym) they should perform in or-
der to achieve all their goals (e.g., earn more money, improve
Figure 2: Interface and instructions for Experiment 1.
relationship with boss, get ﬁt, etc.) with as few activities as
possible because their time is limited. The story highlighted
that some activities achieve multiple goals at the same time.
Next, participants completed a simple practice trial involving
only two means and two goals. Once they had solved this task
successfully, participants proceeded to the Set-Cover prob-
lem they had been assigned to. When participants moused
over an activity, this interface highlighted the goals it would
achieve and the corresponding edges of the graph in green.
Goals that had already been achieved and activities that had
already been selected were highlighted by checkmarks, and
the number of selected activities was shown at the bottom of
the screen. Participants were asked to help a person achieve
their 24 goals with as few activities as possible, and they were
motivated by the prospect of earning a ﬁnancial bonus of $2
for achieving all goals as efﬁciently as possible. After submit-
ting their solution, participants completed an exit question-
naire asking them for basic demographic information (age,
gender, and primary language), and four 9-point Likert scales
(anchors: “not at all”, “somewhat”, and “extremely”) measur-
ing the perceived difﬁculty of the task, motivation to achieve
all goals, motivation to ﬁnd the optimal solution, and their
motivation to ﬁnish the task as quickly as possible. We ex-
cluded 30 participants (4 :6%)because their responses did not
achieve all goals suggesting that they did not follow the in-
structions.
Results We found that treewidth alone explained 44 :90%
of the variance in the frequency with which people found the
optimal solution across the 20 graphs ( F(1;18) =14:20;p=
0:0012): as we increased the treewidth of the graph the per-
centage of participants who discovered the optimal solution
decreased signiﬁcantly ( r=0:59;p=0:0058) from more
than 90% on the graph with treewidth 5 to only about 30%
on the graph with treewidth 14. We found that the aver-age solution quality was negatively correlated with treewidth
(r=0:44;p=0:0525) suggesting that our participants
achieved fewer goals for goal systems with higher treewidth.
Treewidth explained 17 :59% of the variance in the median re-
sponse time across problems ( F(1;18) =3:86;p=0:0650):
the median amount of time people took to solve the problems
tended to increase with treewidth ( r=0:3426) but this effect
was not statistically signiﬁcant ( p=0:1393), and when we re-
stricted this analysis to correct solutions the correlation was
r=0:3825 ( p=0:1297). Perceived difﬁculty also tended to
increase with treewidth ( r=0:37) but this correlation was
not statistically signiﬁcant ( p=0:1062). Our participants
were highly motivated to ﬁnd the optimal solution (average
rating 7 :910:06 out of 9). Thus it appears unlikely that
their motivation was a bottleneck to their performance. Fur-
thermore, motivation appeared to be unaffected by treewidth
(r=0:23;p=0:34). Thus the observed differences in per-
formance appear to result from the inherent difﬁculties of the
means selection problems posed by different goal systems.
Of the additional predictors evaluated, we found that graph
diameter, average shortest path, and average graph eccentric-
ity all were signiﬁcantly positively correlated with the fre-
quency of optimal solutions identiﬁed by our participants
(graph diameter: r=0:5691 ;p=0:0088, avg. shortest path:
r=0:6516 ;p=0:0019, avg. eccentricty: r=0:6265 ;p=
0:0031). In addition, the spectral expansion (measured as
dl2) and the size of the graph vertex and edge expansions
showed signiﬁcant negative correlations with the frequency
of optimal solutions (vertex expansion: r=0:5836 ;p=
0:0069, edge expansion: r=0:4552 ;p=0:0437, spectral
expansion: r=0:6280 ;p=0:0030). We also found that
the average shortest path, average graph eccentricity, spectral
expansion, and the size of the graph vertex expansions were
signiﬁcantly correlated with the average participant solution
qualities (avg. shortest path: r=0:4505 ;p=0:0462, avg. ec-
centricity: r=0:4316 ;p=0:0574, spectral expansion: r=
0:4496 ;p=0:0467, vertex expansion: r=0:4636 ;p=
0:0395). Finally, only the cardinality of the graph edge ex-
pansions exhibited a signiﬁcant correlation with the median
response times on the SC task ( r=0:4538 ;p=0:0445).
Experiment 2: Human performance on Maximum
Coverage
Methods Experiment 2 was identical to Experiment 1 ex-
cept for the task: participants were now instructed to achieve
as many goals as possible subject to the constraint that the
person’s limited time does not permit them to complete more
than ﬁve activities. The 20 graphs and ﬁnancial incentives
were the same as in Experiment 1. The interface of Exper-
iment 1 was modiﬁed to prevent participants from selecting
more than ﬁve activities at a time. When a participant at-
tempted to add a sixth activity they were told they would ﬁrst
have to remove one or more of the activities they had already
selected. The cover story and survey were modiﬁed slightly
to match the change in the task. We recruited 545 participants
on Amazon Mechanical Turk. Participants were paid $1 :25
and could earn a bonus of up to $2. The consent form speci-
ﬁed that participants must not have participated in the previ-
ous version of this experiment. We excluded 23 participants
(4:2%) because they had selected fewer than ﬁve means.
Results The frequency with which people found the op-
timal solution decreased signiﬁcantly with treewidth ( r=
0:4828 ;p=0:0311). We found that treewidth alone
explained 20 :41% of the variance in the frequency with
which people found the optimal solution across the 20
graphs ( F(1;18) = 4:62;p=0:0455). We found that
treewidth explained 25 :25% of the variance in solution qual-
ity (F(1;18) =6:08;p=0:0240) which signiﬁcantly deterio-
rated as treewidth increased ( r=0:4972 ;p=0:0257). The
median amount of time people took to solve the problems did
not increase signiﬁcantly with treewidth ( r=0:25;p=0:28)
and treewidth explained only 0 :6% of our participants’ me-
dian response times ( F(1;18) =0:10;p=0:76). When we
restricted the analysis to the time taken by optimal solu-
tions, the relationship was still not statistically signiﬁcant
(r=0:2994 ;p=0:1998; F(1;18) =0:10;p=0:76). Fi-
nally, treewidth explained only 8 :8% of the variance in the
perceived problem difﬁculty across the 20 graphs ( F(1;18) =
1:74;p=0:20), and the correlation between treewidth and
perceived difﬁculty was not statistically signiﬁcant ( r=
0:26;p=0:26). Our participants were highly motivated to
ﬁnd the optimal solution (average rating 8 :110:06 out of
9). Thus it appears unlikely that their motivation was a bottle-
neck to their performance. Furthermore, motivation appeared
to be unaffected by treewidth ( r=0:03;p=0:91). Thus the
observed differences in performance appear to result from the
inherent difﬁculties of the means selection problems posed by
different goal systems.
In addition, we found that both the size of the graph
edge expansion and the graph spectral expansion (measured
again as dl2) were signiﬁcantly negatively correlated with
the frequency of optimal solutions (edge expansion: r=
0:4802 ;p=0:0321, spectral expansion: r=0:4782 ;p=
0:0330), while the average shortest path and average graph
eccentricity showed a signiﬁcant positive relationship (avg
shortest path: r=0:4912 ;p=0:0279, avg. eccentricity:
r=0:4391 ;p=0:0528). In contrast, only the size of the
graph vertex and edge expansions showed a signiﬁcant cor-
relation with the average solution quality (vertex expansion:
r=0:4431 ;p=0:0504, edge expansion: r=0:4832 ;p=
0:0309), suggesting that in general graph treewidth and com-
binatorial expansions may be more robust predictors of hu-
man performance on the MC problem. None of the metrics
surveyed were signiﬁcantly correlated with median partici-
pant response times.
Experiment 3: A more realistic Set-Cover task
While Experiments 1 and 2 capture some of the computa-
tional challenges of goal achievement, the tasks were rela-
tively abstract. Experiment 3 addresses this limitation by as-signing semantic labels to the 24 goals. These labels were
common new-years resolutions such as “get in shape” and
“earn more money”. Similar semantic goals were used in pre-
vious research in goal-system theory (Zhang et al., 2007). We
also used a different interface to avoid possible visualization
effects that arise from graph drawings in the ﬁrst two experi-
ments.
Methods We recruited 600 participants on Amazon Me-
chanical Turk. Participants were paid $0 :38 for about 5min
of work plus a performance-dependent bonus of $0 :50 if they
found an optimal solution. Each participant was randomly
assigned to one of the twenty graph structures used in Exper-
iments 1 and 2. For each graph, the order in which the means
were listed and the order in which the goals were listed was
randomized between participants. The participants’ task was
to achieve all goals with as few means as possible. The graph-
ical interface of the task was changed to reduce visual clutter.
Instead of drawing edges between mean and goals, the goals
achieved by each mean were listed next to it (see Figure 3).
The cover story was similar to the one used in Experiment
1 but the training trial used the new task interface shown in
Figure 3. The consent form required that participants had
not participated in any of our previous goal management ex-
periments. All participants were included in the subsequent
analyses.
Results On a scale from 1 to 9 participants rated their mo-
tivation to ﬁnd a solution that achieves all goals with the
minimal number of means as 7 :38, their motivation to ﬁn-
ish the task as quickly as possible and move on as 4 :35, and
the difﬁculty of the task as 5 :67. We found that treewidth,
the magnitude of the graph spectral expansion, cardinalities
of the graph edge and vertex expansions, average eccentric-
ity, average shortest path, and graph diameter were all sig-
niﬁcantly correlated with the frequency with which human
participants identiﬁed the optimal solution (treewidth: r=
0:756;p=0:0001; avg. eccentricity: r=0:583;p=0:007;
avg. shortest path: r=0:651;p=0:002; graph diameter: r=
0:525;p=0:017; spectral expansion: r:0:708;p=0:0005;
edge expansion: r=0:7;p=0:0006; vertex expansion:
r=0:7303 ;p=0:0003). Similarly, treewidth, the mag-
nitude of the graph spectral expansion, cardinalities of the
graph edge and vertex expansions, average eccentricity, and
average shortest path were all signiﬁcantly correlated with
the average solution quality of human responses (treewidth:
r=0:60;p=0:005; avg. eccentricity: r=0:4872 ;p=
0:0293; avg. shortest path: r=0:5243 ;p=0:0176; spectral
expansion: r=0:5308 ;p=0:0160; edge expansion: r=
0:5670 ;p=0:0091; vertex expansion: r=0:6047 ;p=
0:0047). None of the features were signiﬁcantly correlated
with median participant response times.
Figure 3: Graphical Interface of Experiment 3.
Conclusions
We demonstrated that people’s performance in Maximum
Coverage and Set Cover can be reliably predicted from graph
theoretic measures for the tree-likeness of the goal system,
such as treewidth and expansion. Our data support the con-
clusion that tree-like goal systems are easier for people to
handle. More generally, our results imply that parameters
that are used in theoretical computer science to differentiate
between hard and easy instances can be leveraged to predict
human performance in NP-hard tasks.
One limitation of our experiments is that their complete,
explicit representation of goals, means, and the connections
between them is a simplifying idealization. In real life, people
are often unaware of some of their goals and means, as well
as some of the connections between goals and means. For
example, maintaining goal systems of moderate size in work-
ing memory when solving means selection problems is likely
to be nontrivial. Hence real-life representations of goals are
likely to make means selection problems as those discussed
here even more challenging to solve. Although such memory
problems are not directly related to how treelike the goal sys-
tem is, they are nevertheless consistent with our hypothesis
that the cognitive difﬁculty of means selection is an impor-
tant limiting factor for people’s ability to achieve their goals.
In conclusion, our results suggest that even highly moti-
vated people will likely fall short of achieving all their goals
when they have to consider many goals and means in paral-
lel. Our analyses provide a novel approach to predicting how
likely people are to succeed in these settings, with implica-
tions for the design of goal systems that make it easier for
people to meet their objectives.
References
Alber, J., & Niedermeier, R. (2002). Improved tree decomposition
based algorithms for domination-like problems. In Latin ameri-
can symposium on theoretical informatics (pp. 613–627).
Atkinson, J. W., & Birch, D. (1970). The dynamics of action . Ox-
ford: John Wiley.Baumeister, R. F., Heatherton, T. F., & Tice, D. M. (1994). Losing
control: How and why people fail at self-regulation. San Diego:
Academic Press.
Carruthers, S., Masson, M. E., & Stege, U. (2012). Human perfor-
mance on hard non-euclidean graph problems: vertex cover. The
Journal of Problem Solving ,5(1), 5.
Clementi, A. E., & Trevisan, L. (1999). Improved non-
approximability results for minimum vertex cover with density
constraints. Theoretical Computer Science ,225(1), 113–128.
Feige, U. (1998). A threshold of ln n for approximating set cover.
Journal of the ACM ,45(4), 634–652.
Gollwitzer, P. M. (1999). Implementation intentions: strong effects
of simple plans. American Psychologist ,54(7), 493–503.
Iwata, Y . (2016). Linear-time kernelization for feedback vertex set.
arXiv preprint arXiv:1608.01463 .
Kleinberg, J., & Tardos, ´E. (2006). Algorithm design . Pearson
Education India.
Kloks, T. (1994). Treewidth: computations and approximations
(V ol. 842). Springer Science & Business Media.
Kruglanski, A. W., Shah, J. Y ., Fishbach, A., Friedman, R., Chun,
W. Y ., & Sleeth-Keppler, D. (2002). A theory of goal systems.
Advances in Experimental Social Psychology ,34, 331–378.
Leyton-Brown, K., Nudelman, E., & Shoham, Y . (2009). Empirical
hardness models: Methodology and a case study on combinatorial
auctions. Journal of the ACM ,56(4), 22.
Little, B. R., & Gee, T. (2007). The methodology of personal
projects analysis: Four modules and a funnel. In B. R. Little,
K. Salmela-Aro, & S. D. Philipps (Eds.), Personal project pur-
suit: Goals, action, and human ﬂourishing. New York: Psychol-
ogy Press.
MacGregor, J. N., & Ormerod, T. (1996). Human performance
on the traveling salesman problem. Perception & Psychophysics ,
58(4), 527–539.
MacGregor, J. N., Ormerod, T. C., & Chronicle, E. (2000). A
model of human performance on the traveling salesperson prob-
lem. Memory & Cognition ,28(7), 1183–1190.
Miller, G. A., Galanter, E., & Pribram, K. H. (1960). Plans and the
structure of behavior . New York: Adams Bannister Cox.
Mischel, W., Shoda, Y ., & Rodriguez, M. L. (1989). Delay of grati-
ﬁcation in children. Science ,244(4907), 933–938.
Muraven, M., & Slessareva, E. (2003). Mechanisms of self-control
failure: Motivation and limited resources. Personality and Social
Psychology Bulletin ,29(7), 894–906.
Newell, A., & Simon, H. A. (1972). Human problem solving . En-
glewood Cliffs: Prentice-Hall.
Robertson, N., & Seymour, P. D. (1986). Graph minors. ii. algorith-
mic aspects of tree-width. Journal of algorithms ,7(3), 309–322.
Thagard, P., & Millgram, E. (1995). Inference to the best plan: A
coherence theory of decision. In A. Ram & D. B. Leake (Eds.),
Goal driven learning (pp. 439–454). MIT Press.
Van Rooij, I. (2008). The tractable cognition thesis. Cognitive
Science ,32(6), 939–984.
Vazirani, V. V . (2013). Approximation algorithms . Springer Science
& Business Media.
Wahlstr ¨om, M. (2014). Half-integrality, LP-branching and FPT
algorithms. In Proceedings of the twenty-ﬁfth annual acm-siam
symposium on discrete algorithms (pp. 1762–1781).
Zhang, Y ., Fishbach, A., & Kruglanski, A. W. (2007). The dilution
model: how additional goals undermine the perceived instrumen-
tality of a shared path. Journal of Personality and Social Psychol-
ogy,92(3), 389–401.

David Bourgin
Falk Lieder
Daniel Reichman
Nimrod Talmon
Tom Griffiths


Trade Regulation Rule on Commercial Surveillance and Data Security Rulemaking.

Trade Regulation Rule on Commercial Surveillance and Data Security Rulemaking
We are pleased to submit a public comment in response to the trade regulation rule on
commercial surveillance and data security.
This comment is structured in response to the following description of unfair practices covered
under Section 5 of the Federal Trade Commission (FTC) Act of 1914:
“Generally, a practice is unfair under Section 5 if (1) it causes or is likely to cause substantial
injury, (2) the injury is not reasonably avoidable by consumers, and (3) the injury is not
outweighed by benefits to consumers or competition.”
1
We make two recommendations. Firstly, because AI systems change the behavior of
populations exposed to them, harms that stem from systemic behavior changes (e.g.,
spread of disinformation on social media) must be measured in addition to direct
algorithmic harms. Secondly, that any federal agencies that procure vendored AI systems
submit annual reports that describe the foreseeable impacts of the system on consumer
behaviors. These policies are necessary to ensure the Section 5 mandate remains
enforceable as automated systems increasingly make strategic decisions based on
human data. The types of harms covered by Section 5 will become both increasingly
common and difficult to evaluate without requisite rulemaking.
Today, regulators and policymakers focus on litigating isolated algorithmic harms, such as model
bias or privacy violations. But this agenda neglects the persistent effects of AI systems on
consumer populations. Meanwhile, leading research labs increasingly concentrate on such
persistent effects by attempting to specify the purpose or “objective” of AI systems in ways that
limit negative outcomes.
2
,
3
,
4
,
5
,
6
Our comment highlights the need to reconcile these agendas through new forms of rulemaking
so that the FTC can take an
ex ante
approach to AI-enabled
consumer vulnerabilities. In
particular, the FTC has a duty to track foreseeable harms that result from how systems make
decisions based on consumer data. This tracking should be achieved by documenting how AI
systems impact consumer behaviors over time.
The argument of our comment proceeds as follows:
6
Hendrycks, Dan, et al. "Unsolved problems in ML safety."
arXiv preprint arXiv:2109.13916
(2021).
5
Stray, Jonathan. "Beyond engagement: Aligning algorithmic recommendations with prosocial goals."
Partnership on AI. https://www.partnershiponai.org/beyond-engagement-aligning-algorithmic-
recommendations-with-prosocial-goals
(2021).
4
Zhuang, Simon, and Dylan Hadfield-Menell. "Consequences of misaligned AI."
Advances in Neural
Information Processing Systems
33 (2020): 15763-15773.
3
Andrus, McKane, et al. "AI development for the public interest: From abstraction traps to sociotechnical
risks."
2020 IEEE International Symposium on Technology
and Society (ISTAS)
. IEEE, 2020.
2
Russell, Stuart. "Human-compatible artificial intelligence."
Human-Like Machine Intelligence
(2021):
3-23.
1
Quoted from here: https://www.regulations.gov/document/FTC-2022-0053-0001
1.
Existing AI regulations focus on removing bias
or unfairness from model outputs,
rather than tracking the outcomes generated by the system itself over time.
Existing policies to regulate the performance of machine learning systems tend to focus
on the accuracy or bias of their classifications. As an example, the EU General Data Protection
Regulation enshrines a “right to explanation” for algorithmic decisions.
7
Within this paradigm,
“harm” is defined in terms of system errors that result in injury or damage. For instance, a
self-driving car might misrecognize stop signs or lane markers, causing it to take actions that are
unsafe.
These regulatory techniques are limited, because they focus on singularly impactful
decision points. Modifying such points generates solutions that may work in the short term, but
leaves the behavior of the system as a whole untouched. Many consequential effects of AI
systems emerge over time as they change how humans make decisions or interact with each
other. For example, a social media algorithm may successfully recommend content to users in
ways that keep them on site, but at the cost of generating habit-forming effects that make them
addicted to its feed.
8
-
9
Meanwhile, self-driving cars
may not just learn to navigate roads safely,
but also cluster in ways that limit the flow of traffic.
10
By affecting human drivers’ access to
critical intersections, these fleets will impact access to public roads themselves. In both cases,
the system’s operation has cumulative effects that companies often fail to track or mitigate.
Structural harms arise whenever these effects tamper with individual rights or
protections, like online freedom of speech or keeping public transit accessible.
11
Those
protections may not have been properly addressed in advance of the system’s deployment. Or
they may be abused by the system itself, as in the above example of social media addiction.
Either way, outcomes remain poorly documented.
12
Their
effects are difficult for both experts to
diagnose and laypeople to perceive, as they require a birds’ eye view of how the system
interacts with social contexts over time.
2.
Documentation is neither standardized nor legally
required for the operation of highly
capable AI systems. Moreover, existing approaches fail to capture or account for the
emergent effects generated by deployed AI systems.
12
Roberts, Sarah T.
Behind the screen
. Yale University
Press, 2019.
11
For more discussion of structural harms see:
https://www.technologyreview.com/2022/08/09/1057171/social-media-polluting-society-moderation-alone-
wont-fix-the-problem/
10
See for example recent behavior exhibited by Cruise vehicles in San Francisco:
https://www.cnbc.com/2022/07/01/self-driving-cars-from-gms-cruise-block-san-francisco-streets.html
9
Allcott, Hunt, et al. "The welfare effects of social media."
American Economic Review
110.3 (2020):
629-76.
8
See for example Allcott, Hunt, Matthew Gentzkow, and Lena Song. "Digital addiction."
American
Economic Review
112.7 (2022): 2424-63.
7
Selbst, Andrew, and Julia Powles. "“Meaningful Information” and the Right to Explanation."
Conference
on Fairness, Accountability and Transparency
. PMLR,
2018.
Machine learning practitioners are now developing documentation protocols to address
the structural harms that arise from system behaviors. One of these tools is the datasheet, a
framework for documenting the specific dataset available to a system. These datasheets may
include the features on which the dataset was labeled, the motivation for that labeling strategy,
the intended use of the dataset in question, and known limitations or missing data.
13
Another
important example is model cards, a framework for documenting the model that has been
learned to complete some machine learning task. Model cards may include the known accuracy
thresholds of the model on specific tasks such as facial recognition (e.g., error rates for men vs.
women), the organization responsible for its training, metrics of interest, and known ethical
considerations.
14
Both these approaches are invaluable for understanding the capabilities of particular
systems and are necessary for the robust documentation of potential harms. However they are
not sufficient for tackling the emergent effects of AI systems, for two reasons.
First, they do not track how distinct components of the entire system architecture interact
with each other over time. For example, a self-driving car may learn unusual swerving behaviors
based on how its vision and navigation subsystems interact, impacting traffic even if it doesn’t
hit anything.
15
Datasheets and model cards fail to
capture such effects, which may be opaque to
designers as well as consumers.
Second is the lack of enforceable standards for good performance in safety-critical
settings. This is pivotal in any rulemaking context, but is particularly urgent for AI systems today.
Beyond the discretion of in-house machine learning teams, there are no technical barriers to
deploying model classifiers that are known to be deeply flawed. In that absence, companies like
Hugging Face have stepped up to serve as a one-stop shop for documentation of popular
machine learning resources.
16
However, there is no
equivalent for social media applications
whose algorithms are proprietary and secretive. While voluntary commitments to transparency
are valuable, consumer rights require more than acts of good faith from designers. What is
needed is a reporting protocol for structural harms so that design priorities remain aligned with
critical consumer protections.
3.
While research on the full sociotechnical effects
of system harms is ongoing, their
potential for more significant risks has been established.
Self-driving cars that can alter the flow of traffic will change how roads work. Over time,
social media algorithms that affect users’ tastes will transform how people communicate.
Improving the accuracy of a predictive model does nothing to reveal the types of structural harm
16
https://huggingface.co/
15
For a theoretical explanation of this problem, see
Gilbert, Thomas Krendl. "
Mapping the Political
Economy of Reinforcement Learning Systems: The Case of Autonomous Vehicles
."
Simons Institute
Newsletter
(2021).
14
Mitchell, Margaret, et al. "
Model cards for model
reporting
."
Proceedings of the ACM Conference on
Fairness, Accountability, and Transparency
, Atlanta
2019.
13
Gebru, Timnit, et al. "
Datasheets for datasets
."
Communications of the ACM
64.12 (2021): 86-92.
implicit in these disruptions. Instead, regulators need the means to anticipate and evaluate AI
system outcomes before they happen.
Fortunately, technical methods can generate insights into possible outcomes that make
potential harms more foreseeable. For example, some AI systems are able to learn continuous
behaviors by taking multiple actions or “decisions” in sequence. Consider a YouTube algorithm
that learns how users watch content differently based on its own past recommendations, and
then factors that into its future ones.
To automatically choose actions in these settings, practitioners often use techniques
such as
reinforcement learning
(RL) or planning. These
techniques enable AI systems to learn
how to take actions that change their environment according to their objectives (in contrast to
simply e.g. classifying a datapoint). By optimizing for long-term outcomes, these techniques can
in theory learn more complex and more desirable behaviors than single-step classifiers. For
example, using RL in social media applications seems to increase long-term engagement
metrics and reduce how much clickbait content is shown to users (because clickbait is bad for
user retention).
17
However, the increased power of RL also raises the stakes for potential harm. Firstly, it is
often hard to understand the systems’ actions even under close scrutiny:
18
the solutions that
these systems converge to might be more complex than what humans can easily reason about.
For example, these techniques are what enabled AI bots to beat the world champions at Chess
and Go. Still today, research continues to unpack and understand the strategies of a famous AI
system for these games that was released in 2017.
19
If systems based on these approaches
were to take systematically harmful actions, it could be hard to determine the nature of the
failure even after the fact, let alone anticipate it.
Moreover,
when
functioning
in
human
contexts,
such
systems
will
have
incentives
to
manipulate
people
they
interact
with.
20
This
is
especially
concerning
in
the
context
of
social
media,
21
,
22
where
an
increasing
number
of
companies
are
using
RL.
23
,
24
RL’s
potential
to
shape
long-term
behavioral
outcomes
beyond
short-term
effects
mirrors
the
monopolistic
strategies
24
Gauci, Jason, et al. "Horizon: Facebook's open source applied reinforcement learning platform." arXiv
preprint arXiv:1811.00260 (2018).
23
Chen, Minmin, et al. "Top-k off-policy correction for a REINFORCE recommender system." Proceedings
of the Twelfth ACM International Conference on Web Search and Data Mining. 2019.
22
Evans, Charles and Atoosa Kasirzadeh. “User Tampering in Reinforcement Learning Recommender
Systems.” ArXiv abs/2109.04083 (2021): n. pag.
21
Carroll, Micah et al. “Estimating and Penalizing Preference Shift in Recommender Systems.”
Proceedings of the 15th ACM Conference on Recommender Systems (2021).
20
Krueger, David et al. “Hidden Incentives for Auto-Induced Distributional Shift.” ArXiv abs/2009.09153
(2020): n. pag.
19
McGrath, Thomas, et al. "Acquisition of chess knowledge in alphazero." Proceedings of the National
Academy of Sciences 119.47 (2022): e2206625119.
18
Puiutta, Erika, and Eric Veith. "Explainable reinforcement learning: A survey." International
cross-domain conference for machine learning and knowledge extraction. Springer, Cham, 2020.
17
Using RL on YouTube saw the largest single launch improvement in 2 years,
https://www.youtube.com/watch?v=HEqQ2_1XRTs&t=83s
used
by
Amazon
and
other
Big
Tech
companies.
25
Given
their
lack
of
interpretability,
it
might
be
hard to systematically prove that such systems are being (or not being) manipulative.
These
concerns
extend
also
to
other
types
of
systems
beyond
RL.
26
For
example,
the
ways
that
RL
explicitly
incorporates
sequential
decisions
mirror
system
failures
observed
in
electrical
grids
and
transportation
networks.
27
Meanwhile,
rent
setting
algorithms
may
be
artificially
and
systematically
driving
up
rent
prices
and
stifling
competition.
28
Whether
the
system
uses
RL
or
not,
sequential
decision
criteria
may
introduce
preventable
types
of
consumer harm within human domains.
At
present,
regulators
rely
on
the
benevolence
of
the
companies
involved
to
assess,
monitor,
and
limit
consumer
harms.
This
kind
of
monitoring
is
against
companies’
economic
interests
because
automated
decision
making
techniques
such
as
RL
can
lead
to
significant
increases
in
revenue.
29
Companies
have
little
incentive
to
leave
a
paper
trail
which
could
expose
them
to
legal
liability.
One
remaining
option
might
be
to
rely
on
external
audits,
but
structural harms are difficult to assess from outside the companies themselves.
4.
The FTC must codify rules that mitigate the potential
safety risks to consumers from
automated systems.
Algorithmic harms remain poorly documented. Whether or not a given system generates
concrete injuries, its impacts remain hard to measure and difficult to weigh against its benefit to
consumers. This means that automated decision-making can generate harms that are not
reasonably avoidable by consumers. Absent appropriate oversight, they must be seen as unfair
practices according to criterion (2) of Section 5 of the FTC Act, and possibly also criteria (1) and
(3) depending on the scale and intensity of their effects.
As leading experts on the likely impacts of advanced AI capabilities, we believe such
harms are likely to become both more common and more impactful in the coming years.
Recently the Digital Services Act has drawn attention to systemic risks created by large online
platforms
30
, and the EU AI Act expands this lens to
include other kinds of AI systems.
31
While
these developments are encouraging, existing standards and protocols remain stuck in a static,
ex post
accountability regime that cannot keep pace
with the emerging risks.
31
https://www.brookings.edu/research/the-eu-ai-act-will-have-global-impact-but-a-limited-brussels-effect/#fo
otnote-2
30
https://eur-lex.europa.eu/legal-content/EN/TXT/?uri=CELEX%3A32022R2065&qid=1666857835014
29
Chen, Minmin, et al. "Top-k off-policy correction for a REINFORCE recommender system." Proceedings
of the Twelfth ACM International Conference on Web Search and Data Mining. 2019.
28
https://www.propublica.org/article/yieldstar-rent-increase-realpage-rent
27
Gilbert, Thomas Krendl, et al. "Choices, Risks, and Reward Reports: Charting Public Policy for
Reinforcement Learning Systems."
arXiv preprint arXiv:2202.05716
(2022).
26
Jiang, Ray, et al. "Degenerate feedback loops in recommender systems." Proceedings of the 2019
AAAI/ACM Conference on AI, Ethics, and Society. 2019.
25
Khan, Lina M. "Amazon's antitrust paradox."
Yale
lJ
126 (2016): 710.
It follows that the FTC must implement trade rules and protocols to better document
automated decisions over time. These rules should reflect how specific types of sequential
decisions tie back to social impacts. For example, social media harms often derive from the use
of metrics like engagement that are at best unreliable proxies for real-world outcomes that are
difficult to encode or directly optimize.
32
,
33
In transportation,
improving traffic throughput may
depend on how automated vehicles route along critical points of a road network.
34
What matters is that both risks and benefits will arise from how automated systems shift
prevailing social norms. These norms include accepted patterns of behavior, rules of conduct,
and information flow between stakeholders.
35
It follows
that new rules should also monitor how
consumer data is collected and retained for use in sequential decisions. These rules should
strive to reconcile the inherent capabilities of the system with established consumer protections.
Such documentation would be a small burden for companies to maintain, while also providing a
paper trail to hold them accountable once harms do occur.
5.
To better investigate the consumer impacts of automated
decisions, the FTC should
support AI documentation that tracks systems’ effects over time.
Present AI documentation techniques like Model Cards and Datasheets provide strictly
ex post
evaluation of AI components that have already
been built. But to apply the FTC Act,
there is a need for
regularly updated
documentation
that tracks whether 1) potential consumer
injuries qualify as substantial, and 2) whether or not they are outweighed by other benefits.
Combining
ex ante
and
ex post
documentation of AI
systems would help diagnose
distinct risks in specific use cases. If regularly updated and publicly available, it would also
support cohesive evaluation of system components and comparison of designers’ expectations
with post-deployment performance. One such framework is Reward Reports, which track how
system behaviors stem from the sequential decisions it was designed to make.
36
Reward
Reports and related efforts could inform how the FTC adopts new standards for consumer
harms based on Section 5.
Since the passage of the National Environmental Policy Act, federal agencies have been
required to submit reports called Environmental Impact Statements (EISs) that explain the likely
36
Gilbert, Thomas Krendl, et al. "Reward Reports for Reinforcement Learning."
arXiv preprint
arXiv:2204.10817
(2022).
35
Nissenbaum, Helen. "Privacy in context."
Privacy
in Context
. Stanford University Press, 2009.
34
Wu, Cathy, Alexandre M. Bayen, and Ankur Mehta. "Stabilizing traffic with autonomous vehicles."
2018
IEEE International Conference on Robotics and Automation (ICRA)
. IEEE, 2018.
33
Bengani, Pri, Stray, Jonathan, and Thorburn, Luke. “What’s right and what’s wrong with optimizing for
engagement?”
https://medium.com/understanding-recommenders/whats-right-and-what-s-wrong-with-optimizing-for-eng
agement-5abaac021851
32
Milli, Smitha, Luca Belli, and Moritz Hardt. "From optimizing engagement to measuring value."
Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency
. 2021.
consequences of their proposed projects on the environment.
37
EISs are a clear model for
documenting AI systems. In particular, we recommend that any federal agencies that rely on
vendored AI systems submit annual Reward Reports that explain the likely consequences of the
system for consumers. If Reward Reports were regularly issued in like manner to EISs, the
FTC’s Section 5 mandate would be much easier to enact and enforce.
Sincerely,
Thomas Krendl Gilbert
Postdoctoral Fellow, Cornell Tech
Micah Carroll
Ph.D. Candidate, UC Berkeley
37
Caldwell, Lynton Keith.
The National Environmental
Policy Act: an agenda for the future
. Indiana
University Press, 1998.


Voluntary safety commitments provide an escape from over-regulation in AI development.

Voluntary safety commitments provide an escape from over-regulation in AI development
=====================================================================================

Abstract
--------

With the introduction of Artificial Intelligence (AI) and related technologies in our daily lives, fear and anxiety about their misuse as well as their inherent biases, incorporated during their creation, have led to a demand for [governance](/topics/social-sciences/governance "Learn more about governance from ScienceDirect's AI-generated Topic Pages") and associated regulation. Yet regulating an innovation process that is not well understood may stifle this process and reduce benefits that society may gain from the generated technology, even under the best intentions. Instruments to shed light on such processes are thus needed as they can ensure that imposed policies achieve the ambitions for which they were designed. Starting from a game-theoretical model that captures the fundamental dynamics of a race for domain supremacy using AI technology, we show how socially unwanted outcomes may be produced when sanctioning is applied unconditionally to risk-taking, i.e. potentially unsafe, behaviours. We demonstrate here the potential of a regulatory approach that combines a voluntary commitment approach reminiscent of soft law, wherein technologists have the freedom of choice between independently pursuing their course of actions or establishing binding agreements to act safely, with either a peer or governmental sanctioning system of those that do not abide by what they pledged. As commitments are binding and sanctioned, they go beyond the classic view of soft law, akin more closely to actual law-enforced regulation. Overall, this work reveals how voluntary but sanctionable commitments generate socially beneficial outcomes in all scenarios envisageable in a short-term race towards domain supremacy through AI technology. These results provide an original dynamic systems perspective of the governance potential of enforceable soft law techniques or co-regulatory mechanisms, showing how they may impact the ambitions of developers in the context of the AI-based applications.

Introduction
------------

With the rapid advancement of AI and related technologies, there has been significant fear and anxiety about their potential misuse as well as the social and ethical consequences that may result from biases within the design of such systems [[1], [2], [3], [4]]. While expectations associated with these advanced technologies increase and monetary profits stimulate rapid deployment, there is a serious risk for taking unethical or risky short cuts to enter a market first with the next innovation, ignoring safety checks and ethical development procedures. As different disagreeable examples have emerged [5], governments and regulating bodies have been catching up by debating new forms and frameworks for regulating this technology [[6], [7], [8], [9]]; notably, the recent EU draft proposal for AI regulation based on identifiable risks [10,11]. Such debates have produced proposals for mechanisms on how to avoid, mediate, or regulate the development and deployment of AI [6,7,9,[12], [13], [14], [15], [16], [17], [18]]. Essentially, regulatory measures such as restrictions and incentives are proposed to limit harmful and risky practices in order to promote beneficial designs [6]. Examples of such approaches [6] include financially supporting the research into beneficial AI [19] and making AI companies pay fines when found liable for the consequences of harmful AI [20].

Although such regulatory measures may provide solutions for particular scenarios, one needs to ensure that they do not overshoot their targets, leading to a stifling of novel innovations, hindering investments into the development into novel directions as they may be perceived to be too risky [21,22]. Worries have been expressed by different organisations/societies that too strict policies may unnecessarily affect the benefits and societal advances that novel AI technologies may have to offer [23]. Regulations affect moreover big and small tech companies differently: A highly regulated domain makes it more difficult for small new start-ups, introducing an inequality and dominance of the market by a few big players [22]. It has been emphasised that neither over-regulation nor a laissez-faire approach suffices when aiming to regulate AI technologies [24]. In order to find a balanced answer, one clearly needs to have first an understanding of how a competitive development dynamic actually could work and how governance choices impact this dynamic, a task well-suited for dynamic systems or agent-based models.

As an intermediate step in regulating novel technologies, less formal and more flexible tools, referred to as soft law, are frequently introduced [25,26], most often by the stake-holders themselves. Soft law involves a series of instruments that are often voluntary in nature and not directly enforceable by governments (which contrasts it from hard or traditional law) [27,28]. Examples are professional guidelines, private standards, codes of conduct, and best practices. Yet, to ensure that these soft law measures are followed, supporting mechanisms such as indirect enforcement by entities like insurance companies, journal publishers, grant funding agencies, and governmental enforcement programs against unfair or deceptive business practices, need to be put in place [28]. For technology with a global impact, it remains to be seen if such simple incentives to follow a set of rules of conduct are sufficient. A globally enforceable system [27] involving also the needs of society, may be required here: just those defined in case of well-defined international/national waters or borders; specific treaties on fishing; war rules of engagement; types of weapon development and deployment; airplanes’ flying aptness, etc.

Starting from a baseline game theoretical model, referred to as the DSAIR model [29] (a model to examine the dynamics of a domain supremacy race), which defines the process through which multiple stake-holders aim for market supremacy, we demonstrate first that unconditional sanctioning will negatively influence social welfare in certain conditions of a short-term race towards domain supremacy through AI technology. Afterwards, we examine an alternative mechanism for resolving the issue, leading to less detrimental effects than unconditional sanctioning. Our approach is to allow technologists or race participants to voluntarily commit themselves to safe innovation procedures, signaling to others their intentions. Specifically, this bottom-up, binding agreement (or commitment) is established for those who want to take a safe choice, with sanctioning applied to violators of such an agreement. The latter differentiates our approach from soft-law [26,28]; voluntary participating in the safety commitment also implies accepting the possibility to face hard repercussions when violating the commitment. As will be shown in the results, a soft-law, voluntary approach, needs to be combined with a form of enforcement, either peer or institutional, to be effective. Enforcement against involuntary actions may be detrimental to an innovation process like the AI revolution we are experiencing at the moment.

**Previous Developments.** As was shown in Ref. [29] participants either follow safety precautions (the SAFE option) or ignore them (the UNSAFE option) in each step of the development process of the DSAIR model. The main assumption in the model was that it requires more time and more effort to comply with the precautionary requirements, making the SAFE option not only costlier, but also slower compared to the UNSAFE option. Accordingly, it was assumed that in playing SAFE, participants must pay a cost *c* > 0, whereas playing UNSAFE costs nothing. Furthermore, whenever playing UNSAFE, the development speed differs and is *s* > 1 whereas in playing SAFE the speed is simply normalized to 1. Decisions to act SAFE or UNSAFE in AI development are repeated until one or more teams attain the designated objective, which can be translated into having completed *W* development steps, on average [29]. As a result, they earn a large benefit or prize *B* (e.g. windfall profits [16]), equally shared among those reaching the target at the same time. A development disaster or a setback may however come to occur with some probability, which is presumed to increase with the number of times that safety requirements were ignored by the winning team(s) at each step. Whenever a disaster of this kind occurs, all the benefits of a risk-taking participant are lost. This risk probability is denoted by *p**r* (see the Models and Methods section for more details).

It was observed in the DSAIR model that, in case the time-scale to reach the target is short, so that over the whole of the development process the average of the accumulated benefit is much smaller than the final benefit *B*, only for a certain window of parameter settings societal interest conflicts with the individual ones: In that region, individual unsafe behaviour dominates, despite that safe development would lead to a larger collective outcome or social welfare (cf. region **II** in Fig. 1). From the regulatory perspective, it is only region **II** that thus requires governance in order to promote or enforce safe actions in order to avoid any disaster that may occur during the technology development race.

A peer punishment mechanism against unsafe behaviour was proposed in order to mediate the behavior in that region (without affecting the desirable safe outcome in region **I**) [30]. It however may lead to a reduction of the societal welfare that can be obtained in region **III** (see Fig. 1) where the desired unsafe (risk-taking/innovative) behaviour becomes significantly reduced whenever punishment is not very costly to the punisher while strongly affecting the punished. The problem with applying system-wide sanctioning for any risk-taking behavior is that it does not take into account the region wherein the AI innovation is taking place, as was visualised by Fig. 1. That is, if it is the case that a development race falls in region **III** (low risk and innovation is collectively beneficial and preferred), unsafe behaviour should not be punished as it is beneficial for the overall social welfare. To enact such a region-dependent targeted punishment scheme, one would require the ability to estimate exactly the risk level associated with each AI development scenario, i.e. knowing beforehand the risk as well as the speed of development. Clearly, that may not be easy due to the lack of data on how such an AI innovation dynamics works. This is especially true in the early stages of the development and adoption of many new technologies, which has become known as the so-called double-bind problem [31]: the impact of technologies, including AI ones cannot be predicted until it becomes a reality, and controlling or changing it is difficult or no longer possible at that point. Additionally, for all its benefits, the large-scale adoption of AI technologies can lead to enormous and unforeseen threats, as pointed out in Ref. [2].

In the remainder of this manuscript, we will focus on a DSAIR model with two participants. In Ref. [29] the results for more than two participants were also provided, showing that the conclusions remained stable, with the exception that the region **I** disappears with an increasing number of race participants.

**Goals.** In this article an alternative solution is proposed that circumvents the need of being able to estimate correctly the speed of development and risk in order to appropriately regulate UNSAFE behavior: race participants can choose whether or not to establish a bilateral commitment to act safely, which also exposes them to a sanction in case they do not uphold their commitment. While, on the one had, the development teams are allowed to work in an UNSAFE manner without repercussions if they do not commit; a prior agreement allows, on the other hand, the safety compliant participants to identify easily unsafe ones while having also the capacity to punish the dishonest ones (who might also be sanctioned by an external party such as a regulating institution).

Our results reveal that this freedom of choice, if enabled through a prior bilateral commitment to SAFE actions, can, on the one hand, avoid over-regulating unsafe behaviour and, on the other, improve significantly the desired safety outcome in the dilemma zone when compared to punishment alone. Interestingly, this type of binding pledges has been argued to be of relevance within other types of global conflicts and dilemmas, such as environmental governance [[32], [33], [34]].

Section snippets
----------------

Models and Methods
------------------

We first recall the DSAIR model with pairwise interactions, previously introduced in Ref. [29], whose understanding is required before one can fully grasp the novel concepts and solutions that are proposed here. Concretely, we extend the prior work with the option to bilaterally, but voluntarily, commit to acting safely, plus the associated punishment of violations of such commitments when one is found out. As will be shown, this novelty leads to surprisingly beneficial outcomes in the

Results
-------

We first focus on analysing the system of self-enforcing bilateral commitment (PP) then showing the results of bilateral commitment complemented with institutional enforcement (IP).

Discussion and conclusions
--------------------------

This paper proposes and analyses, using a multi-agent and population dynamics modelling approach, a novel solution that promotes desirable safety compliance in a technology development race, while at the same time avoiding stifling beneficial innovation due to over-regulation. We base our study on a previously proposed EGT model [29] that describes the dynamics of a competition between safety and risk-taking (unsafe) behaviours, within an (alleged) race for supremacy through AI in a marketable

Acknowledgements
----------------

The authors acknowledge the feedback from Mireille Hildebrandt and Gregory Lewkowicz on issues related with soft law and the necessity of "harder" law in AI governance. T.A.H., L.M.P., T.L. are supported by Future of Life Institute grant RFP2-154. T.A.H. is also supported by a Leverhulme Research Fellowship (RF-2020-603/9). LM.P. support by NOVA-LINCS (UIDB/04516/2020) with the financial support of FCT-Funda\c c\~ao para a Ci\^encia e a Tecnologia, Portugal, through national funds. F.C.S.

References (63)
---------------

* R. de Neufville *et al.*### [Collective action on artificial intelligence: a primer and review](/science/article/pii/S0160791X2100124X)

### Technol. Soc.

(2021)
* X.-P. Chen *et al.*### [The effects of communication and commitment in a public goods social dilemma](/science/article/pii/S0749597884710909)

### Organ. Behav. Hum. Decis. Process.

(1994)
* E.F. Domingos *et al.*### [Timing uncertainty in collective risk dilemmas encourages group reciprocation and polarization](/science/article/pii/S2589004220309494)

### iScience

(2020)
* M.C. Couto *et al.*### [Governance of risky public goods under graduated punishment](/science/article/pii/S0022519320302782)

### J. Theor. Biol.

(2020)
* S. Wang *et al.*### [Exploring optimal institutional incentives for public cooperation](/science/article/pii/S1007570419302357)

### Commun. Nonlinear Sci. Numer. Simulat.

(2019)
* N. Bostrom### Superintelligence: Paths, Dangers, Strategies

(2014)
* A. Holzinger *et al.*### Digital transformation for sustainable development goals (sdgs)-a security, safety and privacy perspective on ai
* C. Stix *et al.*### Bridging the Gap: the Case for an ‘incompletely Theorized Agreement’on Ai Policy

(2020)
* A. Tzachor *et al.*### Artificial intelligence in a crisis needs ethics with urgency

### Nat. Machine Intell.

(2020)
* M. Coeckelbergh### AI Ethics

(2020)

- S.D. Baum### On the promotion of safe and socially beneficial artificial intelligence

### AI Soc.

(2017)
- S. Cave *et al.*### An AI race for strategic advantage: rhetoric and risks
- M. Taddeo *et al.*### Regulate artificial intelligence to avert cyber arms race

### Nature

(2018)
- ### White Paper on Artificial Intelligence – an European Approach to Excellence and Trust

(2020)
- K. Stöger *et al.*### Medical artificial intelligence: the european legal perspective

### Commun. ACM

(2021)
- A. Askell *et al.*### The Role of Cooperation in Responsible AI Development

(2019)
- E.M. Geist### It's already too late to stop the ai arms race: we must manage it instead

### Bull. At. Sci.

(2016)
- T.A. Han *et al.*### Modelling and influencing the AI bidding war: a research agenda
- P. Nemitz### Constitutional democracy and technology in the age of artificial intelligence

### Phil. Trans. Math. Phys. Eng. Sci.

(2018)
- C. O'Keefe *et al.*### The windfall clause: distributing the benefits of ai for the common good
- C. Shulman *et al.*### Arms control and intelligence explosions
- R. Vinuesa *et al.*### The role of artificial intelligence in achieving the sustainable development goals

### Nat. Commun.

(2020)
- J.O. McGinnis(2010)
- J.K. Gurney### Sue My Car Not Me: Products Liability and Accidents Involving Autonomous Vehicles

(2013)
- G.K. Hadfield### Rules for a Flat World: Why Humans Invented Law and How to Reinvent it for a Complex Global Economy

(2017)
- K.-F. Lee### AI Superpowers: China, Silicon Valley, and the New World Order

(2018)
- ### Civil Society Calls for AI Red Lines in the European Union's Artificial Intelligence Proposal

(2021)
- D. Dawson *et al.*### Artificial Intelligence: Australia's Ethics Framework

(2019)
- C.I. Gutierrez *et al.*### Preliminary Results of a Global Database on the Soft Law Governance of Artificial Intelligence

(2020)
- R. Hagemann *et al.*### Soft law for hard problems: the governance of emerging technologies in an uncertain future

### Colo. Tech. LJ

(2018)
- B. Frydman *et al.*### Les codes de conduite: source du droit global?

### Rev. Trimest. Droits Homme

(2009)
View more referencesCited by (15)
-------------

* ### [Electric vehicle charging infrastructure investment strategy analysis: State-owned versus private parking lots](/science/article/pii/S0967070X23001774)

2023, Transport PolicyShow abstractThis paper analytically studies the optimal charging infrastructure investment decisions in parking lots with state-owned firms and private firms, which aims to alleviate range anxiety and facilitate the mass adoption of electric vehicles. The investment decisions are made based on the rational choice in the short term and are assumed to be static. However, the investment in charging infrastructure is a long-term strategy and the decisions might evolve over time. As a result, an evolutionary game theoretic approach is proposed to analytically explore the instability of the investment decisions and derive the optimal investment strategies. We first construct a basic model with a payoff matrix of public and private parking lots and formulate the dynamic replicator equations to find the equilibrium points. We further explore the evolutionary stable strategy (ESS) with its conditions, and analyze the impact of key factors on the strategy evolution. In the basic model, we find that the investment ratio of the state-owned firms and private firms is crucial to the ESS and its original parking revenue has no impact on the investment decisions. To discover the impact of the uncertainty of investments, we next reformulate the model and extend the original payoff matrix by incorporating the perceived value function based on prospect theory. The ESS and its relevant conditions are also derived. We observe that two models have the same ESS in most cases when there is a marked difference in their profitability. When the difference in their profitability is small, the perceived return on investment plays a pivotal role in their investment decisions. Finally, managerial implications are discussed.
* ### [Effects of interconnections among corruption, institutional punishment, and economic factors on the evolution of cooperation](/science/article/pii/S0096300322001357)

2022, Applied Mathematics and ComputationCitation Excerpt :In the future work, we can thus consider different types of institutional incentive strategies such as institutional reward and the combination of punishment and reward. These incentives have been proved to be effective in solving social dilemmas [57,58] and regulating technology safety [59,60]. Thus it is worth studying the interplay among different institutional incentives, corruption, and the evolution of cooperation, and further revealing how these incentives might influence bribery decision making.

Show abstractThe view that altruistic punishment plays an important role in supporting public cooperation among human beings and other species has been widely accepted by the public. However, the positive role of altruistic punishment in enhancing cooperation will be undermined if corruption is considered. Recently, behavioral experiments have confirmed this finding and further investigated the effects of the leader’s punitive power and the economic potential. Nevertheless, there are relatively few studies focusing on how these factors affect the evolution of cooperation from a theoretical perspective. Here, we combine institutional punishment public goods games with bribery games to investigate the effects of the above factors on the evolution of cooperation. Theoretical and numerical results reveal that the existence of corruption will reduce the level of cooperation when cooperators are more inclined to provide bribes. In addition, we demonstrate that stronger leader and richer economic potential are both important to enhance cooperation. In particular, when defectors are more inclined to provide bribes, stronger leaders can sustain the contributions of public goods from cooperators if the economic potential is weak.
* ### [Beneficence Signaling in AI Development Dynamics](https://doi.org/10.48550/arXiv.2305.02561)

2023, arXiv
* ### [On the number of equilibria of the replicator-mutator dynamics for noisy social dilemmas](https://doi.org/10.48550/arXiv.2303.16558)

2023, arXiv
* ### [BOTH EYES OPEN: VIGILANT INCENTIVES HELP REGULATORY MARKETS IMPROVE AI SAFETY](https://doi.org/10.48550/arXiv.2303.03174)

2023, arXiv
* ### [Artificial intelligence development races in heterogeneous settings](https://doi.org/10.1038/s41598-022-05729-3)

2022, Scientific Reports

[View all citing articles on Scopus](http://www.scopus.com/scopus/inward/citedby.url?partnerID=10&rel=3.0.0&eid=2-s2.0-85122086823&md5=c1236fdf6088bd57b7abbe0b6ede39b)Recommended articles (6)
------------------------

* Research article### [Probability of discoveries between research fields to explain scientific and technological change](/science/article/pii/S0160791X2200015X)

Technology in Society, Volume 68, 2022, Article 101874Show abstractOne of the fundamental problems in science is the understanding of discovery and innovation process to explain scientific and technological change in society. The study confronts this problem here by analyzing the different probability of unique (singleton) and multiple discoveries between research fields to clarify the inventive behavior in science and technology. The method here supposes that the occurrence of discoveries between research fields follows the Poisson distribution. This probabilistic model is applied to determine the probability of singletons and multiple discoveries, and inventions in medicine and physics that are critical research fields to science and society. We find that, over a period of ten years, the probability that at least one discovery occurs in physics and medicine is a rather similar event (11.1%), whereas the probability that at least two discoveries occur in these research fields is less than 1%, i.e., a very rare event. Results also suggest that the probability of patented inventions in medicine is higher than physics. The different probability of patented inventions in medicine can be mainly due to more opportunistic scholars, scientific institutions and stakeholders, associated with university-industry collaborations to develop inventions and commercialize innovations to solve health problems in society. These findings show the property of *heterogeneity* in inventive process between research fields and *specificity* of scientific and technological paths based on nature and scientific ecosystem of each discipline. These characteristics contribute theoretically to extend the theories of scientific change and practically to improve the decision making of policymakers for better allocating resources and supporting scientific planning having a positive societal impact of science.
* Research article### [Identification and prioritization of connected vehicle technologies for sustainable development in Iran](/science/article/pii/S0160791X21003043)

Technology in Society, Volume 68, 2022, Article 101829Show abstractThe Internet of Things (IoT) is a new generation of technologies that not only improves efficiency and productivity but also fosters development across various industries. Given rapid population growth and its negative effects on society and the environment, the application of IoT technologies may be regarded as an opportunity to mitigate those negative impacts and increase sustainability. Among various innovations in wireless communication technologies, connected vehicle technologies (CVT) have received increased attention in recent years. However, there is a dearth of studies on CVT and their influences on the sustainability of transportation systems. Therefore, the identification of the most important applications of CVT in the context of sustainable development can provide useful insights for managers and policy-makers to improve the efficiency of transportation systems. To this end, 18 sustainability indicators in transportation systems are identified based on the literature review. The fuzzy DEMATEL-based analytical network process DANP is used to assess interrelationships among sustainability indicators and to determine the importance of each indicator. Then, fuzzy VIKOR is used to rank the CVT applications that would improve sustainability in transportation systems. The findings show that loss reduction, decision support systems, welfare applications, tourism, and entertainment have higher priorities than other applications of CVT in transportation. Finally, we discuss future research directions and managerial implementations of CVT for moving toward sustainable cities.
* Research article### [A comparative study of research and development related to nanotechnology in Egypt, Nigeria and South Africa](/science/article/pii/S0160791X2200029X)

Technology in Society, Volume 68, 2022, Article 101888Show abstractIt has been proven that nanotechnology has a wide range of applications. This makes it a very significant technology for the future. Because of its scope, nanotechnology is multidisciplinary, requires specialized personnel, and is highly capital intensive. Thus, many nations are increasing their respective activities in fundamental research in nanotechnology intending to develop products and/or processes based on their respective strategies, as shown in more than 60 countries globally, Africa inclusive. This study assesses the current status of nanotechnology in Egypt, Nigeria, and South Africa, focusing mainly on research and development (R & D). Malaysia was selected as a control for comparison with the selected African countries and the top-ranking nations in the factors used. The assessment related to R&D in nanotechnology was conducted based on background information, strategic plans, institutions and researchers involved, publications, and nano-based products. Results obtained indicate that South Africa has the highest ranking (66%) in R & D related to nanotechnology when compared to Egypt (59%) and Nigeria (38%). But the achievement is low for all the countries studied when compared to Malaysia (86%), used as a control, and others that have extremely high rankings in the factors studied. By implication, all the countries studied need to invest more, at different rates, in nano-based R & D to achieve their stated plans in this competitive global world.
* Research article### [Time pacing of product development: The influence of goal clarity and autonomy](/science/article/pii/S0160791X22000380)

Technology in Society, Volume 68, 2022, Article 101897Show abstractAlthough previous research has speculated that successful new product development processes are always time-paced, the execution of time pacing is difficult. In multigenerational product development, the development cycle of an entire product line is long and highly uncertain. Therefore, we explore how to ensure that product development moves along time pacing as viewed from the inner workings of the organization. Our research examines the relationship between minimal structures and the time pacing of product development. The results show that both the relative autonomy dimension and the interactive dimension of minimal structures have positive effects on the time pacing of product development. Furthermore, our study also shows a positive relationship between time pacing and multigenerational product development performance. We performed an empirical study and tested our hypotheses using data on Chinese firms. Our research contributes to the research of paradox management, time pacing, and multigenerational products.
* Research article### [Science-technology intermediary and innovation in China: Evidence from State Administration for Market Regulation, 2000–2019](/science/article/pii/S0160791X22000057)

Technology in Society, Volume 68, 2022, Article 101864Show abstractThis research robustly investigates and verifies the impact of science-technology intermediary (STI) on regional innovation for panel data covering 31 provinces in China from 2000 to 2019. Overall, we find that STI's effects on regional innovation are positive, no matter in the static or dynamic model, reflecting that more innovation-related entities may be motivated to plug into a STI-centered network due to increasingly positive externality, which in turn strengthens STI's innovation-enhancing effect over time. Giving consideration to both development and transaction costs in innovation activities, this study explores three mechanisms of the STI-innovation nexus. (1) STI reduces transaction cost, which then stimulates collaborations of innovation. (2) STI facilitates specialization in innovation-related resources' optimal redistribution, improving regional capacity to successfully convert existing elements into innovation outputs. (3) STI strengthens the capacity to absorb the knowledge spillover from neighboring provinces and foreign countries. All arguments herein are supported for the case of China.
* Research article### [Exploring tensions in Holocaust museums’ modes of commemoration and interaction on social media](/science/article/pii/S0160791X22000306)

Technology in Society, Volume 68, 2022, Article 101889Show abstractDigital technologies and social media platforms have been used in museum communication for over a decade now, and Holocaust museums have increasingly adopted them in their modes of commemoration and provision of educational content. Nevertheless, very limited research has been conducted into the potential of social media as new memory ecologies. In this exploratory study, we conceive social media platforms as socio-technical-ecological systems whereby users develop and engage with memory practices of the Holocaust. We adopt a networked socio-ecological approach to analyse how a sample of Holocaust museums (N = 69) develop practices of digital Holocaust memory in social media. The institutions are analysed in terms of “size” (small, medium, or large), how they differ in their attitudes towards these practices, and to what extent they circulate Holocaust memory on social media. The study adopts multiple quantitative approaches and combines the results of a survey with a set of social media metrics analysing how museums engage on Facebook, Twitter, Instagram, and YouTube in terms of generated content, interactivity, popularity, and type of content. Results show that museums have an overall positive attitude towards social media although some concerns were expressed, mostly by smaller institutions; they tend to use mostly Facebook, Instagram and YouTube, and to share educational content and information about the museum's activities. However, despite a tendency to aggregate a large number of fans and followers, especially in the case of larger institutions, interaction with users remains limited. Prospects for more interactive participation and its implications are also discussed.
[View full text](/science/article/pii/S0160791X21003183)

The Anh Han
Tom Lenaerts
Francisco C
Santos
Luís Moniz Pereira