Inferring strategies from observations in long iterated Prisoner’s dilemma experiments.

[Download PDF](/articles/s41598-022-11654-2.pdf)

### Subjects

* [Computational science](/subjects/computational-science)
* [Scientific data](/subjects/scientific-data)
* [Social evolution](/subjects/social-evolution)

Abstract
--------

While many theoretical studies have revealed the strategies that could lead to and maintain cooperation in the Iterated Prisoner’s dilemma, less is known about what human participants actually do in this game and how strategies change when being confronted with anonymous partners in each round. Previous attempts used short experiments, made different assumptions of possible strategies, and led to very different conclusions. We present here two long treatments that differ in the partner matching strategy used, i.e. fixed or shuffled partners. Here we use unsupervised methods to cluster the players based on their actions and then Hidden Markov Model to infer what the memory-one strategies are in each cluster. Analysis of the inferred strategies reveals that fixed partner interaction leads to behavioral self-organization. Shuffled partners generate subgroups of memory-one strategies that remain entangled, apparently blocking the self-selection process that leads to fully cooperating participants in the fixed partner treatment. Analyzing the latter in more detail shows that AllC, AllD, TFT- and WSLS-like behavior can be observed. This study also reveals that long treatments are needed as experiments with less than 25 rounds capture mostly the learning phase participants go through in these kinds of experiments.

Introduction
------------

The emergence of cooperation in human populations has been an object of study in many different disciplines ranging from social sciences[1](#ref-CR1 "Rand, D. G. & Nowak, M. A. Human cooperation. Trends Cognit. Sci. 17, 413–425.
                  https://doi.org/10.1016/j.tics.2013.06.003

                 (2013)."),[2](#ref-CR2 "Nowak, M. A. Five Rules for the Evolution of Cooperation. Science 314, 1560–1563,
                  https://doi.org/10.1126/science.1133755

                 (2006)."),[3](/articles/s41598-022-11654-2#ref-CR3 "Gracia-Lázaro, C., Cuesta, J. A., Sánchez, A. & Moreno, Y. Human behavior in Prisoner’s dilemma experiments suppresses network reciprocity. Sci. Rep. 2, 325.
                  https://doi.org/10.1038/srep00325

                 (2012)."), physics and complex systems[4](/articles/s41598-022-11654-2#ref-CR4 "Santos, F. C., Santos, M. D. & Pacheco, J. M. Social diversity promotes the emergence of cooperation in public goods games. Nature 454, 213–216,
                  https://doi.org/10.1038/nature06940

                 (2008)."),[5](/articles/s41598-022-11654-2#ref-CR5 "Perc, M. et al. Statistical physics of human cooperation. Phys. Rep. 687, 1–51.
                  https://doi.org/10.1016/j.physrep.2017.05.004

                 (2017).") to biology[6](/articles/s41598-022-11654-2#ref-CR6 "Ashlock, D., Ashlock, W. & Umphry, G. An Exploration of differential utility in iterated Prisoner’s dilemma. In 2006 IEEE Symposium on Computational Intelligence and Bioinformatics and Computational Biology, 1–8,
                  https://doi.org/10.1109/CIBCB.2006.330946

                 (2006)."),[7](/articles/s41598-022-11654-2#ref-CR7 "Dugatkin, L. A. Do guppies play TIT FOR TAT during predator inspection visits?. Behav. Ecol. Sociobiol. 23, 395–399.
                  https://doi.org/10.1007/BF00303714

                 (1988).") and computer science[8](/articles/s41598-022-11654-2#ref-CR8 "Fernández-Domingos, E. et al. Delegation to autonomous agents promotes cooperation in collective-risk dilemmas. arXiv:2103.07710 [cs] (2021).
                  ArXiv: 2103.07710

                ."), among many others. While cooperation is widespread in human societies, its origins remain unclear, although many mechanisms have been proposed to explain its evolutionary advantage[2](/articles/s41598-022-11654-2#ref-CR2 "Nowak, M. A. Five Rules for the Evolution of Cooperation. Science 314, 1560–1563,
                  https://doi.org/10.1126/science.1133755

                 (2006)."). In fact, many social situations pose a dilemma, in which, while being cooperative towards society is beneficial for the population as a whole, free-riding on the efforts of others may generate substantial individual gains[9](/articles/s41598-022-11654-2#ref-CR9 "Dawes, R. M. Social dilemmas. Annu. Rev. Psychol. 31, 169–193,
                  https://doi.org/10.1146/annurev.ps.31.020180.001125

                 (1980)."),[10](/articles/s41598-022-11654-2#ref-CR10 "Lange, P. V., Balliet, D. P., Parks, C. D. & Vugt, Mv. Social Dilemmas: Understanding Human Cooperation (Oxford University Press, 2014)."). While many studies have explored which individual behaviors can promote cooperation in these social dilemmas[8](/articles/s41598-022-11654-2#ref-CR8 "Fernández-Domingos, E. et al. Delegation to autonomous agents promotes cooperation in collective-risk dilemmas. arXiv:2103.07710 [cs] (2021).
                  ArXiv: 2103.07710

                ."),[11](/articles/s41598-022-11654-2#ref-CR11 "Han, T. A. The Emergence of Commitments and Cooperation. In  Intention Recognition, Commitment and Their Roles in the Evolution of Cooperation: From Artificial Intelligence Techniques to Evolutionary Game Theory Models (ed. Han, T. A.) 109–121 (Springer, 2013),
                  https://doi.org/10.1007/978-3-642-37512-5_7

                ."),[12](/articles/s41598-022-11654-2#ref-CR12 "Rand, D. G., Ohtsuki, H. & Nowak, M. A. Direct reciprocity with costly punishment: Generous tit-for-tat prevails. J. Theor. Biol. 256, 45–57.
                  https://doi.org/10.1016/j.jtbi.2008.09.015

                 (2009).") or aimed to find how much human decision-making aligns with those cooperative strategies[1](/articles/s41598-022-11654-2#ref-CR1 "Rand, D. G. & Nowak, M. A. Human cooperation. Trends Cognit. Sci. 17, 413–425.
                  https://doi.org/10.1016/j.tics.2013.06.003

                 (2013)."),[13](/articles/s41598-022-11654-2#ref-CR13 "Baek, S. K., Jeong, H.-C., Hilbe, C. & Nowak, M. A. Comparing reactive and memory-one strategies of direct reciprocity. Sci. Rep. 6, 25676,
                  https://doi.org/10.1038/srep25676

                 (2016)."), not so much is known about which strategies effectively can be extracted from experimental game-theoretical data, which often do not end in full cooperation. Identifying the actual decision-making schemes is essential, especially if one wants to understand why cooperation is not achieving anticipated levels, to grasp the incentives needed to promote beneficial outcomes or how artificial systems may need to be designed so that they align with human pro-social behavior.

Using a data science approach we aim to provide an answer to such questions here, focusing on a new cohort of game theoretical experiments within the well-known framework of the pairwise Iterated Prisoner’s dilemma (IPD)[14](/articles/s41598-022-11654-2#ref-CR14 "Axelrod, R. The evolution of strategies in the iterated prisoner’s dilemma. In Genetic Algorithms and Simulated Annealing 32–41 (Morgan Kaufmann Publishers, 1987)."). In the IPD, when both players cooperate (C) in a round, they both get a reward (represented by *R*), while if one of them defects (D), and the other cooperates they get a payoff *T* and *S* respectively. If both defect, they both get a payoff *P*. The dilemma emerges when \(T>R>P>S\), with \(2R>T+S\). The payoff is accumulated by the interacting players at each round until the game ends. In this iterated version of the one-shot PD there are a plethora of possible equilibrium outcomes[15](/articles/s41598-022-11654-2#ref-CR15 "García, J. & van Veelen, M. No strategy can win in the repeated Prisoner’s dilemma: Linking game theory and computer simulations. Front. Robot. AI 5,
                  https://doi.org/10.3389/frobt.2018.00102

                 (2018)."), including cooperative ones[16](/articles/s41598-022-11654-2#ref-CR16 "Fudenberg, D. & Maskin, E. The folk theorem in repeated games with discounting or with incomplete information. Econometrica 54, 533–554,
                  https://doi.org/10.2307/1911307

                 (1986).").

The literature has reported on many strategies that can induce cooperation in the IPD with fixed partners. These results were produced mainly through models and simulations. In the famous first Axelrod tournament[17](/articles/s41598-022-11654-2#ref-CR17 "Axelrod, R. & Hamilton, W. D. The evolution of cooperation. Science 211, 1390–1396,
                  https://doi.org/10.1126/science.7466396

                 (1981)."), for instance, Tit-for-Tat (TFT) emerged as the most successful strategy. TFT starts by signaling the intention to cooperate and afterwards mimics the previous action of the opponent, and it has been argued to be a good representation of reciprocal behavior in human societies. Reciprocity has been thoroughly researched as one of the most important mechanisms to favour cooperation[1](/articles/s41598-022-11654-2#ref-CR1 "Rand, D. G. & Nowak, M. A. Human cooperation. Trends Cognit. Sci. 17, 413–425.
                  https://doi.org/10.1016/j.tics.2013.06.003

                 (2013)."),[2](/articles/s41598-022-11654-2#ref-CR2 "Nowak, M. A. Five Rules for the Evolution of Cooperation. Science 314, 1560–1563,
                  https://doi.org/10.1126/science.1133755

                 (2006)."),[18](/articles/s41598-022-11654-2#ref-CR18 "Trivers, R. L. The evolution of reciprocal altruism. Q. Rev. Biol. 46, 35–57,
                  https://doi.org/10.1086/406755

                 (1971). Publisher: The University of Chicago Press."). More generally, conditionally cooperative strategies appear to generate the right set of opportunities for cooperative behavior to spread in large populations[19](#ref-CR19 "Nowak, M. Stochastic strategies in the Prisoner’s dilemma. Theor. Popul. Biol. 38, 93–112.
                  https://doi.org/10.1016/0040-5809(90)90005-G

                 (1990)."),[20](#ref-CR20 "Reuben, E. & Suetens, S. Revisiting strategic versus non-strategic cooperation. Exp. Econ. 15, 24–43.
                  https://doi.org/10.1007/s10683-011-9286-4

                 (2012)."),[21](/articles/s41598-022-11654-2#ref-CR21 "Fernández-Domingos, E. et al. Timing uncertainty in collective risk dilemmas encourages group reciprocation and polarization. iScience 23, 101752,
                  https://doi.org/10.1016/j.isci.2020.101752

                 (2020)."), highlighting the importance of context and past experiences in the effectiveness of cooperative strategies. Reciprocity thus justifies, from an evolutionary perspective, the existence of altruistic and pro-social behaviors[22](/articles/s41598-022-11654-2#ref-CR22 "Gurven, M. & Winking, J. Collective action in action: Prosocial behavior in and out of the laboratory. Am. Anthropol. 110, 179–190.
                  https://doi.org/10.1111/j.1548-1433.2008.00024.x

                 (2008).").

However successful TFT was to explain aspects such as direct reciprocity and conditional cooperation, this strategy is known to dissolve into mutual defection in the presence of execution errors, i.e. when participants fail to keep the implicit cooperative agreement at a given round[23](/articles/s41598-022-11654-2#ref-CR23 "Wu, J. & Axelrod, R. How to Cope with Noise in the Iterated Prisoner’s dilemma. J. Conflict Resolut. 39, 183–189,
                  https://doi.org/10.1177/0022002795039001008

                 (1995)."), especially in heterogeneous populations[24](/articles/s41598-022-11654-2#ref-CR24 "Nowak, M. A. & Sigmund, K. Tit for tat in heterogeneous populations. Nature 355, 250–253,
                  https://doi.org/10.1038/355250a0

                 (1992)."). Attempts to repair this flaw lead to the introduction of Generous-Tit-for-Tat (GTFT)[25](/articles/s41598-022-11654-2#ref-CR25 "Wedekind, C. & Milinski, M. Human cooperation in the simultaneous and the alternating Prisoner’s dilemma: Pavlov versus Generous Tit-for-Tat. In Proceedings of the National Academy of Sciences 93, 2686–2689,
                  https://doi.org/10.1073/pnas.93.7.2686

                 (1996).") which cooperates if the opponent cooperates, but if the opponent defects it sometimes “forgives” and continues to cooperate. This strategy is also a reactive strategy, but unlike TFT, it is stochastic, because the player’s next action is now given with some probability. Yet, GTFT has its own problems leading to the introduction of the Win-Stay-Lose-Shift (WSLS)[26](/articles/s41598-022-11654-2#ref-CR26 "Kraines, D. & Kraines, V. Learning to cooperate with Pavlov an adaptive strategy for the iterated Prisoner’s dilemma with noise. Theory Decis. 35, 107–150.
                  https://doi.org/10.1007/BF01074955

                 (1993).") strategy that repeats the action from the previous round if the player is happy with the obtained payoff (*T* and *R*), otherwise changes to the opposite action (when the payoff is *P* or *S*). More strategies have been proposed and analyzed since then (see Martinez-Vaquero et al[27](/articles/s41598-022-11654-2#ref-CR27 "Martinez-Vaquero, L. A., Cuesta, J. A. & Sánchez, A. Generosity pays in the presence of direct reciprocity: A comprehensive study of 2 × 2 repeated games. PLOS ONE 7, e35135,
                  https://doi.org/10.1371/journal.pone.0035135

                 (2012).") for a comprehensive study).

From an experimental angle, only a few works have examined how participant actions in the IPD may be translated into relevant strategies used by humans, while also studying what factors may affect the observed behavior. Some experiments with animals showed that guppies[7](/articles/s41598-022-11654-2#ref-CR7 "Dugatkin, L. A. Do guppies play TIT FOR TAT during predator inspection visits?. Behav. Ecol. Sociobiol. 23, 395–399.
                  https://doi.org/10.1007/BF00303714

                 (1988)."), sticklebacks[28](/articles/s41598-022-11654-2#ref-CR28 "Milinski, M. TIT for TAT in sticklebacks and the evolution of cooperation. Nature 325, 433–435.
                  https://doi.org/10.1038/325433a0

                 (1987).") and tree swallows[29](/articles/s41598-022-11654-2#ref-CR29 "Lombardo, M. P. Mutual restraint in tree swallows: A test of the TIT for TAT model of reciprocity. Science (New York, N.Y.) 227, 1363–1365.
                  https://doi.org/10.1126/science.227.4692.1363

                 (1985)."), exhibit a TFT-like behavior. In human behavioral economic experiments, both the theoretical GTFT and WSLS appear to align with the decision-process of the subjects[30](/articles/s41598-022-11654-2#ref-CR30 "Milinski, M. & Wedekind, C. Working memory constrains human cooperation in the Prisoner’s dilemma. In Proceedings of the National Academy of Sciences 95, 13755–13758,
                  https://doi.org/10.1073/pnas.95.23.13755

                 (1998)."). Dal Bó et al[31](/articles/s41598-022-11654-2#ref-CR31 "Dal Bó, P. & Frechette, G. Strategy choice in the infinitely repeated prisoners’ dilemma. Discussion Papers, Research Unit: Economics of Change SP II 2013-311, WZB Berlin Social Science Center (2013).") showed in another IPD experiment that, when given the choice among theoretical strategies, subjects preferred strategies like Always Defect (AllD), TFT, and Grim Trigger (starts cooperating, cooperates as long as both players have cooperated in the last round, and defects otherwise) and did not use WSLS, which was shown to have more “desirable” properties, such as not defecting forever after a deviation.

While matching theoretical strategies to experiments or asking people to select their preferred strategy provides insight into how they relate to human preferences, it does not immediately reveal how humans actually decide to act in the IPD while playing freely, since their strategies can change with time and respond to different factors, such as their opponents’ actions or simply by learning the game and reach their own equilibrium. Alternatively, their strategy may be directly inferred from the data itself, using algorithmic models[32](/articles/s41598-022-11654-2#ref-CR32 "Breiman, L. Statistical modeling: The two cultures (with comments and a rejoinder by the author). Stat. Sci. 16, 199–231,
                  https://doi.org/10.1214/ss/1009213726

                 (2001)."). This approach was taken to determine the strategies in an Ultimatum game experiment, where binary decision trees were used to model the strategies[33](/articles/s41598-022-11654-2#ref-CR33 "Engle-Warnick, J. Inferring strategies from observed actions: A nonparametric, binary tree classification approach. J. Econ. Dyn. Control 27, 2151–2170.
                  https://doi.org/10.1016/S0165-1889(02)00119-7

                 (2003).") or symbolic regression[34](/articles/s41598-022-11654-2#ref-CR34 "Duffy, J. & Engle-Warnick, J. Using Symbolic Regression to Infer Strategies from Experimental Data. In Evolutionary Computation in Economics and Finance (ed. Chen, S.-H.) Studies in Fuzziness and Soft Computing, 61–82 (Physica-Verlag HD, 2002),
                  https://doi.org/10.1007/978-3-7908-1784-3_4

                ."). A similar approach was taken to infer the behavior of participants in Trust games[35](/articles/s41598-022-11654-2#ref-CR35 "Engle-Warnick, J. & Slonim, R. L. Inferring repeated-game strategies from actions: Evidence from trust game experiments. Econ. Theory 28, 603–632.
                  https://doi.org/10.1007/s00199-005-0633-6

                 (2006)."),[36](/articles/s41598-022-11654-2#ref-CR36 "Engle-Warnick, J. & Slonim, R. L. The evolution of strategies in a repeated trust game. J. Econ. Behav. Organ. 55, 553–573.
                  https://doi.org/10.1016/j.jebo.2003.11.008

                 (2004).") or in a market simulation, where strategies were inferred using using Bayesian inference[37](/articles/s41598-022-11654-2#ref-CR37 "Engle-Warnick, J. & Ruffle, B. J. The Strategies Behind Their Actions: A New Method to Infer Repeated-Game Strategies and an Application to Buyer Behavior. SSRN Scholarly Paper ID 300500, Social Science Research Network, Rochester, NY (2002).
                  https://doi.org/10.2139/ssrn.300500

                ."),[38](/articles/s41598-022-11654-2#ref-CR38 "Kleiman-Weiner, M., Tenenbaum, J. B. & Zhou, P. Non-parametric Bayesian inference of strategies in repeated games. Econ. J. 21, 298–315,
                  https://doi.org/10.1111/ectj.12112

                 (2018)."). Yet so far, no inference of strategic models has been performed on IPD, a caveat this work is overcoming.

We present thus, on the one hand, the results of a behavioral economics experiment that investigates how subjects act in the IPD and, on the other hand, the strategies that can be extracted from the data. The experiment consists of two treatments: i) one where the subjects play the IPD with a fixed partner over a large, unspecified number of rounds, which will be called the *fixed partners* or *FP* treatment; and ii) a second long IPD treatment where the opponent changes each round, which we call *shuffled partners* or *SP* treatment. The objective of collecting the data over a large number of rounds was to understand what the effect is of a long experiment on the level of cooperation in the IPD and to study how the inferred behavior differs over time and between a setting where partnerships can be established or one where one is repeatedly confronted by strangers[39](#ref-CR39 "Grujić, J., Röhl, T., Semmann, D., Milinski, M. & Traulsen, A. Consistent strategy updating in spatial and non-spatial behavioral experiments does not promote cooperation in social networks. PLOS ONE 7, e47718,
                  https://doi.org/10.1371/journal.pone.0047718

                 (2012)."),[40](#ref-CR40 "Andreoni, J. & Croson, R. Partners versus strangers: Random rematching in public goods experiments. Handbook Exp. Econ. Results 1, 776–783 (2001)."),[41](/articles/s41598-022-11654-2#ref-CR41 "Gächter, S. Conditional cooperation: Behavioral regularities from the lab and the field and their policy implications. In Economics and Psychology: A Promising New Cross-disciplinary Field, CESifo seminar series, 19–50 (MIT Press, 2007).").

It is not clear from the literature how many rounds of the IPD are needed to observe a stabilization in the human decision-making process, meaning that the learning phase has passed and people are acting according to a clearly defined strategy. Some works have studied the strategies subjects play for ten rounds[42](/articles/s41598-022-11654-2#ref-CR42 "Heuer, L. & Orland, A. Cooperation in the Prisoner’s dilemma: An experimental comparison between pure and mixed strategies. R. Soc. Open Sci. 6, 182142,
                  https://doi.org/10.1098/rsos.182142

                 (2019)."), another one that studied the evolution of cooperation in the IPD lasted on average 1.96 and 4.42 rounds, for their two treatments[43](/articles/s41598-022-11654-2#ref-CR43 "Dal Bó, P. & Fréchette, G. R. The evolution of cooperation in infinitely repeated games: Experimental evidence. Am. Econ. Rev. 101, 411–429.
                  https://doi.org/10.1257/aer.101.1.411

                 (2011)."), another work studied strategies in the IPD with noise lasted on average 8 rounds, where they noticed a considerable strategic diversity, suggesting the subjects did not learn the game completely[44](/articles/s41598-022-11654-2#ref-CR44 "Fudenberg, D., Rand, D. G. & Dreber, A.Slow to anger and fast to forgive: Cooperation in an uncertain world. Am. Econ. Rev. 102, 720–749.
                  https://doi.org/10.1257/aer.102.2.720

                 (2012)."), others, a range of 10 to 35 rounds[45](/articles/s41598-022-11654-2#ref-CR45 "Fleiß, J. & Leopold-Wildburger, U. Once nice, always nice? Results on factors influencing nice behavior from an iterated Prisoner’s dilemma experiment. Syst. Res. Behav. Sci. 31, 327–334,
                  https://doi.org/10.1002/sres.2194

                 (2014)."), 15 rounds[46](/articles/s41598-022-11654-2#ref-CR46 "Majolo, B. et al. Human friendship favours cooperation in the Iterated Prisoner’s dilemma. Behaviour 143, 1383–1395,
                  https://doi.org/10.1163/156853906778987506

                 (2006).") and 100 rounds[47](/articles/s41598-022-11654-2#ref-CR47 "Liu, P.-P. Learning about a Reciprocating Opponent in an Iterated Prisoner’s Dilemma. In State University of New York at Stony Brook (State University of New York at Stony Brook, 2014)."), to cite a few. Yet, the latter focuses on an interaction with a fixed theoretically defined agent. To study the evolution in the decision-making process, we need to know how long the subjects need to learn the game among themselves in different points in time. Given this insight, one can then use the algorithmic modeling approach to infer the actual behaviors, comparing them over time and over different treatment settings, as is the case here.

As experience determines the behavior of each participant, which we will call context, we will use unsupervised clustering techniques to identify the contexts that are found in both IPD treatment data and then determine which strategic algorithmic models may be inferred from the data in each of the contexts (with the context being defined by their own actions and their opponents’ in the previous round). We focus in this work on memory-one strategies since the experiments only informed participants of what happened in a previous round. Moreover, it has been shown that for indefinitely repeated games, players’ payoffs with memory-one strategies are the same regardless if their co-players use longer-memory strategies[48](/articles/s41598-022-11654-2#ref-CR48 "Press, W. H. & Dyson, F. J. Iterated Prisoner’s dilemma contains strategies that dominate any evolutionary opponent. Proc. Natl. Acad. Sci. USA 109, 10409–10413.
                  https://doi.org/10.1073/pnas.1206569109

                 (2012)."), making it difficult to discern the significance of the longer-memory ones.

Our hypothesis is that different behaviors will be inferred from the differing context experiences. Iterating the game for a sufficient time, as specified earlier, will thus allow to clearly discern their short and long-term behavior. We designed the experiment to last 100 rounds, without explicitly informing this hard limit to the participants, which is much longer than previous experiments[43](#ref-CR43 "Dal Bó, P. & Fréchette, G. R. The evolution of cooperation in infinitely repeated games: Experimental evidence. Am. Econ. Rev. 101, 411–429.
                  https://doi.org/10.1257/aer.101.1.411

                 (2011)."),[44](#ref-CR44 "Fudenberg, D., Rand, D. G. & Dreber, A.Slow to anger and fast to forgive: Cooperation in an uncertain world. Am. Econ. Rev. 102, 720–749.
                  https://doi.org/10.1257/aer.102.2.720

                 (2012)."),[45](/articles/s41598-022-11654-2#ref-CR45 "Fleiß, J. & Leopold-Wildburger, U. Once nice, always nice? Results on factors influencing nice behavior from an iterated Prisoner’s dilemma experiment. Syst. Res. Behav. Sci. 31, 327–334,
                  https://doi.org/10.1002/sres.2194

                 (2014).") to cite a few, and investigated how many rounds are actually needed before the learning process ends and the participant behavior appears to become consistent. In other long experiments, this stabilization seemed to appear after 10-20 rounds[49](/articles/s41598-022-11654-2#ref-CR49 "Grujić, J., Eke, B., Cabrales, A., Cuesta, J. A. & Sánchez, A. Three is a crowd in iterated Prisoner’s dilemmas: Experimental evidence on reciprocal behavior. Sci. Rep. 2, 638.
                  https://doi.org/10.1038/srep00638

                 (2012)."), which will be examined in more in a detailed analysis here.

The participant behavior is modeled using Hidden Markov Models (HMM)[50](/articles/s41598-022-11654-2#ref-CR50 "Rabiner, L. A tutorial on hidden Markov models and selected applications in speech recognition. In Proceedings of the IEEE 77, 257–286,
                  https://doi.org/10.1109/5.18626

                 (1989)."), and its parameters are trained using hmmlearn[51](/articles/s41598-022-11654-2#ref-CR51 "Weiss, R. et al. Hmmlearn: Unsupervised learning and inference of Hidden Markov Models (2016).
                  https://github.com/hmmlearn/hmmlearn

                .") on the treatment data separated in behavioral clusters, while simultaneously trying to find the preferred minimal HMM structure (number of states and transition structure) to achieve this. The resulting HMM models are both simple and transparent, containing enough modeling power to represent subjects’ strategies in the IPD while also being generative. The latter is of interest as the inferred strategies could be directly be used within the context of theoretical simulations assessing their performance, which is left for future work.

Methods
-------

### Experimental data collection

As explained in the Introduction, data was collected for two treatments wherein participants played long IPD in two different pairwise configurations, i.e. fixed partners (FP) and shuffled partners (SP). The data from these experiments were collected in Brussels, Belgium, at the Brussels Experimental Economics Laboratory (BEEL), part of the Vrije Universiteit Brussel (VUB). All experiments followed the relevant guidelines and regulations of data protection and experiments with human participants and were approved by the Ethical Commission for Human Sciences at the VUB (ECHW2015\_3). Moreover, all participants gave consent to the experiment by signing a consent form after the instructions of the experiment were read and all questions the participants had were answered and addressed.

Table [1](/articles/s41598-022-11654-2#Tab1) shows the payoff matrix containing the per round rewards used in the two treatments. For both treatments, participants could observe the actions of their partner in the previous round, even when this partner changed from the previous to the current round. More information about the experimental sessions and details about the data collected can be found in the Supplementary Information.

**Table 1 Payoff matrix for both IPD treatments.**[Full size table](/articles/s41598-022-11654-2/tables/1)For each participant in both treatments, we collected their choices, as is visualized by panel A in Fig. [1](/articles/s41598-022-11654-2#Fig1). The combination of two actions, i.e. *CC*, *CD*, *DC* or *DD* (action format: player-opponent, e.g. *CD* means the focal player cooperated and their opponent defected in the previous round), provides a context for the next round, as participants will get this information when making their decision. Each context can now be combined with the action of a participant after observing that context and this combination, as shown in panel B of Fig. [1](/articles/s41598-022-11654-2#Fig1), can now be translated into one of eight values. The entire sequence of actions of a participant and the associated contexts can thus be transformed into a new sequence of numbers that represent the *conditional actions* of each participant. This new sequence will be used to train an HMM, representing in its emission probabilities in every state the conditional response, i.e. cooperate or defect, based on the context they experienced. We also collect for each context, the probability of observing a conditionally cooperative action, as visualized in Figs. [4](/articles/s41598-022-11654-2#Fig4) and [6](/articles/s41598-022-11654-2#Fig6)in the Results section.

**Figure 1**[![figure 1](//media.springernature.com/lw685/springer-static/image/art%3A10.1038%2Fs41598-022-11654-2/MediaObjects/41598_2022_11654_Fig1_HTML.png)](/articles/s41598-022-11654-2/figures/1)From treatment data to HMM. (**A**) At the top, the first 10 rounds between a player and her opponent, In the red box, the actions of round 2, which represent the context for round 3. (**B**) The action sequence mapping, with the green box in panel A representing the action sequence (*CD*)*D* for player A and the blue box (*DC*)*C* also in panel A for player B in rounds 5 and 7 respectively. (**C**) The transformed sequence for players A and B into integers so the HMM with the mapping table in panel B. (**D**) The resulting sequences are grouped given their sub-cluster and then processed by the HMM in panel (**E**). In this example, the resulting HMM is a model with two states *s*1 and *s*2 with transition probabilities given by the yellow arrows with emission probabilities in the boxes below. (**F**) HMM visualization. Note that all the emission probabilities ≤ 0.05 and all transition probabilities ≤ 0.01 were not taken into account for readability. The resulting HMM emission table is colored green from the hidden state *s*1.

[Full size image](/articles/s41598-022-11654-2/figures/1)### Clustering contexts and context-dependent behaviors

People act according to their preferences, which includes how they think others should behave. In order to understand the strategies that are being used one needs to explicitly consider the context wherein they happen. Thus, to find the strategies in the FP and SP treatments, we first separate participants according to their experiences. Once this is complete one can question whether each participant displays the same response for the contexts they experience. This allows one to correctly grasp how humans act in the IPD. As mentioned in the introduction, we focus in this work on memory-one strategies. There are two reasons for this focus: 1) as Press et al. mentioned, in indefinitely repeated games memory-one strategies have the same payoff against those with a longer memory[48](/articles/s41598-022-11654-2#ref-CR48 "Press, W. H. & Dyson, F. J. Iterated Prisoner’s dilemma contains strategies that dominate any evolutionary opponent. Proc. Natl. Acad. Sci. USA 109, 10409–10413.
                  https://doi.org/10.1073/pnas.1206569109

                 (2012)."), making it problematic to discern them. This argument holds for our experiments as the participants did not know how many rounds the experiment would take. 2) The experimental interface (see Supplementary Information) only reminded participants of their and their opponent’s actions in the last round. It may be that participants remember actions from two or more rounds in the past, yet they were not triggered to do so. Our methodology nevertheless allows for the identification of strategies with longer memories, but we have left that for future work as new experiments and more extensive analysis would be required to estimate the importance of each longer-memory strategy against the memory-one’s that are studied here.

To identify the context groups and the behaviors within those groups, a cluster analysis was performed here: we first clustered the subjects based on the number of times of (*CC*), (*CD*), (*DC*) and (*DD*) happened, providing a *contextual clustering*. Subsequently, another clustering was performed on the number of cooperative actions in the sequence for each of the previous action combinations, i.e. (*CC*)*C*, (*CD*)*C*, (*DC*)*C* and (*DD*)*C*, providing thus a *behavioral clustering per context*. These variables are used to generate the t-distributed stochastic neighbor embedding (t-SNE)[52](/articles/s41598-022-11654-2#ref-CR52 "Maaten, Lvd & Hinton, G. Visualizing data using t-SNE. J. Mach. Learn. Res. 9, 2579–2605 (2008).") plot in Fig. [3](/articles/s41598-022-11654-2#Fig3) and Supplementary Fig. [s3](/articles/s41598-022-11654-2#MOESM1).

Different clustering approaches were considered (e.g. K-means, Hierarchical Clustering, and Network Modularity analysis, as can be observed in Supplementary Information, see section 1.3 for more information on the Modularity Network clustering) but in the end, K-means is sufficient as the other approaches generated similar results (see for Supplementary Fig. [S1](/articles/s41598-022-11654-2#MOESM1), [S3](/articles/s41598-022-11654-2#MOESM1) top row, for a comparison with Hierarchical Clustering and Fig. [S4](/articles/s41598-022-11654-2#MOESM1) for Modularity Network Clustering). The implementations were done in Scikit learn[53](/articles/s41598-022-11654-2#ref-CR53 "Pedregosa, F. et al. Scikit-learn: Machine learning in python. J. Mach. Learn. Res. 12, 2825–2830 (2011)."). To determine the optimal number of clusters, we used the “elbow” method by Santopaa et al[54](/articles/s41598-022-11654-2#ref-CR54 "Satopaa, V., Albrecht, J., Irwin, D. & Raghavan, B. Finding a “Kneedle” in a Haystack: Detecting Knee Points in System Behavior. In 2011 31st International Conference on Distributed Computing Systems Workshops, 166–171,
                  https://doi.org/10.1109/ICDCSW.2011.20

                 (2011). ISSN: 2332-5666.") which plots a curve with the sum of squared distances of samples to their closest cluster center and chooses the minimum best number of clusters that minimize the inertia. See Supplementary Figs. [S1](/articles/s41598-022-11654-2#MOESM1) and [S2](/articles/s41598-022-11654-2#MOESM1) for the results produced but the “elbow” method.

To evaluate the quality of the results generated by clustering algorithms, we use the silhouette measure[55](/articles/s41598-022-11654-2#ref-CR55 "Rousseeuw, P. J. Silhouettes: A graphical aid to the interpretation and validation of cluster analysis. J. Comput. Appl. Math. 20, 53–65.
                  https://doi.org/10.1016/0377-0427(87)90125-7

                 (1987)."), which measures the mean intra-cluster Euclidean distance and the mean nearest-cluster Euclidean distance for each observation, with 1 is being the best score and values near zero indicate that some clusters might be overlapping, while negative values indicate observations assigned to a wrong cluster.

### Inferring Hidden Markov Models

Given the context clusters and the context-dependent behavioral sub-clusters, a HMM, using the conditional action sequences (see Fig. [1](/articles/s41598-022-11654-2#Fig1)), is produced for each sub-cluster. As shown in the figure, a HMM is composed of a number of states (*s*1 and *s*2 in the figure) which are connected by transitions (yellow arrows). The HMM is a probabilistic Markov Model in which each observation of a sequence is produced by a hidden (non-observable) state[50](/articles/s41598-022-11654-2#ref-CR50 "Rabiner, L. A tutorial on hidden Markov models and selected applications in speech recognition. In Proceedings of the IEEE 77, 257–286,
                  https://doi.org/10.1109/5.18626

                 (1989)."). We used a multinomial model for HMM with a sequential structure, this means that the model has states connected from left to right, where a transition to another state can only be made in that direction. Returns to a previous hidden state are thus not possible. As also shown in the figure, the transformed sequence of conditional actions is used to train the HMM. The procedure to determine the optimal number of hidden states is specified in Algorithm 1 in the Supplementary Information.

To train the HMM, the hmmlearn library[51](/articles/s41598-022-11654-2#ref-CR51 "Weiss, R. et al. Hmmlearn: Unsupervised learning and inference of Hidden Markov Models (2016).
                  https://github.com/hmmlearn/hmmlearn

                .") was used. To visualize the resulting models the GraphViz package for Python[56](/articles/s41598-022-11654-2#ref-CR56 "Bank, S. G.
                  https://github.com/xflr6/graphviz

                 (2021).") was used. We expect the participant choices to be stochastic since subjects were not instructed to use any strategy in particular, but the action they considered was best according to their expectations. For this reason, we expect the inferred strategies to be noisy. For visualization, we mapped the number sequences back to human-readable triplets, and all the emission probabilities in the HMM less than 0.05 were discarded, as shown in Fig. [1](/articles/s41598-022-11654-2#Fig1)\*.

### Evolution of the strategies

To analyze how the players changed and adapted their strategy over all the 100 rounds, the same procedure of the two-fold clustering and HMM modeling was performed on four different round windows, i.e. from round 1-25, 26-50, 51-75, and finally rounds 76-100). The objective is to assess whether the strategy stabilizes over time and to examine how the treatment, i.e. fixed or shuffled partners, affect the results.

### Ethical approval

Ethical approval by reference number ECHW2015\_3 was obtained from the Ethical Commission for Human Sciences at the Vrije Universiteit Brussel to perform this experiment. All experiments were performed in accordance with the European Union GDPR guidelines and regulations, and the study was conducted in accordance with the Declaration of Helsinki. All the participants in the study had to give their informed consent prior to the participation. All the data of the experiment has been anonymized and cannot be linked to any participant.

Results
-------

### Specific contextual subgroups with associated behavioral responses emerge in each treatment

Before identifying the different clusters in the data provided by both treatments (see “[Methods](/articles/s41598-022-11654-2#Sec2)” section and Supplementary Information for details), we first examine the distribution of contexts experienced by participants during the experiment. Figure [2](/articles/s41598-022-11654-2#Fig2) shows this distribution while also revealing the fraction of times each context led to a cooperative response by a participant. In the case of the FP treatment, mutual cooperation (*CC*) or mutual defection (*DD*) is observed most often and was matched with the expected response (i.e. either \((CC) \rightarrow C\) or \((DD)\rightarrow D\), which we write as (*CC*)*C* and (*DD*)*D* respectively). In the SP treatment, mutual defection *DD* and the anticipated response *D* was most prevalent. Although a similar cooperation probability was shared by the two treatments (see Supplementary Table [S3](/articles/s41598-022-11654-2#MOESM1)) in the case of *DC* and *CD*, there was a higher probability of cooperation in FP than in SP in the *DC* context, i.e. responding positively to an act of cooperation of the co-player. Overall, the figure shows that by shuffling partners cooperation is reduced, as anticipated by prior experiments[39](/articles/s41598-022-11654-2#ref-CR39 "Grujić, J., Röhl, T., Semmann, D., Milinski, M. & Traulsen, A. Consistent strategy updating in spatial and non-spatial behavioral experiments does not promote cooperation in social networks. PLOS ONE 7, e47718,
                  https://doi.org/10.1371/journal.pone.0047718

                 (2012)."). Also, it already hints at different clusters that could be inferred from both data sets.

**Figure 2**[![figure 2](//media.springernature.com/lw685/springer-static/image/art%3A10.1038%2Fs41598-022-11654-2/MediaObjects/41598_2022_11654_Fig2_HTML.png)](/articles/s41598-022-11654-2/figures/2)Probability of each context and the cooperation rate per treatment. In the x-axis, the context of each decision is shown, i.e. the actions of the previous round. The green bar in each chart shows the fraction of subsequent cooperative actions when experiencing the particular context.

[Full size image](/articles/s41598-022-11654-2/figures/2)Clustering the treatments on the contextual information that each participant experiences (the frequency of contexts *CC*, *CD*, *DC* and *DD*) with K-means (see “[Methods](/articles/s41598-022-11654-2#Sec2)” section) singled out 3 groups in both FP and also 3 in SP. As reported in “[Methods](/articles/s41598-022-11654-2#Sec2)” section and visualized in Supplementary Fig. [S1](/articles/s41598-022-11654-2#MOESM1), different algorithms (network modularity and hierarchical clustering) were tested, revealing that a similar number of clusters was obtained in each case. Cluster quality is assessed using silhouette scores, which are 0.5445 and 0.4201 respectively for the FP and SP treatment. These results indicated that a relatively good separation is found between the different contextual clusters.

Panel A in Fig. [3](/articles/s41598-022-11654-2#Fig3) shows the composition of each cluster in terms of its context. In the FP treatment (A, top row), three different groups are identified: Cluster A containing the players who experience mutual cooperation *CC* most often, cluster B where they experience mutual defection *DD* most, and then cluster C where experiences are mixed. These results were anticipated, given the differences shown in Fig. [2](/articles/s41598-022-11654-2#Fig2). In SP (bottom row Fig. [3](/articles/s41598-022-11654-2#Fig3)A) mutual defection is favored over mutual cooperation in two of the clusters, i.e. clusters D and E. Nonetheless, there are sufficient differences between them which are captured by our clustering approach.

**Figure 3**[![figure 3](//media.springernature.com/lw685/springer-static/image/art%3A10.1038%2Fs41598-022-11654-2/MediaObjects/41598_2022_11654_Fig3_HTML.png)](/articles/s41598-022-11654-2/figures/3)Probability of cooperation given each context per cluster (using K-Means). (**A**) The top row shows the context composition of each cluster in the FP treatment, the bottom row does the same for each SP cluster. (**B**) tSNE visualization of the different clusters A–F.

[Full size image](/articles/s41598-022-11654-2/figures/3)The tSNE plot (see “[Methods](/articles/s41598-022-11654-2#Sec2)” section) in panel B of Fig. [3](/articles/s41598-022-11654-2#Fig3) allows for the exploration of these differences and compares the clusters identified in both treatments. This plot reveals that clusters A and C, which are more cooperative in experienced contexts and responses, clearly differentiate themselves from the rest, and one can find them at completely the other side of experiences and responses to those belonging to cluster B of the FP treatment. There seems nonetheless to be an overlap between some members cluster C and those in F in the SP treatment, yet most C members form a group on themselves. The experiences and behaviors of members of the E cluster, on the other hand, are close to those in B, which one can also observe when comparing the two bar charts. Finally, cluster D consists of participants that have experiences and responses that lie in between all others, separating the more defective and cooperative spectrum.

Given these observations, in the following sections, we show how we can infer the behaviors adopted by participants in each contextual cluster. It is important to note that not considering explicitly the experiences of the participants and focusing directly on the conditional behavior may result in grouping together participants that have encountered a different distribution of contexts. It is therefore essential to first partition the participants in function of their contexts distribution and then determine how they differ in their choices given these experiences.

### Fixed partners promotes behavioral self-selection to cooperative or defecting behaviors

Table [2](/articles/s41598-022-11654-2#Tab2) reports the number of behavioral sub-clusters for each contextual cluster (see “[Methods](/articles/s41598-022-11654-2#Sec2)” section): in total, 8 subgroups were inferred from the raw data of the FP treatment, i.e., 2 in cluster A, 3 in cluster B, and 3 in cluster C. Supplementary Fig. [S3](/articles/s41598-022-11654-2#MOESM1) shows a tSNE visualization of how the treatment data is divided into clusters and sub-clusters for FP. As before, the best settings for *K* in K-means clustering were produced using the “elbow” method[54](/articles/s41598-022-11654-2#ref-CR54 "Satopaa, V., Albrecht, J., Irwin, D. & Raghavan, B. Finding a “Kneedle” in a Haystack: Detecting Knee Points in System Behavior. In 2011 31st International Conference on Distributed Computing Systems Workshops, 166–171,
                  https://doi.org/10.1109/ICDCSW.2011.20

                 (2011). ISSN: 2332-5666.") (see “[Methods](/articles/s41598-022-11654-2#Sec2)” section and Supplementary Fig. [S2](/articles/s41598-022-11654-2#MOESM1)).

**Table 2 Summary of the clustering results using K-Means.**[Full size table](/articles/s41598-022-11654-2/tables/2)The behaviors in each contextual cluster are captured each time by two plots: i) a first plot that shows the likelihood of cooperation when experiencing a given context in a contextual cluster, and ii) a second plot that provides the inferred HMM in the behavioral sub-cluster (see “[Methods](/articles/s41598-022-11654-2#Sec2)” section).

For cluster A, one can observe the difference in response when it comes to a *DD* situation, when comparing both sub-clusters (results in red): Individuals in cluster A.0 are more likely to cooperate again, than those in A.1 (see Fig. [4](/articles/s41598-022-11654-2#Fig4)). Moreover, the resulting HMM for cluster A.0 shows how an initial strategy of reciprocating defection ((*CD*)*D*), leads to mutual cooperation ((*CC*)*C*). In addition, an increase in forgiving defective behavior ((*CD*)*C*) can be observed in Cluster A.1 (see HMM for A.1 in Fig. [5](/articles/s41598-022-11654-2#Fig5)), which is present less often in cluster A.0.

**Figure 4**[![figure 4](//media.springernature.com/lw685/springer-static/image/art%3A10.1038%2Fs41598-022-11654-2/MediaObjects/41598_2022_11654_Fig4_HTML.png)](/articles/s41598-022-11654-2/figures/4)Probability of cooperation given each context per cluster for the FP treatment. In each plot, the conditional probability of cooperating for each context is given. The bars represent the binomial error. Each contextual cluster is divided into sub-clusters that were identified using the frequency of cooperation given a context.

[Full size image](/articles/s41598-022-11654-2/figures/4)**Figure 5**[![figure 5](//media.springernature.com/lw685/springer-static/image/art%3A10.1038%2Fs41598-022-11654-2/MediaObjects/41598_2022_11654_Fig5_HTML.png)](/articles/s41598-022-11654-2/figures/5)Hidden Markov Models for the FP treatment. Here, the eight sub-clusters found in FP have a HMM that describes each sub-cluster’s strategy for the data over all rounds. Bold rectangles represent the initial state, while the others represent subsequent hidden states. Symbols with a probability lower than 0.05 are not shown, as well as the transition probability between states lower or equal to 0.01.

[Full size image](/articles/s41598-022-11654-2/figures/5)Although cluster B is divided into three sub-clusters, there are actually two essential ones in terms of the number of participants they represent (see both Fig. [4](/articles/s41598-022-11654-2#Fig4) and the green HMM in Fig. [5](/articles/s41598-022-11654-2#Fig5)). This next FP cluster mainly experienced mutual defection, and it is situated at the opposite of the behaviors present in the other two FP clusters (see Fig. [3](/articles/s41598-022-11654-2#Fig3)B). Looking at B.1 and B.2 in both figures, participants in sub-cluster B.1 appear to respond more often with cooperation than those in sub-cluster B.2 (see Fig. [4](/articles/s41598-022-11654-2#Fig4), center panel), given that the triplets (*CD*)*C* and (*DD*)*C* occur at a higher frequency and that unconditional defection (*DC*)*D* occurs with a high frequency in sub-cluster B.2.

Finally, in cluster C of the FP treatment, participants experienced a mixture of contexts, with a preference for mutually cooperative interactions. Clustering this group and inferring the corresponding HMM still revealed some differences (see again Fig. [4](/articles/s41598-022-11654-2#Fig4), Cluster C and Fig. [5](/articles/s41598-022-11654-2#Fig5), blue group). Sub-cluster C.0 presents different behavior, members of this sub-cluster unconditionally cooperated despite their opponent’s defection ((*CD*)*C*). Sub-cluster C.1 had a mixed strategy of reciprocating their opponent’s previous action, but their rate of exploitation ((*CC*)*D* = 0.33 and (*DC*)*D* = 0.07) is higher than other sub-clusters. Lastly, the sub-cluster C.2 was mainly matching their opponent’s previous action.

### While stranger interactions induce defection, cooperative aspirations remain

In the SP treatment, a larger variety of behaviors can be observed, i.e. 11 in total spread again over 3 contextual clusters. Figures [6](/articles/s41598-022-11654-2#Fig6) and [7](/articles/s41598-022-11654-2#Fig7) immediately reveal some interesting information about the participants’ behavior in the three clusters found for the SP treatment. Over all three clusters, i.e, D, E, and F, we can observe, through their associated HMM models, that they have different degrees of mutual defection (*DD*), with E containing the most (as also was clear from Fig. [3](/articles/s41598-022-11654-2#Fig3), bottom row, panel A). Moreover, the HMM models for E reveal a high tendency to defect even when the co-player acts cooperatively, even more than the defecting cluster B in FP. The opposite seems to occur in cluster F, where mutual cooperation is the highest and players are more likely to ignore a unilateral defection of the co-player.

**Figure 6**[![figure 6](//media.springernature.com/lw685/springer-static/image/art%3A10.1038%2Fs41598-022-11654-2/MediaObjects/41598_2022_11654_Fig6_HTML.png)](/articles/s41598-022-11654-2/figures/6)Probability of cooperation given each context for the Shuffled Partners treatment. In the x-axis, it is shown the probability of cooperating given that in the previous round there was a certain context. The bars represent the binomial error. Each cluster is divided into sub-cluster that were divided using the frequency of cooperation given a context.

[Full size image](/articles/s41598-022-11654-2/figures/6)**Figure 7**[![figure 7](//media.springernature.com/lw685/springer-static/image/art%3A10.1038%2Fs41598-022-11654-2/MediaObjects/41598_2022_11654_Fig7_HTML.png)](/articles/s41598-022-11654-2/figures/7)Hidden Markov Models for the Shuffled Partners treatment. Here, the eleven sub-clusters found in SP have a HMM that describes each sub-cluster’s strategy. Symbols with a probability less than 0.05 were not shown, as well as the transition between states less or equal to 0.01.

[Full size image](/articles/s41598-022-11654-2/figures/7)Cluster D on the other hand, appears to represent intermediate behavior with players still more mutually defecting but the frequency of some other response patterns has increased, which is also visible in Fig. [3](/articles/s41598-022-11654-2#Fig3)B. Sub-cluster D.1 shows to be more conditionally cooperative ((*CC*)*C* and (*DC*)*C*) than the other sub-clusters in D, i.e. with probability 0.3 (see Fig. [7](/articles/s41598-022-11654-2#Fig7), cluster D). Players in sub-cluster D.3 and E.2 appear to try to signal their co-players to cooperate given the high frequency of (*DD*)*C* in that subgroup to establish cooperation. The cluster D.0 is very similar to the sub-cluster D.2, where the difference is how forgiving and relentless they appear to be in relation to their co-player (see frequency of (*CD*)*C* and (*DC*)*D* in D.0 versus D.2).

### Decision-making simplifies over time and outcomes are decided early

Although the HMM models in Figs. [5](/articles/s41598-022-11654-2#Fig5) and [7](/articles/s41598-022-11654-2#Fig7) provide detailed insights into the behaviors observed over a complete (long) IPD experiment, we need to understand whether these behaviors are consistent over time, or whether early decision-making differs from downstream decision-making, which would indicate a learning effect in the experiment. To achieve this goal, the treatment data for each participant is divided into four parts, each consisting of 25 rounds. For each window of 25 rounds, a separate HMM is trained, collectively visualized in the Supplementary Fig. [S8](/articles/s41598-022-11654-2#MOESM1) for the FP treatment and Supplementary Fig. [S9](/articles/s41598-022-11654-2#MOESM1) for the FP treatment.

For FP one can observe that each strategy seems to converge with time to a dominating decision-making pattern. For example, the differences we observed between sub-clusters A.0 and A.1 are determined by how they act in the first 25 rounds, where it seems participants were still exploring their options. For example, participants in sub-cluster A.1, show a mix of unconditional cooperation ((*CD*)*C*), unconditional defection ((*DC*)*D*) and conditional behaviors ((*CD*)*D*, (*DD*)*D* and (*CC*)*C*). After 25 rounds, they coordinated on mutual cooperation for the remainder of the game. Cluster A.0 appears to have led to cooperation thanks to an initial reciprocal behavior ((*CD*)*D*) that was followed by being resolving mistakes if they occur ((*DC*)*C*). Something similar, but then for defection, happens in SP (see Supplementary Fig. [S9](/articles/s41598-022-11654-2#MOESM1)), where sub-clusters E.0 and E.1 differs essentially in how they react to contexts in the first 25 rounds, yet converge to the same behavioral pattern in the end.

The evolution of sub-cluster B.0 underlines the importance to have long enough experiments: as can be observed, these two participants started out exploring actions in the first 25 rounds, then mutually defected in the next 50 rounds, and finally found a way to cooperate in the last 25. The other two sub-clusters, B.1 and B.2 increase their probability to reciprocate defection (*DD*)*D*, but participants in sub-cluster B.1 showed from rounds 26-75 a willingness to establish cooperation (\((CD)C = 0.24\) and 0.12 respectively) and ended up acting conditionally in the last quarter of the game.

Additionally, although cluster C in FP appeared to have less well-defined behaviors associated with it, one can see that C.0 and C.2 are in fact also very specific in their HMM description. Only for C.1 do we see many different contexts and responses in the emission probabilities of the HMM, yet this abundance of responses remains consistent over all rounds. A very similar situation happens in SP (see Supplementary Fig. [S9](/articles/s41598-022-11654-2#MOESM1)), where even though in some clusters the strategies became more concise by having less diversity of contexts (see for instance cluster D.0). Yet, the majority of behaviors appear to remain rather stable over time, which is especially clear for cluster E. Overall, the HMM models in the figure reveal explicitly how having new partners at each round induces noise in the decision-process and hinders participants to converge to a clear strategy, relevant for the experiences they have.

### FP Participants organize according to their behavioral clusters

So clusters A and B in the FP treatment consist mostly of (forgiving as well as strict) cooperators and defectors respectively. It appears to indicate that behavioral self-selection occurred among the participants in FP. In complex systems and evolutionary game theory, a self-selection mechanism (also called self-organization) occurs when parts of the system appear to reach a stable state[57](/articles/s41598-022-11654-2#ref-CR57 "Yackinous, W. S. Chapter 5-Overview of an Ecological System Dynamics Framework. In Understanding Complex Ecosystem Dynamics (ed. Yackinous, W. S.) 83–91 (Academic Press, 2015),
                  https://doi.org/10.1016/B978-0-12-802031-9.00005-X

                ."). In this case, self-organization brings them to either of the extreme cases, i.e. mostly cooperation or mostly defection. This phenomenon could be happening because of the possibility for imitation of the choices by their neighbors in FP, as argued by Mahmoodi et al.[58](/articles/s41598-022-11654-2#ref-CR58 "Mahmoodi, K., West, B. J. & Grigolini, P. Self-organizing Complex Networks: Individual versus global rules. Front. Physiol. 8, 478.
                  https://doi.org/10.3389/fphys.2017.00478

                 (2017)."). Members of the third cluster are in some sense still in between, either switching between matching choices or trying to outsmart the co-player. More information about this analysis can be found in the Supplementary information, visualized in Supplementary Fig. [S5](/articles/s41598-022-11654-2#MOESM1).

The behavioral self-selection in FP is clearly a consequence of the fixed relationships in that treatment. In that case, previous actions play a more significant role than in the case of the SP treatment. This hypothesis is confirmed by the information obtained in the short questionnaire at the end of the experiment: each participant was asked if her decision was influenced by what the other player did in the previous round (see Supplementary Information for more details). In the SP treatment, 50 players responded that they were influenced (52.08^) by the other’s actions. Yet, in the FP treatment, 74 participants responded “yes” to the same question (80.43%). This means that FP participant actions were shaped strongly by actions in the last round, while in SP this shaping did not take place as almost half of the participants did not care about what their opponents did, leaving them with a different approach to decide between cooperating or defect.

### Conditional strategies like TFT and WSLS are observed but not consistently used

How do the strategies inferred from the treatment data relate to those proposed in the literature for the origins of cooperative behavior? To answer this question, the HMM results in Supplementary Figs. [S8](/articles/s41598-022-11654-2#MOESM1) and [S9](/articles/s41598-022-11654-2#MOESM1) need to be considered as the theoretical strategies have been analyzed mostly in fixed partners interactions. Focusing on cluster A in FP first, one can derive that both sub-clusters in A may be associated with TFT-like or a WSLS-like behavior. Considering A.0, one can observe from round 1-25 that this cluster is associated with a reciprocal strategy: It starts out by defecting when the co-player defects (i.e. (*CD*)*D*) but still cooperating when she does ((*DC*)*C*), leading quickly to mutual cooperation ((*CC*)*C*) for the remainder of the FP treatment. Given the strong presence of (*CD*)*D*, this behavior is almost like a form of TFT, around 70% of them started cooperating in the first round as Supplementary Fig. [S6](/articles/s41598-022-11654-2#MOESM1) shows, suggesting that participants of this sub-cluster followed the “starting nice” principle as the classical TFT dictates[14](/articles/s41598-022-11654-2#ref-CR14 "Axelrod, R. The evolution of strategies in the iterated prisoner’s dilemma. In Genetic Algorithms and Simulated Annealing 32–41 (Morgan Kaufmann Publishers, 1987).").

Behaviors in A.1 also end up in mutual cooperation but achieve this in a different, yet less clear, manner: one could say the participants in A.1 also use a form of reciprocation when the co-player defects but cooperation appears to have been promoted while being generous ((*CD*)*C* in combination with (*CD*)*D*) and signaling to go back to cooperation when both defect ((*DD*)*C*). This behavior is very much associated with the idea of WSLS (Win-Stay, Lose-Shift) or Pavlov as explained by Kraines and Kraines[26](/articles/s41598-022-11654-2#ref-CR26 "Kraines, D. & Kraines, V. Learning to cooperate with Pavlov an adaptive strategy for the iterated Prisoner’s dilemma with noise. Theory Decis. 35, 107–150.
                  https://doi.org/10.1007/BF01074955

                 (1993)."): When winning ((*DC*) and (*CC*) contexts) continue with the same action, when loosing ((*CD*) and (*DD*)) switch. This makes sense in the way that this strategy was designed to promote cooperation in defective environments, and to mimic the stochasticity of biological and social interactions[26](/articles/s41598-022-11654-2#ref-CR26 "Kraines, D. & Kraines, V. Learning to cooperate with Pavlov an adaptive strategy for the iterated Prisoner’s dilemma with noise. Theory Decis. 35, 107–150.
                  https://doi.org/10.1007/BF01074955

                 (1993)."). Although non-WSLS responses are still present, it appears that together they were sufficient to induce cooperation.

So in the case of fixed partnership interaction, we see successful cases of TFT- and WSLS-like behavior but more often the “implementations” of these strategies did not lead to cooperation, most likely because they were not rigorously applied, as opposed to the case when working with theoretical models. We also see cases of non-conditional behavior, for example in sub-cluster E.0 and E.1 in rounds 26 to 100 where the participants in these clusters defect unconditionally (*DC*)*D* and (*DD*)*D*, which resembles the theoretical strategy AllD. The contrary also holds, where for example, sub-cluster F.0 unconditionally cooperates (*CD*)*C* and (*CC*)*C* during the whole game.

Discussion
----------

As seen in the Introduction, many approaches can be taken to study the strategies in the IPD. In this work, we clustered participants based on the context they experience. This is an important difference from other works since it allows us to see the nuances between people reacting to the same situation. We choose this approach because the act of either cooperating or defecting can mean different things depending on the context of the action. It is not the same to defect to exploit your opponent than to defect to reciprocate your opponents’ actions. Moreover, it is not the same to cooperate with a fixed opponent as to cooperate with a stranger, that has been interacting with strangers as well. We used unsupervised methods which make no prior assumptions about what strategies players could use, for this, we tested several methods with the same goal to group participants with similar experiences, and we confirmed there is a significant overlap between different approaches.

A second level clustering (behavioral clustering), on how often they cooperated is necessary to analyze the individual differences given their opponents’ actions, and how they signal their intentions in each situation. In fact, in the sub-clusters we found, we could identify players that faced the same context but behaved differently. This means that to understand the strategies people use in the IPD, it is necessary to see what was happening around them in the past and observe how they play in such situations.

Another factor that has an effect on people’s strategies in the IPD is their opponent structure. Since we had two treatments, one with fixed co-players and one with shuffled ones, we could see how they act and what strategies they use given each context. Not only players in FP achieved more cooperation, but their strategies looked similar to their co-players. In other words, they were better at self-organizing and the intra-cluster behavior was very similar, as opposed to players of the SP treatment, who had to interact with members of other sub-clusters, who at the same time, had different strategies in mind.

One last hypothesis that we were able to prove here was the learning effect on people’s strategies in the IPD. By dividing the analysis into windows of 25 rounds, we could see that the initial quarter was used by the players to explore and try different options, we could even see this effect in a chaotic environment such as SP, where establishing an action plan can be difficult due to its opponent structure.

Using Hidden Markov Models proved to be a handful tool to visualize the strategies and approximate how these underlying factors (in this case, represented as hidden states) shape participants’ behavior. Taking a probabilistic approach, rather than a rigid, deterministic one accounts for the stochastic nature of our interactions.

Our findings have implications on how strategies in the IPD are studied: first, performing long experiments allowed us to observe that the participants may not have one but many strategies throughout the game before reaching an equilibrium. This means that when we talk about “how do we really act” we need to account for an exploration stage and a stabilization point. Second, we found that the opponent structure has an effect on people’s behavior, and therefore, their strategies. Whether they are more adaptive to their opponents’ actions or persistent based on their initial expectations and preferences, this depends on how the game is designed, different strategies might arise. For this reason, we believe that a logical step towards developing these results further is to analyze these strategies in other scenarios, such as N-player IPD, as opposed to the pairwise setting we presented in this paper. Previous work has stated that cooperation could be hindered[49](/articles/s41598-022-11654-2#ref-CR49 "Grujić, J., Eke, B., Cabrales, A., Cuesta, J. A. & Sánchez, A. Three is a crowd in iterated Prisoner’s dilemmas: Experimental evidence on reciprocal behavior. Sci. Rep. 2, 638.
                  https://doi.org/10.1038/srep00638

                 (2012)."), but it is uncertain whether they will still be able to reach an equilibrium in their strategies as fast as the pairwise game, and from the strategic point of view, their rationale of what constitutes exploitation or reciprocation changes in this case. Moreover, as we mentioned in the “[Methods](/articles/s41598-022-11654-2#Sec2)” section, our methodology can also be applied to more complex strategies such as Memory-2 or higher. Even though subjects may as well be using more sophisticated strategies, the data is not sufficient to really draw conclusions about this. New experiments including a comparative analysis would be required to estimate the presence of such longer-memory strategies, while also assessing their importance relative to the memory-one ones. As this would lead to a paper in itself, this research is beyond the scope of the current work.

Although we could observe some identifiable strategies thanks to the theoretical work behind the IPD, it is still challenging to determine and predict human behavior in these games, and this was represented by the big diversity of options on each HMM. And while taking a probabilistic approach to model decision-making might help us to understand our heuristics, this will continue to be an open issue in research on human cooperation, since all models have their limitations and there are many other scenarios to take into account. Another factor that is well known is the noise that not only our actions produce, but also our perception of what others are doing and their expectations, which they may also have about us. This is a caveat of human interaction to overcome and we are still trying to understand through these experiments.

Data availability
-----------------

The experimental data and other supplementary information are available at <https://doi.org/10.5061/dryad.37pvmcvmk>. The code to reproduce the results and plots can be found at the Zenodo repository.

References
----------

1. Rand, D. G. & Nowak, M. A. Human cooperation. *Trends Cognit. Sci.* **17**, 413–425. <https://doi.org/10.1016/j.tics.2013.06.003> (2013).

[Article](https://doi.org/10.1016%2Fj.tics.2013.06.003)
 [Google Scholar](http://scholar.google.com/scholar_lookup?&title=Human%20cooperation&journal=Trends%20Cognit.%20Sci.&doi=10.1016%2Fj.tics.2013.06.003&volume=17&pages=413-425&publication_year=2013&author=Rand%2CDG&author=Nowak%2CMA)
2. Nowak, M. A. Five Rules for the Evolution of Cooperation. *Science* **314**, 1560–1563, <https://doi.org/10.1126/science.1133755> (2006).
3. Gracia-Lázaro, C., Cuesta, J. A., Sánchez, A. & Moreno, Y. Human behavior in Prisoner’s dilemma experiments suppresses network reciprocity. *Sci. Rep.* **2**, 325. <https://doi.org/10.1038/srep00325> (2012).

[Article](https://doi.org/10.1038%2Fsrep00325)
 [ADS](http://adsabs.harvard.edu/cgi-bin/nph-data_query?link_type=ABSTRACT&bibcode=2012NatSR...2E.325G)
 [CAS](/articles/cas-redirect/1:CAS:528:DC%2BC38XpsFygt7o%3D)
 [PubMed](http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?cmd=Retrieve&db=PubMed&dopt=Abstract&list_uids=22439103)
 [PubMed Central](http://www.ncbi.nlm.nih.gov/pmc/articles/PMC3309394)
 [Google Scholar](http://scholar.google.com/scholar_lookup?&title=Human%20behavior%20in%20Prisoner%E2%80%99s%20dilemma%20experiments%20suppresses%20network%20reciprocity&journal=Sci.%20Rep.&doi=10.1038%2Fsrep00325&volume=2&publication_year=2012&author=Gracia-L%C3%A1zaro%2CC&author=Cuesta%2CJA&author=S%C3%A1nchez%2CA&author=Moreno%2CY)
4. Santos, F. C., Santos, M. D. & Pacheco, J. M. Social diversity promotes the emergence of cooperation in public goods games. *Nature* **454**, 213–216, <https://doi.org/10.1038/nature06940> (2008).
5. Perc, M. *et al.* Statistical physics of human cooperation. *Phys. Rep.* **687**, 1–51. <https://doi.org/10.1016/j.physrep.2017.05.004> (2017).

[Article](https://doi.org/10.1016%2Fj.physrep.2017.05.004)
 [ADS](http://adsabs.harvard.edu/cgi-bin/nph-data_query?link_type=ABSTRACT&bibcode=2017PhR...687....1P)
 [MathSciNet](http://www.ams.org/mathscinet-getitem?mr=3670080)
 [MATH](http://www.emis.de/MATH-item?1366.80006)
 [Google Scholar](http://scholar.google.com/scholar_lookup?&title=Statistical%20physics%20of%20human%20cooperation&journal=Phys.%20Rep.&doi=10.1016%2Fj.physrep.2017.05.004&volume=687&pages=1-51&publication_year=2017&author=Perc%2CM)
6. Ashlock, D., Ashlock, W. & Umphry, G. An Exploration of differential utility in iterated Prisoner’s dilemma. In *2006 IEEE Symposium on Computational Intelligence and Bioinformatics and Computational Biology*, 1–8, <https://doi.org/10.1109/CIBCB.2006.330946> (2006).
7. Dugatkin, L. A. Do guppies play TIT FOR TAT during predator inspection visits?. *Behav. Ecol. Sociobiol.* **23**, 395–399. <https://doi.org/10.1007/BF00303714> (1988).

[Article](https://doi.org/10.1007%2FBF00303714)
 [Google Scholar](http://scholar.google.com/scholar_lookup?&title=Do%20guppies%20play%20TIT%20FOR%20TAT%20during%20predator%20inspection%20visits%3F&journal=Behav.%20Ecol.%20Sociobiol.&doi=10.1007%2FBF00303714&volume=23&pages=395-399&publication_year=1988&author=Dugatkin%2CLA)
8. Fernández-Domingos, E. *et al.* Delegation to autonomous agents promotes cooperation in collective-risk dilemmas. arXiv:2103.07710 [cs] (2021). [ArXiv: 2103.07710](http://arxiv.org/abs/2103.07710).
9. Dawes, R. M. Social dilemmas. *Annu. Rev. Psychol.* **31**, 169–193, <https://doi.org/10.1146/annurev.ps.31.020180.001125> (1980).
10. Lange, P. V., Balliet, D. P., Parks, C. D. & Vugt, Mv. *Social Dilemmas: Understanding Human Cooperation* (Oxford University Press, 2014).

[Google Scholar](http://scholar.google.com/scholar_lookup?&title=Social%20Dilemmas%3A%20Understanding%20Human%20Cooperation&publication_year=2014&author=Lange%2CPV&author=Balliet%2CDP&author=Parks%2CCD&author=Vugt%2CMv)
11. Han, T. A. The Emergence of Commitments and Cooperation. In *Intention Recognition, Commitment and Their Roles in the Evolution of Cooperation: From Artificial Intelligence Techniques to Evolutionary Game Theory Models* (ed. Han, T. A.) 109–121 (Springer, 2013), <https://doi.org/10.1007/978-3-642-37512-5_7>.
12. Rand, D. G., Ohtsuki, H. & Nowak, M. A. Direct reciprocity with costly punishment: Generous tit-for-tat prevails. *J. Theor. Biol.* **256**, 45–57. <https://doi.org/10.1016/j.jtbi.2008.09.015> (2009).

[Article](https://doi.org/10.1016%2Fj.jtbi.2008.09.015)
 [ADS](http://adsabs.harvard.edu/cgi-bin/nph-data_query?link_type=ABSTRACT&bibcode=2009JThBi.256...45R)
 [MathSciNet](http://www.ams.org/mathscinet-getitem?mr=2973422)
 [PubMed](http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?cmd=Retrieve&db=PubMed&dopt=Abstract&list_uids=18938180)
 [MATH](http://www.emis.de/MATH-item?1400.91054)
 [Google Scholar](http://scholar.google.com/scholar_lookup?&title=Direct%20reciprocity%20with%20costly%20punishment%3A%20Generous%20tit-for-tat%20prevails&journal=J.%20Theor.%20Biol.&doi=10.1016%2Fj.jtbi.2008.09.015&volume=256&pages=45-57&publication_year=2009&author=Rand%2CDG&author=Ohtsuki%2CH&author=Nowak%2CMA)
13. Baek, S. K., Jeong, H.-C., Hilbe, C. & Nowak, M. A. Comparing reactive and memory-one strategies of direct reciprocity. *Sci. Rep.* **6**, 25676, <https://doi.org/10.1038/srep25676> (2016).
14. Axelrod, R. The evolution of strategies in the iterated prisoner’s dilemma. In *Genetic Algorithms and Simulated Annealing* 32–41 (Morgan Kaufmann Publishers, 1987).
15. García, J. & van Veelen, M. No strategy can win in the repeated Prisoner’s dilemma: Linking game theory and computer simulations. *Front. Robot. AI* **5**, <https://doi.org/10.3389/frobt.2018.00102> (2018).
16. Fudenberg, D. & Maskin, E. The folk theorem in repeated games with discounting or with incomplete information. *Econometrica* **54**, 533–554, <https://doi.org/10.2307/1911307> (1986).
17. Axelrod, R. & Hamilton, W. D. The evolution of cooperation. *Science* **211**, 1390–1396, <https://doi.org/10.1126/science.7466396> (1981).
18. Trivers, R. L. The evolution of reciprocal altruism. *Q. Rev. Biol.* **46**, 35–57, <https://doi.org/10.1086/406755> (1971). Publisher: The University of Chicago Press.
19. Nowak, M. Stochastic strategies in the Prisoner’s dilemma. *Theor. Popul. Biol.* **38**, 93–112. <https://doi.org/10.1016/0040-5809(90)90005-G> (1990).

[Article](https://doi.org/10.1016%2F0040-5809%2890%2990005-G)
 [MathSciNet](http://www.ams.org/mathscinet-getitem?mr=1069150)
 [MATH](http://www.emis.de/MATH-item?0726.92015)
 [Google Scholar](http://scholar.google.com/scholar_lookup?&title=Stochastic%20strategies%20in%20the%20Prisoner%E2%80%99s%20dilemma&journal=Theor.%20Popul.%20Biol.&doi=10.1016%2F0040-5809%2890%2990005-G&volume=38&pages=93-112&publication_year=1990&author=Nowak%2CM)
20. Reuben, E. & Suetens, S. Revisiting strategic versus non-strategic cooperation. *Exp. Econ.* **15**, 24–43. <https://doi.org/10.1007/s10683-011-9286-4> (2012).

[Article](https://doi.org/10.1007%2Fs10683-011-9286-4)
 [Google Scholar](http://scholar.google.com/scholar_lookup?&title=Revisiting%20strategic%20versus%20non-strategic%20cooperation&journal=Exp.%20Econ.&doi=10.1007%2Fs10683-011-9286-4&volume=15&pages=24-43&publication_year=2012&author=Reuben%2CE&author=Suetens%2CS)
21. Fernández-Domingos, E. *et al.* Timing uncertainty in collective risk dilemmas encourages group reciprocation and polarization. *iScience* **23**, 101752, <https://doi.org/10.1016/j.isci.2020.101752> (2020).
22. Gurven, M. & Winking, J. Collective action in action: Prosocial behavior in and out of the laboratory. *Am. Anthropol.* **110**, 179–190. <https://doi.org/10.1111/j.1548-1433.2008.00024.x> (2008).

[Article](https://doi.org/10.1111%2Fj.1548-1433.2008.00024.x)
 [Google Scholar](http://scholar.google.com/scholar_lookup?&title=Collective%20action%20in%20action%3A%20Prosocial%20behavior%20in%20and%20out%20of%20the%20laboratory&journal=Am.%20Anthropol.&doi=10.1111%2Fj.1548-1433.2008.00024.x&volume=110&pages=179-190&publication_year=2008&author=Gurven%2CM&author=Winking%2CJ)
23. Wu, J. & Axelrod, R. How to Cope with Noise in the Iterated Prisoner’s dilemma. *J. Conflict Resolut.* **39**, 183–189, <https://doi.org/10.1177/0022002795039001008> (1995).
24. Nowak, M. A. & Sigmund, K. Tit for tat in heterogeneous populations. *Nature* **355**, 250–253, <https://doi.org/10.1038/355250a0> (1992).
25. Wedekind, C. & Milinski, M. Human cooperation in the simultaneous and the alternating Prisoner’s dilemma: Pavlov versus Generous Tit-for-Tat. In *Proceedings of the National Academy of Sciences* **93**, 2686–2689, <https://doi.org/10.1073/pnas.93.7.2686> (1996).
26. Kraines, D. & Kraines, V. Learning to cooperate with Pavlov an adaptive strategy for the iterated Prisoner’s dilemma with noise. *Theory Decis.* **35**, 107–150. <https://doi.org/10.1007/BF01074955> (1993).

[Article](https://doi.org/10.1007%2FBF01074955)
 [MathSciNet](http://www.ams.org/mathscinet-getitem?mr=1238490)
 [MATH](http://www.emis.de/MATH-item?0800.90794)
 [Google Scholar](http://scholar.google.com/scholar_lookup?&title=Learning%20to%20cooperate%20with%20Pavlov%20an%20adaptive%20strategy%20for%20the%20iterated%20Prisoner%E2%80%99s%20dilemma%20with%20noise&journal=Theory%20Decis.&doi=10.1007%2FBF01074955&volume=35&pages=107-150&publication_year=1993&author=Kraines%2CD&author=Kraines%2CV)
27. Martinez-Vaquero, L. A., Cuesta, J. A. & Sánchez, A. Generosity pays in the presence of direct reciprocity: A comprehensive study of 2 × 2 repeated games. *PLOS ONE* **7**, e35135, <https://doi.org/10.1371/journal.pone.0035135> (2012).
28. Milinski, M. TIT for TAT in sticklebacks and the evolution of cooperation. *Nature* **325**, 433–435. <https://doi.org/10.1038/325433a0> (1987).

[Article](https://doi.org/10.1038%2F325433a0)
 [ADS](http://adsabs.harvard.edu/cgi-bin/nph-data_query?link_type=ABSTRACT&bibcode=1987Natur.325..433M)
 [CAS](/articles/cas-redirect/1:STN:280:DyaL2s7hsFyitg%3D%3D)
 [PubMed](http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?cmd=Retrieve&db=PubMed&dopt=Abstract&list_uids=3808044)
 [Google Scholar](http://scholar.google.com/scholar_lookup?&title=TIT%20for%20TAT%20in%20sticklebacks%20and%20the%20evolution%20of%20cooperation&journal=Nature&doi=10.1038%2F325433a0&volume=325&pages=433-435&publication_year=1987&author=Milinski%2CM)
29. Lombardo, M. P. Mutual restraint in tree swallows: A test of the TIT for TAT model of reciprocity. *Science (New York, N.Y.)* **227**, 1363–1365. <https://doi.org/10.1126/science.227.4692.1363> (1985).

[Article](https://doi.org/10.1126%2Fscience.227.4692.1363)
 [ADS](http://adsabs.harvard.edu/cgi-bin/nph-data_query?link_type=ABSTRACT&bibcode=1985Sci...227.1363L)
 [CAS](/articles/cas-redirect/1:STN:280:DC%2BC3cvjsVegtA%3D%3D)
 [Google Scholar](http://scholar.google.com/scholar_lookup?&title=Mutual%20restraint%20in%20tree%20swallows%3A%20A%20test%20of%20the%20TIT%20for%20TAT%20model%20of%20reciprocity&journal=Science%20%28New%20York%2C%20N.Y.%29&doi=10.1126%2Fscience.227.4692.1363&volume=227&pages=1363-1365&publication_year=1985&author=Lombardo%2CMP)
30. Milinski, M. & Wedekind, C. Working memory constrains human cooperation in the Prisoner’s dilemma. In *Proceedings of the National Academy of Sciences* **95**, 13755–13758, <https://doi.org/10.1073/pnas.95.23.13755> (1998).
31. Dal Bó, P. & Frechette, G. Strategy choice in the infinitely repeated prisoners’ dilemma. Discussion Papers, Research Unit: Economics of Change SP II 2013-311, WZB Berlin Social Science Center (2013).
32. Breiman, L. Statistical modeling: The two cultures (with comments and a rejoinder by the author). *Stat. Sci.* **16**, 199–231, <https://doi.org/10.1214/ss/1009213726> (2001).
33. Engle-Warnick, J. Inferring strategies from observed actions: A nonparametric, binary tree classification approach. *J. Econ. Dyn. Control* **27**, 2151–2170. <https://doi.org/10.1016/S0165-1889(02)00119-7> (2003).

[Article](https://doi.org/10.1016%2FS0165-1889%2802%2900119-7)
 [MathSciNet](http://www.ams.org/mathscinet-getitem?mr=1990789)
 [MATH](http://www.emis.de/MATH-item?1179.91089)
 [Google Scholar](http://scholar.google.com/scholar_lookup?&title=Inferring%20strategies%20from%20observed%20actions%3A%20A%20nonparametric%2C%20binary%20tree%20classification%20approach&journal=J.%20Econ.%20Dyn.%20Control&doi=10.1016%2FS0165-1889%2802%2900119-7&volume=27&pages=2151-2170&publication_year=2003&author=Engle-Warnick%2CJ)
34. Duffy, J. & Engle-Warnick, J. Using Symbolic Regression to Infer Strategies from Experimental Data. In *Evolutionary Computation in Economics and Finance* (ed. Chen, S.-H.) Studies in Fuzziness and Soft Computing, 61–82 (Physica-Verlag HD, 2002), <https://doi.org/10.1007/978-3-7908-1784-3_4>.
35. Engle-Warnick, J. & Slonim, R. L. Inferring repeated-game strategies from actions: Evidence from trust game experiments. *Econ. Theory* **28**, 603–632. <https://doi.org/10.1007/s00199-005-0633-6> (2006).

[Article](https://doi.org/10.1007%2Fs00199-005-0633-6)
 [MathSciNet](http://www.ams.org/mathscinet-getitem?mr=2224858)
 [MATH](http://www.emis.de/MATH-item?1118.91018)
 [Google Scholar](http://scholar.google.com/scholar_lookup?&title=Inferring%20repeated-game%20strategies%20from%20actions%3A%20Evidence%20from%20trust%20game%20experiments&journal=Econ.%20Theory&doi=10.1007%2Fs00199-005-0633-6&volume=28&pages=603-632&publication_year=2006&author=Engle-Warnick%2CJ&author=Slonim%2CRL)
36. Engle-Warnick, J. & Slonim, R. L. The evolution of strategies in a repeated trust game. *J. Econ. Behav. Organ.* **55**, 553–573. <https://doi.org/10.1016/j.jebo.2003.11.008> (2004).

[Article](https://doi.org/10.1016%2Fj.jebo.2003.11.008)
 [Google Scholar](http://scholar.google.com/scholar_lookup?&title=The%20evolution%20of%20strategies%20in%20a%20repeated%20trust%20game&journal=J.%20Econ.%20Behav.%20Organ.&doi=10.1016%2Fj.jebo.2003.11.008&volume=55&pages=553-573&publication_year=2004&author=Engle-Warnick%2CJ&author=Slonim%2CRL)
37. Engle-Warnick, J. & Ruffle, B. J. The Strategies Behind Their Actions: A New Method to Infer Repeated-Game Strategies and an Application to Buyer Behavior. SSRN Scholarly Paper ID 300500, Social Science Research Network, Rochester, NY (2002). <https://doi.org/10.2139/ssrn.300500>.
38. Kleiman-Weiner, M., Tenenbaum, J. B. & Zhou, P. Non-parametric Bayesian inference of strategies in repeated games. *Econ. J.* **21**, 298–315, <https://doi.org/10.1111/ectj.12112> (2018).
39. Grujić, J., Röhl, T., Semmann, D., Milinski, M. & Traulsen, A. Consistent strategy updating in spatial and non-spatial behavioral experiments does not promote cooperation in social networks. *PLOS ONE* **7**, e47718, <https://doi.org/10.1371/journal.pone.0047718> (2012).
40. Andreoni, J. & Croson, R. Partners versus strangers: Random rematching in public goods experiments. *Handbook Exp. Econ. Results* **1**, 776–783 (2001).

[Article](https://doi.org/10.1016%2FS1574-0722%2807%2900082-0)
 [Google Scholar](http://scholar.google.com/scholar_lookup?&title=Partners%20versus%20strangers%3A%20Random%20rematching%20in%20public%20goods%20experiments&journal=Handbook%20Exp.%20Econ.%20Results&doi=10.1016%2FS1574-0722%2807%2900082-0&volume=1&pages=776-783&publication_year=2001&author=Andreoni%2CJ&author=Croson%2CR)
41. Gächter, S. Conditional cooperation: Behavioral regularities from the lab and the field and their policy implications. In *Economics and Psychology: A Promising New Cross-disciplinary Field*, CESifo seminar series, 19–50 (MIT Press, 2007).
42. Heuer, L. & Orland, A. Cooperation in the Prisoner’s dilemma: An experimental comparison between pure and mixed strategies. *R. Soc. Open Sci.* **6**, 182142, <https://doi.org/10.1098/rsos.182142> (2019).
43. Dal Bó, P. & Fréchette, G. R. The evolution of cooperation in infinitely repeated games: Experimental evidence. *Am. Econ. Rev.* **101**, 411–429. <https://doi.org/10.1257/aer.101.1.411> (2011).

[Article](https://doi.org/10.1257%2Faer.101.1.411)
 [Google Scholar](http://scholar.google.com/scholar_lookup?&title=The%20evolution%20of%20cooperation%20in%20infinitely%20repeated%20games%3A%20Experimental%20evidence&journal=Am.%20Econ.%20Rev.&doi=10.1257%2Faer.101.1.411&volume=101&pages=411-429&publication_year=2011&author=B%C3%B3%2CP&author=Fr%C3%A9chette%2CGR)
44. Fudenberg, D., Rand, D. G. & Dreber, A.Slow to anger and fast to forgive: Cooperation in an uncertain world. *Am. Econ. Rev.* **102**, 720–749. <https://doi.org/10.1257/aer.102.2.720> (2012).

[Article](https://doi.org/10.1257%2Faer.102.2.720)
 [Google Scholar](http://scholar.google.com/scholar_lookup?&title=Slow%20to%20anger%20and%20fast%20to%20forgive%3A%20Cooperation%20in%20an%20uncertain%20world&journal=Am.%20Econ.%20Rev.&doi=10.1257%2Faer.102.2.720&volume=102&pages=720-749&publication_year=2012&author=Fudenberg%2CD&author=Rand%2CDG&author=Dreber%2CA)
45. Fleiß, J. & Leopold-Wildburger, U. Once nice, always nice? Results on factors influencing nice behavior from an iterated Prisoner’s dilemma experiment. *Syst. Res. Behav. Sci.* **31**, 327–334, <https://doi.org/10.1002/sres.2194> (2014).
46. Majolo, B. *et al.* Human friendship favours cooperation in the Iterated Prisoner’s dilemma. *Behaviour* **143**, 1383–1395, <https://doi.org/10.1163/156853906778987506> (2006).
47. Liu, P.-P. Learning about a Reciprocating Opponent in an Iterated Prisoner’s Dilemma. In *State University of New York at Stony Brook* (State University of New York at Stony Brook, 2014).
48. Press, W. H. & Dyson, F. J. Iterated Prisoner’s dilemma contains strategies that dominate any evolutionary opponent. *Proc. Natl. Acad. Sci. USA* **109**, 10409–10413. <https://doi.org/10.1073/pnas.1206569109> (2012).

[Article](https://doi.org/10.1073%2Fpnas.1206569109)
 [ADS](http://adsabs.harvard.edu/cgi-bin/nph-data_query?link_type=ABSTRACT&bibcode=2012PNAS..10910409P)
 [PubMed](http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?cmd=Retrieve&db=PubMed&dopt=Abstract&list_uids=22615375)
 [PubMed Central](http://www.ncbi.nlm.nih.gov/pmc/articles/PMC3387070)
 [MATH](http://www.emis.de/MATH-item?1264.91009)
 [Google Scholar](http://scholar.google.com/scholar_lookup?&title=Iterated%20Prisoner%E2%80%99s%20dilemma%20contains%20strategies%20that%20dominate%20any%20evolutionary%20opponent&journal=Proc.%20Natl.%20Acad.%20Sci.%20USA&doi=10.1073%2Fpnas.1206569109&volume=109&pages=10409-10413&publication_year=2012&author=Press%2CWH&author=Dyson%2CFJ)
49. Grujić, J., Eke, B., Cabrales, A., Cuesta, J. A. & Sánchez, A. Three is a crowd in iterated Prisoner’s dilemmas: Experimental evidence on reciprocal behavior. *Sci. Rep.* **2**, 638. <https://doi.org/10.1038/srep00638> (2012).

[Article](https://doi.org/10.1038%2Fsrep00638)
 [ADS](http://adsabs.harvard.edu/cgi-bin/nph-data_query?link_type=ABSTRACT&bibcode=2012NatSR...2E.638G)
 [CAS](/articles/cas-redirect/1:CAS:528:DC%2BC3sXht1ygtb3M)
 [PubMed](http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?cmd=Retrieve&db=PubMed&dopt=Abstract&list_uids=22962633)
 [PubMed Central](http://www.ncbi.nlm.nih.gov/pmc/articles/PMC3435562)
 [Google Scholar](http://scholar.google.com/scholar_lookup?&title=Three%20is%20a%20crowd%20in%20iterated%20Prisoner%E2%80%99s%20dilemmas%3A%20Experimental%20evidence%20on%20reciprocal%20behavior&journal=Sci.%20Rep.&doi=10.1038%2Fsrep00638&volume=2&publication_year=2012&author=Gruji%C4%87%2CJ&author=Eke%2CB&author=Cabrales%2CA&author=Cuesta%2CJA&author=S%C3%A1nchez%2CA)
50. Rabiner, L. A tutorial on hidden Markov models and selected applications in speech recognition. In *Proceedings of the IEEE* **77**, 257–286, <https://doi.org/10.1109/5.18626> (1989).
51. Weiss, R. *et al.* Hmmlearn: Unsupervised learning and inference of Hidden Markov Models (2016). <https://github.com/hmmlearn/hmmlearn>.
52. Maaten, Lvd & Hinton, G. Visualizing data using t-SNE. *J. Mach. Learn. Res.* **9**, 2579–2605 (2008).

[MATH](http://www.emis.de/MATH-item?1225.68219)
 [Google Scholar](http://scholar.google.com/scholar_lookup?&title=Visualizing%20data%20using%20t-SNE&journal=J.%20Mach.%20Learn.%20Res.&volume=9&pages=2579-2605&publication_year=2008&author=Maaten%2CLvd&author=Hinton%2CG)
53. Pedregosa, F. *et al.* Scikit-learn: Machine learning in python. *J. Mach. Learn. Res.* **12**, 2825–2830 (2011).

[MathSciNet](http://www.ams.org/mathscinet-getitem?mr=2854348)
 [MATH](http://www.emis.de/MATH-item?1280.68189)
 [Google Scholar](http://scholar.google.com/scholar_lookup?&title=Scikit-learn%3A%20Machine%20learning%20in%20python&journal=J.%20Mach.%20Learn.%20Res.&volume=12&pages=2825-2830&publication_year=2011&author=Pedregosa%2CF)
54. Satopaa, V., Albrecht, J., Irwin, D. & Raghavan, B. Finding a “Kneedle” in a Haystack: Detecting Knee Points in System Behavior. In *2011 31st International Conference on Distributed Computing Systems Workshops*, 166–171, <https://doi.org/10.1109/ICDCSW.2011.20> (2011). ISSN: 2332-5666.
55. Rousseeuw, P. J. Silhouettes: A graphical aid to the interpretation and validation of cluster analysis. *J. Comput. Appl. Math.* **20**, 53–65. <https://doi.org/10.1016/0377-0427(87)90125-7> (1987).

[Article](https://doi.org/10.1016%2F0377-0427%2887%2990125-7)
 [MATH](http://www.emis.de/MATH-item?0636.62059)
 [Google Scholar](http://scholar.google.com/scholar_lookup?&title=Silhouettes%3A%20A%20graphical%20aid%20to%20the%20interpretation%20and%20validation%20of%20cluster%20analysis&journal=J.%20Comput.%20Appl.%20Math.&doi=10.1016%2F0377-0427%2887%2990125-7&volume=20&pages=53-65&publication_year=1987&author=Rousseeuw%2CPJ)
56. Bank, S. G. <https://github.com/xflr6/graphviz> (2021).
57. Yackinous, W. S. Chapter 5-Overview of an Ecological System Dynamics Framework. In *Understanding Complex Ecosystem Dynamics* (ed. Yackinous, W. S.) 83–91 (Academic Press, 2015), <https://doi.org/10.1016/B978-0-12-802031-9.00005-X>.
58. Mahmoodi, K., West, B. J. & Grigolini, P. Self-organizing Complex Networks: Individual versus global rules. *Front. Physiol.* **8**, 478. <https://doi.org/10.3389/fphys.2017.00478> (2017).

[Article](https://doi.org/10.3389%2Ffphys.2017.00478)
 [PubMed](http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?cmd=Retrieve&db=PubMed&dopt=Abstract&list_uids=28736534)
 [PubMed Central](http://www.ncbi.nlm.nih.gov/pmc/articles/PMC5500654)
 [Google Scholar](http://scholar.google.com/scholar_lookup?&title=Self-organizing%20Complex%20Networks%3A%20Individual%20versus%20global%20rules&journal=Front.%20Physiol.&doi=10.3389%2Ffphys.2017.00478&volume=8&publication_year=2017&author=Mahmoodi%2CK&author=West%2CBJ&author=Grigolini%2CP)

[Download references](https://citation-needed.springer.com/v2/references/10.1038/s41598-022-11654-2?format=refman&flavour=references)

Acknowledgements
----------------

E.M. and T.L benefit from the support by the Flemish Government through the AI Research Program and by TAILOR, a project funded by the EU Horizon 2020 research and innovation program under GA No 952215. E.F.D is supported by an F.N.R.S Chargê de Recherche position, grant number 40005955. J.G. was supported by the FWO - Research Foundation Flanders. T.L. is furthermore supported by the F.N.R.S. project with grant number 31257234, the F.W.O. project with grant nr. G.0391.13N, the FuturICT 2.0 ([www.futurict2.eu](http://www.futurict2.eu)) project funded by the FLAG-ERA JCT 2016 and the Service Public de Wallonie Recherche under grant n° 2010235-ARIAC by DigitalWallonia4.ai.

Funding
-------

The funding agency had no role in the design of the study; in the collection, analyses, or interpretation of data; in the writing of the manuscript, or in the decision to publish the results.

Author information
------------------

Author notes1. These authors contributed equally: Eladio Montero-Porras and Jelena Grujić.

### Authors and Affiliations

1. Artificial Intelligence Laboratory, Vrije Universiteit Brussel, Brussels, 1050, Belgium

Eladio Montero-Porras, Jelena Grujić, Elias Fernández Domingos & Tom Lenaerts
2. Machine Learning Group, Université Libre de Bruxelles, Brussels, 1050, Belgium

Elias Fernández Domingos & Tom Lenaerts
3. Center for Human-Compatible AI, UC Berkeley, Berkeley, 94702, USA

Tom Lenaerts
4. FARI Institute, Université Libre de Bruxelles-Vrije Universiteit Brussel, 1050, Brussels, Belgium

Tom Lenaerts

Authors1. Eladio Montero-Porras[View author publications](/search?author=Eladio%20Montero-Porras)You can also search for this author in
 [PubMed](http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?cmd=search&term=Eladio%20Montero-Porras) [Google Scholar](http://scholar.google.co.uk/scholar?as_q=&num=10&btnG=Search+Scholar&as_epq=&as_oq=&as_eq=&as_occt=any&as_sauthors=%22Eladio%20Montero-Porras%22&as_publication=&as_ylo=&as_yhi=&as_allsubj=all&hl=en)
2. Jelena Grujić[View author publications](/search?author=Jelena%20Gruji%C4%87)You can also search for this author in
 [PubMed](http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?cmd=search&term=Jelena%20Gruji%C4%87) [Google Scholar](http://scholar.google.co.uk/scholar?as_q=&num=10&btnG=Search+Scholar&as_epq=&as_oq=&as_eq=&as_occt=any&as_sauthors=%22Jelena%20Gruji%C4%87%22&as_publication=&as_ylo=&as_yhi=&as_allsubj=all&hl=en)
3. Elias Fernández Domingos[View author publications](/search?author=Elias%20Fern%C3%A1ndez%20Domingos)You can also search for this author in
 [PubMed](http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?cmd=search&term=Elias%20Fern%C3%A1ndez%20Domingos) [Google Scholar](http://scholar.google.co.uk/scholar?as_q=&num=10&btnG=Search+Scholar&as_epq=&as_oq=&as_eq=&as_occt=any&as_sauthors=%22Elias%20Fern%C3%A1ndez%20Domingos%22&as_publication=&as_ylo=&as_yhi=&as_allsubj=all&hl=en)
4. Tom Lenaerts[View author publications](/search?author=Tom%20Lenaerts)You can also search for this author in
 [PubMed](http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?cmd=search&term=Tom%20Lenaerts) [Google Scholar](http://scholar.google.co.uk/scholar?as_q=&num=10&btnG=Search+Scholar&as_epq=&as_oq=&as_eq=&as_occt=any&as_sauthors=%22Tom%20Lenaerts%22&as_publication=&as_ylo=&as_yhi=&as_allsubj=all&hl=en)
### Contributions

J.G. designed, performed the experiments, and collected the data. E.M., J.G, T.L., and E.F analyzed the results. E.M. and J.G. performed the clustering and developed the models. E.M., E.F., and T.L. wrote the manuscript. All authors read and approved the manuscript.

### Corresponding author

Correspondence to
 [Eladio Montero-Porras](mailto:eladio.montero@vub.be).

Ethics declarations
-------------------

### Competing interests

The authors declare no competing interests.

Additional information
----------------------

### Publisher's note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary Information
-------------------------

### [Supplementary Information.](https://static-content.springer.com/esm/art%3A10.1038%2Fs41598-022-11654-2/MediaObjects/41598_2022_11654_MOESM1_ESM.pdf)

Rights and permissions
----------------------

**Open Access** This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit <http://creativecommons.org/licenses/by/4.0/>.

[Reprints and Permissions](https://s100.copyright.com/AppDispatchServlet?title=Inferring%20strategies%20from%20observations%20in%20long%20iterated%20Prisoner%E2%80%99s%20dilemma%20experiments&author=Eladio%20Montero-Porras%20et%20al&contentID=10.1038%2Fs41598-022-11654-2&copyright=The%20Author%28s%29&publication=2045-2322&publicationDate=2022-05-09&publisherName=SpringerNature&orderBeanReset=true&oa=CC%20BY)

This article is cited by
------------------------

* ###
[Individual and situational factors influence cooperative choices in the decision-making process](https://doi.org/10.1007/s12144-023-04348-z)

	+ Rosa Angela Fabio
	+ Valentina Romeo
	+ Chiara Calabrese*Current Psychology* (2023)
* ###
[Fast deliberation is related to unconditional behaviour in iterated Prisoners’ Dilemma experiments](https://doi.org/10.1038/s41598-022-24849-4)

	+ Eladio Montero-Porras
	+ Tom Lenaerts
	+ Jelena Grujic*Scientific Reports* (2022)

Comments
--------

By submitting a comment you agree to abide by our [Terms](/info/tandc.html) and [Community Guidelines](/info/community-guidelines.html). If you find something abusive or that does not comply with our terms or guidelines please flag it as inappropriate.

Eladio Montero-Porras
Jelena Grujić
Elias Fernández Domingos & Tom Lenaerts