Cognitive prostheses for goal achievement.

### Subjects

* [Computer science](/subjects/computer-science)
* [Human behaviour](/subjects/human-behaviour)
* [Information technology](/subjects/information-technology)
* [Psychology](/subjects/psychology)

Abstract
--------

Procrastination takes a considerable toll on people’s lives, the economy and society at large. Procrastination is often a consequence of people’s propensity to prioritize their immediate experiences over the long-term consequences of their actions. This suggests that aligning immediate rewards with long-term values could be a promising way to help people make more future-minded decisions and overcome procrastination. Here we develop an approach to decision support that leverages artificial intelligence and game elements to restructure challenging sequential decision problems in such a way that it becomes easier for people to take the right course of action. A series of four increasingly realistic experiments suggests that this approach can enable people to make better decisions faster, procrastinate less, complete their work on time and waste less time on unimportant tasks. These findings suggest that our method is a promising step towards developing cognitive prostheses that help people achieve their goals.

[Access through your institution](https://wayf.springernature.com?redirect_uri=https%3A%2F%2Fwww.nature.com%2Farticles%2Fs41562-019-0672-9)

[Buy or subscribe](#access-options)

This is a preview of subscription content, [access via your institution](https://wayf.springernature.com?redirect_uri=https%3A%2F%2Fwww.nature.com%2Farticles%2Fs41562-019-0672-9)

 if (window.dataLayer) {
 window.dataLayer.push({
 content: { article: { relevantArticlesCount: 0 }}
 })
 }

Access options
--------------

[Access through your institution](https://wayf.springernature.com?redirect_uri=https%3A%2F%2Fwww.nature.com%2Farticles%2Fs41562-019-0672-9)

[Access through your institution](https://wayf.springernature.com?redirect_uri=https%3A%2F%2Fwww.nature.com%2Farticles%2Fs41562-019-0672-9)

[Change institution](https://wayf.springernature.com?redirect_uri=https%3A%2F%2Fwww.nature.com%2Farticles%2Fs41562-019-0672-9)

[Buy or subscribe](#access-options)

/\* style specs start \*/
style{display:none!important}.LiveAreaSection-193358632 \*{align-content:stretch;align-items:stretch;align-self:auto;animation-delay:0s;animation-direction:normal;animation-duration:0s;animation-fill-mode:none;animation-iteration-count:1;animation-name:none;animation-play-state:running;animation-timing-function:ease;azimuth:center;backface-visibility:visible;background-attachment:scroll;background-blend-mode:normal;background-clip:borderBox;background-color:transparent;background-image:none;background-origin:paddingBox;background-position:0 0;background-repeat:repeat;background-size:auto auto;block-size:auto;border-block-end-color:currentcolor;border-block-end-style:none;border-block-end-width:medium;border-block-start-color:currentcolor;border-block-start-style:none;border-block-start-width:medium;border-bottom-color:currentcolor;border-bottom-left-radius:0;border-bottom-right-radius:0;border-bottom-style:none;border-bottom-width:medium;border-collapse:separate;border-image-outset:0s;border-image-repeat:stretch;border-image-slice:100%;border-image-source:none;border-image-width:1;border-inline-end-color:currentcolor;border-inline-end-style:none;border-inline-end-width:medium;border-inline-start-color:currentcolor;border-inline-start-style:none;border-inline-start-width:medium;border-left-color:currentcolor;border-left-style:none;border-left-width:medium;border-right-color:currentcolor;border-right-style:none;border-right-width:medium;border-spacing:0;border-top-color:currentcolor;border-top-left-radius:0;border-top-right-radius:0;border-top-style:none;border-top-width:medium;bottom:auto;box-decoration-break:slice;box-shadow:none;box-sizing:border-box;break-after:auto;break-before:auto;break-inside:auto;caption-side:top;caret-color:auto;clear:none;clip:auto;clip-path:none;color:initial;column-count:auto;column-fill:balance;column-gap:normal;column-rule-color:currentcolor;column-rule-style:none;column-rule-width:medium;column-span:none;column-width:auto;content:normal;counter-increment:none;counter-reset:none;cursor:auto;display:inline;empty-cells:show;filter:none;flex-basis:auto;flex-direction:row;flex-grow:0;flex-shrink:1;flex-wrap:nowrap;float:none;font-family:initial;font-feature-settings:normal;font-kerning:auto;font-language-override:normal;font-size:medium;font-size-adjust:none;font-stretch:normal;font-style:normal;font-synthesis:weight style;font-variant:normal;font-variant-alternates:normal;font-variant-caps:normal;font-variant-east-asian:normal;font-variant-ligatures:normal;font-variant-numeric:normal;font-variant-position:normal;font-weight:400;grid-auto-columns:auto;grid-auto-flow:row;grid-auto-rows:auto;grid-column-end:auto;grid-column-gap:0;grid-column-start:auto;grid-row-end:auto;grid-row-gap:0;grid-row-start:auto;grid-template-areas:none;grid-template-columns:none;grid-template-rows:none;height:auto;hyphens:manual;image-orientation:0deg;image-rendering:auto;image-resolution:1dppx;ime-mode:auto;inline-size:auto;isolation:auto;justify-content:flexStart;left:auto;letter-spacing:normal;line-break:auto;line-height:normal;list-style-image:none;list-style-position:outside;list-style-type:disc;margin-block-end:0;margin-block-start:0;margin-bottom:0;margin-inline-end:0;margin-inline-start:0;margin-left:0;margin-right:0;margin-top:0;mask-clip:borderBox;mask-composite:add;mask-image:none;mask-mode:matchSource;mask-origin:borderBox;mask-position:0 0;mask-repeat:repeat;mask-size:auto;mask-type:luminance;max-height:none;max-width:none;min-block-size:0;min-height:0;min-inline-size:0;min-width:0;mix-blend-mode:normal;object-fit:fill;object-position:50% 50%;offset-block-end:auto;offset-block-start:auto;offset-inline-end:auto;offset-inline-start:auto;opacity:1;order:0;orphans:2;outline-color:initial;outline-offset:0;outline-style:none;outline-width:medium;overflow:visible;overflow-wrap:normal;overflow-x:visible;overflow-y:visible;padding-block-end:0;padding-block-start:0;padding-bottom:0;padding-inline-end:0;padding-inline-start:0;padding-left:0;padding-right:0;padding-top:0;page-break-after:auto;page-break-before:auto;page-break-inside:auto;perspective:none;perspective-origin:50% 50%;pointer-events:auto;position:static;quotes:initial;resize:none;right:auto;ruby-align:spaceAround;ruby-merge:separate;ruby-position:over;scroll-behavior:auto;scroll-snap-coordinate:none;scroll-snap-destination:0 0;scroll-snap-points-x:none;scroll-snap-points-y:none;scroll-snap-type:none;shape-image-threshold:0;shape-margin:0;shape-outside:none;tab-size:8;table-layout:auto;text-align:initial;text-align-last:auto;text-combine-upright:none;text-decoration-color:currentcolor;text-decoration-line:none;text-decoration-style:solid;text-emphasis-color:currentcolor;text-emphasis-position:over right;text-emphasis-style:none;text-indent:0;text-justify:auto;text-orientation:mixed;text-overflow:clip;text-rendering:auto;text-shadow:none;text-transform:none;text-underline-position:auto;top:auto;touch-action:auto;transform:none;transform-box:borderBox;transform-origin:50% 50%0;transform-style:flat;transition-delay:0s;transition-duration:0s;transition-property:all;transition-timing-function:ease;vertical-align:baseline;visibility:visible;white-space:normal;widows:2;width:auto;will-change:auto;word-break:normal;word-spacing:normal;word-wrap:normal;writing-mode:horizontalTb;z-index:auto;-webkit-appearance:none;-moz-appearance:none;-ms-appearance:none;appearance:none;margin:0}.LiveAreaSection-193358632{width:100%}.LiveAreaSection-193358632 .login-option-buybox{display:block;width:100%;font-size:17px;line-height:30px;color:#222;padding-top:30px;font-family:Harding,Palatino,serif}.LiveAreaSection-193358632 .additional-access-options{display:block;font-weight:700;font-size:17px;line-height:30px;color:#222;font-family:Harding,Palatino,serif}.LiveAreaSection-193358632 .additional-login>li:not(:first-child)::before{transform:translateY(-50%);content:"";height:1rem;position:absolute;top:50%;left:0;border-left:2px solid #999}.LiveAreaSection-193358632 .additional-login>li:not(:first-child){padding-left:10px}.LiveAreaSection-193358632 .additional-login>li{display:inline-block;position:relative;vertical-align:middle;padding-right:10px}.BuyBoxSection-683559780{display:flex;flex-wrap:wrap;flex:1;flex-direction:row-reverse;margin:-30px -15px 0}.BuyBoxSection-683559780 .box-inner{width:100%;height:100%}.BuyBoxSection-683559780 .readcube-buybox{background-color:#f3f3f3;flex-shrink:1;flex-grow:1;flex-basis:255px;background-clip:content-box;padding:0 15px;margin-top:30px}.BuyBoxSection-683559780 .subscribe-buybox{background-color:#f3f3f3;flex-shrink:1;flex-grow:4;flex-basis:300px;background-clip:content-box;padding:0 15px;margin-top:30px}.BuyBoxSection-683559780 .subscribe-buybox-nature-plus{background-color:#f3f3f3;flex-shrink:1;flex-grow:4;flex-basis:100%;background-clip:content-box;padding:0 15px;margin-top:30px}.BuyBoxSection-683559780 .title-readcube,.BuyBoxSection-683559780 .title-buybox{display:block;margin:0;margin-right:10%;margin-left:10%;font-size:24px;line-height:32px;color:#222;padding-top:30px;text-align:center;font-family:Harding,Palatino,serif}.BuyBoxSection-683559780 .title-asia-buybox{display:block;margin:0;margin-right:5%;margin-left:5%;font-size:24px;line-height:32px;color:#222;padding-top:30px;text-align:center;font-family:Harding,Palatino,serif}.BuyBoxSection-683559780 .asia-link{color:#069;cursor:pointer;text-decoration:none;font-size:1.05em;font-family:-apple-system,BlinkMacSystemFont,"Segoe UI",Roboto,Oxygen-Sans,Ubuntu,Cantarell,"Helvetica Neue",sans-serif;line-height:1.05em6}.BuyBoxSection-683559780 .access-readcube{display:block;margin:0;margin-right:10%;margin-left:10%;font-size:14px;color:#222;padding-top:10px;text-align:center;font-family:-apple-system,BlinkMacSystemFont,"Segoe UI",Roboto,Oxygen-Sans,Ubuntu,Cantarell,"Helvetica Neue",sans-serif;line-height:20px}.BuyBoxSection-683559780 .access-asia-buybox{display:block;margin:0;margin-right:5%;margin-left:5%;font-size:14px;color:#222;padding-top:10px;text-align:center;font-family:-apple-system,BlinkMacSystemFont,"Segoe UI",Roboto,Oxygen-Sans,Ubuntu,Cantarell,"Helvetica Neue",sans-serif;line-height:20px}.BuyBoxSection-683559780 .access-buybox{display:block;margin:0;margin-right:10%;margin-left:10%;font-size:14px;color:#222;opacity:.8px;padding-top:10px;text-align:center;font-family:-apple-system,BlinkMacSystemFont,"Segoe UI",Roboto,Oxygen-Sans,Ubuntu,Cantarell,"Helvetica Neue",sans-serif;line-height:20px}.BuyBoxSection-683559780 .price-buybox{display:block;font-size:30px;color:#222;font-family:-apple-system,BlinkMacSystemFont,"Segoe UI",Roboto,Oxygen-Sans,Ubuntu,Cantarell,"Helvetica Neue",sans-serif;padding-top:30px;text-align:center}.BuyBoxSection-683559780 .price-buybox-to{display:block;font-size:30px;color:#222;font-family:-apple-system,BlinkMacSystemFont,"Segoe UI",Roboto,Oxygen-Sans,Ubuntu,Cantarell,"Helvetica Neue",sans-serif;text-align:center}.BuyBoxSection-683559780 .price-info-text{font-size:16px;padding-right:10px;color:#222;font-family:-apple-system,BlinkMacSystemFont,"Segoe UI",Roboto,Oxygen-Sans,Ubuntu,Cantarell,"Helvetica Neue",sans-serif}.BuyBoxSection-683559780 .price-value{font-size:30px;font-family:-apple-system,BlinkMacSystemFont,"Segoe UI",Roboto,Oxygen-Sans,Ubuntu,Cantarell,"Helvetica Neue",sans-serif}.BuyBoxSection-683559780 .price-per-period{font-family:-apple-system,BlinkMacSystemFont,"Segoe UI",Roboto,Oxygen-Sans,Ubuntu,Cantarell,"Helvetica Neue",sans-serif}.BuyBoxSection-683559780 .price-from{font-size:14px;padding-right:10px;color:#222;font-family:-apple-system,BlinkMacSystemFont,"Segoe UI",Roboto,Oxygen-Sans,Ubuntu,Cantarell,"Helvetica Neue",sans-serif;line-height:20px}.BuyBoxSection-683559780 .issue-buybox{display:block;font-size:13px;text-align:center;color:#222;font-family:-apple-system,BlinkMacSystemFont,"Segoe UI",Roboto,Oxygen-Sans,Ubuntu,Cantarell,"Helvetica Neue",sans-serif;line-height:19px}.BuyBoxSection-683559780 .no-price-buybox{display:block;font-size:13px;line-height:18px;text-align:center;padding-right:10%;padding-left:10%;padding-bottom:20px;padding-top:30px;color:#222;font-family:-apple-system,BlinkMacSystemFont,"Segoe UI",Roboto,Oxygen-Sans,Ubuntu,Cantarell,"Helvetica Neue",sans-serif}.BuyBoxSection-683559780 .vat-buybox{display:block;margin-top:5px;margin-right:20%;margin-left:20%;font-size:11px;color:#222;padding-top:10px;padding-bottom:15px;text-align:center;font-family:-apple-system,BlinkMacSystemFont,"Segoe UI",Roboto,Oxygen-Sans,Ubuntu,Cantarell,"Helvetica Neue",sans-serif;line-height:17px}.BuyBoxSection-683559780 .tax-buybox{display:block;width:100%;color:#222;padding:20px 16px;text-align:center;font-family:-apple-system,BlinkMacSystemFont,"Segoe UI",Roboto,Oxygen-Sans,Ubuntu,Cantarell,"Helvetica Neue",sans-serif;line-height:NaNpx}.BuyBoxSection-683559780 .button-container{display:flex;padding-right:20px;padding-left:20px;justify-content:center}.BuyBoxSection-683559780 .button-container>\*{flex:1px}.BuyBoxSection-683559780 .button-container>a:hover,.Button-505204839:hover,.Button-1078489254:hover,.Button-2496381730:hover{text-decoration:none}.BuyBoxSection-683559780 .readcube-button{background:#fff;margin-top:30px}.BuyBoxSection-683559780 .button-asia{background:#069;border:1px solid #069;border-radius:0;cursor:pointer;display:block;padding:9px;outline:0;text-align:center;text-decoration:none;min-width:80px;margin-top:75px}.BuyBoxSection-683559780 .button-label-asia,.ButtonLabel-3869432492,.ButtonLabel-3296148077,.ButtonLabel-1651148777{display:block;color:#fff;font-size:17px;line-height:20px;font-family:-apple-system,BlinkMacSystemFont,"Segoe UI",Roboto,Oxygen-Sans,Ubuntu,Cantarell,"Helvetica Neue",sans-serif;text-align:center;text-decoration:none;cursor:pointer}.Button-505204839,.Button-1078489254,.Button-2496381730{background:#069;border:1px solid #069;border-radius:0;cursor:pointer;display:block;padding:9px;outline:0;text-align:center;text-decoration:none;min-width:80px;max-width:320px;margin-top:10px}.Button-505204839 .readcube-label,.Button-1078489254 .readcube-label,.Button-2496381730 .readcube-label{color:#069}
/\* style specs end \*/Access Nature and 54 other Nature Portfolio journals

Get Nature+, our best-value online-access subscription

$29.99 / 30 days

cancel any time

[Learn more](https://shop-amers.nature.com/products/plus)Subscribe to this journal

Receive 12 digital issues and online access to articles

$119.00 per year

only $9.92 per issue

[Learn more](/nathumbehav/subscribe)Rent or buy this article

Prices vary by article type

from$1.95

to$39.95

[Learn more](//www.nature.com/articles/s41562-019-0672-9.epdf?no_publisher_access=1&r3_referer=nature)Prices may be subject to local taxes which are calculated during checkout

### Additional access options:

* [Log in](https://idp.nature.com/authorize/natureuser?client_id=grover&redirect_uri=https%3A%2F%2Fwww.nature.com%2Farticles%2Fs41562-019-0672-9)
* [Learn about institutional subscriptions](https://www.springernature.com/gp/librarians/licensing/license-options)
* [Read our FAQs](https://support.nature.com/en/support/home)
* [Contact customer support](https://www.springernature.com/gp/contact)

**Fig. 1: Experiment 1.**![](//media.springernature.com/m312/springer-static/image/art%3A10.1038%2Fs41562-019-0672-9/MediaObjects/41562_2019_672_Fig1_HTML.png)**Fig. 2: Conditions of Experiment 1.**![](//media.springernature.com/m312/springer-static/image/art%3A10.1038%2Fs41562-019-0672-9/MediaObjects/41562_2019_672_Fig2_HTML.png)**Fig. 3: Experiment 2.**![](//media.springernature.com/m312/springer-static/image/art%3A10.1038%2Fs41562-019-0672-9/MediaObjects/41562_2019_672_Fig3_HTML.png)**Fig. 4: Conditions of Experiment 2.**![](//media.springernature.com/m312/springer-static/image/art%3A10.1038%2Fs41562-019-0672-9/MediaObjects/41562_2019_672_Fig4_HTML.png)**Fig. 5: Screenshot from Experiment 3.**![](//media.springernature.com/m312/springer-static/image/art%3A10.1038%2Fs41562-019-0672-9/MediaObjects/41562_2019_672_Fig5_HTML.png)**Fig. 6: Proportions of participants in Experiment 3 who completed all assignments by the deadline.**![](//media.springernature.com/m312/springer-static/image/art%3A10.1038%2Fs41562-019-0672-9/MediaObjects/41562_2019_672_Fig6_HTML.png)

Data availability
-----------------

The data that support the findings of this study are available at <https://osf.io/h7vqy/>.

Code availability
-----------------

The code used to conduct Experiment 1, Experiment 2 and its follow-up experiments is available at <https://osf.io/h7vqy/>. The code used to conduct Experiment 3 and Experiment 4 is available on GitHub at <https://github.com/BrowenChen/Cognitive-Tools-for-Self-Mastery>. The code used to analyse the data is available at <https://osf.io/h7vqy/> and in the [Supplementary Software](/articles/s41562-019-0672-9#MOESM3) file.

References
----------

1. Ariely, D. *Predictably iIrrational*. (Harper Collins, 2008).
2. Tversky, A. & Kahneman, D. Judgment under uncertainty: heuristics and biases. *Science* **185**, 1124–1131 (1974).

[Article](https://doi.org/10.1126%2Fscience.185.4157.1124)
 [CAS](/articles/cas-redirect/1:STN:280:DC%2BC3cvls12ruw%3D%3D)
 [Google Scholar](http://scholar.google.com/scholar_lookup?&title=Judgment%20under%20uncertainty%3A%20heuristics%20and%20biases&journal=Science&doi=10.1126%2Fscience.185.4157.1124&volume=185&pages=1124-1131&publication_year=1974&author=Tversky%2CA&author=Kahneman%2CD)
3. Gilovich, T., Griffin, D. & Kahneman, D. *Heuristics and Biases: The Psychology of Intuitive Judgment* (Cambridge Univ. Press, 2002).
4. Todd, P. M. & Gigerenzer, G. Environments that make us smart: ecological rationality. *Curr. Dir. Psychol. Sci.* **16**, 167–171 (2007).

[Article](https://doi.org/10.1111%2Fj.1467-8721.2007.00497.x)
 [Google Scholar](http://scholar.google.com/scholar_lookup?&title=Environments%20that%20make%20us%20smart%3A%20ecological%20rationality&journal=Curr.%20Dir.%20Psychol.%20Sci.&doi=10.1111%2Fj.1467-8721.2007.00497.x&volume=16&pages=167-171&publication_year=2007&author=Todd%2CPM&author=Gigerenzer%2CG)
5. Oaksford, M. & Chater, N. A rational analysis of the selection task as optimal data selection. *Psychol. Rev.* **101**, 608–631 (1994).

[Article](https://doi.org/10.1037%2F0033-295X.101.4.608)
 [Google Scholar](http://scholar.google.com/scholar_lookup?&title=A%20rational%20analysis%20of%20the%20selection%20task%20as%20optimal%20data%20selection&journal=Psychol.%20Rev.&doi=10.1037%2F0033-295X.101.4.608&volume=101&pages=608-631&publication_year=1994&author=Oaksford%2CM&author=Chater%2CN)
6. Chater, N. & Oaksford, M. Ten years of the rational analysis of cognition. *Trends Cogn. Sci.* **3**, 57–65 (1999).

[Article](https://doi.org/10.1016%2FS1364-6613%2898%2901273-X)
 [CAS](/articles/cas-redirect/1:STN:280:DC%2BC2sbgsVejsw%3D%3D)
 [Google Scholar](http://scholar.google.com/scholar_lookup?&title=Ten%20years%20of%20the%20rational%20analysis%20of%20cognition&journal=Trends%20Cogn.%20Sci.&doi=10.1016%2FS1364-6613%2898%2901273-X&volume=3&pages=57-65&publication_year=1999&author=Chater%2CN&author=Oaksford%2CM)
7. Todd, P. M. & Gigerenzer, G. *Ecological Rationality: Intelligence in the World*. (Oxford Univ. Press, New York, 2012).
8. Griffiths, T. L., Lieder, F. & Goodman, N. D. Rational use of cognitive resources: levels of analysis between the computational and the algorithmic. *Top. Cogn. Sci.* **7**, 217–229 (2015).

[Article](https://doi.org/10.1111%2Ftops.12142)
 [Google Scholar](http://scholar.google.com/scholar_lookup?&title=Rational%20use%20of%20cognitive%20resources%3A%20levels%20of%20analysis%20between%20the%20computational%20and%20the%20algorithmic&journal=Top.%20Cogn.%20Sci.&doi=10.1111%2Ftops.12142&volume=7&pages=217-229&publication_year=2015&author=Griffiths%2CTL&author=Lieder%2CF&author=Goodman%2CND)
9. Gigerenzer, G. & Hoffrage, U. How to improve Bayesian reasoning without instruction: frequency formats. *Psychol. Rev.* **102**, 684–704 (1995).

[Article](https://doi.org/10.1037%2F0033-295X.102.4.684)
 [Google Scholar](http://scholar.google.com/scholar_lookup?&title=How%20to%20improve%20Bayesian%20reasoning%20without%20instruction%3A%20frequency%20formats&journal=Psychol.%20Rev.&doi=10.1037%2F0033-295X.102.4.684&volume=102&pages=684-704&publication_year=1995&author=Gigerenzer%2CG&author=Hoffrage%2CU)
10. Hoffrage, U., Lindsey, S., Hertwig, R. & Gigerenzer, G. Communicating statistical information. *Science* **290**, 2261–2262 (2000).

[Article](https://doi.org/10.1126%2Fscience.290.5500.2261)
 [CAS](/articles/cas-redirect/1:STN:280:DC%2BD3M%2Fnt1OqtQ%3D%3D)
 [Google Scholar](http://scholar.google.com/scholar_lookup?&title=Communicating%20statistical%20information&journal=Science&doi=10.1126%2Fscience.290.5500.2261&volume=290&pages=2261-2262&publication_year=2000&author=Hoffrage%2CU&author=Lindsey%2CS&author=Hertwig%2CR&author=Gigerenzer%2CG)
11. Gigerenzer, G. & Edwards, A. Simple tools for understanding risks: from innumeracy to insight. *BMJ* **327**, 741 (2003).

[Article](https://doi.org/10.1136%2Fbmj.327.7417.741)
 [Google Scholar](http://scholar.google.com/scholar_lookup?&title=Simple%20tools%20for%20understanding%20risks%3A%20from%20innumeracy%20to%20insight&journal=BMJ&doi=10.1136%2Fbmj.327.7417.741&volume=327&publication_year=2003&author=Gigerenzer%2CG&author=Edwards%2CA)
12. Thaler, R. H. & Sunstein, C. R. *Nudge: Improving Decisions about Health, Wealth, and Happiness*. (Yale Univ. Press, New Haven, 2008).
13. Johnson, E. J. et al. Beyond nudges: tools of a choice architecture. *Mark. Lett.* **23**, 487–504 (2012).

[Article](https://doi.org/10.1007%2Fs11002-012-9186-1)
 [Google Scholar](http://scholar.google.com/scholar_lookup?&title=Beyond%20nudges%3A%20tools%20of%20a%20choice%20architecture&journal=Mark.%20Lett.&doi=10.1007%2Fs11002-012-9186-1&volume=23&pages=487-504&publication_year=2012&author=Johnson%2CEJ)
14. Aronson, J. E., Liang, T.-P. & Turban, E. *Decision Support Systems and Intelligent Systems* (Pearson Prentice-Hall, 2005).
15. Power, D. J., Sharda, R. & Burstein, F. in *Wiley Encyclopedia of Management* Vol. 7 (eds Straub, D. & Welke, R.) 1–11 (Wiley, 2015).
16. Aviv, Y. & Pazgal, A. A partially observed Markov decision procBhatnagar, S., Fernández-Gaucherand, E., Fu, M. C., He, Y. & Marcus, S. I. A Markov decision process model for capacity expansion and allocation. InIEEE Conference on Decision and Control Vol. 2 (ed Altman, E.) 1380–1385 (IEEE, 1999).ess for dynamic pricing. *Manag. Sci.* **51**, 1400–1416 (2005).

[Article](https://doi.org/10.1287%2Fmnsc.1050.0393)
 [Google Scholar](http://scholar.google.com/scholar_lookup?&title=A%20partially%20observed%20Markov%20decision%20procBhatnagar%2C%20S.%2C%20Fern%C3%A1ndez-Gaucherand%2C%20E.%2C%20Fu%2C%20M.%20C.%2C%20He%2C%20Y.%20%26%20Marcus%2C%20S.%20I.%20A%20Markov%20decision%20process%20model%20for%20capacity%20expansion%20and%20allocation.%20InIEEE%20Conference%20on%20Decision%20and%20Control%20Vol.%202%20%28ed%20Altman%2C%20E.%29%201380%E2%80%931385%20%28IEEE%2C%201999%29.ess%20for%20dynamic%20pricing&journal=Manag.%20Sci.&doi=10.1287%2Fmnsc.1050.0393&volume=51&pages=1400-1416&publication_year=2005&author=Aviv%2CY&author=Pazgal%2CA)
17. Song, H., Liu, C.-C., Lawarrée, J. & Dahlgren, R. W. Optimal electricity supply bidding by Markov decision process. *IEEE Trans. Power Syst.* **15**, 618–624 (2000).

[Article](https://doi.org/10.1109%2F59.867150)
 [Google Scholar](http://scholar.google.com/scholar_lookup?&title=Optimal%20electricity%20supply%20bidding%20by%20Markov%20decision%20process&journal=IEEE%20Trans.%20Power%20Syst.&doi=10.1109%2F59.867150&volume=15&pages=618-624&publication_year=2000&author=Song%2CH&author=Liu%2CC-C&author=Lawarr%C3%A9e%2CJ&author=Dahlgren%2CRW)
18. Gadomski, A. M., Bologna, S., Costanzo, G. D., Perini, A. & Schaerf, M. Towards intelligent decision support systems for emergency managers: the IDA approach. *Int. J. Risk Assess. Manag.* **2**, 224–242 (2001).

[Article](https://doi.org/10.1504%2FIJRAM.2001.001507)
 [Google Scholar](http://scholar.google.com/scholar_lookup?&title=Towards%20intelligent%20decision%20support%20systems%20for%20emergency%20managers%3A%20the%20IDA%20approach&journal=Int.%20J.%20Risk%20Assess.%20Manag.&doi=10.1504%2FIJRAM.2001.001507&volume=2&pages=224-242&publication_year=2001&author=Gadomski%2CAM&author=Bologna%2CS&author=Costanzo%2CGD&author=Perini%2CA&author=Schaerf%2CM)
19. Bhatnagar, S., Fernández-Gaucherand, E., Fu, M. C., He, Y. & Marcus, S. I. A Markov decision process model for capacity expansion and allocation. In *IEEE Conference on Decision and Control* Vol. 2 (ed Altman, E.) 1380–1385 (IEEE, 1999).
20. Nunes, L. G. N., de Carvalho, S. V. & de Cássia Meneses Rodrigues, R. Markov decision process applied to the control of hospital elective admissions. *Artif. Intell. Med.* **47**, 159–171 (2009).

[Article](https://doi.org/10.1016%2Fj.artmed.2009.07.003)
 [Google Scholar](http://scholar.google.com/scholar_lookup?&title=Markov%20decision%20process%20applied%20to%20the%20control%20of%20hospital%20elective%20admissions&journal=Artif.%20Intell.%20Med.&doi=10.1016%2Fj.artmed.2009.07.003&volume=47&pages=159-171&publication_year=2009&author=Nunes%2CLGN&author=Carvalho%2CSV&author=C%C3%A1ssia%20Meneses%20Rodrigues%2CR)
21. Ainslie, G. & Haslam, N. in *Choice over Time* (eds Loewenstein, G. & Elster, J.) 57–92 (Russell Sage Foundation, 1992).
22. Myerson, J. & Green, L. Discounting of delayed rewards: models of individual choice. *J. Exp. Anal. Behav.* **64**, 263–276 (1995).

[Article](https://doi.org/10.1901%2Fjeab.1995.64-263)
 [CAS](/articles/cas-redirect/1:STN:280:DC%2BC3cnitlyrsA%3D%3D)
 [Google Scholar](http://scholar.google.com/scholar_lookup?&title=Discounting%20of%20delayed%20rewards%3A%20models%20of%20individual%20choice&journal=J.%20Exp.%20Anal.%20Behav.&doi=10.1901%2Fjeab.1995.64-263&volume=64&pages=263-276&publication_year=1995&author=Myerson%2CJ&author=Green%2CL)
23. Huys, Q. J. M. et al. Bonsai trees in your head: how the Pavlovian system sculpts goal-directed choices by pruning decision trees. *PLoS Comput. Biol.* **8**, e1002410 (2012).

[Article](https://doi.org/10.1371%2Fjournal.pcbi.1002410)
 [CAS](/articles/cas-redirect/1:CAS:528:DC%2BC38XktF2htLw%3D)
 [Google Scholar](http://scholar.google.com/scholar_lookup?&title=Bonsai%20trees%20in%20your%20head%3A%20how%20the%20Pavlovian%20system%20sculpts%20goal-directed%20choices%20by%20pruning%20decision%20trees&journal=PLoS%20Comput.%20Biol.&doi=10.1371%2Fjournal.pcbi.1002410&volume=8&publication_year=2012&author=Huys%2CQJM)
24. Steel, P. The nature of procrastination: a meta-analytic and theoretical review of quintessential self-regulatory failure. *Psychol. Bull.* **133**, 65–94 (2007).

[Article](https://doi.org/10.1037%2F0033-2909.133.1.65)
 [Google Scholar](http://scholar.google.com/scholar_lookup?&title=The%20nature%20of%20procrastination%3A%20a%20meta-analytic%20and%20theoretical%20review%20of%20quintessential%20self-regulatory%20failure&journal=Psychol.%20Bull.&doi=10.1037%2F0033-2909.133.1.65&volume=133&pages=65-94&publication_year=2007&author=Steel%2CP)
25. Mischel, W., Shoda, Y. & Rodriguez, M. L. Delay of gratification in children. *Science* **244**, 933–938 (1989).

[Article](https://doi.org/10.1126%2Fscience.2658056)
 [CAS](/articles/cas-redirect/1:STN:280:DyaL1M3ms1yqug%3D%3D)
 [Google Scholar](http://scholar.google.com/scholar_lookup?&title=Delay%20of%20gratification%20in%20children&journal=Science&doi=10.1126%2Fscience.2658056&volume=244&pages=933-938&publication_year=1989&author=Mischel%2CW&author=Shoda%2CY&author=Rodriguez%2CML)
26. Deterding, S., Dixon, D., Khaled, R. & Nacke, L. From game design elements to gamefulness: defining gamification. In *Proc. of the 15th International Academic MindTrek Conference: Envisioning Future Media Environments* (eds Lugmayr, A., Franssila, H., Safran, C. & Hammouda, I.) 9–15 (ACM, 2011).
27. Hamari, J., Koivisto, J. & Sarsa, H. Does gamification work? A literature review of empirical studies on gamification. In *Proc. of the 47th Hawaii International Conference on System Sciences* (ed Sprague, R. H., Jr.) 3025–3034 (IEEE, 2014).
28. Callan, R. C., Bauer, K. N. & Landers, R. N. in *Gamification in Education and Business* (eds Reiners, T. & Wood, L. C.) 553–568 (Springer, 2015).
29. Devers, C. J. & Gurung, R. A. in *Gamification in Education and Business* (eds Reiners, T. & Wood, L. C.) 417–430 (Springer, 2015).
30. Sutton, R. S. & Barto, A. G. *Reinforcement Learning: An Introduction*. (MIT Press, 1998).
31. Callaway, F. et al. A resource-rational analysis of human planning. In *Proc. of the 40th Annual Conference of the Cognitive Science Society* (eds Kalish, C. et al.) 178–183 (Cognitive Science Society, 2018).
32. Baumeister, R. F., Heatherton, T. F. & Tice, D. M. *Losing Control: How and Why People Fail at Self-Regulation* (Academic, 1994).
33. Ng, A. Y., Harada, D. & Russell, S. Policy invariance under reward transformations: theory and application to reward shaping. In *Proc. of the 16th Annual International Conference on Machine Learning* (eds Bratko, I. & Dzeroski, S.) 278–287 (Morgan Kaufmann, 1999).
34. Rieley, J. B. Are your employees gaming the system? *Natl Product. Rev.* **19**, 1–6 (2000).

[Article](https://doi.org/10.1002%2Fnpr.4040190302)
 [Google Scholar](http://scholar.google.com/scholar_lookup?&title=Are%20your%20employees%20gaming%20the%20system%3F&journal=Natl%20Product.%20Rev.&doi=10.1002%2Fnpr.4040190302&volume=19&pages=1-6&publication_year=2000&author=Rieley%2CJB)
35. Bevan, G. & Hood, C. What’s measured is what matters: targets and gaming in the english public health care system. *Public Adm.* **84**, 517–538 (2006).

[Article](https://doi.org/10.1111%2Fj.1467-9299.2006.00600.x)
 [Google Scholar](http://scholar.google.com/scholar_lookup?&title=What%E2%80%99s%20measured%20is%20what%20matters%3A%20targets%20and%20gaming%20in%20the%20english%20public%20health%20care%20system&journal=Public%20Adm.&doi=10.1111%2Fj.1467-9299.2006.00600.x&volume=84&pages=517-538&publication_year=2006&author=Bevan%2CG&author=Hood%2CC)
36. Thom, J., Millen, D. & DiMicco, J. Removing gamification from an enterprise SNS. In *Proc. of the ACM 2012 Conference on Computer Supported Cooperative Work* (eds Poltrock, S., Simone, C., Grudin, J., Mark, G. & Riedl, J.) 1067–1070 (ACM, 2012).
37. Lieder, F., Callaway, F., Krueger, P., Griffiths, T. & Gul, S. Discovering and teaching optimal planning strategies. In *KOGWIS2018: Computational Approaches to Cognitive Science* (eds Rothkopf, C. et al.) 38 (GK, 2018).
38. Simon, H. A. Rational choice and the structure of the environment. *Psychol. Rev.* **63**, 129 (1956).

[Article](https://doi.org/10.1037%2Fh0042769)
 [CAS](/articles/cas-redirect/1:STN:280:DyaG28%2FmtlSktw%3D%3D)
 [Google Scholar](http://scholar.google.com/scholar_lookup?&title=Rational%20choice%20and%20the%20structure%20of%20the%20environment&journal=Psychol.%20Rev.&doi=10.1037%2Fh0042769&volume=63&publication_year=1956&author=Simon%2CHA)
39. Gagné, M. & Deci, E. L. Self-determination theory and work motivation. *J. Organ. Behav.* **26**, 331–362 (2005).

[Article](https://doi.org/10.1002%2Fjob.322)
 [Google Scholar](http://scholar.google.com/scholar_lookup?&title=Self-determination%20theory%20and%20work%20motivation&journal=J.%20Organ.%20Behav.&doi=10.1002%2Fjob.322&volume=26&pages=331-362&publication_year=2005&author=Gagn%C3%A9%2CM&author=Deci%2CEL)
40. Steel, P. & König, C. J. Integrating theories of motivation. *Acad. Manag. Rev.* **31**, 889–913 (2006).

[Article](https://doi.org/10.5465%2Famr.2006.22527462)
 [Google Scholar](http://scholar.google.com/scholar_lookup?&title=Integrating%20theories%20of%20motivation&journal=Acad.%20Manag.%20Rev.&doi=10.5465%2Famr.2006.22527462&volume=31&pages=889-913&publication_year=2006&author=Steel%2CP&author=K%C3%B6nig%2CCJ)
41. Puterman, M. L. *Markov Decision Processes: Discrete Stochastic Dynamic Programming* (John Wiley & Sons, 2014).
42. Krueger, P. M., Lieder, F. & Griffiths, T. L. Enhancing metacognitive reinforcement learning using reward structures and feedback. In *Proc. of the 39th Annual Conference of the Cognitive Science Society* (eds Gunzelmann, G. et al.) 2469–2474 (Cognitive Science Society, 2017).
43. Hertwig, R. & Grüne-Yanoff, T. Nudging and boosting: steering or empowering good decisions. *Perspect. Psychol. Sci.* **12**, 973–986 (2017).

[Article](https://doi.org/10.1177%2F1745691617702496)
 [Google Scholar](http://scholar.google.com/scholar_lookup?&title=Nudging%20and%20boosting%3A%20steering%20or%20empowering%20good%20decisions&journal=Perspect.%20Psychol.%20Sci.&doi=10.1177%2F1745691617702496&volume=12&pages=973-986&publication_year=2017&author=Hertwig%2CR&author=Gr%C3%BCne-Yanoff%2CT)
44. Hertwig, R. When to consider boosting: some rules for policy-makers. *Behav. Public Policy* **1**, 143–161 (2017).

[Article](https://doi.org/10.1017%2Fbpp.2016.14)
 [Google Scholar](http://scholar.google.com/scholar_lookup?&title=When%20to%20consider%20boosting%3A%20some%20rules%20for%20policy-makers&journal=Behav.%20Public%20Policy&doi=10.1017%2Fbpp.2016.14&volume=1&pages=143-161&publication_year=2017&author=Hertwig%2CR)
45. Keramati, M., Smittenaar, P., Dolan, R. J. & Dayan, P. Adaptive integration of habits into depth-limited planning defines a habitual-goal–directed spectrum. *Proc. Natl Acad. Sci. USA* **113**, 12868–12873 (2016).

[Article](https://doi.org/10.1073%2Fpnas.1609094113)
 [CAS](/articles/cas-redirect/1:CAS:528:DC%2BC28XhslalsL3J)
 [Google Scholar](http://scholar.google.com/scholar_lookup?&title=Adaptive%20integration%20of%20habits%20into%20depth-limited%20planning%20defines%20a%20habitual-goal%E2%80%93directed%20spectrum&journal=Proc.%20Natl%20Acad.%20Sci.%20USA&doi=10.1073%2Fpnas.1609094113&volume=113&pages=12868-12873&publication_year=2016&author=Keramati%2CM&author=Smittenaar%2CP&author=Dolan%2CRJ&author=Dayan%2CP)
46. Michaelsen, P., Johansson, L.-O. & Hedesström, M. Experiencing nudges: autonomy, intrusion and choice satisfaction as judged by people themselves. In *Society for Judgment and Decision Making: The 2017 38th Annual Conference* (eds Shu, S. et al.) 30 (SJDM, 2017).
47. Dillard, J. P. & Shen, L. On the nature of reactance and its role in persuasive health communication. *Commun. Monogr.* **72**, 144–168 (2005).

[Article](https://doi.org/10.1080%2F03637750500111815)
 [Google Scholar](http://scholar.google.com/scholar_lookup?&title=On%20the%20nature%20of%20reactance%20and%20its%20role%20in%20persuasive%20health%20communication&journal=Commun.%20Monogr.&doi=10.1080%2F03637750500111815&volume=72&pages=144-168&publication_year=2005&author=Dillard%2CJP&author=Shen%2CL)

[Download references](https://citation-needed.springer.com/v2/references/10.1038/s41562-019-0672-9?format=refman&flavour=references)

Acknowledgements
----------------

Preliminary versions of Experiments 1 and 2 were presented at the 38th Annual Meeting of the Cognitive Science Society. This material has been substantially revised and expanded for the present Article. We would like to thank J. R. Nill and E. Q. Zhang for their contributions to developing the to-do list gamification app, P. Michaelsen for sharing his self-report measures of autonomy and intrusion, T. Ntounis for technical support, and E. Kon, R. Antonova, P. Krueger, M. Pacer, D. Reichman, S. Russell and J. Suchow for feedback and discussion. This work was supported by grant number 1757269 from the National Science Foundation, grant number ONR MURI N00014-13-1-0341, and a grant from the Templeton World Charity Foundation to T.L.G. The funders had no role in study design, data collection and analysis, decision to publish or preparation of the manuscript.

Author information
------------------

### Authors and Affiliations

1. Max Planck Institute for Intelligent Systems, Tübingen, Germany

Falk Lieder
2. Department of Electrical Engineering and Computer Science, University of California, Berkeley, CA, USA

Owen X. Chen
3. Department of Computer Science, Princeton University, Princeton, NJ, USA

Paul M. Krueger & Thomas L. Griffiths
4. Department of Psychology, Princeton University, Princeton, NJ, USA

Thomas L. Griffiths

Authors1. Falk Lieder[View author publications](/search?author=Falk%20Lieder)You can also search for this author in
 [PubMed](http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?cmd=search&term=Falk%20Lieder) [Google Scholar](http://scholar.google.co.uk/scholar?as_q=&num=10&btnG=Search+Scholar&as_epq=&as_oq=&as_eq=&as_occt=any&as_sauthors=%22Falk%20Lieder%22&as_publication=&as_ylo=&as_yhi=&as_allsubj=all&hl=en)
2. Owen X. Chen[View author publications](/search?author=Owen%20X.%20Chen)You can also search for this author in
 [PubMed](http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?cmd=search&term=Owen%20X.%20Chen) [Google Scholar](http://scholar.google.co.uk/scholar?as_q=&num=10&btnG=Search+Scholar&as_epq=&as_oq=&as_eq=&as_occt=any&as_sauthors=%22Owen%20X.%20Chen%22&as_publication=&as_ylo=&as_yhi=&as_allsubj=all&hl=en)
3. Paul M. Krueger[View author publications](/search?author=Paul%20M.%20Krueger)You can also search for this author in
 [PubMed](http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?cmd=search&term=Paul%20M.%20Krueger) [Google Scholar](http://scholar.google.co.uk/scholar?as_q=&num=10&btnG=Search+Scholar&as_epq=&as_oq=&as_eq=&as_occt=any&as_sauthors=%22Paul%20M.%20Krueger%22&as_publication=&as_ylo=&as_yhi=&as_allsubj=all&hl=en)
4. Thomas L. Griffiths[View author publications](/search?author=Thomas%20L.%20Griffiths)You can also search for this author in
 [PubMed](http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?cmd=search&term=Thomas%20L.%20Griffiths) [Google Scholar](http://scholar.google.co.uk/scholar?as_q=&num=10&btnG=Search+Scholar&as_epq=&as_oq=&as_eq=&as_occt=any&as_sauthors=%22Thomas%20L.%20Griffiths%22&as_publication=&as_ylo=&as_yhi=&as_allsubj=all&hl=en)
### Contributions

F.L. and T.L.G. designed the research. F.L., P.M.K. and O.C. performed the research. F.L. and P.M.K. analysed the data. F.L. and T.L.G. wrote the paper.

### Corresponding author

Correspondence to
 [Falk Lieder](mailto:falk.lieder@tuebingen.mpg.de).

Ethics declarations
-------------------

### Competing interests

The authors declare no competing interests.

Additional information
----------------------

**Peer review information:** Primary Handling Editor: Stavroula Kousta.

**Publisher’s note:** Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary information
-------------------------

### [Supplementary Information](https://static-content.springer.com/esm/art%3A10.1038%2Fs41562-019-0672-9/MediaObjects/41562_2019_672_MOESM1_ESM.pdf)

Supplementary Figs. 1–21, Supplementary Tables 1–5, Supplementary Methods and Supplementary References.

### [Reporting Summary](https://static-content.springer.com/esm/art%3A10.1038%2Fs41562-019-0672-9/MediaObjects/41562_2019_672_MOESM2_ESM.pdf)

### [Supplementary Software](https://static-content.springer.com/esm/art%3A10.1038%2Fs41562-019-0672-9/MediaObjects/41562_2019_672_MOESM3_ESM.zip)

Analysis code.

Rights and permissions
----------------------

[Reprints and Permissions](https://s100.copyright.com/AppDispatchServlet?title=Cognitive%20prostheses%20for%20goal%20achievement&author=Falk%20Lieder%20et%20al&contentID=10.1038%2Fs41562-019-0672-9&copyright=The%20Author%28s%29%2C%20under%20exclusive%20licence%20to%20Springer%20Nature%20Limited&publication=2397-3374&publicationDate=2019-08-19&publisherName=SpringerNature&orderBeanReset=true)

Falk Lieder
Owen X
Chen
Paul M
Krueger
Thomas L
Griffiths


Bounded Rationality in Las Vegas: Probabilistic Finite Automata Play Multi-Armed Bandits.

Bounded Rationality in Las Vegas: Probabilistic Finite Automata Play
Multi-Armed Bandits
Xinming Liu
Computer Science Dept.
Cornell University
Ithaca, NY 14853
xl379@cornell.eduJoseph Y. Halpern
Computer Science Dept.
Cornell University 414 Gates Hall
Ithaca, NY 14853
halpern@cs.cornell.edu
Abstract
While traditional economics assumes that hu-
mans are fully rational agents who always
maximize their expected utility, in practice, we
constantly observe apparently irrational behav-
ior. One explanation is that people have limited
computational power, so that they are, quite ra-
tionally, making the best decisions they can,
given their computational limitations. To test
this hypothesis, we consider the multi-armed
bandit (MAB) problem. We examine a simple
strategy for playing an MAB that can be imple-
mented easily by a probabilistic ﬁnite automa-
ton (PFA). Roughly speaking, the PFA sets cer-
tain expectations, and plays an arm as long as it
meets them. If the PFA has sufﬁciently many
states, it performs near-optimally. Its perfor-
mance degrades gracefully as the number of
states decreases. Moreover, the PFA acts in
a “human-like” way, exhibiting a number of
standard human biases, like an optimism bias
and a negativity bias .
1 INTRODUCTION
Behavioral economists have argued for years that the tra-
ditional model of homo economicus —an agent who is
always rational and behaves optimally—is misguided.
There is a lot of experimental work backing up their
claims (see, e.g., (Thaler 2015)). Recent work has ar-
gued that perhaps the behavior that we observe can best
be explained by thinking of agents as rational (i.e., try-
ing to behave optimally), but not able to due to compu-
tational limitations; that is, they are doing the best they
can, given their computational limitations.
In this paper, following a tradition that goes back Ru-
binstein (1986) and Neyman (1985), we model computa-
Proceedings of the 36thConference on Uncertainty in Artiﬁcial
Intelligence (UAI) , PMLR volume 124, 2020.tionally bounded agents as probabilistic ﬁnite automata
(PFAs). We can think of the number of states of the au-
tomaton as a proxy for how computationally bounded the
agent is. Neyman (1985) showed that cooperation can
arise if PFAs play a ﬁnitely-repeated prisoner’s dilemma;
work on this topic has continued to attract attention (see
Papadimitriou and Yannakakis (1994) and the references
therein). Wilson (2015) considered a decision problem
where an agent must decide whether nature is in state 0
or state 1, after getting signals that are correlated with
nature’s state. She characterized an optimal n-state PFA
for making this decision, and showed that it exhibited
“human-like” behavior; speciﬁcally, it ignored evidence
(something a Bayesian would never do), and exhibited
what could be viewed as a ﬁrst-impression bias and con-
ﬁrmation bias. Halpern, Pass, and Seeman (2012) con-
sidered a similar problem in a dynamic setting, where the
state of nature could change (slowly) over time. Again,
they showed that a simple PFA both performed well
and exhibited the kind of behavior humans exhibited in
games studied by Erev, Ert, and Roth (2010).
We continue this line of work, and try to understand the
behavior of computationally bounded agents playing a
multi-armed bandit (e.g., playing slot machines in Las
Vegas). Our ﬁrst step in doing this is to understand the
extent to which optimal play can be approximated by a
PFA without worrying about the number of states used.
There are a number of notions of optimal that we could
consider. Here we focus on arguably the simplest one:
we compare the expected average payoff of the automa-
ton after it runs for Nsteps to the expected average pay-
off of always pulling the optimal arm of the bandit. We
also assume that the possible payoff of each arm is either
1 or 0 (i.e. success or failure), so the expected payoff of
an arm is just the probability of getting a 1.
There are well-known protocols that use Bayesian meth-
ods (e.g., Thompson Sampling (Thompson 1933)) that
approach optimal play in the limit; however, these ap-
proaches are computationally expensive. We show that
they have to be. No approach that can be implemented
by a PFA can perform optimally. Indeed, for all PFAs,
there exists an  > 0such that as the number of steps
gets large, the ratio of the expected payoff of the automa-
ton to the expected payoff of the optimal arm is at most
1. That is, a PFA must be off by some  >0from
optimal play (although we can make as small as we like
by allowing sufﬁciently many states). Among families of
ﬁnite automata that have near-optimal payoff, we are in-
terested in ones that (a) make efﬁcient use of their states
(so, for a ﬁxed number Mof states, have high expected
payoff), (b) converge to near-optimal behavior quickly,
and (c) use simple “human-like” heuristics.
A standard approach to dealing with multi-armed bandit
problem is one we call explore-then-exploit . We simply
test each arm Ntimes (where Nis a parameter), and
from then on play the best arm (i.e., the one with the
highest average reward). If the bandit has Karms, then
we need roughly O(NKlog(N) log(K))states, since we
need to keep track of the possible tuples of outcomes of
the tests, as well as two counters, one to keep track of
which arm is being tested, and the other to keep track of
how many times we have played it.
We can greatly reduce the number of states by essentially
using an elimination tournament. We ﬁrst compare arm
1 against arm 2, eliminate the worse arm, run the winner
against arm 3, eliminate the worse arm, run the winner
against arm 4, and so on. The way we compare arm
iandjis straightforward: we alternate playing iand
jand use a counter to keep track of the relative num-
ber of successes of i. If the counter hits an appropriate
thresholdM(so thatihas hadMmore successes than
j),iis the winner; if the counter hits M, thenjis the
winner. To do this, we needK
2
2(2M+ 1)2K2M
states: we need to keep track of which arms are being
played, which arm is currently moving, and the counter.
In choosing M, we need to balance out the desire not to
mistakenly eliminate a good arm (which is more likely
to happen the smaller that Mis) with the desire not to
“waste” too much time in ﬁnding the right arm (since
the payoff while we are doing that may not be so high,
particularly if we are playing two arms whose success
probabilities are equal but not very high). We deal with
this by stopping a comparison after an expected number
Nof steps. (We implement this by stopping the compar-
ison with probability 1=N, which does not require any
extra states.) As we shall see, this approach, which we
call the elimination tournament , does extremely well.
The-greedy protocol is a slight variant of this approach:
Again, we test for the ﬁrst Nsteps, and then play the
current best arm with probability 1and a random
arm with probability . But this requires inﬁnitely manystates, since we must keep track of the fraction of suc-
cesses for all arms to determine the current best arm.
Clearly neither approach is optimal. With positive prob-
ability, both explore-then-exploit and the elimination-
tournament protocols will choose a non-optimal arm;
from then on it is not getting the optimal reward.
The-greedy protocol gets a non-optimal reward with
(roughly) probability . While we can make all these ap-
proaches arbitrarily close to optimal by choosing the pa-
rametersN,, andMappropriately, they do not satisfy
our third criterion: they don’t seem to be what people are
doing. The -greedy approach and explore-then-exploit
require an agent to keep track of large amounts of in-
formation, while the elimination tournament alternates
between arms at every step, which may have nontrivial
costs. (Imagine a gambler in Las Vegas who wants to
compare two arms that are at opposite ends of a large
room. Will he really walk back and forth?)
We instead consider an approach that takes as its starting
point earlier work by Rao (2017), who considered only
two-armed bandits, where, just as for us, each arm has
a payoff inf0;1g. She deﬁned a family of PFAs that
act like “approximate Bayesians”. More precisely, each
arm has an associated rank that represents a coarse esti-
mate of the arm’s payoff probability. Rao plays the arms
repeatedly (using complicated rules to determine which
arm to play next) in order to estimate the success proba-
bility of each arm, and then chooses the best arm.
While we use ranks, we use them in a very different way
from Rao. We take as our inspiration Simon’s notion of
satisﬁcing (Simon 1956). The idea is that an arm will be
accepted if its success probability is above some thresh-
old. In the words of Gigerenzer and Gaissmaier (2015):
“Set an aspiration level, search through alternatives se-
quentially, and stop search as soon as an alternative is
found that satisﬁes the level.” (We remark that the im-
portance of the aspiration level goes back to the 1930s in
the psychology literature, and has been studied at length
since then; see, e.g., the highly-cited work of Lewin et
al. (1944).) But how do we determine the aspiration
level? This is a nontrivial issue. Selten (1998) and Si-
mon (1982) (both Nobel prize winners) discuss this is-
sue at length. As Gigerenzer and Gaissmaier (2015) ob-
serve, “The aspiration level need not be ﬁxed, but can
be dynamically adjusted to feedback.” In our setting, it
is relatively straightforward: we use an optimism bias
(Sharot 2011). We start with a high aspiration level (suc-
cess probability) p, and run a tournament as above be-
tween each arm kand a “virtual arm” that has success
probabilityp. Since this is a virtual arm, we are essen-
tially comparing the performance of each arm kto our
expectation. If arm kdoes not meet our expectation, then
we go to the next arm. If no arm meets our expectation,
we adjust the aspiration level according to this feedback,
by lowering it. This requires KMm states, where M,
as before, is the counter used to keep track of the relative
performance of the arm being tested and mis the number
of ranks. We call this the aspiration-level approach.
We get good performance by taking mK, so the
aspiration-level approach uses essentially the same num-
ber of states as the elimination tournament. Moreover,
as we show by simulation, its performance approach de-
grades gracefully as the number of states decreases. Even
with relatively few states, it compares quite favorably to
the-greedy approach and to Thompson Sampling, al-
though they require inﬁnitely many states. More impor-
tantly from our perspective, the aspiration-level approach
is quite human-like. We have already mentioned how it
incorporates satisﬁcing, the adjustment of expectations
according to feedback, and an optimism bias. But there
is more. Whereas the elimination-tournament approach
treats the two arms that it is comparing symmetrically,
the aspiration-level approach does not. If the virtual arm
wins, it just means that we try another arm. Moreover,
especially initially, we expect the virtual arm to win be-
cause we start out with a high aspiration level. On the
other hand, if an actual arm wins, that is the arm we use
from then on. Thus, we want to be relatively quick to
reject an arm, and slow to accept. This can be be viewed
as a negativity bias (Kanouse and Hanson 1972): neg-
ative outcomes have a greater effect than positive out-
comes. The focus on recent behavior can be viewed
as implementing an availability heuristic (Tversky and
Kahneman 1973): people tend to heavily weight their
judgments toward more recent or available information.
Finally, a short run of good luck can have a signiﬁcant
inﬂuence, causing an arm to be played for a long time
(or even played forever, if it is enough to get it accepted).
People are well-known to label some arms as “lucky” and
keep playing them long after the evidence has indicated
otherwise. This can also be viewed as an instance of the
status quo bias (Samuelson and Zeckhauser 1998): peo-
ple are much more likely to stick with the current state of
affairs (provided they think it is reasonably good).
2 MULTI-ARMED BANDITS
This section provides the necessary background for the
rest of the paper. In particular, we (1) brieﬂy review
multi-armed bandits, (2) deﬁne the notion of optimality
we consider, and (3) prove that a PFA cannot be optimal.2.1 THE MULTI-ARMED BANDIT PROBLEM
The multi-armed bandit (MAB) problem is a standard
way of modeling the tradeoff between exploitation and
exploration. An agent has Karms that she can pull. Each
arm offers a set of possible rewards, each obtained with
some probability. The agent does not know the probabil-
ities in advance, but can learn them by playing the arm
sufﬁciently often. Formally, a K-armed bandit is a tu-
ple distribution over rewards for arm k. Letkbe the
expected reward of arm i, fori=k;:::;K . The best
expected reward of Bis denoted
B= maxkfkg.
We assume for simplicity in this paper that the possible
rewards of an arm are either 0 or 1. With this assump-
tion,kis the probability of getting a 1 with arm k. We
can easily modify the protocol to deal with a ﬁnite set of
possible rewards, as long as the set of possible rewards
is known in advance. We also assume for now that the
distributions Rkdo not vary over time.
2.2 OPTIMAL PROTOCOLS FOR MAB
PROBLEMS
We are interested in protocols that play MABs (almost)
optimally. Formally, a protocol is a (possibly random-
ized) function from history to actions. We focus on
one particular simple notion of optimality here, which
informally amounts to approaching the average reward
of the best arm. To make this precise, given a pro-
tocolP, letaP;B
t be a random variable that denotes
the arm played by protocol Pat thetth step. Thus,
aP;B
tis the expected reward of arm aP;B
t. It is easy to
see that the expected cumulative reward of protocol P
when run for Nsteps on MAB BisCum (P;B;N ) =PN
t=1E[aP;B
t]. Since the reward for playing the opti-
mal arm of MAB BforNsteps isN
B, the expected
regret is the difference between the cumulative reward
ofPand the optimal reward: Reg(P;B;N ) =N
B
Cum (P;B;N ). Finally, the averageN-step regret ofP
onBisAReg (P;B;N ) =Reg(P;B;N )=N. We say
thatPisoptimal iflimN!1AReg (P;B;N ) = 0 for
all MABsB.
As we observed in the introduction, neither explore-
then-exploit nor the -greedy protocol is optimal in this
sense. There are Bayesian approaches that are optimal.
We brieﬂy discuss one: Thompson Sampling (Thomp-
son 1933). Roughly speaking, at each step, this proto-
col computes the probability of each arm being optimal,
given the observations. It then chooses arm kwith a
probability proportional to its current estimate that kis
the optimal arm. It is not hard to show that, with proba-
bility 1, the probability of a non-optimal arm being cho-
sen goes to 0. (By way of contrast, the probability of a
non-optimal arm being chosen at any given step with the
-greedy protocol is a constant: at least (K1)=K, if
there areKarms.)
As shown by Kaufman, Kordan, and Munos (2012),
Thompson Sampling is optimal in an even stronger
sense than what we have considered so far. Tak-
ing TS to denote Thompson Sampling, not only
do we have limN!1Reg(TS;B;N )=N = 0 , but
there is a constant c
B(that depends on the MAB
B, but has been completely characterized) such that
limN!1Reg(TS;N )=log(N) =c
B. Moreover, this
is optimal; as shown by Lai and Robbins (1985), for all
protocolsPsatisfying a minimal technical condition, we
must have limN!1G(P;B;N )=log(N)c
B. That
means that Thompson Sampling approaches optimal be-
havior as quickly as possible, and its cumulative regret
grows only logarithmically. We mention this because we
will be comparing the performance of our approach to
that of Thompson Sampling later.
2.3 PROBABILISTIC FINITE AUTOMATA AND
NON-OPTIMALITY
As we said in the introduction, we are interested in
resource-bounded agents playing MABs, and we model
resource-boundedness using PFAs. A PFA is just like a
deterministic ﬁnite automaton, except that the transitions
are probabilistic. We also want our automata to produce
an output (an arm to pull, or no arm), rather than accept-
ing a language, so, technically, we are looking at what
have been called probabilistic ﬁnite automata with out-
putorprobabilistic transducers . (This is also the case for
all the earlier papers that considered PFAs playing games
or making decisions, such as (Halpern, Pass, and Seeman
2012; Papadimitriou and Yannakakis 1994; Rubinstein
1986; Wilson 2015).) Formally, a PFA with output is a
tuple (Q;q 0;;O;
; ), where
Qis a ﬁnite set of states ;
q02Qis the initial state;
is the input alphabet (in our case this will consist
of the observations “arm khad rewardj” forj2
f0;1g);
Ois the output alphabet (in our case this will be
“k”, which is interpreted as playing arm k, fork2
f1;:::;Kg);

:Q!(O)is a probabilistic action function (as
usual, (X)denotes the set of probability distribu-
tions onX);
:Q!(Q)is a probabilistic transition
function.Intuitively, the automaton starts in state q0and plays an
arm according to distribution
(q0). It then observes the
outcomeoof pulling the arm (an element of ) and then
transitions to a state q0(according to (q0;o)). It then
plays arm
(q0), and so on.
It is easy to see that the explore-then-exploit protocol
can be implemented by a ﬁnite automaton. On the other
hand, the-greedy protocol and Thompson Sampling
cannot. That is because they keep track of the total num-
ber of times each arm kwas played, and the fraction of
those times that a reward of 1 was obtained with k. This
requires inﬁnitely many states.
We claim that no protocol implemented by a PFA can be
optimal. To prove this, we need some deﬁnitions.
Deﬁnition 2.1. AK-arm MABB= (1;:::;K)is
generic if (1)
B<1, (2)i6=jfori6=j, and (3) if
K= 2, then min(1;2)>0.
Note the if we put the obvious uniform distribution on the
set ofK-armed bandits (identifying a K-armed bandit
with aK-vector of real numbers), then the set of generic
MABs has probability 1.
Deﬁnition 2.2. B0= (0
1;:::;0
K)is apermutation of
B= (1;:::;K)if there is some permutation of the
indices such that k=0
(k).
Theorem 2.1. For all PFAs Mand all generic MABs
B, there exists some M;B>0(that, as the no-
tation suggests, depends on both MandB) and
an MABB0that is a permutation of Bsuch that
limN!1Reg(M;B0;N)=NM;B.
Before giving the proof, we can explain why we must
consider generic MABs and permutations. To understand
why we consider permutations, suppose that Malways
plays arm 1. If it so happens that arm 1 is the best arm for
B, thenMgets the optimal reward with input B. But it
will not get the optimal reward for a permutation of Bfor
which arm 1 is not the best arm. It is not hard to see that if

B= 1, then there exists a PFA Mthat gets the optimal
reward given input Bor any of its permutations: Mjust
plays an arm until it does not get a payoff of 1, then goes
on to the next arm. Sooner or later Mwill play an arm
that always gets a reward of 1. A similar PFA also gets
the optimal reward if K= 2given an input B= (1;2)
such thatk= 0 for some arm k: it alternates between
the arms until it ﬁnds an arm that gives reward 1, and
sticks with that arm. Finally, if 1==K, then no
matter what arm Mplays, it will get the optimal reward
onBand all of its permutations. The requirement that
allks are distinct is actually stronger than we need, but
since slight perturbations of the rewards of an arm sufﬁce
to make all rewards distinct, we use it here for simplicity.
Proof. Given a PFA Mand a nontrivial MAB B, there
are two possibilities: (1) there is some state qthat can be
reached from the start state q0with positive probability
and an armksuch that, after reaching state q,Mplays
armkfrom then on, no matter what it observes; (2) there
is no such state q. Note that the ﬁrst case is what happens
with explore-then-exploit. After the exploration phase,
the same arm is played over and over. The second case
is more like Thompson Sampling or -greedy; there is
always some positive probability that a given arm kwill
be played.
For case (1), let o1;:::;oTbe a sequence of observations
that, with positive probability, leads Mto a stateqafter
which it always plays arm k. If the arm that Mplays
in stateqis not the best arm of B, let=
Bk,
and letM;B be the probability with which o1;:::;oT
is observed when running Mon inputB. Clearly,
limN!1Reg(M;B;N )=NM;B. And ifk=,
consider a permutation B0= (0
1;:::;0
K)such that
0
j= 0 if and only if j= 0 (i.e., the permutation is
the identity on all arms jsuch thatj= 0) such that
0
k6=
B0=
B. It is still the case that o1;:::;oTcan
be observed with some positive probability M;B0when
runningMon inputB0. Taking0=
Bj, we have
limN!1Reg(M;B;N )=N0M;B0.
For case (2), no matter what state q M is in, with some
probabilityq>0,Mplays a non-optimal arm at qor
moves to another state q0and plays a non-optimal arm
there. Let
M= minqq. SinceMhas only ﬁnitely
many states, >0. Given as input an MAB B, let
Bbe the difference between the
Band the probabil-
ity that the second-best arm returns 1. (Here we are
using the fact that all arms have different probabilities
of returning 1.) Let XM;B;T be a random variable that
represents the reward received on the Tth step that M
is run on input B. Our discussion shows that, for all
T, we must have E(XT+XT+1)2
B
MB,
since with probability at least
M, one ofXTorXT+1
is at leastBless than
B. SinceReg(M;B; 2N)=
2N
BP2N
T=1XM;B;T2N
BN(2
B
NB),
it follows that Reg(M;B; 2N)=2N
BB=2. This
gives us the desired result.
3 AN ALMOST-OPTIMAL FAMILY OF
PFAS FOR MAB PROBLEMS
In this section, we introduce the aspiration-level proto-
col more formally. We start by reviewing Rao’s (2017)
approach to dealing with 2-armed bandits, since our ap-
proach uses some of the same ideas.3.1 RAO’S APPROACH
With only ﬁnitely many states, a PFA cannot keep track
of the exact success rate of each arm in an MAB. Thus,
it needs to keep a ﬁnite representation of the success
rate. Rao’s idea was to use a ﬁnite set of possible
ranks to encode the agent’s belief about the relative
goodness of each arm. There are mpossible ranks,
f1;:::;mg, wheremis a parameter of the protocol.
Thus, Rao’s PFA has m2possible states, which have
the form (r1;r2)(since Rao considers only 2-armed ban-
dits), where r1;r22f1;:::;mg.
Rao assumes that the initial state of the PFA has the form
(n;n)for somen2f1;:::;mg; the exact choice does
not matter. Thus, initially, the two arms are assumed to
be equally good. Of course, if an agent has some prior
reason to believe that one arm is better than the other,
then the initial state can encode this belief.
The action function
is deﬁned as follows: If the higher-
ranked arm has the highest possible rank ( m) and the
other arm does not, then the higher-ranked arm is played.
Otherwise, similar in spirit to Thompson Sampling, the
next arm to play is chosen according to a probability that
depends on the difference between the ranks of the arms
(jr1r2j) and how far the arm’s ranks are from aver-
age (jr1m=2j+jr2m=2j). The two numbers are
then combined using two further parameters (called
andCby Rao) of the protocol. We refer the reader to
(Rao 2017) for the technical detail and intuition.
Finally, the transition function is deﬁned as follows: the
rank of the arm last played goes up with some probabil-
ity (if it is not already m) if a payoff of 1 is observed and
goes down with some probability (if it is not already 1) if
a payoff of 0 is observed. The rank of the arm not played
does not changed. The exact probability of a state change
depends on a quantity that Rao calls the inertia , which is
determined by the ranks of the arms, and two other pa-
rameters of the protocol, called andCtby Rao. Intu-
itively, the inertia characterizes the resistance to a change
in rank. The less frequently an arm has been played, the
higher its associated inertia will be, so its rank is updated
with a lower probability. Again, we refer the reader to
(Rao 2017) for details.
3.2 THE ASPIRATION-LEVEL PROTOCOL
We want to deﬁne a family of PFAs for K-armed MABs.
We continue to use Rao’s idea of associating with each
arm a rank. The naive extension would thus require
O(mK)states. For large K, this is quite unreasonable.
So we assume that the PFA focuses only one arm at a
time, comparing it to a “virtual” arm whose success prob-
ability can be thought of as the agent’s aspiration level
(Lewin, Dembo, Festinger, and Sears 1944). The ﬁrst
arm that meets the agent’s aspirations is the arm that is
played from then on. As we mentioned in the introduc-
tion, this can be viewed as satisﬁcing (Simon 1956). Not
only does this approach use signiﬁcantly fewer states, it
seems more like what people do.
Rao’s protocol has another feature that renders it an im-
plausible model of human behavior. It uses a number
of parameters ( m;;C;;Ct) to trade off exploitation
and exploration; the best choice of parameter settings de-
pends on the application domain. Moreover, these pa-
rameters are combined in a nontrivial way (using, for ex-
ample, exponentiation). It is hard to believe that people
would take the trouble (or have enough experience) to
learn the appropriate parameter settings for a particular
domain, nor are they likely to be willing to do the com-
putations needed to use them.
We thus signiﬁcantly simplify the action function and
transition function. As we said, we use the idea of a
tournament, but we play the current arm against a “vir-
tual arm”, whose success probability is determined by
the aspiration level, which is rank. If there are mranks,
then a rank of r2f1;:::;mgcan be thought of as repre-
senting the interval of probability [(r1)=m;r=m ]. We
thus take the success probability of a virtual arm with
aspiration level rto be (r:5)=m, the midpoint of the
interval. If we compare arm ito the virtual arm using a
counter. Suppose that we get a success with arm i(i.e.,
1 is observed). Since we expect the virtual arm to have
a success with probability (r:5)=m, we increase the
counter by 1 with probability 1(r:5)=m(since this is
the probability that the virtual arm had a failure, so that
armihad one more success than the virtual arm), and
leave the counter unchanged with probability (r:5)=m
(since, with this probability, both the virtual arm and arm
ihad a success). Similarly, if there is a failure with arm
i, we decrease the counter with probability (r:5)=m
and leave it unchanged with probability 1(r:5)=m.
We use two thresholds M1andM2to decide when to
end the comparison. If the counter reaches M1, then
we declare the current arm ibeing considered to have
won the tournament; intuitively, its success probability
is higher than that of the virtual arm. From then on we
play armi. If the counter reaches M2, then the virtual
arm has won the comparison. We (temporarily) elimi-
nate armk, and compare the virtual arm to arm k+ 1if
k <K . We discuss what happens if k=Kshortly, but
ﬁrst note that there is no analogue to the parameter N
of the elimination-tournament protocol here. The con-
cern in the elimination-tournament protocol is that we
are comparing two arms iandjthat have roughly equal,
but not very good success probabilities. Then the tour-nament will go on for a long time, but not give a high
reward. With the aspiration-level protocol, if arm khas a
success probability that is essentially the same as that of
the virtual arm, although the comparison may go on for
a long time, the agent is getting a cumulative reward that
essentially matches expectations, so there is no pressure
to stop the comparison.
Ifk=K, then the virtual arm did better than all arms
with this aspiration level. That means that our expecta-
tions are too high, so we lower the aspiration level from
rtor1, and retest all arms.
As discussed in the introduction, we do not assume that
M1=M2. The implications of an arm iwinning the
comparison against the virtual arm are much different
than the implications of the virtual arm winning. In the
former case, we play arm ifrom then on; in the latter
case, we just continue looking for another (hopefully bet-
ter) arm. Because the implications are so different, it
turns out that we want to take M1signiﬁcantly larger
thanM2. (Our experiments suggest that M1= 20 and
M2=3are good choices, along with m= 100 ; see
Section 4.2.)
One other issue: if the actual best success probability
is low (say, .2) and there are 100 ranks, it will take a
long time before the aspiration level is set appropriately.
During this time, the cumulative regret is increasing. To
speed up the process of ﬁnding the “right” aspiration
level, we can do a quick preprocessing phase to ﬁnd the
right range, and then explore more carefully. Speciﬁ-
cally, ifm= 100 , in the preprocessing phase, when we
reset the rank, we decrease it by 10 (in general, we de-
crease it bypm) rather than decreasing it by 1. We also
use smaller values of M1andM2(say,M1= 5 and
M2=1rather thanM1= 20 andM2=3). If
an armibeats the virtual arm when the aspiration level
r= 60 , we go back to the previous setting of aspira-
tion levelr= 70 , and do a more careful search start-
ing from there, now decreasing the aspiration level by 1,
and usingM1= 20 andM2=3. This preprocessing
phase allows us to home in on the appropriate expecta-
tions quickly. Again, besides being more efﬁcient, this
seems to be the type of thing that people do.
With this background, we are ready to de-
ﬁne our family of PFAs. For ease of presen-
tation, we do not use a preprocessing phase.
Formally, we have a family MK;m;M 1;M2=
(QK;m;M 1;M2;qm;K;OK;
K;K;m;M 1;M2) of
PFAs, indexed by 4 parameters: Kis the total number
of arms,mis the number of possible ranks for each arm,
andM1andM2are the upper and lower thresholds for
the counter. We assume that Kis given as part of the
input; we discuss how m,M1, andM2are chosen in the
next section. Not only do we have fewer parameters than
Rao, as we shall see, they are easier to set (and easier to
explain and understand). In more detail, the components
of the tuple are as follows:
A stateq2QK;m;M 1;M2has the form (r;k;c ),
where 1rm,1kK, andM2< c
M1. Intuitively, a state (r;k;c )says that the current
aspiration level is r, we are testing arm k, and the
counter that keeps track of the relative success rate
of armkcompared to the virtual arm is at c.
We take the initial state q0to be (m;1;0): we start
by setting the aspiration level to m(the highest level
possible), testing arm 1, and have the counter at 0.
Kconsists of observations of the form (k;h),
wherek2f1;:::;Kgandh2f0;1g. We observe
the outcome of playing arm k, which is a reward of
either 0 or 1.
OK=f1;:::;Kg: we can play any arm.
The action function
Kat a state (r;k;c )plays arm
k.
The transition function K;m;M 1;M2proceeds as fol-
lows. In state (r;k;c ), ifc=M1, the state does
not change. (We have chosen ias the arm to play
from then on.) If c < M 1, given an observation
(h;k), ifh= 1 (a success was observed), the new
state is (r;i;c0), wherec0=c+ 1with probability
1(r:5)=m, and otherwise c0=c. Ifh= 0and
c > M 2+ 1, then the new state is (r;i;c0), where
c0=c1with probability (r:5)=m, and other-
wise is unchanged. If c=M2+ 1, then with proba-
bility (r:5)=m, the new state is (r1;1;0)(the
aspiration level is lowered and we start over com-
paring the virtual arm to all the arms, starting with
arm 1); otherwise the state is unchanged.
4 EXPERIMENTS
4.1 PERFORMANCE METRICS
We use simulations to test the performance of various
protocols. In the simulations, we consider an MAB B
withKarms, whose true success probabilities are uni-
formly distributed in [0, ], whereis a random num-
ber in [0,1]. If we had just assumed that the success
probabilities were uniformly distributed in [0,1], then the
probability of there being an arm in the [0.9,1] interval is
10:9K, which is approximately 0.995 for K= 50 .
Indeed, the probability of there being an arm in the inter-
val[:99;1], is about 0.4. Not only does this seem unrea-
sonable in practice, this assumption would make it tooeasy to set the right aspiration level in our approach (i.e.,
it would hide some real-world difﬁculty). The assump-
tion that the success probabilities are bounded by for
a randomly-chosen seems more reasonable. While as-
suming that the success probabilities are uniformly dis-
tributed in [0;]may not be so reasonable, our results
remain essentially unchanged even if the success proba-
bilities are chosen adversarially, and the uniform distri-
bution is much easier to generate.
We focus on two metrics when it comes to measuring the
performance of a protocol: (1) the expected cumulative
regret of a protocol Pas a function of the number of steps
played (which roughly depends on how long it takes to
ﬁnd the best arm) and (2) the expected average regret in
the limit (i.e., limN!1AReg (P;B;N )), which essen-
tially measures the gap between the success probability
of the arm chosen by protocol Pand the success proba-
bility of the optimal arm of B. We take the expectation
over MABsBgenerated as discussed above. Essentially,
we want a protocol that gets to the best arm quickly and
accurately.
4.2 PARAMETER SETTINGS IN THE
ASPIRATION-LEVEL PROTOCOL
There are three parameter settings for the aspiration-level
protocol: the number of ranks m, and the thresholds M1
andM2for winning and losing a comparison against the
virtual arm. We examine the effect of different choices
here.
The largermis, the ﬁner distinctions we will be able
to make between the arms that we are testing. Roughly
speaking, if the virtual arm has rank rand the virtual
arm performed better than all arms when the aspiration
level wasr+ 1, we would expect that all arms have suc-
cess probability less than (r+:5)=m, and that an arm
with success probability greater than (r:5)=mwill beat
the virtual arm. However, this arm can have probability
as much as 1=mless than the arm with highest success
probability. By taking mlarger, we thus minimize the
expected gap between the success probability of the arm
chosen and the best arm.
We consider an MAB BwithK= 50 arms and run sim-
ulations. As expected, the larger mis, the smaller the
gap, but there are diminishing returns. Figure 1 shows
that, with other parameters ﬁxed ( M1= 20;M 2= 3),
there is signiﬁcant improvement in going from m= 50
tom= 100 ; but the marginal improvement drops off
quickly. This is no signiﬁcant difference between m=
100and larger values such as m= 200 orm= 500 .
The corresponding gaps between the success probability
of the arm chosen and the success probability of the op-
timal arm of B, averaged over 100 repetitions, are 0.020,
0.007, 0.0068, 0.0065, respectively. In addition, since
we start optimistically by initializing the aspiration level
at the highest possible rank, when mis larger, it takes
longer to get the right aspiration level and hence the cu-
mulative regret is larger, as shown in Figure 1. Consid-
ering both performance metrics as mentioned above, we
choosem= 100 in the later simulations.
Figure 1: Cumulative regret for different m.
Once we ﬁx m, we now examine the choices of M1and
M2. The parameters M1andM2determine the condi-
tions of winning and losing: if counter gets to M1, then
the current arm beats the “virtual arm” and is therefore
chosen as the best arm; if the counter gets to M2, then
the current arm loses the tournament with the “virtual
arm” and we move to a new arm. If all Karms lose the
tournament, we decrease the aspiration level by 1 and
restart the tournament. We want it to be easier for the
“virtual arm” to win, since the consequences are lower
in that case (the protocol ends if we declare arm ia
winner, whereas we keep going if the “virtual arm” is
a winner). Therefore, it makes sense to have an asym-
metry and choose M1greater than M2. We again con-
sider an MAB BwithK= 50 arms andm= 100
ﬁxed. As shown in Figure 2, the cumulative regret in-
creases asM1andM2get larger. However, the gap be-
tween the success probability of the arm chosen by pro-
tocolPand the success probability of the optimal arm
ofB, decreases. The corresponding gaps, averaged over
100 repetitions, are 0.014, 0.007, 0.005, 0.004, respec-
tively. Since the number of states in the aspiration-level
protocol isKm(M1+M2), there is a tradeoff between
accuracy and the number of states required. Taking into
account state-efﬁciency, accuracy, and the expected cu-
mulative regret, we choose M1= 20 andM2= 3.
Both Figure 1 and Figure 2 show that the performance of
the aspiration-level protocol degrades quite gracefully as
we take smaller values of m,M1, andM2(which is how
we would have to deal with having fewer states).
Figure 2: Cumulative regret for different M1andM2.
4.3 PARAMETER SETTINGS FOR THE
ELIMINATION TOURNAMENT
The elimination-tournament protocol has two parame-
ters:M(the point at which an arm is declared a winner in
the two-way comparison) and N(recall that 1=Nis the
probability that an arm is declared in the two-way com-
parison if no arm is dominant and has Mmore successes
than the other). Thus, after an expected number of at
mostN(K1)steps, the elimination-tournament proto-
col has reduced to one arm. We clearly want MandNto
be large enough to give the protocol time to select a rel-
atively good arm. However, we don’t want to stick with
bad arms for too long, since this will lead to larger cumu-
lative regret. Figure 3 shows the cumulative regrets for
different choices of NandM, for an MAB with K= 50
arms. For the choices of (N;M )considered—(1000,10),
(1000,20), (1000, 100), (100,10), (100,20)—the gaps,
averaged 100 repetitions, are 0.01, 0.007, 0.006, 0.03,
0.03, respectively. Both N= 1000;M = 20 and
N= 1000;M= 100 give similarly good performance
in terms of the expected average regret, but the latter
leads to larger expected cumulative regret. Therefore,
forK= 50 , we choose N= 1000 andM= 20 .
Figure 3: Cumulative regret for different NandM.
4.4 COMPARING PROTOCOLS
Based on the simulations above, to minimize the number
of states used while maintaining relatively good perfor-
mance, forK= 50 , we choose the parameters m=
100;M 1= 20;M 2= 3 for the aspiration-level pro-
tocol andM= 20 ,N= 1000 for the elimination-
tournament protocol, and compare these two ﬁnite-state
protocols to the -greedy protocol and Thompson Sam-
pling, which are inﬁnite-state protocols. With these
choices, the aspiration-level protocol uses 115,000 states,
while the elimination-tournament protocol uses just over
100,000. While this may seem to be a a lot of states,
they can be encoded using 17 bits. Given the number of
neurons in a human brain, this should not be a problem.
We can greatly reduce the cumulative regret for the
aspiration-level protocol by a preprocessing phase, as
suggested earlier. For K= 50 arms and the aspiration-
level protocol with m= 100;M 1= 20;M 2= 3, we ﬁrst
use a preprocessing phase to get a rough idea of what the
true highest success probability might be. We use the
parameters suggested earlier, decreasing the aspiration
level by 10 after testing all the arms in the preprocessing,
and use thresholds M0
1=5andM0
2=1. We use
this two-phase approach for the aspiration-level protocol
in the following simulation.
We consider MABs with K= 50 arms, and see how
the elimination-tournament protocol, the aspiration-level
protocol,-greedy, and Thompson sampling perform.
Not surprisingly, Thompson sampling performs best, and
has logarithmic cumulative regret, whereas the other
three protocols have linear cumulative regret. After
50,000 steps, the expected difference between the suc-
cess probability of the arm chosen and that of the op-
timal arm for these protocols are 0.007, 0.008, 0.025,
0.003, respectively. Interestingly, both the aspiration-
level protocol and the elimination-tournament protocol
eventually outperform -greedy, although the latter re-
quires inﬁnitely many states.
Figure 4: Cumulative regret over time.5 DISCUSSION
We have introduced two ﬁnite-state protocols for playing
MABs, the aspiration-level protocol and the elimination-
tournament protocol. Both perform quite well in prac-
tice, while using relatively few states. In cases where
switching between arms incurs a signiﬁcant cost, the
aspiration-level protocol is a better choice.
Recall that the main motivation for this study was under-
standing human behavior. The fact that the aspiration-
level protocol exhibits such human-like behavior, includ-
ing adjusting aspiration levels according to feedback, an
optimism bias, a negativity bias, and a status quo bias,
as well as a focus on recent behavior, suggests that hu-
mans are not being so irrational. Note that these biases
are emphasized if the number of states is decreased. For
example, if an agent decreases M2, the threshold for re-
jecting an arm in a two-way comparison with the virtual
arm, in response to having fewer states, this increases the
negativity bias. Decreasing M1increases the likelihood
that an agent will continue to play an apparently “lucky
arm”. The impact of decreasing Min the elimination-
tournament protocol is similar. The bottom line is that
these protocols exhibit apparently irrational behavior for
quite rational reasons! At the same time, they may be of
interest even for those not interested in modeling human
behavior, since they have quite good performance, even
with relatively few states.
We have focused here on a static setting, where the prob-
abilities do not change over time. We could easily mod-
ify our PFA to deal with the dynamic setting by simply
resetting the tournaments from time to time. More in-
terestingly, we would like to apply these ideas to a more
game-theoretic setting, such as the wildlife poaching set-
ting considered by Kar et al. (2015), where rangers are
trying to protect rhinos from poachers. We hope to re-
port on that in future work.
Acknowledgements
This research was supported by MURI (MultiUniversity
Research Initiative) under grant W911NF-19-1-0217, by
the ARO under grant W911NF-17-1-0592, by the NSF
under grants IIS-1703846 and IIS-1718108, and by a
grant from the Open Philosophy Foundation. We thank
Alice Chen for her preliminary work and comments on
an earlier version of the manuscript. We also thank four
anonymous reviewers for their feedback.
References
Erev, I., E. Ert, and A. E. Roth (2010). A choice
prediction competition for market entry games:
An introduction. Games and Economic Behav-
ior 1 (1), 117–136.
Gigerenzer, G. and W. Gaissmaier (2015). Decision
making: Nonrational theories. In J. D. Wright
(Ed.), International Encyclopedia of the Social
and Behavioral Sciences (2nd Edition) , pp. 911–
916.
Halpern, J. Y ., R. Pass, and L. Seeman (2012). I’m
doing as well as I can: modeling people as ra-
tional ﬁnite automata. In Proc. Twenty-Sixth Na-
tional Conference on Artiﬁcial Intelligence (AAAI
’12), pp. 1917–1923.
Kanouse, D. E. and L. Hanson (1972). Negativity
in evaluations. In E. E. Jones, D. E. Kanouse,
S. Valins, H. H. Kelley, R. E. Nisbett, and
B. Weiner (Eds.), Attribution: Perceiving the
Causes of Behavior . Morristown, NJ: General
Learning Press.
Kar, D., F. Fang, F. Delle Fave, N. Sintov, and
M. Tambe (2015). “Game of thrones”: when hu-
man behavior models compete in repeated Stack-
elberg security games. In Proc. 2015 International
Conference on Autonomous Agents and Multia-
gent Systems , pp. 1381–1390.
Kaufmann, E., N. Korda, and R. Munos (2012).
Thompson sampling: an asymptotically optimal
ﬁnite-time analysis. In N. H. Bshouty, G. Stoltz,
N. Vayatis, and T. Zeugmann (Eds.), Algorith-
mic Learning Theory )ALT 2012) , LNCS, V ol-
ume 7568, pp. 199–213. Springer.
Lai, T. L. and H. Robbins (1985). Asymptotically ef-
ﬁcient adaptive allocation rules. Advances in Ap-
plied Mathematics 6 (1), 4–22.
Lewin, K., T. Dembo, L. Festinger, and P. S.
Sears (1944). Level of aspiration. In J. M. Hunt
(Ed.), Personality and the Behavior Disorders , pp.
333–378. Cambridge, MA: Ronald Press.
Neyman, A. (1985). Bounded complexity justiﬁes co-
operation in ﬁnitely repeated prisoner’s dilemma.
Economic Letters 19 , 227–229.
Papadimitriou, C. H. and M. Yannakakis (1994). On
complexity as bounded rationality. In Proc. 26th
ACM Symposium on Theory of Computing , pp.
726–733.
Rao, A. (2017). A ﬁnite memory automaton for two-
armed Bernoulli bandit problems. In Proc. Thirty-
First National Conference on Artiﬁcial Intelli-gence (AAAI ’17) , pp. 4981–4982. The full pa-
per is available at http://raoariel.github.io/raoariel-
fma.pdf.
Rubinstein, A. (1986). Finite automata play the re-
peated prisoner’s dilemma. Journal of Economic
Theory 39 , 83–96.
Samuelson, W. and R. Zeckhauser (1998). Status quo
bias in decision making. Journal of Risk and Un-
certainty 1 , 7–59.
Selten, R. (1998). Aspiration adaptation theory. Jour-
nal of Mathematical Psychology 42 , 191–214.
Sharot, T. (2011). The Optimism Bias: A Tour of the
Irrationally Positive Brain . New York, NY: Pan-
theon Books.
Simon, H. A. (1956). Rational choice and the structure
of the environment. Psychological Review 63 (2),
129–138.
Simon, H. A. (1982). Models of bounded rationality .
Cambridge, MA: MIT Press.
Thaler, R. (2015). Misbehaving: The Making of Be-
havioral Economics . New York, NY: W. W. Nor-
ton and Company.
Thompson, W. R. (1933). On the likelihood that one
unknown probability exceeds another in view of
the evidence of two samples. Biometrika 25 (3–4),
285–294.
Tversky, A. and D. Kahneman (1973). Availability:
a heuristic for judging frequency and probability.
Cognitive Psychology 5 , 207–232.
Wilson, A. (2015). Bounded memory and biases in in-
formation processing. Econometrica 82 (6), 2257–
2294.