Probabilistic machine learning and artificial intelligence

### Subjects

* [Computer science](/subjects/computer-science)
* [Mathematics and computing](/subjects/mathematics-and-computing)
* [Neuroscience](/subjects/neuroscience)

Abstract
--------

How can a machine learn from experience? Probabilistic modelling provides a framework for understanding what learning is, and has therefore emerged as one of the principal theoretical and practical approaches for designing machines that learn from data acquired through experience. The probabilistic framework, which describes how to represent and manipulate uncertainty about models and predictions, has a central role in scientific data analysis, machine learning, robotics, cognitive science and artificial intelligence. This Review provides an introduction to this framework, and discusses some of the state-of-the-art advances in the field, namely, probabilistic programming, Bayesian optimization, data compression and automatic model discovery.

[Access through your institution](https://wayf.springernature.com?redirect_uri=https%3A%2F%2Fwww.nature.com%2Farticles%2Fnature14541)

[Buy or subscribe](#access-options)

This is a preview of subscription content, [access via your institution](https://wayf.springernature.com?redirect_uri=https%3A%2F%2Fwww.nature.com%2Farticles%2Fnature14541)

 if (window.dataLayer) {
 window.dataLayer.push({
 content: { article: { relevantArticlesCount: 3 }}
 })
 }

Relevant articles
-----------------

Open Access articles citing this article.

* ###
[Modeling the rutting performance of asphalt pavements: a review](https://doi.org/10.1186/s43065-023-00082-9)

	+ Yong Deng
	+ & Xianming Shi
*Journal of Infrastructure Preservation and Resilience*
Open Access
28 July 2023
* ###
[Machine learning framework with feature selection approaches for thyroid disease classification and associated risk factors identification](https://doi.org/10.1186/s43067-023-00101-5)

	+ Azrin Sultana
	+ & Rakibul Islam
*Journal of Electrical Systems and Information Technology*
Open Access
16 June 2023
* ###
[Predictive factors for degenerative lumbar spinal stenosis: a model obtained from a machine learning algorithm technique](https://doi.org/10.1186/s12891-023-06330-z)

	+ Janan Abbas
	+ , Malik Yousef
	+ … Kamal Hamoud
*BMC Musculoskeletal Disorders*
Open Access
23 March 2023

Access options
--------------

[Access through your institution](https://wayf.springernature.com?redirect_uri=https%3A%2F%2Fwww.nature.com%2Farticles%2Fnature14541)

[Access through your institution](https://wayf.springernature.com?redirect_uri=https%3A%2F%2Fwww.nature.com%2Farticles%2Fnature14541)

[Change institution](https://wayf.springernature.com?redirect_uri=https%3A%2F%2Fwww.nature.com%2Farticles%2Fnature14541)

[Buy or subscribe](#access-options)

/\* style specs start \*/
style{display:none!important}.LiveAreaSection-193358632 \*{align-content:stretch;align-items:stretch;align-self:auto;animation-delay:0s;animation-direction:normal;animation-duration:0s;animation-fill-mode:none;animation-iteration-count:1;animation-name:none;animation-play-state:running;animation-timing-function:ease;azimuth:center;backface-visibility:visible;background-attachment:scroll;background-blend-mode:normal;background-clip:borderBox;background-color:transparent;background-image:none;background-origin:paddingBox;background-position:0 0;background-repeat:repeat;background-size:auto auto;block-size:auto;border-block-end-color:currentcolor;border-block-end-style:none;border-block-end-width:medium;border-block-start-color:currentcolor;border-block-start-style:none;border-block-start-width:medium;border-bottom-color:currentcolor;border-bottom-left-radius:0;border-bottom-right-radius:0;border-bottom-style:none;border-bottom-width:medium;border-collapse:separate;border-image-outset:0s;border-image-repeat:stretch;border-image-slice:100%;border-image-source:none;border-image-width:1;border-inline-end-color:currentcolor;border-inline-end-style:none;border-inline-end-width:medium;border-inline-start-color:currentcolor;border-inline-start-style:none;border-inline-start-width:medium;border-left-color:currentcolor;border-left-style:none;border-left-width:medium;border-right-color:currentcolor;border-right-style:none;border-right-width:medium;border-spacing:0;border-top-color:currentcolor;border-top-left-radius:0;border-top-right-radius:0;border-top-style:none;border-top-width:medium;bottom:auto;box-decoration-break:slice;box-shadow:none;box-sizing:border-box;break-after:auto;break-before:auto;break-inside:auto;caption-side:top;caret-color:auto;clear:none;clip:auto;clip-path:none;color:initial;column-count:auto;column-fill:balance;column-gap:normal;column-rule-color:currentcolor;column-rule-style:none;column-rule-width:medium;column-span:none;column-width:auto;content:normal;counter-increment:none;counter-reset:none;cursor:auto;display:inline;empty-cells:show;filter:none;flex-basis:auto;flex-direction:row;flex-grow:0;flex-shrink:1;flex-wrap:nowrap;float:none;font-family:initial;font-feature-settings:normal;font-kerning:auto;font-language-override:normal;font-size:medium;font-size-adjust:none;font-stretch:normal;font-style:normal;font-synthesis:weight style;font-variant:normal;font-variant-alternates:normal;font-variant-caps:normal;font-variant-east-asian:normal;font-variant-ligatures:normal;font-variant-numeric:normal;font-variant-position:normal;font-weight:400;grid-auto-columns:auto;grid-auto-flow:row;grid-auto-rows:auto;grid-column-end:auto;grid-column-gap:0;grid-column-start:auto;grid-row-end:auto;grid-row-gap:0;grid-row-start:auto;grid-template-areas:none;grid-template-columns:none;grid-template-rows:none;height:auto;hyphens:manual;image-orientation:0deg;image-rendering:auto;image-resolution:1dppx;ime-mode:auto;inline-size:auto;isolation:auto;justify-content:flexStart;left:auto;letter-spacing:normal;line-break:auto;line-height:normal;list-style-image:none;list-style-position:outside;list-style-type:disc;margin-block-end:0;margin-block-start:0;margin-bottom:0;margin-inline-end:0;margin-inline-start:0;margin-left:0;margin-right:0;margin-top:0;mask-clip:borderBox;mask-composite:add;mask-image:none;mask-mode:matchSource;mask-origin:borderBox;mask-position:0 0;mask-repeat:repeat;mask-size:auto;mask-type:luminance;max-height:none;max-width:none;min-block-size:0;min-height:0;min-inline-size:0;min-width:0;mix-blend-mode:normal;object-fit:fill;object-position:50% 50%;offset-block-end:auto;offset-block-start:auto;offset-inline-end:auto;offset-inline-start:auto;opacity:1;order:0;orphans:2;outline-color:initial;outline-offset:0;outline-style:none;outline-width:medium;overflow:visible;overflow-wrap:normal;overflow-x:visible;overflow-y:visible;padding-block-end:0;padding-block-start:0;padding-bottom:0;padding-inline-end:0;padding-inline-start:0;padding-left:0;padding-right:0;padding-top:0;page-break-after:auto;page-break-before:auto;page-break-inside:auto;perspective:none;perspective-origin:50% 50%;pointer-events:auto;position:static;quotes:initial;resize:none;right:auto;ruby-align:spaceAround;ruby-merge:separate;ruby-position:over;scroll-behavior:auto;scroll-snap-coordinate:none;scroll-snap-destination:0 0;scroll-snap-points-x:none;scroll-snap-points-y:none;scroll-snap-type:none;shape-image-threshold:0;shape-margin:0;shape-outside:none;tab-size:8;table-layout:auto;text-align:initial;text-align-last:auto;text-combine-upright:none;text-decoration-color:currentcolor;text-decoration-line:none;text-decoration-style:solid;text-emphasis-color:currentcolor;text-emphasis-position:over right;text-emphasis-style:none;text-indent:0;text-justify:auto;text-orientation:mixed;text-overflow:clip;text-rendering:auto;text-shadow:none;text-transform:none;text-underline-position:auto;top:auto;touch-action:auto;transform:none;transform-box:borderBox;transform-origin:50% 50%0;transform-style:flat;transition-delay:0s;transition-duration:0s;transition-property:all;transition-timing-function:ease;vertical-align:baseline;visibility:visible;white-space:normal;widows:2;width:auto;will-change:auto;word-break:normal;word-spacing:normal;word-wrap:normal;writing-mode:horizontalTb;z-index:auto;-webkit-appearance:none;-moz-appearance:none;-ms-appearance:none;appearance:none;margin:0}.LiveAreaSection-193358632{width:100%}.LiveAreaSection-193358632 .login-option-buybox{display:block;width:100%;font-size:17px;line-height:30px;color:#222;padding-top:30px;font-family:Harding,Palatino,serif}.LiveAreaSection-193358632 .additional-access-options{display:block;font-weight:700;font-size:17px;line-height:30px;color:#222;font-family:Harding,Palatino,serif}.LiveAreaSection-193358632 .additional-login>li:not(:first-child)::before{transform:translateY(-50%);content:"";height:1rem;position:absolute;top:50%;left:0;border-left:2px solid #999}.LiveAreaSection-193358632 .additional-login>li:not(:first-child){padding-left:10px}.LiveAreaSection-193358632 .additional-login>li{display:inline-block;position:relative;vertical-align:middle;padding-right:10px}.BuyBoxSection-683559780{display:flex;flex-wrap:wrap;flex:1;flex-direction:row-reverse;margin:-30px -15px 0}.BuyBoxSection-683559780 .box-inner{width:100%;height:100%}.BuyBoxSection-683559780 .readcube-buybox{background-color:#f3f3f3;flex-shrink:1;flex-grow:1;flex-basis:255px;background-clip:content-box;padding:0 15px;margin-top:30px}.BuyBoxSection-683559780 .subscribe-buybox{background-color:#f3f3f3;flex-shrink:1;flex-grow:4;flex-basis:300px;background-clip:content-box;padding:0 15px;margin-top:30px}.BuyBoxSection-683559780 .subscribe-buybox-nature-plus{background-color:#f3f3f3;flex-shrink:1;flex-grow:4;flex-basis:100%;background-clip:content-box;padding:0 15px;margin-top:30px}.BuyBoxSection-683559780 .title-readcube,.BuyBoxSection-683559780 .title-buybox{display:block;margin:0;margin-right:10%;margin-left:10%;font-size:24px;line-height:32px;color:#222;padding-top:30px;text-align:center;font-family:Harding,Palatino,serif}.BuyBoxSection-683559780 .title-asia-buybox{display:block;margin:0;margin-right:5%;margin-left:5%;font-size:24px;line-height:32px;color:#222;padding-top:30px;text-align:center;font-family:Harding,Palatino,serif}.BuyBoxSection-683559780 .asia-link{color:#069;cursor:pointer;text-decoration:none;font-size:1.05em;font-family:-apple-system,BlinkMacSystemFont,"Segoe UI",Roboto,Oxygen-Sans,Ubuntu,Cantarell,"Helvetica Neue",sans-serif;line-height:1.05em6}.BuyBoxSection-683559780 .access-readcube{display:block;margin:0;margin-right:10%;margin-left:10%;font-size:14px;color:#222;padding-top:10px;text-align:center;font-family:-apple-system,BlinkMacSystemFont,"Segoe UI",Roboto,Oxygen-Sans,Ubuntu,Cantarell,"Helvetica Neue",sans-serif;line-height:20px}.BuyBoxSection-683559780 .access-asia-buybox{display:block;margin:0;margin-right:5%;margin-left:5%;font-size:14px;color:#222;padding-top:10px;text-align:center;font-family:-apple-system,BlinkMacSystemFont,"Segoe UI",Roboto,Oxygen-Sans,Ubuntu,Cantarell,"Helvetica Neue",sans-serif;line-height:20px}.BuyBoxSection-683559780 .access-buybox{display:block;margin:0;margin-right:10%;margin-left:10%;font-size:14px;color:#222;opacity:.8px;padding-top:10px;text-align:center;font-family:-apple-system,BlinkMacSystemFont,"Segoe UI",Roboto,Oxygen-Sans,Ubuntu,Cantarell,"Helvetica Neue",sans-serif;line-height:20px}.BuyBoxSection-683559780 .price-buybox{display:block;font-size:30px;color:#222;font-family:-apple-system,BlinkMacSystemFont,"Segoe UI",Roboto,Oxygen-Sans,Ubuntu,Cantarell,"Helvetica Neue",sans-serif;padding-top:30px;text-align:center}.BuyBoxSection-683559780 .price-buybox-to{display:block;font-size:30px;color:#222;font-family:-apple-system,BlinkMacSystemFont,"Segoe UI",Roboto,Oxygen-Sans,Ubuntu,Cantarell,"Helvetica Neue",sans-serif;text-align:center}.BuyBoxSection-683559780 .price-info-text{font-size:16px;padding-right:10px;color:#222;font-family:-apple-system,BlinkMacSystemFont,"Segoe UI",Roboto,Oxygen-Sans,Ubuntu,Cantarell,"Helvetica Neue",sans-serif}.BuyBoxSection-683559780 .price-value{font-size:30px;font-family:-apple-system,BlinkMacSystemFont,"Segoe UI",Roboto,Oxygen-Sans,Ubuntu,Cantarell,"Helvetica Neue",sans-serif}.BuyBoxSection-683559780 .price-per-period{font-family:-apple-system,BlinkMacSystemFont,"Segoe UI",Roboto,Oxygen-Sans,Ubuntu,Cantarell,"Helvetica Neue",sans-serif}.BuyBoxSection-683559780 .price-from{font-size:14px;padding-right:10px;color:#222;font-family:-apple-system,BlinkMacSystemFont,"Segoe UI",Roboto,Oxygen-Sans,Ubuntu,Cantarell,"Helvetica Neue",sans-serif;line-height:20px}.BuyBoxSection-683559780 .issue-buybox{display:block;font-size:13px;text-align:center;color:#222;font-family:-apple-system,BlinkMacSystemFont,"Segoe UI",Roboto,Oxygen-Sans,Ubuntu,Cantarell,"Helvetica Neue",sans-serif;line-height:19px}.BuyBoxSection-683559780 .no-price-buybox{display:block;font-size:13px;line-height:18px;text-align:center;padding-right:10%;padding-left:10%;padding-bottom:20px;padding-top:30px;color:#222;font-family:-apple-system,BlinkMacSystemFont,"Segoe UI",Roboto,Oxygen-Sans,Ubuntu,Cantarell,"Helvetica Neue",sans-serif}.BuyBoxSection-683559780 .vat-buybox{display:block;margin-top:5px;margin-right:20%;margin-left:20%;font-size:11px;color:#222;padding-top:10px;padding-bottom:15px;text-align:center;font-family:-apple-system,BlinkMacSystemFont,"Segoe UI",Roboto,Oxygen-Sans,Ubuntu,Cantarell,"Helvetica Neue",sans-serif;line-height:17px}.BuyBoxSection-683559780 .tax-buybox{display:block;width:100%;color:#222;padding:20px 16px;text-align:center;font-family:-apple-system,BlinkMacSystemFont,"Segoe UI",Roboto,Oxygen-Sans,Ubuntu,Cantarell,"Helvetica Neue",sans-serif;line-height:NaNpx}.BuyBoxSection-683559780 .button-container{display:flex;padding-right:20px;padding-left:20px;justify-content:center}.BuyBoxSection-683559780 .button-container>\*{flex:1px}.BuyBoxSection-683559780 .button-container>a:hover,.Button-1078489254:hover,.Button-2496381730:hover{text-decoration:none}.BuyBoxSection-683559780 .readcube-button{background:#fff;margin-top:30px}.BuyBoxSection-683559780 .button-asia{background:#069;border:1px solid #069;border-radius:0;cursor:pointer;display:block;padding:9px;outline:0;text-align:center;text-decoration:none;min-width:80px;margin-top:75px}.BuyBoxSection-683559780 .button-label-asia,.ButtonLabel-3296148077,.ButtonLabel-1651148777{display:block;color:#fff;font-size:17px;line-height:20px;font-family:-apple-system,BlinkMacSystemFont,"Segoe UI",Roboto,Oxygen-Sans,Ubuntu,Cantarell,"Helvetica Neue",sans-serif;text-align:center;text-decoration:none;cursor:pointer}.Button-1078489254,.Button-2496381730{background:#069;border:1px solid #069;border-radius:0;cursor:pointer;display:block;padding:9px;outline:0;text-align:center;text-decoration:none;min-width:80px;max-width:320px;margin-top:10px}.Button-1078489254 .readcube-label,.Button-2496381730 .readcube-label{color:#069}
/\* style specs end \*/Subscribe to this journal

Receive 51 print issues and online access

$199.00 per year

only $3.90 per issue

[Learn more](/nature/subscribe)Rent or buy this article

Prices vary by article type

from$1.95

to$39.95

[Learn more](//www.nature.com/articles/nature14541.epdf?no_publisher_access=1&r3_referer=nature)Prices may be subject to local taxes which are calculated during checkout

### Additional access options:

* [Log in](https://idp.nature.com/authorize/natureuser?client_id=grover&redirect_uri=https%3A%2F%2Fwww.nature.com%2Farticles%2Fnature14541)
* [Learn about institutional subscriptions](https://www.springernature.com/gp/librarians/licensing/license-options)
* [Read our FAQs](https://support.nature.com/en/support/home)
* [Contact customer support](https://www.springernature.com/gp/contact)

**Figure 1: Bayesian inference.**![](//media.springernature.com/m312/springer-static/image/art%3A10.1038%2Fnature14541/MediaObjects/41586_2015_BFnature14541_Fig1_HTML.jpg)**Figure 2: Probabilistic programming.**![](//media.springernature.com/m312/springer-static/image/art%3A10.1038%2Fnature14541/MediaObjects/41586_2015_BFnature14541_Fig2_HTML.jpg)**Figure 3: Bayesian optimization.**![](//media.springernature.com/m312/springer-static/image/art%3A10.1038%2Fnature14541/MediaObjects/41586_2015_BFnature14541_Fig3_HTML.jpg)**Figure 4: The Automatic Statistician.**![](//media.springernature.com/m312/springer-static/image/art%3A10.1038%2Fnature14541/MediaObjects/41586_2015_BFnature14541_Fig4_HTML.jpg)

References
----------

1. Russell, S. & Norvig, P. *Artificial Intelligence: a Modern Approach* (Prentice–Hall, 1995).

[MATH](http://www.emis.de/MATH-item?0835.68093)
 [Google Scholar](http://scholar.google.com/scholar_lookup?&title=Artificial%20Intelligence%3A%20a%20Modern%20Approach&publication_year=1995&author=Russell%2CS&author=Norvig%2CP)
2. Thrun, S., Burgard, W. & Fox, D. *Probabilistic Robotics* (MIT Press, 2006).

[MATH](http://www.emis.de/MATH-item?1081.68703)
 [Google Scholar](http://scholar.google.com/scholar_lookup?&title=Probabilistic%20Robotics&publication_year=2006&author=Thrun%2CS&author=Burgard%2CW&author=Fox%2CD)
3. Bishop, C. M. *Pattern Recognition and Machine Learning* (Springer, 2006).

[MATH](http://www.emis.de/MATH-item?1107.68072)
 [Google Scholar](http://scholar.google.com/scholar_lookup?&title=Pattern%20Recognition%20and%20Machine%20Learning&publication_year=2006&author=Bishop%2CCM)
4. Murphy, K. P. *Machine Learning: A Probabilistic Perspective* (MIT Press, 2012).

[MATH](http://www.emis.de/MATH-item?1295.68003)
 [Google Scholar](http://scholar.google.com/scholar_lookup?&title=Machine%20Learning%3A%20A%20Probabilistic%20Perspective&publication_year=2012&author=Murphy%2CKP)
5. Hinton, G. et al. Deep neural networks for acoustic modeling in speech recognition: the shared views of four research groups. *IEEE Signal Process. Mag.* **29**, 82–97 (2012).

[ADS](http://adsabs.harvard.edu/cgi-bin/nph-data_query?link_type=ABSTRACT&bibcode=2012ISPM...29...82H)
 [Google Scholar](http://scholar.google.com/scholar_lookup?&title=Deep%20neural%20networks%20for%20acoustic%20modeling%20in%20speech%20recognition%3A%20the%20shared%20views%20of%20four%20research%20groups&journal=IEEE%20Signal%20Process.%20Mag.&volume=29&pages=82-97&publication_year=2012&author=Hinton%2CG)
6. Krizhevsky, A., Sutskever, I. & Hinton, G. E. ImageNet classification with deep convolutional neural networks. In *Proc. Advances in Neural Information Processing Systems 25* 1097–1105 (2012).

[Google Scholar](http://scholar.google.com/scholar_lookup?&title=Proc.%20Advances%20in%20Neural%20Information%20Processing%20Systems%2025&pages=1097-1105&publication_year=2012&author=Krizhevsky%2CA&author=Sutskever%2CI&author=Hinton%2CGE)
7. Sermanet, P. et al. Overfeat: integrated recognition, localization and detection using convolutional networks. In *Proc. International Conference on Learning Representations* <http://arxiv.org/abs/1312.6229> (2014).

[Google Scholar](http://scholar.google.com/scholar_lookup?&title=Proc.%20International%20Conference%20on%20Learning%20Representations&publication_year=2014&author=Sermanet%2CP)
8. Bengio, Y., Ducharme, R., Vincent, P. & Janvin, C. A neural probabilistic language model. *J. Mach. Learn. Res.* **3**, 1137–1155 (2003).

[MATH](http://www.emis.de/MATH-item?1061.68157)
 [Google Scholar](http://scholar.google.com/scholar_lookup?&title=A%20neural%20probabilistic%20language%20model&journal=J.%20Mach.%20Learn.%20Res.&volume=3&pages=1137-1155&publication_year=2003&author=Bengio%2CY&author=Ducharme%2CR&author=Vincent%2CP&author=Janvin%2CC)
9. Ghahramani, Z. Bayesian nonparametrics and the probabilistic approach to modelling. *Phil. Trans. R. Soc. A* **371**, 20110553 (2013). **A review of Bayesian non-parametric modelling written for a general scientific audience.**

[ADS](http://adsabs.harvard.edu/cgi-bin/nph-data_query?link_type=ABSTRACT&bibcode=2012RSPTA.37110553G)
 [PubMed](http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?cmd=Retrieve&db=PubMed&dopt=Abstract&list_uids=23277609)
 [MATH](http://www.emis.de/MATH-item?1353.62033)
 [Google Scholar](http://scholar.google.com/scholar_lookup?&title=Bayesian%20nonparametrics%20and%20the%20probabilistic%20approach%20to%20modelling&journal=Phil.%20Trans.%20R.%20Soc.%20A&volume=371&publication_year=2013&author=Ghahramani%2CZ)
10. Jaynes, E. T. *Probability Theory: the Logic of Science* (Cambridge Univ. Press, 2003).

[MATH](http://www.emis.de/MATH-item?1045.62001)
 [Google Scholar](http://scholar.google.com/scholar_lookup?&title=Probability%20Theory%3A%20the%20Logic%20of%20Science&publication_year=2003&author=Jaynes%2CET)
11. Koller, D. & Friedman, N. *Probabilistic Graphical Models: Principles and Techniques* (MIT Press, 2009). **This is an encyclopaedic text on probabilistic graphical models spanning many key topics.**

[MATH](http://www.emis.de/MATH-item?1183.68483)
 [Google Scholar](http://scholar.google.com/scholar_lookup?&title=Probabilistic%20Graphical%20Models%3A%20Principles%20and%20Techniques&publication_year=2009&author=Koller%2CD&author=Friedman%2CN)
12. Cox, R. T. *The Algebra of Probable Inference* (Johns Hopkins Univ. Press, 1961).

[MATH](http://www.emis.de/MATH-item?0103.11404)
 [Google Scholar](http://scholar.google.com/scholar_lookup?&title=The%20Algebra%20of%20Probable%20Inference&publication_year=1961&author=Cox%2CRT)
13. Van Horn, K. S. Constructing a logic of plausible inference: a guide to Cox's theorem. *Int. J. Approx. Reason.* **34**, 3–24 (2003).

[MathSciNet](http://www.ams.org/mathscinet-getitem?mr=2017777)
 [MATH](http://www.emis.de/MATH-item?1069.68104)
 [Google Scholar](http://scholar.google.com/scholar_lookup?&title=Constructing%20a%20logic%20of%20plausible%20inference%3A%20a%20guide%20to%20Cox%27s%20theorem&journal=Int.%20J.%20Approx.%20Reason.&volume=34&pages=3-24&publication_year=2003&author=Van%20Horn%2CKS)
14. De Finetti, B. La prévision: ses lois logiques, ses sources subjectives. In *Annales de l'institut Henri Poincaré* [in French] **7**, 1–68 (1937).

[Google Scholar](http://scholar.google.com/scholar_lookup?&title=Annales%20de%20l%27institut%20Henri%20Poincar%C3%A9&pages=1-68&publication_year=1937&author=De%20Finetti%2CB)
15. Knill, D. & Richards, W. Perception as Bayesian inference (Cambridge Univ.Press, 1996).
16. Griffiths, T. L. & Tenenbaum, J. B. Optimal predictions in everyday cognition. *Psychol. Sci.* **17**, 767–773 (2006).

[PubMed](http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?cmd=Retrieve&db=PubMed&dopt=Abstract&list_uids=16984293)
 [Google Scholar](http://scholar.google.com/scholar_lookup?&title=Optimal%20predictions%20in%20everyday%20cognition&journal=Psychol.%20Sci.&volume=17&pages=767-773&publication_year=2006&author=Griffiths%2CTL&author=Tenenbaum%2CJB)
17. Wolpert, D. M., Ghahramani, Z. & Jordan, M. I. An internal model for sensorimotor integration. *Science* **269**, 1880–1882 (1995).

[ADS](http://adsabs.harvard.edu/cgi-bin/nph-data_query?link_type=ABSTRACT&bibcode=1995Sci...269.1880W)
 [CAS](/articles/cas-redirect/1:CAS:528:DyaK2MXotlKqtbw%3D)
 [PubMed](http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?cmd=Retrieve&db=PubMed&dopt=Abstract&list_uids=7569931)
 [Google Scholar](http://scholar.google.com/scholar_lookup?&title=An%20internal%20model%20for%20sensorimotor%20integration&journal=Science&volume=269&pages=1880-1882&publication_year=1995&author=Wolpert%2CDM&author=Ghahramani%2CZ&author=Jordan%2CMI)
18. Tenenbaum, J. B., Kemp, C., Griffiths, T. L. & Goodman, N. D. How to grow a mind: statistics, structure, and abstraction. *Science* **331**, 1279–1285 (2011).

[ADS](http://adsabs.harvard.edu/cgi-bin/nph-data_query?link_type=ABSTRACT&bibcode=2011Sci...331.1279T)
 [MathSciNet](http://www.ams.org/mathscinet-getitem?mr=2807660)
 [CAS](/articles/cas-redirect/1:CAS:528:DC%2BC3MXivVWkt7o%3D)
 [PubMed](http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?cmd=Retrieve&db=PubMed&dopt=Abstract&list_uids=21393536)
 [MATH](http://www.emis.de/MATH-item?1226.68087)
 [Google Scholar](http://scholar.google.com/scholar_lookup?&title=How%20to%20grow%20a%20mind%3A%20statistics%2C%20structure%2C%20and%20abstraction&journal=Science&volume=331&pages=1279-1285&publication_year=2011&author=Tenenbaum%2CJB&author=Kemp%2CC&author=Griffiths%2CTL&author=Goodman%2CND)
19. Marcus, G. F. & Davis, E. How robust are probabilistic models of higher-level cognition? *Psychol. Sci.* **24**, 2351–2360 (2013).

[PubMed](http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?cmd=Retrieve&db=PubMed&dopt=Abstract&list_uids=24084039)
 [Google Scholar](http://scholar.google.com/scholar_lookup?&title=How%20robust%20are%20probabilistic%20models%20of%20higher-level%20cognition%3F&journal=Psychol.%20Sci.&volume=24&pages=2351-2360&publication_year=2013&author=Marcus%2CGF&author=Davis%2CE)
20. Goodman, N. D. et al. Relevant and robust a response to Marcus and Davis (2013). *Psychol. Sci.* **26**, 539–541 (2015).

[PubMed](http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?cmd=Retrieve&db=PubMed&dopt=Abstract&list_uids=25749699)
 [Google Scholar](http://scholar.google.com/scholar_lookup?&title=Relevant%20and%20robust%20a%20response%20to%20Marcus%20and%20Davis%20%282013%29&journal=Psychol.%20Sci.&volume=26&pages=539-541&publication_year=2015&author=Goodman%2CND)
21. Doya, K., Ishii, S., Pouget, A. & Rao, R. P. N. *Bayesian Brain: Probabilistic Approaches to Neural Coding* (MIT Press, 2007).

[MATH](http://www.emis.de/MATH-item?1137.91015)
 [Google Scholar](http://scholar.google.com/scholar_lookup?&title=Bayesian%20Brain%3A%20Probabilistic%20Approaches%20to%20Neural%20Coding&publication_year=2007&author=Doya%2CK&author=Ishii%2CS&author=Pouget%2CA&author=Rao%2CRPN)
22. Deneve, S. Bayesian spiking neurons I: inference. *Neural Comput.* **20**, 91–117 (2008).

[MathSciNet](http://www.ams.org/mathscinet-getitem?mr=2368392)
 [PubMed](http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?cmd=Retrieve&db=PubMed&dopt=Abstract&list_uids=18045002)
 [MATH](http://www.emis.de/MATH-item?1148.92004)
 [Google Scholar](http://scholar.google.com/scholar_lookup?&title=Bayesian%20spiking%20neurons%20I%3A%20inference&journal=Neural%20Comput.&volume=20&pages=91-117&publication_year=2008&author=Deneve%2CS)
23. Neal, R. M. *Probabilistic Inference Using Markov Chain Monte Carlo Methods*. Report No. CRG-TR-93–1 [http://www.cs.toronto.edu/∼radford/review.abstract.html](http://www.cs.toronto.edu/~radford/review.abstract.html) (Univ. Toronto, 1993).

[Google Scholar](http://scholar.google.com/scholar_lookup?&title=Probabilistic%20Inference%20Using%20Markov%20Chain%20Monte%20Carlo%20Methods&publication_year=1993&author=Neal%2CRM)
24. Jordan, M., Ghahramani, Z., Jaakkola, T. & Saul, L. An introduction to variational methods in graphical models. *Mach. Learn.* **37**, 183–233 (1999).

[MATH](http://www.emis.de/MATH-item?0945.68164)
 [Google Scholar](http://scholar.google.com/scholar_lookup?&title=An%20introduction%20to%20variational%20methods%20in%20graphical%20models&journal=Mach.%20Learn.&volume=37&pages=183-233&publication_year=1999&author=Jordan%2CM&author=Ghahramani%2CZ&author=Jaakkola%2CT&author=Saul%2CL)
25. Doucet, A., de Freitas, J. F. G. & Gordon, N. J. *Sequential Monte Carlo Methods in Practice* (Springer, 2000).

[MATH](http://www.emis.de/MATH-item?0967.00022)
 [Google Scholar](http://scholar.google.com/scholar_lookup?&title=Sequential%20Monte%20Carlo%20Methods%20in%20Practice&publication_year=2000&author=Doucet%2CA&author=de%20Freitas%2CJFG&author=Gordon%2CNJ)
26. Minka, T. P. Expectation propagation for approximate Bayesian inference. In *Proc. Uncertainty in Artificial Intelligence* **17** 362–369 (2001).

[Google Scholar](http://scholar.google.com/scholar_lookup?&title=Proc.%20Uncertainty%20in%20Artificial%20Intelligence&pages=362-369&publication_year=2001&author=Minka%2CTP)
27. Neal, R. M. In *Handbook of Markov Chain Monte Carlo* (eds Brooks, S., Gelman, A., Jones, G. & Meng, X.-L.) (Chapman & Hall/CRC, 2010).

[Google Scholar](http://scholar.google.com/scholar_lookup?&title=Handbook%20of%20Markov%20Chain%20Monte%20Carlo&publication_year=2010&author=Neal%2CRM)
28. Girolami, M. & Calderhead, B. Riemann manifold Langevin and Hamiltonian Monte Carlo methods. *J. R. Stat. Soc. Series B Stat. Methodol.* **73**, 123–214 (2011).

[MathSciNet](http://www.ams.org/mathscinet-getitem?mr=2814492)
 [MATH](http://www.emis.de/MATH-item?1411.62071)
 [Google Scholar](http://scholar.google.com/scholar_lookup?&title=Riemann%20manifold%20Langevin%20and%20Hamiltonian%20Monte%20Carlo%20methods&journal=J.%20R.%20Stat.%20Soc.%20Series%20B%20Stat.%20Methodol.&volume=73&pages=123-214&publication_year=2011&author=Girolami%2CM&author=Calderhead%2CB)
29. Sutskever, I., Vinyals, O. & Le, Q. V. Sequence to sequence learning with neural networks. In *Proc. Advances in Neural Information Processing Systems 27*, 3104–3112 (2014).

[Google Scholar](http://scholar.google.com/scholar_lookup?&title=Proc.%20Advances%20in%20Neural%20Information%20Processing%20Systems%2027&pages=3104-3112&publication_year=2014&author=Sutskever%2CI&author=Vinyals%2CO&author=Le%2CQV)
30. Neal, R. M. in *Maximum Entropy and Bayesian Methods* 197–211 (Springer, 1992).

[Google Scholar](http://scholar.google.com/scholar_lookup?&title=Maximum%20Entropy%20and%20Bayesian%20Methods&pages=197-211&publication_year=1992&author=Neal%2CRM)
31. Orbanz, P. & Teh, Y. W. in *Encyclopedia of Machine Learning* 81–89 (Springer, 2010).

[Google Scholar](http://scholar.google.com/scholar_lookup?&title=Encyclopedia%20of%20Machine%20Learning&pages=81-89&publication_year=2010&author=Orbanz%2CP&author=Teh%2CYW)
32. Hjort, N., Holmes, C., Müller, P. & Walker, S. (eds). *Bayesian Nonparametrics* (Cambridge Univ. Press, 2010).

[MATH](http://www.emis.de/MATH-item?1192.62080)
 [Google Scholar](http://scholar.google.com/scholar_lookup?&title=Bayesian%20Nonparametrics&publication_year=2010&author=Hjort%2CN&author=Holmes%2CC&author=M%C3%BCller%2CP&author=Walker%2CS)
33. Rasmussen, C. E. & Williams, C. K. I. *Gaussian Processes for Machine Learning* (MIT Press, 2006). **This is a classic monograph on Gaussian processes, relating them to kernel methods and other areas of machine learning.**

[MATH](http://www.emis.de/MATH-item?1177.68165)
 [Google Scholar](http://scholar.google.com/scholar_lookup?&title=Gaussian%20Processes%20for%20Machine%20Learning&publication_year=2006&author=Rasmussen%2CCE&author=Williams%2CCKI)
34. Lu, C. & Tang, X. Surpassing human-level face verification performance on LFW with GaussianFace. In *Proc. 29th AAAI Conference on Artificial Intelligence* <http://arxiv.org/abs/1404.3840> (2015).

[Google Scholar](http://scholar.google.com/scholar_lookup?&title=Proc.%2029th%20AAAI%20Conference%20on%20Artificial%20Intelligence&publication_year=2015&author=Lu%2CC&author=Tang%2CX)
35. Ferguson, T. S. A Bayesian analysis of some nonparametric problems. *Ann. Stat.* **1**, 209–230 (1973).

[MathSciNet](http://www.ams.org/mathscinet-getitem?mr=350949)
 [MATH](http://www.emis.de/MATH-item?0255.62037)
 [Google Scholar](http://scholar.google.com/scholar_lookup?&title=A%20Bayesian%20analysis%20of%20some%20nonparametric%20problems&journal=Ann.%20Stat.&volume=1&pages=209-230&publication_year=1973&author=Ferguson%2CTS)
36. Teh, Y. W., Jordan, M. I., Beal, M. J. & Blei, D. M. Hierarchical Dirichlet processes. *J. Am. Stat. Assoc.* **101**, 1566–1581 (2006).

[MathSciNet](http://www.ams.org/mathscinet-getitem?mr=2279480)
 [CAS](/articles/cas-redirect/1:CAS:528:DC%2BD2sXms1OjtQ%3D%3D)
 [MATH](http://www.emis.de/MATH-item?1171.62349)
 [Google Scholar](http://scholar.google.com/scholar_lookup?&title=Hierarchical%20Dirichlet%20processes&journal=J.%20Am.%20Stat.%20Assoc.&volume=101&pages=1566-1581&publication_year=2006&author=Teh%2CYW&author=Jordan%2CMI&author=Beal%2CMJ&author=Blei%2CDM)
37. Kemp, C., Tenenbaum, J. B., Griffiths, T. L., Yamada, T. & Ueda, N. Learning systems of concepts with an infinite relational model. In *Proc. 21st National Conference on Artificial Intelligence* 381–388 (2006).

[Google Scholar](http://scholar.google.com/scholar_lookup?&title=Proc.%2021st%20National%20Conference%20on%20Artificial%20Intelligence&pages=381-388&publication_year=2006&author=Kemp%2CC&author=Tenenbaum%2CJB&author=Griffiths%2CTL&author=Yamada%2CT&author=Ueda%2CN)
38. Medvedovic, M. & Sivaganesan, S. Bayesian infinite mixture model based clustering of gene expression profiles. *Bioinformatics* **18**, 1194–1206 (2002).

[CAS](/articles/cas-redirect/1:CAS:528:DC%2BD38XmvVCqsb4%3D)
 [PubMed](http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?cmd=Retrieve&db=PubMed&dopt=Abstract&list_uids=12217911)
 [Google Scholar](http://scholar.google.com/scholar_lookup?&title=Bayesian%20infinite%20mixture%20model%20based%20clustering%20of%20gene%20expression%20profiles&journal=Bioinformatics&volume=18&pages=1194-1206&publication_year=2002&author=Medvedovic%2CM&author=Sivaganesan%2CS)
39. Rasmussen, C. E., De la Cruz, B. J., Ghahramani, Z. & Wild, D. L. Modeling and visualizing uncertainty in gene expression clusters using Dirichlet process mixtures. *Trans. Comput. Biol. Bioinform.* **6**, 615–628 (2009).

[CAS](/articles/cas-redirect/1:CAS:528:DC%2BD1MXhsFOlsrjL)
 [Google Scholar](http://scholar.google.com/scholar_lookup?&title=Modeling%20and%20visualizing%20uncertainty%20in%20gene%20expression%20clusters%20using%20Dirichlet%20process%20mixtures&journal=Trans.%20Comput.%20Biol.%20Bioinform.&volume=6&pages=615-628&publication_year=2009&author=Rasmussen%2CCE&author=De%20la%20Cruz%2CBJ&author=Ghahramani%2CZ&author=Wild%2CDL)
40. Griffiths, T. L. & Ghahramani, Z. The Indian buffet process: an introduction and review. *J. Mach. Learn. Res.* **12**, 1185–1224 (2011). **This article introduced a new class of Bayesian non-parametric models for latent feature modelling.**

[MathSciNet](http://www.ams.org/mathscinet-getitem?mr=2804598)
 [MATH](http://www.emis.de/MATH-item?1280.62038)
 [Google Scholar](http://scholar.google.com/scholar_lookup?&title=The%20Indian%20buffet%20process%3A%20an%20introduction%20and%20review&journal=J.%20Mach.%20Learn.%20Res.&volume=12&pages=1185-1224&publication_year=2011&author=Griffiths%2CTL&author=Ghahramani%2CZ)
41. Adams, R. P., Wallach, H. & Ghahramani, Z. Learning the structure of deep sparse graphical models. In *Proc. 13th International Conference on Artificial Intelligence and Statistics* (eds Teh, Y. W. & Titterington, M.) 1–8 (2010).

[Google Scholar](http://scholar.google.com/scholar_lookup?&title=Proc.%2013th%20International%20Conference%20on%20Artificial%20Intelligence%20and%20Statistics&pages=1-8&publication_year=2010&author=Adams%2CRP&author=Wallach%2CH&author=Ghahramani%2CZ)
42. Miller, K., Jordan, M. I. & Griffiths, T. L. Nonparametric latent feature models for link prediction. In *Proc. Advances in Neural Information Processing Systems* 1276–1284 (2009).

[Google Scholar](http://scholar.google.com/scholar_lookup?&title=Proc.%20Advances%20in%20Neural%20Information%20Processing%20Systems&pages=1276-1284&publication_year=2009&author=Miller%2CK&author=Jordan%2CMI&author=Griffiths%2CTL)
43. Hinton, G. E., McClelland, J. L. & Rumelhart, D. E. in *Parallel Distributed Processing: Explorations in the Microstructure of Cognition: Foundations* 77–109 (MIT Press, 1986).

[Google Scholar](http://scholar.google.com/scholar_lookup?&title=Parallel%20Distributed%20Processing%3A%20Explorations%20in%20the%20Microstructure%20of%20Cognition%3A%20Foundations&pages=77-109&publication_year=1986&author=Hinton%2CGE&author=McClelland%2CJL&author=Rumelhart%2CDE)
44. Neal, R. M. *Bayesian Learning for Neural Networks* (Springer, 1996). **This text derived MCMC-based Bayesian inference in neural networks and drew important links to Gaussian processes.**

[MATH](http://www.emis.de/MATH-item?0888.62021)
 [Google Scholar](http://scholar.google.com/scholar_lookup?&title=Bayesian%20Learning%20for%20Neural%20Networks&publication_year=1996&author=Neal%2CRM)
45. Koller, D., McAllester, D. & Pfeffer, A. Effective Bayesian inference for stochastic programs. In *Proc. 14th National Conference on Artificial Intelligence* 740–747 (1997).

[Google Scholar](http://scholar.google.com/scholar_lookup?&title=Proc.%2014th%20National%20Conference%20on%20Artificial%20Intelligence&pages=740-747&publication_year=1997&author=Koller%2CD&author=McAllester%2CD&author=Pfeffer%2CA)
46. Goodman, N. D. & Stuhlmüller, A. *The Design and Implementation of Probabilistic Programming Languages*. Available at <http://dippl.org> (2015).

[Google Scholar](http://scholar.google.com/scholar_lookup?&title=The%20Design%20and%20Implementation%20of%20Probabilistic%20Programming%20Languages&publication_year=2015&author=Goodman%2CND&author=Stuhlm%C3%BCller%2CA)
47. Pfeffer, A. *Practical Probabilistic Programming* (Manning, 2015).

[Google Scholar](http://scholar.google.com/scholar_lookup?&title=Practical%20Probabilistic%20Programming&publication_year=2015&author=Pfeffer%2CA)
48. Freer, C., Roy, D. & Tenenbaum, J. B. in *Turing's Legacy* (ed. Downey, R.) 195–252 (2014).

[Google Scholar](http://scholar.google.com/scholar_lookup?&title=Turing%27s%20Legacy&pages=195-252&publication_year=2014&author=Freer%2CC&author=Roy%2CD&author=Tenenbaum%2CJB)
49. Marjoram, P., Molitor, J., Plagnol, V. & Tavaré, S. Markov chain Monte Carlo without likelihoods. *Proc. Natl Acad. Sci. USA* **100**, 15324–15328 (2003).

[ADS](http://adsabs.harvard.edu/cgi-bin/nph-data_query?link_type=ABSTRACT&bibcode=2003PNAS..10015324M)
 [CAS](/articles/cas-redirect/1:CAS:528:DC%2BD2cXhtVektw%3D%3D)
 [PubMed](http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?cmd=Retrieve&db=PubMed&dopt=Abstract&list_uids=14663152)
 [Google Scholar](http://scholar.google.com/scholar_lookup?&title=Markov%20chain%20Monte%20Carlo%20without%20likelihoods&journal=Proc.%20Natl%20Acad.%20Sci.%20USA&volume=100&pages=15324-15328&publication_year=2003&author=Marjoram%2CP&author=Molitor%2CJ&author=Plagnol%2CV&author=Tavar%C3%A9%2CS)
50. Mansinghka, V., Kulkarni, T. D., Perov, Y. N. & Tenenbaum, J. Approximate Bayesian image interpretation using generative probabilistic graphics programs. In *Proc. Advances in Neural Information Processing Systems 26* 1520–1528 (2013).

[Google Scholar](http://scholar.google.com/scholar_lookup?&title=Proc.%20Advances%20in%20Neural%20Information%20Processing%20Systems%2026&pages=1520-1528&publication_year=2013&author=Mansinghka%2CV&author=Kulkarni%2CTD&author=Perov%2CYN&author=Tenenbaum%2CJ)
51. Bishop, C. M. Model-based machine learning. *Phil. Trans. R. Soc. A* **371**, 20120222 (2013). **This article is a very clear tutorial exposition of probabilistic modelling.**

[ADS](http://adsabs.harvard.edu/cgi-bin/nph-data_query?link_type=ABSTRACT&bibcode=2012RSPTA.37120222B)
 [MathSciNet](http://www.ams.org/mathscinet-getitem?mr=3005666)
 [PubMed](http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?cmd=Retrieve&db=PubMed&dopt=Abstract&list_uids=23277612)
 [MATH](http://www.emis.de/MATH-item?1353.68229)
 [Google Scholar](http://scholar.google.com/scholar_lookup?&title=Model-based%20machine%20learning&journal=Phil.%20Trans.%20R.%20Soc.%20A&volume=371&publication_year=2013&author=Bishop%2CCM)
52. Lunn, D. J., Thomas, A., Best, N. & Spiegelhalter, D. WinBUGS — a Bayesian modelling framework: concepts, structure, and extensibility. *Stat. Comput.* **10**, 325–337 (2000). **This reports an early probabilistic programming framework widely used in statistics.**

[Google Scholar](http://scholar.google.com/scholar_lookup?&title=WinBUGS%20%E2%80%94%20a%20Bayesian%20modelling%20framework%3A%20concepts%2C%20structure%2C%20and%20extensibility&journal=Stat.%20Comput.&volume=10&pages=325-337&publication_year=2000&author=Lunn%2CDJ&author=Thomas%2CA&author=Best%2CN&author=Spiegelhalter%2CD)
53. Stan Development Team. *Stan Modeling Language Users Guide and Reference Manual, Version 2.5.0*. <http://mc-stan.org/> (2014).
54. Fischer, B. & Schumann, J. AutoBayes: a system for generating data analysis programs from statistical models. *J. Funct. Program.* **13**, 483–508 (2003).

[MathSciNet](http://www.ams.org/mathscinet-getitem?mr=1996539)
 [MATH](http://www.emis.de/MATH-item?1037.68044)
 [Google Scholar](http://scholar.google.com/scholar_lookup?&title=AutoBayes%3A%20a%20system%20for%20generating%20data%20analysis%20programs%20from%20statistical%20models&journal=J.%20Funct.%20Program.&volume=13&pages=483-508&publication_year=2003&author=Fischer%2CB&author=Schumann%2CJ)
55. Minka, T. P., Winn, J. M., Guiver, J. P. & Knowles, D. A. *Infer.NET 2.4*. <http://research.microsoft.com/infernet> (Microsoft Research, 2010).

[Google Scholar](http://scholar.google.com/scholar_lookup?&title=Infer.NET%202.4&publication_year=2010&author=Minka%2CTP&author=Winn%2CJM&author=Guiver%2CJP&author=Knowles%2CDA)
56. Wingate, D., Stuhlmüller, A. & Goodman, N. D. Lightweight implementations of probabilistic programming languages via transformational compilation. In *Proc. International Conference on Artificial Intelligence and Statistics* 770–778 (2011).

[Google Scholar](http://scholar.google.com/scholar_lookup?&title=Proc.%20International%20Conference%20on%20Artificial%20Intelligence%20and%20Statistics&pages=770-778&publication_year=2011&author=Wingate%2CD&author=Stuhlm%C3%BCller%2CA&author=Goodman%2CND)
57. Pfeffer, A. IBAL: a probabilistic rational programming language. In *Proc. International Joint Conference on Artificial Intelligence* 733–740 (2001).

[Google Scholar](http://scholar.google.com/scholar_lookup?&title=Proc.%20International%20Joint%20Conference%20on%20Artificial%20Intelligence&pages=733-740&publication_year=2001&author=Pfeffer%2CA)
58. Milch, B. et al. BLOG: probabilistic models with unknown objects. In *Proc. 19th International Joint Conference on Artificial Intelligence* 1352–1359 (2005).

[Google Scholar](http://scholar.google.com/scholar_lookup?&title=Proc.%2019th%20International%20Joint%20Conference%20on%20Artificial%20Intelligence&pages=1352-1359&publication_year=2005&author=Milch%2CB)
59. Goodman, N., Mansinghka, V., Roy, D., Bonawitz, K. & Tenenbaum, J. Church: a language for generative models. In *Proc. Uncertainty in Artificial Intelligence 22* 23 (2008). **This is an influential paper introducing the Turing-complete probabilistic programming language Church.**

[Google Scholar](http://scholar.google.com/scholar_lookup?&title=Proc.%20Uncertainty%20in%20Artificial%20Intelligence%2022&publication_year=2008&author=Goodman%2CN&author=Mansinghka%2CV&author=Roy%2CD&author=Bonawitz%2CK&author=Tenenbaum%2CJ)
60. Pfeffer, A. *Figaro: An Object-Oriented Probabilistic Programming Language*. Tech. Rep. (Charles River Analytics, 2009).

[Google Scholar](http://scholar.google.com/scholar_lookup?&title=Figaro%3A%20An%20Object-Oriented%20Probabilistic%20Programming%20Language&publication_year=2009&author=Pfeffer%2CA)
61. Mansinghka, V., Selsam, D. & Perov, Y. Venture: a higher-order probabilistic programming platform with programmable inference. Preprint at <http://arxiv.org/abs/1404.0099> (2014).
62. Wood, F., van de Meent, J. W. & Mansinghka, V. A new approach to probabilistic programming inference. In *Proc. 17th International Conference on Artificial Intelligence and Statistics* 1024–1032 (2014).

[Google Scholar](http://scholar.google.com/scholar_lookup?&title=Proc.%2017th%20International%20Conference%20on%20Artificial%20Intelligence%20and%20Statistics&pages=1024-1032&publication_year=2014&author=Wood%2CF&author=van%20de%20Meent%2CJW&author=Mansinghka%2CV)
63. Li, L., Wu, Y. & Russell, S. J. *SWIFT: Compiled Inference for Probabilistic Programs*. Report No. UCB/EECS-2015–12 (Univ. California, Berkeley, 2015).

[Google Scholar](http://scholar.google.com/scholar_lookup?&title=SWIFT%3A%20Compiled%20Inference%20for%20Probabilistic%20Programs&publication_year=2015&author=Li%2CL&author=Wu%2CY&author=Russell%2CSJ)
64. Bergstra, J. et al. Theano: a CPU and GPU math expression compiler. In *Proc. 9th Python in Science Conference* <http://conference.scipy.org/proceedings/scipy2010/> (2010).

[Google Scholar](http://scholar.google.com/scholar_lookup?&title=Proc.%209th%20Python%20in%20Science%20Conference&publication_year=2010&author=Bergstra%2CJ)
65. Kushner, H. A new method of locating the maximum point of an arbitrary multipeak curve in the presence of noise. *J. Basic Eng.* **86**, 97–106 (1964).

[Google Scholar](http://scholar.google.com/scholar_lookup?&title=A%20new%20method%20of%20locating%20the%20maximum%20point%20of%20an%20arbitrary%20multipeak%20curve%20in%20the%20presence%20of%20noise&journal=J.%20Basic%20Eng.&volume=86&pages=97-106&publication_year=1964&author=Kushner%2CH)
66. Jones, D. R., Schonlau, M. & Welch, W. J. Efficient global optimization of expensive black-box functions. *J. Glob. Optim.* **13**, 455–492 (1998).

[MathSciNet](http://www.ams.org/mathscinet-getitem?mr=1673460)
 [MATH](http://www.emis.de/MATH-item?0917.90270)
 [Google Scholar](http://scholar.google.com/scholar_lookup?&title=Efficient%20global%20optimization%20of%20expensive%20black-box%20functions&journal=J.%20Glob.%20Optim.&volume=13&pages=455-492&publication_year=1998&author=Jones%2CDR&author=Schonlau%2CM&author=Welch%2CWJ)
67. Brochu, E., Cora, V. M. & de Freitas, N. A tutorial on Bayesian optimization of expensive cost functions, with application to active user modeling and hierarchical reinforcement learning. Preprint at <http://arXiv.org/abs/1012.2599> (2010).
68. Hennig, P. & Schuler, C. J. Entropy search for information-efficient global optimization. *J. Mach. Learn. Res.* **13**, 1809–1837 (2012).

[MathSciNet](http://www.ams.org/mathscinet-getitem?mr=2956343)
 [MATH](http://www.emis.de/MATH-item?1432.65073)
 [Google Scholar](http://scholar.google.com/scholar_lookup?&title=Entropy%20search%20for%20information-efficient%20global%20optimization&journal=J.%20Mach.%20Learn.%20Res.&volume=13&pages=1809-1837&publication_year=2012&author=Hennig%2CP&author=Schuler%2CCJ)
69. Hernández-Lobato, J. M., Hoffman, M. W. & Ghahramani, Z. Predictive entropy search for efficient global optimization of black-box functions. In *Proc. Advances in Neural Information Processing Systems* 918–926 (2014).

[Google Scholar](http://scholar.google.com/scholar_lookup?&title=Proc.%20Advances%20in%20Neural%20Information%20Processing%20Systems&pages=918-926&publication_year=2014&author=Hern%C3%A1ndez-Lobato%2CJM&author=Hoffman%2CMW&author=Ghahramani%2CZ)
70. Snoek, J., Larochelle, H. & Adams, R. P. Practical Bayesian optimization of machine learning algorithms. In *Proc. Advances in Neural Information Processing Systems* 2960–2968 (2012).

[Google Scholar](http://scholar.google.com/scholar_lookup?&title=Proc.%20Advances%20in%20Neural%20Information%20Processing%20Systems&pages=2960-2968&publication_year=2012&author=Snoek%2CJ&author=Larochelle%2CH&author=Adams%2CRP)
71. Thornton, C., Hutter, F., Hoos, H. H. & Leyton-Brown, K. Auto-WEKA: combined selection and hyperparameter optimization of classification algorithms. In *Proc. 19th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining* 847–855 (2013).

[Google Scholar](http://scholar.google.com/scholar_lookup?&title=Proc.%2019th%20ACM%20SIGKDD%20International%20Conference%20on%20Knowledge%20Discovery%20and%20Data%20Mining&pages=847-855&publication_year=2013&author=Thornton%2CC&author=Hutter%2CF&author=Hoos%2CHH&author=Leyton-Brown%2CK)
72. Robbins, H. Some aspects of the sequential design of experiments. *Bull. Amer. Math. Soc.* **55**, 527–535 (1952).

[MathSciNet](http://www.ams.org/mathscinet-getitem?mr=50246)
 [MATH](http://www.emis.de/MATH-item?0049.37009)
 [Google Scholar](http://scholar.google.com/scholar_lookup?&title=Some%20aspects%20of%20the%20sequential%20design%20of%20experiments&journal=Bull.%20Amer.%20Math.%20Soc.&volume=55&pages=527-535&publication_year=1952&author=Robbins%2CH)
73. Deisenroth, M. P. & Rasmussen, C. E. PILCO: a model-based and data-efficient approach to policy search. In *Proc. 28th International Conference on Machine Learning* 465–472 (2011).

[Google Scholar](http://scholar.google.com/scholar_lookup?&title=Proc.%2028th%20International%20Conference%20on%20Machine%20Learning&pages=465-472&publication_year=2011&author=Deisenroth%2CMP&author=Rasmussen%2CCE)
74. Poupart, P. in *Encyclopedia of Machine Learning* 90–93 (Springer, 2010).

[Google Scholar](http://scholar.google.com/scholar_lookup?&title=Encyclopedia%20of%20Machine%20Learning&pages=90-93&publication_year=2010&author=Poupart%2CP)
75. Diaconis, P. in *Statistical Decision Theory and Related Topics IV* 163–175 (Springer, 1988).

[Google Scholar](http://scholar.google.com/scholar_lookup?&title=Statistical%20Decision%20Theory%20and%20Related%20Topics%20IV&pages=163-175&publication_year=1988&author=Diaconis%2CP)
76. O'Hagan, A. Bayes-Hermite quadrature. *J. Statist. Plann. Inference* **29**, 245–260 (1991).

[MathSciNet](http://www.ams.org/mathscinet-getitem?mr=1144171)
 [MATH](http://www.emis.de/MATH-item?0829.65024)
 [Google Scholar](http://scholar.google.com/scholar_lookup?&title=Bayes-Hermite%20quadrature&journal=J.%20Statist.%20Plann.%20Inference&volume=29&pages=245-260&publication_year=1991&author=O%27Hagan%2CA)
77. Shannon, C. & Weaver, W. *The Mathematical Theory of Communication* (Univ. Illinois Press, 1949).

[MATH](http://www.emis.de/MATH-item?0041.25804)
 [Google Scholar](http://scholar.google.com/scholar_lookup?&title=The%20Mathematical%20Theory%20of%20Communication&publication_year=1949&author=Shannon%2CC&author=Weaver%2CW)
78. MacKay, D. J. C. *Information Theory, Inference, and Learning Algorithms* (Cambridge Univ. Press, 2003).

[MATH](http://www.emis.de/MATH-item?1055.94001)
 [Google Scholar](http://scholar.google.com/scholar_lookup?&title=Information%20Theory%2C%20Inference%2C%20and%20Learning%20Algorithms&publication_year=2003&author=MacKay%2CDJC)
79. Wood, F., Gasthaus, J., Archambeau, C., James, L. & Teh, Y. W. The sequence memoizer. *Commun. ACM* **54**, 91–98 (2011). **This article derives a state-of-the-art data compression scheme based on Bayesian nonparametric models.**

[Google Scholar](http://scholar.google.com/scholar_lookup?&title=The%20sequence%20memoizer&journal=Commun.%20ACM&volume=54&pages=91-98&publication_year=2011&author=Wood%2CF&author=Gasthaus%2CJ&author=Archambeau%2CC&author=James%2CL&author=Teh%2CYW)
80. Steinruecken, C., Ghahramani, Z. & MacKay, D. J. C. Improving PPM with dynamic parameter updates. In *Proc. Data Compression Conference* (in the press).
81. Lloyd, J. R., Duvenaud, D., Grosse, R., Tenenbaum, J. B. & Ghahramani, Z. Automatic construction and natural-language description of nonparametric regression models. In *Proc. 28th AAAI Conference on Artificial Intelligence* Preprint at: <http://arxiv.org/abs/1402.4304> (2014). **Introduces the Automatic Statistician, translating learned probabilistic models into reports about data.**

[Google Scholar](http://scholar.google.com/scholar_lookup?&title=Proc.%2028th%20AAAI%20Conference%20on%20Artificial%20Intelligence&publication_year=2014&author=Lloyd%2CJR&author=Duvenaud%2CD&author=Grosse%2CR&author=Tenenbaum%2CJB&author=Ghahramani%2CZ)
82. Grosse, R. B., Salakhutdinov, R. & Tenenbaum, J. B. Exploiting compositionality to explore a large space of model structures. In *Proc. Conference on Uncertainty in Artificial Intelligence* 306–315 (2012).

[Google Scholar](http://scholar.google.com/scholar_lookup?&title=Proc.%20Conference%20on%20Uncertainty%20in%20Artificial%20Intelligence&pages=306-315&publication_year=2012&author=Grosse%2CRB&author=Salakhutdinov%2CR&author=Tenenbaum%2CJB)
83. Schmidt, M. & Lipson, H. Distilling free-form natural laws from experimental data. *Science* **324**, 81–85 (2009).

[ADS](http://adsabs.harvard.edu/cgi-bin/nph-data_query?link_type=ABSTRACT&bibcode=2009Sci...324...81S)
 [CAS](/articles/cas-redirect/1:CAS:528:DC%2BD1MXjvVajsbw%3D)
 [PubMed](http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?cmd=Retrieve&db=PubMed&dopt=Abstract&list_uids=19342586)
 [Google Scholar](http://scholar.google.com/scholar_lookup?&title=Distilling%20free-form%20natural%20laws%20from%20experimental%20data&journal=Science&volume=324&pages=81-85&publication_year=2009&author=Schmidt%2CM&author=Lipson%2CH)
84. Wolstenholme, D. E., O'Brien, C. M. & Nelder, J. A. GLIMPSE: a knowledge-based front end for statistical analysis. *Knowl. Base. Syst.* **1**, 173–178 (1988).

[Google Scholar](http://scholar.google.com/scholar_lookup?&title=GLIMPSE%3A%20a%20knowledge-based%20front%20end%20for%20statistical%20analysis&journal=Knowl.%20Base.%20Syst.&volume=1&pages=173-178&publication_year=1988&author=Wolstenholme%2CDE&author=O%27Brien%2CCM&author=Nelder%2CJA)
85. Hand, D. J. Patterns in statistical strategy. In *Artificial Intelligence and Statistics* (ed Gale, W. A.) (Addison-Wesley Longman, 1986).

[MATH](http://www.emis.de/MATH-item?0808.68083)
 [Google Scholar](http://scholar.google.com/scholar_lookup?&title=Artificial%20Intelligence%20and%20Statistics&publication_year=1986&author=Hand%2CDJ)
86. King, R. D. et al. Functional genomic hypothesis generation and experimentation by a robot scientist. *Nature* **427**, 247–252 (2004).

[ADS](http://adsabs.harvard.edu/cgi-bin/nph-data_query?link_type=ABSTRACT&bibcode=2004Natur.427..247K)
 [CAS](/articles/cas-redirect/1:CAS:528:DC%2BD2cXjtFKjuw%3D%3D)
 [PubMed](http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?cmd=Retrieve&db=PubMed&dopt=Abstract&list_uids=14724639)
 [Google Scholar](http://scholar.google.com/scholar_lookup?&title=Functional%20genomic%20hypothesis%20generation%20and%20experimentation%20by%20a%20robot%20scientist&journal=Nature&volume=427&pages=247-252&publication_year=2004&author=King%2CRD)
87. Welling, M. et al. Bayesian inference with big data: a snapshot from a workshop. *ISBA Bulletin* **21**, <https://bayesian.org/sites/default/files/fm/bulletins/1412.pdf> (2014).

[Google Scholar](http://scholar.google.com/scholar_lookup?&title=ISBA%20Bulletin&publication_year=2014&author=Welling%2CM)
88. Bakker, B. & Heskes, T. Task clustering and gating for Bayesian multitask learning. *J. Mach. Learn. Res.* **4**, 83–99 (2003).

[MATH](http://www.emis.de/MATH-item?1085.68612)
 [Google Scholar](http://scholar.google.com/scholar_lookup?&title=Task%20clustering%20and%20gating%20for%20Bayesian%20multitask%20learning&journal=J.%20Mach.%20Learn.%20Res.&volume=4&pages=83-99&publication_year=2003&author=Bakker%2CB&author=Heskes%2CT)
89. Houlsby, N., Hernández-Lobato, J. M., Huszár, F. & Ghahramani, Z. Collaborative Gaussian processes for preference learning. In *Proc. Advances in Neural Information Processing Systems 26* 2096–2104 (2012).

[Google Scholar](http://scholar.google.com/scholar_lookup?&title=Proc.%20Advances%20in%20Neural%20Information%20Processing%20Systems%2026&pages=2096-2104&publication_year=2012&author=Houlsby%2CN&author=Hern%C3%A1ndez-Lobato%2CJM&author=Husz%C3%A1r%2CF&author=Ghahramani%2CZ)
90. Russell, S. J. & Wefald, E. *Do the Right Thing: Studies in Limited Rationality* (MIT Press, 1991).

[Google Scholar](http://scholar.google.com/scholar_lookup?&title=Do%20the%20Right%20Thing%3A%20Studies%20in%20Limited%20Rationality&publication_year=1991&author=Russell%2CSJ&author=Wefald%2CE)
91. Jordan, M. I. On statistics, computation and scalability. *Bernoulli* **19**, 1378–1390 (2013).

[MathSciNet](http://www.ams.org/mathscinet-getitem?mr=3102908)
 [MATH](http://www.emis.de/MATH-item?1273.62030)
 [Google Scholar](http://scholar.google.com/scholar_lookup?&title=On%20statistics%2C%20computation%20and%20scalability&journal=Bernoulli&volume=19&pages=1378-1390&publication_year=2013&author=Jordan%2CMI)
92. Hoffman, M., Blei, D., Paisley, J. & Wang, C. Stochastic variational inference. *J. Mach. Learn. Res.* **14**, 1303–1347 (2013).

[MathSciNet](http://www.ams.org/mathscinet-getitem?mr=3081926)
 [MATH](http://www.emis.de/MATH-item?1317.68163)
 [Google Scholar](http://scholar.google.com/scholar_lookup?&title=Stochastic%20variational%20inference&journal=J.%20Mach.%20Learn.%20Res.&volume=14&pages=1303-1347&publication_year=2013&author=Hoffman%2CM&author=Blei%2CD&author=Paisley%2CJ&author=Wang%2CC)
93. Hensman, J., Fusi, N. & Lawrence, N. D. Gaussian processes for big data. In *Proc. Conference on Uncertainty in Artificial Intelligence* 244 (UAI, 2013).

[Google Scholar](http://scholar.google.com/scholar_lookup?&title=Proc.%20Conference%20on%20Uncertainty%20in%20Artificial%20Intelligence&publication_year=2013&author=Hensman%2CJ&author=Fusi%2CN&author=Lawrence%2CND)
94. Korattikara, A., Chen, Y. & Welling, M. Austerity in MCMC land: cutting the Metropolis-Hastings budget. In *Proc. 31th International Conference on Machine Learning* 181–189 (2014).

[Google Scholar](http://scholar.google.com/scholar_lookup?&title=Proc.%2031th%20International%20Conference%20on%20Machine%20Learning&pages=181-189&publication_year=2014&author=Korattikara%2CA&author=Chen%2CY&author=Welling%2CM)
95. Paige, B., Wood, F., Doucet, A. & Teh, Y. W. Asynchronous anytime sequential Monte Carlo. In *Proc. Advances in Neural Information Processing Systems 27* 3410–3418 (2014).

[Google Scholar](http://scholar.google.com/scholar_lookup?&title=Proc.%20Advances%20in%20Neural%20Information%20Processing%20Systems%2027&pages=3410-3418&publication_year=2014&author=Paige%2CB&author=Wood%2CF&author=Doucet%2CA&author=Teh%2CYW)
96. Jefferys, W. H. & Berger, J. O. Ockham's Razor and Bayesian Analysis. *Am. Sci.* **80**, 64–72 (1992).

[ADS](http://adsabs.harvard.edu/cgi-bin/nph-data_query?link_type=ABSTRACT&bibcode=1992AmSci..80...64J)
 [Google Scholar](http://scholar.google.com/scholar_lookup?&title=Ockham%27s%20Razor%20and%20Bayesian%20Analysis&journal=Am.%20Sci.&volume=80&pages=64-72&publication_year=1992&author=Jefferys%2CWH&author=Berger%2CJO)
97. Rasmussen, C. E. & Ghahramani, Z. Occam's Razor. In *Neural Information Processing Systems 13* (eds Leen, T. K., Dietterich, T. G., & Tresp, V.) 294–300 (2001).

[Google Scholar](http://scholar.google.com/scholar_lookup?&title=Neural%20Information%20Processing%20Systems%2013&pages=294-300&publication_year=2001&author=Rasmussen%2CCE&author=Ghahramani%2CZ)
98. Rabiner, L. R. A tutorial on hidden Markov models and selected applications in speech recognition. *Proc. IEEE* **77**, 257–286 (1989).

[Google Scholar](http://scholar.google.com/scholar_lookup?&title=A%20tutorial%20on%20hidden%20Markov%20models%20and%20selected%20applications%20in%20speech%20recognition&journal=Proc.%20IEEE&volume=77&pages=257-286&publication_year=1989&author=Rabiner%2CLR)
99. Gelman, A. et al. *Bayesian Data Analysis* 3rd edn (Chapman & Hall/CRC, 2013).

[Google Scholar](http://scholar.google.com/scholar_lookup?&title=Bayesian%20Data%20Analysis&publication_year=2013&author=Gelman%2CA)
100. Lloyd, J. R. & Ghahramani, Z. Statistical model criticism using kernel two sample tests <http://mlg.eng.cam.ac.uk/Lloyd/papers/kernel-model-checking.pdf> (2015).

[Download references](https://citation-needed.springer.com/v2/references/10.1038/nature14541?format=refman&flavour=references)

Acknowledgements
----------------

I acknowledge an EPSRC grant EP/I036575/1, the DARPA PPAML programme, a Google Focused Research Award for the Automatic Statistician and support from Microsoft Research. I am also grateful for valuable input from D. Duvenaud, H. Ge, M. W. Hoffman, J. R. Lloyd, A. Ścibior, M. Welling and D. Wolpert.

Author information
------------------

### Authors and Affiliations

1. Department of Engineering, University of Cambridge, Trumpington Street, Cambridge, CB2 1PZ, UK

Zoubin Ghahramani

Authors1. Zoubin Ghahramani[View author publications](/search?author=Zoubin%20Ghahramani)You can also search for this author in
 [PubMed](http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?cmd=search&term=Zoubin%20Ghahramani) [Google Scholar](http://scholar.google.co.uk/scholar?as_q=&num=10&btnG=Search+Scholar&as_epq=&as_oq=&as_eq=&as_occt=any&as_sauthors=%22Zoubin%20Ghahramani%22&as_publication=&as_ylo=&as_yhi=&as_allsubj=all&hl=en)
### Corresponding author

Correspondence to
 [Zoubin Ghahramani](mailto:zoubin@eng.cam.ac.uk).

Ethics declarations
-------------------

### Competing interests

I serve as an advisor to a number of companies working on machine learning and artificial intelligence.

Additional information
----------------------

Reprints and permissions information is available at [www.nature.com/reprints](http://www.nature.com/reprints).

Rights and permissions
----------------------

[Reprints and Permissions](https://s100.copyright.com/AppDispatchServlet?title=Probabilistic%20machine%20learning%20and%20artificial%20intelligence&author=Zoubin%20Ghahramani&contentID=10.1038%2Fnature14541&copyright=Springer%20Nature%20Limited&publication=0028-0836&publicationDate=2015-05-27&publisherName=SpringerNature&orderBeanReset=true)

This article is cited by
------------------------

* ###
[Modeling the rutting performance of asphalt pavements: a review](https://doi.org/10.1186/s43065-023-00082-9)

	+ Yong Deng
	+ Xianming Shi*Journal of Infrastructure Preservation and Resilience* (2023)
* ###
[Machine learning framework with feature selection approaches for thyroid disease classification and associated risk factors identification](https://doi.org/10.1186/s43067-023-00101-5)

	+ Azrin Sultana
	+ Rakibul Islam*Journal of Electrical Systems and Information Technology* (2023)
* ###
[Predictive factors for degenerative lumbar spinal stenosis: a model obtained from a machine learning algorithm technique](https://doi.org/10.1186/s12891-023-06330-z)

	+ Janan Abbas
	+ Malik Yousef
	+ Kamal Hamoud*BMC Musculoskeletal Disorders* (2023)
* ###
[Uncertainty quantification via a memristor Bayesian deep neural network for risk-sensitive reinforcement learning](https://doi.org/10.1038/s42256-023-00680-y)

	+ Yudeng Lin
	+ Qingtian Zhang
	+ Huaqiang Wu*Nature Machine Intelligence* (2023)
* ###
[Clinical Concept-Based Radiology Reports Classification Pipeline for Lung Carcinoma](https://doi.org/10.1007/s10278-023-00787-z)

	+ Sneha Mithun
	+ Ashish Kumar Jha
	+ Leonard Wee*Journal of Digital Imaging* (2023)

Comments
--------

By submitting a comment you agree to abide by our [Terms](/info/tandc.html) and [Community Guidelines](/info/community-guidelines.html). If you find something abusive or that does not comply with our terms or guidelines please flag it as inappropriate.


ReviewDeep learning in neural networks: An overview
===================================================

Abstract
--------

In recent years, deep [artificial neural networks](/topics/engineering/artificial-neural-network "Learn more about artificial neural networks from ScienceDirect's AI-generated Topic Pages") (including [recurrent](/topics/engineering/recurrent "Learn more about recurrent from ScienceDirect's AI-generated Topic Pages") ones) have won numerous contests in pattern recognition and [machine learning](/topics/computer-science/machine-learning "Learn more about machine learning from ScienceDirect's AI-generated Topic Pages"). This historical survey compactly summarizes relevant work, much of it from the previous millennium. Shallow and Deep Learners are distinguished by the depth of their *credit assignment paths*, which are chains of possibly learnable, causal links between actions and effects. I review deep supervised learning (also recapitulating the history of backpropagation), [unsupervised learning](/topics/computer-science/unsupervised-learning "Learn more about unsupervised learning from ScienceDirect's AI-generated Topic Pages"), [reinforcement learning](/topics/computer-science/reinforcement-learning "Learn more about reinforcement learning from ScienceDirect's AI-generated Topic Pages") & evolutionary computation, and indirect search for short programs encoding deep and large networks.

Section snippets
----------------

Preface
-------

This is the preprint of an invited *Deep Learning* (DL) overview. One of its goals is to assign credit to those who contributed to the present state of the art. I acknowledge the limitations of attempting to achieve this goal. The DL research community itself may be viewed as a continually evolving, deep network of scientists who have influenced each other in complex ways. Starting from recent DL results, I tried to trace back the origins of relevant ideas through the past half century and

Introduction to Deep Learning (DL) in Neural Networks (NNs)
-----------------------------------------------------------

Which modifiable components of a learning system are responsible for its success or failure? What changes to them improve performance? This has been called the *fundamental credit assignment problem* (Minsky, 1963). There are general credit assignment methods for *universal problem solvers* that are time-optimal in various theoretical senses (Section  6.8). The present survey, however, will focus on the narrower, but now commercially important, subfield of *Deep Learning* (DL) in *Artificial Neural*

Event-oriented notation for activation spreading in NNs
-------------------------------------------------------

Throughout this paper, let i,j,k,t,p,q,r denote positive integer variables assuming ranges implicit in the given contexts. Let n,m,T denote positive integer constants.

An NN’s topology may change over time (e.g., Sections 5.3, 5.6.3). At any given moment, it can be described as a finite subset of units (or nodes or neurons) N={u1,u2,…,} and a finite set H⊆N×N of directed edges or connections between nodes. FNNs are acyclic graphs, RNNs cyclic. The first (input) layer is the set of input units, a

Depth of Credit Assignment Paths (CAPs) and of problems
-------------------------------------------------------

To measure whether credit assignment in a given NN application is of the *deep* or *shallow* type, I introduce the concept of *Credit Assignment Paths* or CAPs, which are chains of possibly causal links between the events of Section  2, e.g., from input through hidden to output layers in FNNs, or through transformations over time in RNNs.

Let us first focus on SL. Consider two events xp and xq(1≤p<q≤T). Depending on the application, they may have a *Potential Direct Causal Connection* (PDCC) expressed

Dynamic programming for Supervised/Reinforcement Learning (SL/RL)
-----------------------------------------------------------------

One recurring theme of DL is *Dynamic Programming* (DP) (Bellman, 1957), which can help to facilitate credit assignment under certain assumptions. For example, in SL NNs, backpropagation itself can be viewed as a DP-derived method (Section  5.5). In traditional RL based on strong Markovian assumptions, DP-derived methods can help to greatly reduce problem depth (Section  6.2). DP algorithms are also essential for systems that combine concepts of NNs and graphical models, such as *Hidden Markov*

Supervised NNs, some helped by unsupervised NNs
-----------------------------------------------

The main focus of current practical applications is on *Supervised Learning* (SL), which has dominated recent pattern recognition contests (Sections  5.17 2009: first official competitions won by RNNs, and with MPCNNs, 5.18 2010: plain backprop (+ distortions) on GPU breaks MNIST record, 5.19 2011: MPCNNs on GPU achieve superhuman vision performance, 5.20 2011: Hessian-free optimization for RNNs, 5.21 2012: first contests won on ImageNet, object detection, segmentation, 5.22 2013-: more contests

DL in FNNs and RNNs for Reinforcement Learning (RL)
---------------------------------------------------

So far we have focused on Deep Learning (DL) in supervised or unsupervised NNs. Such NNs learn to perceive/encode/predict/classify patterns or pattern sequences, but they do not learn to act in the more general sense of *Reinforcement Learning* (RL) in unknown environments (see surveys, e.g.,  Kaelbling et al., 1996, Sutton and Barto, 1998, Wiering and van Otterlo, 2012). Here we add a discussion of DL FNNs and RNNs for RL. It will be shorter than the discussion of FNNs and RNNs for SL and UL

Conclusion and outlook
----------------------

*Deep Learning* (DL) in *Neural Networks* (NNs) is relevant for *Supervised Learning* (SL) (Section  5), *Unsupervised Learning* (UL) (Section  5), and *Reinforcement Learning* (RL) (Section  6). By alleviating problems with deep *Credit Assignment Paths* (CAPs, Sections 3, 5.9), UL (Section  5.6.4) cannot only facilitate SL of sequences (Section  5.10) and stationary patterns (Sections 5.7, 5.15), but also RL (Sections 6.4, 4.2). *Dynamic Programming* (DP, Section  4.1) is important for both deep SL

Acknowledgments
---------------

Since 16 April 2014, drafts of this paper have undergone massive open online peer review through public mailing lists including [[email protected]](/cdn-cgi/l/email-protection#ddbeb2b3b3b8bea9b4b2b3b4aea9ae9dbeaef3beb0a8f3b8b9a8), [[email protected]](/cdn-cgi/l/email-protection#3b565716555e4c487b5c54545c575e5c49544e4b4815585456), [[email protected]](/cdn-cgi/l/email-protection#caa9a5a7bae7a4afbfb8a58aa4afbfb8a5a3a4ace4a5b8ad), [[email protected]](/cdn-cgi/l/email-protection#1770727972637e74486765787065767a7a7e7970576e767f78787065786267643974787a), [[email protected]](/cdn-cgi/l/email-protection#b8cad495d4d1cbccf8dfd7d7dfd4dddfcad7cdc8cb96dbd7d5), [[email protected]](/cdn-cgi/l/email-protection#2a43474b4d4f5d4558464e6a4e43415f044e41), *Google*+  *machine learning forum*. Thanks to numerous NN/DL experts for valuable comments. Thanks to SNF, DFG, and the European Commission for partially funding my DL research group in the past quarter-century.

References (888)
----------------

* R. Andrews *et al.*### [Survey and critique of techniques for extracting rules from trained artificial neural networks](/science/article/pii/0950705196819204)

### Knowledge-Based Systems

(1995)
* D. Anguita *et al.*### [Mixing floating- and fixed-point formats for neural network learning on neuroprocessors](/science/article/pii/0165607496000129)

### Microprocessing and Microprogramming

(1996)
* D. Anguita *et al.*### [An efficient implementation of BP on RISC-based workstations](/science/article/pii/0925231294900345)

### Neurocomputing

(1994)
* P. Baldi *et al.*### [Neural networks and principal component analysis: learning from examples without local minima](/science/article/pii/0893608089900142)

### Neural Networks

(1989)
* P. Baldi *et al.*### [The dropout learning algorithm](/science/article/pii/S0004370214000216)

### Artificial Intelligence

(2014)
* F. Biegler-König *et al.*### [A learning algorithm for multilayered neural networks based on linear least squares problems](/science/article/pii/S0893608005800772)

### Neural Networks

(1993)
* V.D. Blondel *et al.*### [A survey of computational complexity results in systems and control](/science/article/pii/S0005109800000509)

### Automatica

(2000)
* A.L. Blum *et al.*### [Training a 3-node neural network is NP-complete](/science/article/pii/S0893608005800103)

### Neural Networks

(1992)
* A. Blumer *et al.*### [Occam’s razor](/science/article/pii/0020019087901141)

### Information Processing Letters

(1987)
* S.M. Bohte *et al.*### [Error-backpropagation in temporally encoded networks of spiking neurons](/science/article/pii/S0925231201006580)

### Neurocomputing

(2002)

- D. Aberdeen### Policy-gradient algorithms for partially observable Markov decision processes

(2003)
- J. Abounadi *et al.*### Learning algorithms for Markov decision processes with average cost

### SIAM Journal on Control and Optimization

(2002)
- H. Akaike### Statistical predictor identification

### Annals of the Institute of Statistical Mathematics

(1970)
- H. Akaike### Information theory and an extension of the maximum likelihood principle
- H. Akaike### A new look at the statistical model identification

### IEEE Transactions on Automatic Control

(1974)
- A. Allender### Application of time-bounded Kolmogorov complexity in complexity theory
- Almeida, L. B. (1987). A learning rule for asynchronous perceptrons with feedback in a combinatorial environment. In...
- L.B. Almeida *et al.*### On-line step size adaptation. Technical report, INESC, 9 Rua Alves Redol, 1000

(1997)
- S. Amari### A theory of adaptive pattern classifiers

### IEEE Transactions on Electronic Computers

(1967)
- S.-I. Amari### Natural gradient works efficiently in learning

### Neural Computation

(1998)
- S. Amari *et al.*### A new learning algorithm for blind signal separation
- S. Amari *et al.*### Statistical theory of learning curves under entropic loss criterion

### Neural Computation

(1993)
- D.J. Amit *et al.*### Dynamics of a recurrent network of spiking neurons before and following learning

### Network: Computation in Neural Systems

(1997)
- G. An### The effects of adding noise during backpropagation training on a generalization performance

### Neural Computation

(1996)
- M.A. Andrade *et al.*### Evaluation of secondary structure of proteins from UV circular dichroism spectra using an unsupervised learning neural network

### Protein Engineering

(1993)
- I. Arel *et al.*### Deep machine learning—a new frontier in artificial intelligence research

### IEEE Computational Intelligence Magazine

(2010)
- T. Ash### Dynamic node creation in backpropagation neural networks

### Connection Science

(1989)
- J.J. Atick *et al.*### Understanding retinal color coding from first principles

### Neural Computation

(1992)
- A.F. Atiya *et al.*### New results on recurrent network training: unifying the algorithms and accelerating convergence

### IEEE Transactions on Neural Networks

(2000)
- J. Ba *et al.*### Adaptive dropout for training deep neural networks
- Baird, H. (1990). Document image defect models. In Proceddings, IAPR workshop on syntactic and structural pattern...
- Baird, L. C. (1995). Residual algorithms: Reinforcement learning with function approximation. In International...
- L. Baird *et al.*### Gradient descent for general reinforcement learning
- B. Bakker### Reinforcement learning with long short-term memory
- B. Bakker *et al.*### Hierarchical reinforcement learning based on subgoal discovery and subpolicy specialization
- Bakker, B., Zhumatiy, V., Gruener, G., & Schmidhuber, J. (2003). A robot that reinforcement-learns to identify and...
- P. Baldi### Gradient descent learning algorithms overview: A general dynamical systems perspective

### IEEE Transactions on Neural Networks

(1995)
- P. Baldi### Autoencoders, unsupervised learning, and deep architectures

### Journal of Machine Learning Research

(2012)
- P. Baldi *et al.*### Exploiting the past and the future in protein secondary structure prediction

### Bioinformatics

(1999)
- P. Baldi *et al.*### Neural networks for fingerprint recognition

### Neural Computation

(1993)
- P. Baldi *et al.*### Hybrid modeling, HMM/NN architectures, and protein applications

### Neural Computation

(1996)
- P. Baldi *et al.*### Learning in linear networks: a survey

### IEEE Transactions on Neural Networks

(1995)
- P. Baldi *et al.*### The principled design of large-scale recursive neural network architectures—DAG-RNNs and the protein structure prediction problem

### Journal of Machine Learning Research

(2003)
- Ballard, D. H. (1987). Modular learning in neural networks. In Proc. AAAI (pp....
- S. Baluja### Population-based incremental learning: A method for integrating genetic search based function optimization and competitive learning. Technical report CMU-CS-94-163

(1994)
- R. Balzer### A 15 year perspective on automatic programming

### IEEE Transactions on Software Engineering

(1985)
- H.B. Barlow### Unsupervised learning

### Neural Computation

(1989)
- H.B. Barlow *et al.*### Finding minimum entropy codes

### Neural Computation

(1989)
- H.G. Barrow### Learning receptive fields
- A.G. Barto *et al.*### Recent advances in hierarchical reinforcement learning

### Discrete Event Dynamic Systems

(2003)
View more referencesCited by (13349)
----------------

* ### [A deep neural network-assisted metamodel for damage detection of trusses using incomplete time-series acceleration](/science/article/pii/S0957417423014690)

2023, Expert Systems with ApplicationsShow abstractIn this article, a deep neural network (DNN)-driven metamodel is first introduced to damage identification of trusses utilizing acceleration signals incompletely measured from limited sensors. This metamodel can automatically learn its damage-sensitive properties itself to construct a better DNN from previously trained poorer ones. Data utilized to build such a model are generated by finite element method (FEM). In which, inputs are the acceleration behavior corresponding to measurement sensors, and outputs are the damage ratios of truss members. The damage site and extent of low-risk members predicted by the current DNN are eradicated by a suggested damage threshold. This helps to dramatically reduce the number of output units to feed into the next DNNs. By repeating such a procedure, the subsequently upgraded DNNs become more precise after several iterations although they only require a simple network architecture, fewer samples, small epochs and less computational effort. To illustrate the efficiency and robustness of the present metamodel, four 2D and 3D trusses with various damage scenarios are investigated. Obtained results indicate that the suggested approach can reliably diagnose both the location and severity of damaged truss members using acceleration behavior measured by a few sensors in a relatively short period of time, even with high noise levels.
* ### [Experimental and numerical investigations on thermal-hydraulic characteristics of supercritical CO<inf>2</inf> flows in printed circuit heat exchangers](/science/article/pii/S1290072923004349)

2023, International Journal of Thermal SciencesShow abstractIn the supercritical CO2 (SCO2) Brayton cycle, the pre-cooler is employed to cool SCO2 to be near the pseudo-critical point state to minimise compression work and achieve high cycle efficiency. The dramatic variation in thermophysical properties of SCO2 makes heat transfer and flow very complex in the pre-cooler, bringing a lot of difficulty to the design of the related components and system. This study attempts to address this issue. The experiment and simulation are carried out to analyse the influences of mass flux, inlet temperature, and pressure on the thermal-hydraulic characteristics of SCO2 flows in a straight-channel printed circuit heat exchanger (PCHE) employed as a pre-cooler, and the underlying heat transfer mechanism is analysed as well. The local heat transfer coefficient (*h*) is higher when the phenomenon that the local temperature of SCO2 (*T*f) is equal to the pseudo-critical temperature (*T*pc) appears in the laminar sublayer than when this phenomenon appears in the buffer layer. The new heat transfer and flow correlations are presented, and the genetic algorithm and back-prediction (GA-BP) neural network is employed to predict the local heat transfer and flow performance. The maximum relative error between the Nusselt number (*Nu*) predicted by the new heat transfer correlation and *Nu* obtained via experiment and simulation is within ±20%. The maximum relative error between the *Nu* predicted by the GA-BP neural network and the *Nu* obtained through experiment and simulation is within ±15%. The maximum relative error between the Fanning friction factor (*f*) predicted by the new friction factor correlation and *f* obtained via experiment and simulation is within ±5%. The maximum relative error between *f* predicted by the GA-BP neural network and *f* obtained through experiment and simulation is within ±4%. The present investigations analyse the thermal-hydraulic characteristics of SCO2 in the pre-cooler prototype and could provide useful guidance for the design of pre-coolers for the SCO2 Brayton cycle.
* ### [A review of machine learning state-of-charge and state-of-health estimation algorithms for lithium-ion batteries](/science/article/pii/S235248472300118X)

2023, Energy ReportsShow abstractVehicle electrification has been proven to be an efficient way to reduce carbon dioxide emissions and solve the energy crisis. Lithium-ion batteries (LiBs) are considered the dominant energy storage medium for electric vehicles (EVs) owing to their high energy density and long lifespan. To maintain a safe, efficient, and stable operating condition for the battery system, we must monitor the state of the battery, especially the state-of-charge (SOC) and state-of-health (SOH). With the development of big data, cloud computing, and other emerging techniques, data-driven machine learning (ML) techniques have attracted attention for their enormous potential in state estimation for LiBs. Therefore, this paper reviews the four most studied types of ML algorithms for SOC and SOH estimation, including shallow neural network (NN), deep learning (DL), support vector machine (SVM), and Gaussian process regression (GPR) methods. The basic principles and uniform flowcharts of different ML algorithms are introduced. Then, the applications of each ML algorithm for state estimation within recent years are comprehensively reviewed and compared in terms of used datasets, input features, hyperparameter selection, performance metrics, advantages, and disadvantages. Based on the investigation, this review discusses the current challenges and prospects from four aspects, aiming to provide some inspiration for developing advanced ML state estimation algorithms.
* ### [Artificial intelligence systems for the design of magic shotgun drugs](/science/article/pii/S2667318522000253)

2023, Artificial Intelligence in the Life SciencesShow abstractDesigning magic shotgun compounds, i.e., compounds hitting multiple targets using artificial intelligence (AI) systems based on machine learning (ML) and deep learning (DL) approaches, has a huge potential to revolutionize drug discovery. Such intelligent systems enable computers to create new chemical structures and predict their multi-target properties at a low cost and in a time-efficient manner. Most examples of AI applied to drug discovery are single-target oriented and there is still a lack of concise information regarding the application of this technology for the discovery of multi-target drugs or drugs with broad-spectrum action. In this review, we focus on current developments in AI systems for the next generation of automated design of multi-target drugs. We discuss how classical ML methods, cutting-edge generative models, and multi-task deep neural networks can help *de novo* design and hit-to-lead optimization of multi-target drugs. Moreover, we present state-of-the-art workflows and highlight some studies demonstrating encouraging experimental results, which pave the way for *de novo* drug design and multi-target drug discovery.
* ### [Angle Measurement Based on Second Harmonic Generation Using Artificial Neural Network](https://doi.org/10.1007/s41871-023-00206-5)

2023, Nanomanufacturing and Metrology
* ### [1D and 2D Chaotic Time Series Prediction Using Hierarchical Reservoir Computing System](https://doi.org/10.1142/S0129156423500143)

2023, International Journal of High Speed Electronics and Systems

[View all citing articles on Scopus](http://www.scopus.com/scopus/inward/citedby.url?partnerID=10&rel=3.0.0&eid=2-s2.0-84910651844&md5=39f622c0b4d58c5e9012b7c30576918)Recommended articles (6)
------------------------

* Research article### [A complex-valued neural dynamical optimization approach and its stability analysis](/science/article/pii/S0893608014002238)

Neural Networks, Volume 61, 2015, pp. 59-67Show abstractIn this paper, we propose a complex-valued neural dynamical method for solving a complex-valued nonlinear convex programming problem. Theoretically, we prove that the proposed complex-valued neural dynamical approach is globally stable and convergent to the optimal solution. The proposed neural dynamical approach significantly generalizes the real-valued nonlinear Lagrange network completely in the complex domain. Compared with existing real-valued neural networks and numerical optimization methods for solving complex-valued quadratic convex programming problems, the proposed complex-valued neural dynamical approach can avoid redundant computation in a double real-valued space and thus has a low model complexity and storage capacity. Numerical simulations are presented to show the effectiveness of the proposed complex-valued neural dynamical approach.
* Research article### [An efficient sampling algorithm with adaptations for Bayesian variable selection](/science/article/pii/S0893608014002202)

Neural Networks, Volume 61, 2015, pp. 22-31Show abstractIn Bayesian variable selection, indicator model selection (IMS) is a class of well-known sampling algorithms, which has been used in various models. The IMS is a class of methods that uses pseudo-priors and it contains specific methods such as Gibbs variable selection (GVS) and Kuo and Mallick’s (KM) method. However, the efficiency of the IMS strongly depends on the parameters of a proposal distribution and the pseudo-priors. Specifically, the GVS determines their parameters based on a pilot run for a full model and the KM method sets their parameters as those of priors, which often leads to slow mixings of them. In this paper, we propose an algorithm that adapts the parameters of the IMS during running. The parameters obtained on the fly provide an appropriate proposal distribution and pseudo-priors, which improve the mixing of the algorithm. We also prove the convergence theorem of the proposed algorithm, and confirm that the algorithm is more efficient than the conventional algorithms by experiments of the Bayesian variable selection.
* Research article### [Structural damage identification based on autoencoder neural networks and deep learning](/science/article/pii/S0141029618302062)

Engineering Structures, Volume 172, 2018, pp. 13-28Show abstractArtificial neural networks are computational approaches based on machine learning to learn and make predictions based on data, and have been applied successfully in diverse applications including structural health monitoring in civil engineering. It is difficult to optimize the weights in the neural networks that have multiple hidden layers due to the vanishing gradient issue. This paper proposes an autoencoder based framework for structural damage identification, which can support deep neural networks and be utilized to obtain optimal solutions for pattern recognition problems of highly non-linear nature, such as learning a mapping between the vibration characteristics and structural damage. Two main components are defined in the proposed framework, namely, dimensionality reduction and relationship learning. The first component is to reduce the dimensionality of the original input vector while preserving the required necessary information, and the second component is to perform the relationship learning between the features with the reduced dimensionality and the stiffness reduction parameters of the structure. Vibration characteristics, such as natural frequencies and mode shapes, are used as the input and the structural damage are considered as the output vector. A pre-training scheme is performed to train the hidden layers in the autoencoders layer by layer, and fine tuning is conducted to optimize the whole network. Numerical and experimental investigations on steel frame structures are conducted to demonstrate the accuracy and efficiency of the proposed framework, comparing with the traditional ANN methods.
* Research article### [Metaheuristic design of feedforward neural networks: A review of two decades of research](/science/article/pii/S0952197617300234)

Engineering Applications of Artificial Intelligence, Volume 60, 2017, pp. 97-116Show abstractOver the past two decades, the feedforward neural network (FNN) optimization has been a key interest among the researchers and practitioners of multiple disciplines. The FNN optimization is often viewed from the various perspectives: the optimization of weights, network architecture, activation nodes, learning parameters, learning environment, etc. Researchers adopted such different viewpoints mainly to improve the FNN's generalization ability. The gradient-descent algorithm such as backpropagation has been widely applied to optimize the FNNs. Its success is evident from the FNN's application to numerous real-world problems. However, due to the limitations of the gradient-based optimization methods, the metaheuristic algorithms including the evolutionary algorithms, swarm intelligence, etc., are still being widely explored by the researchers aiming to obtain generalized FNN for a given problem. This article attempts to summarize a broad spectrum of FNN optimization methodologies including conventional and metaheuristic approaches. This article also tries to connect various research directions emerged out of the FNN optimization practices, such as evolving neural network (NN), cooperative coevolution NN, complex-valued NN, deep learning, extreme learning machine, quantum NN, etc. Additionally, it provides interesting research challenges for future research to cope-up with the present information processing era.
* Research article### [Deep learning on image denoising: An overview](/science/article/pii/S0893608020302665)

Neural Networks, Volume 131, 2020, pp. 251-275Show abstractDeep learning techniques have received much attention in the area of image denoising. However, there are substantial differences in the various types of deep learning methods dealing with image denoising. Specifically, discriminative learning based on deep learning can ably address the issue of Gaussian noise. Optimization models based on deep learning are effective in estimating the real noise. However, there has thus far been little related research to summarize the different deep learning techniques for image denoising. In this paper, we offer a comparative study of deep techniques in image denoising. We first classify the deep convolutional neural networks (CNNs) for additive white noisy images; the deep CNNs for real noisy images; the deep CNNs for blind denoising and the deep CNNs for hybrid noisy images, which represents the combination of noisy, blurred and low-resolution images. Then, we analyze the motivations and principles of the different types of deep learning methods. Next, we compare the state-of-the-art methods on public denoising datasets in terms of quantitative and qualitative analyses. Finally, we point out some potential challenges and directions of future research.
* Research article### [Conceptual Understanding of Convolutional Neural Network- A Deep Learning Approach](/science/article/pii/S1877050918308019)

Procedia Computer Science, Volume 132, 2018, pp. 679-688Show abstractDeep learning has become an area of interest to the researchers in the past few years. Convolutional Neural Network (CNN) is a deep learning approach that is widely used for solving complex problems. It overcomes the limitations of traditional machine learning approaches. The motivation of this study is to provide the knowledge and understanding about various aspects of CNN. This study provides the conceptual understanding of CNN along with its three most common architectures, and learning algorithms. This study will help researchers to have a broad comprehension of CNN and motivate them to venture in this field. This study will be a resource and quick reference for those who are interested in this field.
[View full text](/science/article/pii/S0893608014002135)