![Kenneth Hayworth portrait](https://intelligence.org/wp-content/uploads/2014/09/image00.jpg)Kenneth Hayworth is president of the [Brain Preservation Foundation](http://www.brainpreservation.org/) (BPF), an organization formed to skeptically evaluate cryonic and other potential human preservation technologies by examining how well they preserve the brain’s neural circuitry at the nanometer scale. Hayworth is also a Senior Scientist at the HHMI’s Janelia Farm Research Campus where he is currently researching ways to extend Focused Ion Beam Scanning Electron Microscopy (FIBSEM) of brain tissue to encompass much larger volumes than are currently possible. Hayworth is co-inventor of the ATUM-SEM process for high-throughput volume imaging of neural circuits at the nanometer scale and he designed and built several automated machines to implement this process. Hayworth received his PhD in Neuroscience from the University of Southern California for research into how the human visual system encodes spatial relations among objects. Hayworth is a vocal advocate for brain preservation and mind uploading and, through the BPF’s Brain Preservation Prize, he has challenged scientists and medical researchers to develop a reliable, scientifically verified surgical procedure which can demonstrate long-term ultrastructure preservation across an entire human brain. Once won, Hayworth advocates for the widespread implementation of such a surgical procedure in hospitals. Several research labs are currently attempting to win this prize.

**Luke Muehlhauser**: One interesting feature of your own thinking ([Hayworth 2012](http://www.brainpreservation.org/sites/default/files/ElectronImagingTechnologyForWholeBrainNeuralCircuitMapping_Hayworth2012.pdf)) about whole brain emulation (WBE) is that you are more concerned with modeling high-level cognitive functions accurately than is e.g. [Sandberg (2013)](http://commonsenseatheism.com/wp-content/uploads/2012/09/Sandberg-Feasability-of-Whole-Brain-Emulation.pdf). Whereas Sandberg expects WBE will be achieved by modeling low-level brain function in exact detail (at the level of scale separation, wherever that is), you instead lean heavily on modeling higher-level cognitive processes using a cognitive architecture called [ACT-R](http://en.wikipedia.org/wiki/ACT-R). Is that because you think this will be *easier* than Sandberg’s approach, or for some other reason?

---

**Kenneth Hayworth**: I think the key distinction is that philosophers are focused on whether mind uploading (a term I prefer to WBE) is possible in principle, and, to a lesser extent, on whether it is of such technical difficulty as to put its achievement off so far into the future that its possibility can be safely ignored for today’s planning. With these motivations, philosophers tend to gravitate toward arguments with the fewest possible assumptions, i.e. modeling low-level brain functions in exact detail.

As a practicing cognitive and neuroscientist I have fundamentally different motivations. From my training, I am already totally convinced that the functioning of the brain can be understood at a fully mechanistic level, with sufficient precision to allow for mind uploading. I just want to work toward making mind uploading happen in reality. To do this I need to start with an understanding of the requirements, not based on the fewest assumptions, but instead based on the field’s current best theories.

To use an analogy, before airplanes were invented one could argue that heavier-than-air flying machines must be possible in principle even if it meant copying a bird’s flapping motions etc. This is a sound philosophical argument but not one that engineers like the Wright Brothers would focus on. Instead they needed to know the crucial elements necessary for heavier-than-air flight –lift, drag, thrust. Understanding these allowed them to start building and testing, refining their theories and engineering.

If we want to create the technology to upload someone’s mind into a computer simulation and have that simulation have the same memories, intelligence, personality, and consciousness as the original, then we need to start with a top-level understanding of these cognitive functions. The place to look for these is the field of cognitive science and to highly researched computational models of intelligent behavior such as ACT-R.

I have focused my research on understanding how the computational elements of cognitive models such as ACT-R (symbolic representations, production rules, etc.) are likely mapped onto the neural circuits in the brain. This has led me to concrete predictions on exactly what technologies would be necessary to preserve, scan, and simulate a person’s brain. Those are the technologies I am currently working on.

---

**Luke Muehlhauser:** How widely used or accepted is ACT-R? Just looking through the literature, it doesn’t seem particularly dominant in cognitive neuroscience, though perhaps that’s just because *no* high-level model of its kind is dominant. E.g. ACT-R doesn’t seem to be mentioned in the 1100+ pages of the new *[Oxford Handbook of Cognitive Neuroscience](http://www.amazon.com/Handbook-Cognitive-Neuroscience-Library-Psychology/dp/0195381599/)*, nor in the 1800+ pages of the *[Encyclopedia of Behavioral Neuroscience](http://www.amazon.com/Encyclopedia-Behavioral-Neuroscience-Three--Set-ebook/dp/B00D8GA4XG/).* I rarely see mention of ACT-R by people who aren’t its proponents.

---

**Kenneth Hayworth**: Looking at the [main website for ACT-R](http://act-r.psy.cmu.edu/) I count a total of 1,035 individual publications related to ACT-R theory or using the ACT-R modeling framework. This publication record comes from hundreds of researchers and stretches back decades and continues strong today. ACT-R would not be considered a “cognitive neuroscience” theory since it does not talk about neurons (although there have been attempts to map ACT-R onto a neural framework which I will discuss). Instead, ACT-R is properly classified in the field of Cognitive Science. Of course there is a messy overlap across the fields of computer science, psychology, neuroscience, neurology, cognitive science, artificial intelligence, computational neuroscience, etc. and ACT-R certainly was designed with constraints from these various fields in mind.

Cognitive science can best be distinguished from neuroscience based on the typical level of abstraction of its models of intelligent behavior. Cognitive science is committed to modeling the mind as a computational system in the traditional sense of processing analog and symbolic representations of various complexity via algorithms. In general, cognitive science models of human intelligent behavior are couched in representational and algorithmic form. For example, a model of human attention and memory retention effects in a perceptual experiment might posit that our mind contains 4 buffers which can be used to store symbolic tokens representing individual letters that were briefly flashed on a viewing screen to the subject. Notice that cognitive science models like this do not talk about neurons or even brain regions, yet they make concrete testable predictions about behavioral responses, response timing, error rates, learning rates, etc. –effects which can be tested with great detail and precision in psychophysical experiments.

If you are looking for explanations for how humans understand natural language, or how they solve problems and reason about goals, or, crucially, how the mind creates and maintains a “self model” underlying our consciousness then cognitive science models at this level of abstraction are really the only place to look. Neuroscience should be seen as talking about the implementation details underlying the algorithms and representations assumed by such cognitive science models.

Now cognitive science as a field has been around for a long time and has been generating an enormous variety of small models and algorithmic hypotheses for how humans perform various actions. Way back in 1973 the great cognitive scientist Allen Newell wrote a paper called “You can’t play 20 questions with nature and win” in which he pointed out to the cognitive science community that it needed to strive for “unified theories of cognition” which were designed to explain not just one experimental result but all of them simultaneously. He proposed a particular computational formalism called a “production system” that could provide the basis for such a unification of different cognitive science models. He and others developed that proposal into the widely used SOAR model of human cognition (see his book “Unified Theories of Cognition” for a complete explanation of production systems as a model of the human mind).

There have been many production system models of the human mind based upon this original work but Anderson’s ACT-R theory is currently the standard-bearer of this class. As such, ACT-R should not really be thought of as just another computational model, instead you should think of ACT-R as attempting a summary of all results from the entire field of cognitive science –a summary in the form of a general computational framework for the human mind. Below is a screen capture of the ACT-R website which keeps a list of all publications and ACT-R models generated over the years. You can see that the categories span the entire field of cognitive science (e.g. language processing, learning, perception, memory, problem solving, decision making, etc.)

[![image00](https://intelligence.org/wp-content/uploads/2014/09/image00.png)](https://intelligence.org/wp-content/uploads/2014/09/image00.png)
Also you should note that since ACT-R is a computational framework, models built within the ACT-R framework really “work” in the sense that an ACT-R model of sentence parsing will actually parse sentences.

I hear you when you say “[ACT-R] doesn’t seem particularly dominant in cognitive neuroscience”. This is absolutely true. Most of the neuroscientists I have met have never heard of ACT-R or production systems and usually are unfamiliar with most of the great discoveries in the field of Cognitive Science. This is a real tragedy since Cognitive Science and Neuroscience are properly seen as just two levels of description of the same organ –the brain. Without the abstractions of cognitive science (symbols, declarative memory chunks, production rules, goal buffers, etc.) neuroscience is faced with an insurmountable gap to cross in its attempt to explain how neurons give rise to mind. It would be like having to explain how the computer program Microsoft Word works by describing its operation at the transistor level. It just cannot be done. One must introduce intermediate levels of abstraction (memory buffers, if-then statements, while-loops, etc.) in order to create a theory of Microsoft Word which is understandable. The same is true when it comes to how mind is generated by the brain.

I argue that any complete theory of how the mind is generated by the physical brain must include at least the following four levels of description:

1. Philosophical theories of consciousness and self (Example: Thomas Metzinger’s book “Being No One: The Self-Model Theory of Subjectivity”)
2. Cognitive science models of the human cognitive control architecture (Example: John Anderson’s book “How can the human mind occur in the physical universe” –his most recent overview of ACT-R)
3. Abstract “artificial” neural network architectures of hippocampal memory systems, perceptual feature hierarchies, reinforcement learning pattern recognition networks, etc. (Example: Edmund Rolls’ book “Neural Networks and Brain Function”)
4. Electrical and molecular models of real biological neurons and systems (Example: Kandel, Schwartz and Jessell’s book “Principles of Neural Science”)

If you cannot deftly switch between these levels of description, understanding the core principles of each and how they are built upon one another, then you are unprepared to understand what is required to accomplish whole brain emulation. Simply understanding neuroscience is not enough.

One of the weakest links which exists today in this hierarchy of descriptive levels is the link between #2 and #3. This is why I have been putting considerable effort into showing how the symbolic computations assumed by ACT-R might be implemented by standard artificial neural network models of autoassocative memory etc. My 2012 publication [“Dynamically partitionable autoassociative networks as a solution to the neural binding problem](http://journal.frontiersin.org/Journal/10.3389/fncom.2012.00073/abstract)”  is a first attempt at this. I am currently preparing a new publication and set of neural models designed to make this link even clearer. My goal is to demonstrate to the neuroscience community that the level of abstraction assumed by cognitive science models like ACT-R is not so removed from the neural network models which neuroscientists currently embrace. An effective bridging of this gap between cognitive science models and neural network models has the potential to release a wave of synergy between the fields, in which the top-down constraints of cognitive science can directly inform neuroscience models, and the bottom-up constraints of neuroscience can directly inform cognitive models.

---

**Luke**: How well-specified is ACT-R? In particular, can you give an example of ACT-R making a surprising, novel quantitative prediction that, upon experiment, turned out to be correct? (ACT-R could still be useful even if this hasn’t happened yet, but if there are examples of this happening then it’d be nice to know about them!)

---

**Kenneth**: One difficulty with answering that question is that ACT-R is really a “framework” in which to create models of brain function, it is not a model in and of itself. A researcher typically uses the ACT-R framework to create a model of a particular cognitive task –say modeling how a student learns to solve algebraic expressions. To be successful, the model must make quantitative predictions about the error rates and types of errors, the speed of responses and how this speed increases with practice, etc. Any particular ACT-R model consists of a set of initial “productions” (symbolic pattern matching if-then rules) and “declarative memory chunks” (symbolic memories) which are assumed to have already been learned by the individual. The model will produce intelligent behaviors using these initial production rules and memory chunks. It will also change the weights of the rules and chunks as it learns by trial and error, and it will produce new production rules and new memories as it interacts with its simulated environment.

Now one can ask your question about novel predictions regarding either a particular ACT-R model or regarding the ACT-R framework itself.  To pick one particularly good example of an ACT-R model making “surprising and novel” predictions you might want to look at the paper: “A Central Circuit of the Mind” (Anderson, Fincham, Qin, and Stocco 2008). In that paper (and a host of follow up papers) an ACT-R model of a cognitive task was used to predict not only behavioral responses but fMRI BOLD activation levels and timing in subjects performing the cognitive task. Considering that the ACT-R community spent many years developing models of brain function without fMRI data as a constraint, it is particularly intriguing to see how well those models have fared when viewed against this new type of data.

As for predictions of the ACT-R architecture itself, I would like to point to perhaps ACT-R’s central prediction going straight back to its inception –the prediction that there are two types of learning: “procedural” and “declarative”. This distinction is so well established now (think of studies of the amnesia patient HM) that it is hard to remember that this was anything but settled when ACT-R was invented. In fact, this is the key difference between the ACT-R and the SOAR cognitive architectures. SOAR was built on an incorrect prediction –that all knowledge in the brain was encoded as procedural rules. This incorrect prediction is why SOAR lost favor with the cognitive science community and why ACT-R is still widely used.

You ask “How well-specified is ACT-R?”. This is a good question. The ACT-R architecture has changed significantly over the years as new information has become available. Over the last decade it went through a dramatic simplification in the complexity that was allowed for the pattern matching in production rules. It also has set the values of some of its core performance and learning parameters (for example the time it takes to execute a single production rule is now set to be 50ms). These ACT-R architectural parameters were of course determined by fitting data to behavioral experiments, but since ACT-R is meant to be a theory of the brain as a whole, individual models are not allowed to arbitrarily change these parameters to fit their particular data. In effect, these ACT-R architectural features have become a significant constraint on the degrees of freedom modelers have when proposing new ACT-R models of particular cognitive tasks. The success or failure of these tighter specifications of the ACT-R framework parameters are best judged by how successful it still is for modeling a wide range of behaviors. As for that I would again point to the wide range of publications using ACT-R.

I don’t think one should judge ACT-R on the basis of novel predictions however. It is designed as a summary theory to account for a large body of cognitive science facts. It should, and has, changed when its predictions were found unsound. For example, ACT-R’s original model of production matching was essentially found untenable given what we know about neural networks, so it was jettisoned for a new simpler version.

I think the best way to judge ACT-R is to understand what it is meant to do and what it is not meant to do. ACT-R is used to model cognitive tasks at the computational level, not at the neural implementation level. It is quite specific and well specified at the computational level and has been shown to provide satisfying (tentative) explanations for a wide range of intelligent human behaviors at this level. I have argued that it is our best model of the human mind at this computational level and therefore we as neuroscientists should use it as a starting point for understanding what types of high-level computations the neural circuits of the brain are likely performing.

---

**Luke**: Back to WBE. You were a participant in the 2007 workshop that led to FHI’s [Whole Brain Emulation: A Roadmap](http://www.fhi.ox.ac.uk/brain-emulation-roadmap-report.pdf) report. Would you mind sharing your own estimates on some of the key questions from the report? In particular, Table 4 of the report (p. 39) summarizes possible modeling complications, along with estimates of how likely they are to be necessary for WBE and how hard they would be to implement. Which of those estimates do you disagree with most strongly? (By “WBE” I mean what the report calls success criterion 6a (“social role-fit emulation”), so as to set aside questions of consciousness and personal identity.)

---

**Kenneth**: First I would like to say how fantastic it was for Anders and the FHI to put on that workshop. It is amazing to see how far the field has progressed in the meantime. I agree with most of the main conclusions of that report, but since you asked for what I disagree with I will focus in on those few points on which I have reservations.

The report starts off with a section entitled “Little need for whole-system understanding” where the authors state: “An important hypothesis for WBE is that in order to emulate the brain we do not need to understand the whole system, but rather we just need a database containing all necessary low‐level information about the brain and knowledge of the local update rules that change brain states from moment to moment.” (p. 8)

I agree that it is technically possible to achieve WBE without an overall understanding of how the brain works, but I seriously doubt this is the way it will happen. I believe our neuroscience models, and the experimental techniques to test them, will continue their rapid advance over the next few decades so that we will reach a very deep understanding of the brain’s functioning at all levels well before we have the technology for applying WBE to a human. Under that scenario we will understand exactly which features listed in table 4 of the report are needed, and which are unnecessary.

As a concrete example (meant to be controversial), there are several types of cortical neurons thought to provide general feedforward and feedback inhibition to the cortex’s main excitatory cells. These inhibitory cells effectively regulate the percentage of the excitatory cells that are allowed to be active at a given time (i.e. the ‘sparseness’ of the neural representation).  I doubt that the details of these inhibitory cells will be modeled at all in a future WBE since it would be much easier, and more reliable, in a computer model to enforce this sparseness of firing directly. Now this suggestion will probably strike many of your readers as a particularly risky shortcut, and they may ask “How can we be sure that the detailed functioning of these inhibitory neurons is not crucial to the operation of the mind?” This may turn out to be the case, but my point here is that by the time we have the technology to actually try a human WBE we will know for sure what ‘shortcuts’ are acceptable.  The experiments needed to test these hypotheses are much simpler than WBE itself.

---

**Luke**: In your view, what are some specific, tractable “next research projects” that, if funded, could show substantial progress toward WBE in the next 10 years?

---

**Kenneth**: Over the next ten years progress in WBE is likely to be tied to progress in the field of connectomics (automated electron microscopic (EM) mapping of brain tissue to reveal the precise connectivity between neurons). Since the 2004 invention of the [serial block face scanning electron microscope](http://www.ncbi.nlm.nih.gov/pmc/articles/PMC524270/) (SBFSEM, or SBEM) there has been an explosion of interest in this area. SBFSEM was the first automated device to really show how one could, in principle, “deconstruct brain tissue” and extract its precise wiring diagram. Since then the SBFSEM has undergone dramatic improvements and has been joined by several other automated electron microscopy tools for mapping brain tissue at the nanometer scale ([TEMCA](http://www.nature.com/nature/journal/v471/n7337/full/nature09802.html), [ATUM-SEM](http://journal.frontiersin.org/Journal/10.3389/fncir.2014.00068/abstract), [FIB-SEM](http://www.ncbi.nlm.nih.gov/pmc/articles/PMC3196160/)). Kevin Briggman and Davi Bock (two of the leaders in this field) wrote a great [review article](http://www.ncbi.nlm.nih.gov/pubmed/22119321) in 2012 covering the current state of all of these connectomics imaging technologies. In short, neuroscientists finally have some automated tools which can image the complete synaptic connectivity of the neural circuits they have been studying for years by behavioral and electrophysiological means. Several “high profile” connectomics publications have recently come out ([Nature 2011](http://www.nature.com/nature/journal/v471/n7337/full/nature09802.html), [Nature 2011](http://www.nature.com/nature/journal/v471/n7337/full/nature09818.html), [Nature 2013](http://www.nature.com/nature/journal/v500/n7461/full/nature12450.html), [Nature 2013](http://www.nature.com/nature/journal/v500/n7461/full/nature12346.html), [Nature 2014](http://www.nature.com/nature/journal/v509/n7500/full/nature13240.html)) giving a taste of what can be accomplished with such tools, but these publications represent just the “tip of the iceberg” of what is likely to be a revolution in the way neuroscience is done, a revolution that over the long run will lead to WBE.

There is, however, one thing that is currently holding back the entire field of connectomics – the lack of an automated solution to tracing neural connectivity. Even though there has been fantastic progress in automating 3D EM imaging, each of these connectomics publications required a small army of dedicated human tracers to supplement and “error-correct” the output of today’s inadequate software tracing algorithms. This roadblock has been widely recognized by connectomics practitioners:

“[A]ll cellular-resolution connectomics studies to date have involved thousands to tens-of-thousands of hours of [human labor for neural] reconstruction. Although imaging speed has substantially increased, reconstruction speed is massively lagging behind. For any of the proposed dense-circuit reconstructions (mouse neocortex [column], olfactory bulb, fish brain and human neocortex [column]…), analysis-time estimates are at least one if not several orders of magnitude larger than what has been accomplished to date… requirements for these envisioned projects are around several hundred thousand hours of manual labor per project. These enormous numbers constitute what can be called the analysis gap in cellular connectomics: although imaging larger circuits is becoming more and more feasible, reconstructing them is not.” – Moritz Helmstaedter ([Nature Methods review 2013](http://www.nature.com/nmeth/journal/v10/n6/full/nmeth.2476.html))

A full solution to this “analysis gap” will likely require advances on three fronts:

1. Improved 3D imaging resolution
2. Improved tissue preservation and staining
3. More advanced algorithms

Most of today’s connectomics imaging technologies still rely on physical sectioning of tissue which practically limits the “z” resolution of EM imaging to >20nm. However, the [FIB-SEM](http://www.ncbi.nlm.nih.gov/pmc/articles/PMC3196160/) technique instead uses a focused ion beam to reliably “polish” away a <5nm layer of tissue between each EM imaging step. This means that FIB-SEM datasets can image tissue with voxel resolutions as low as 5x5x5nm. This extra resolution is crucial when trying to follow tubular neuronal processes which can often shrink to <40nm in diameter. This improved FIB-SEM resolution has been [demonstrated](http://www.sciencedirect.com/science/article/pii/S095943881400035X) to dramatically increase the effectiveness of today’s tracing algorithms. I have been working on extending this FIB-SEM technique to make it capable of handling arbitrarily large volumes. I do this by using a heated, oil-lubricated diamond knife to losslessly subdivide a large block of tissue into chunks optimally-sized (~20 microns thick) for high-resolution, parallel FIBSEM imaging. I will have a paper on this coming out later this year.

This higher resolution (FIB-SEM) imaging, along with likely advances in tracing algorithms, should finally be able to achieve the goal of fully-automated tracing of neural tissue –i.e. it should overcome the “analysis gap”, eliminating the reliance on human tracing.

That is, as long as the tissue is optimally preserved and stained for EM imaging. Unfortunately, today’s best tissue preservation and staining protocols are limited to <200 micron thick tissue slabs. This volume limit also represents a fundamental roadblock to connectomics, and there is only one researcher that I know of that has seriously taken up the challenge to remove this limitation –[Shawn Mikula](http://connectomes.org/index.php?p=about-connectomes). He has made [substantial progress](http://connectomes.org/pdf/mikula2012.pdf) toward this goal, but considerable work remains.

So regarding your question for “specific, tractable next research projects”, I would offer up the following: A project to develop a protocol for preserving, staining, and losslessly subdividing an entire mouse brain for random-access FIB-SEM imaging, along with a project to build a “farm” of hundreds of inexpensive FIB-SEM machines capable of imaging neural circuits spanning large regions of this mouse’s brain.

I believe that the Mikula protocol can be extended to provide excellent EM staining for all parts of a mouse brain (Mikula, personal communications). I also believe, with sufficient work and funding, that this mouse protocol could be made compatible with my “lossless subdivision” technique which would allow the entire mouse brain to be quickly and reliably chopped up into little cubes (~50x50x50 microns in size) any of which could be loaded into a FIB-SEM machine for imaging. I also believe, with sufficient funding, that today’s generation of expensive (>1million USD) FIB-SEM machines could be redesigned, and drastically simplified for mass-manufacture.

I think that such a project is indeed tractable over a ten year time frame, and it would in effect provide neuroscientists with a tool (what I have termed a “[connectome observatory](https://www.youtube.com/watch?v=zp1TlhxZQmk)”) to map out the neural circuits involved in vision, memory, motor control, reinforcement learning, etc. all within the same mouse brain. Allowing researchers to see not only an individual region’s circuits, but also the long distance connections among these regions which allow them to act as a coherent whole controlling the mouse’s behavior. I think it is at this level that we will begin to see rudiments of the types of coordinated, symbol-level operations predicted by cognitive models like ACT-R, the types of executive control circuits which in us evolved to underlie our unique intelligence and consciousness.

We must remember and accept that the long-term goal of human whole brain emulation is almost unimaginably difficult. It is ludicrous to expect significant progress toward that goal in the next decade or two. But I believe we should not shrink away from saying that that goal is the one we are ultimately pursuing. I believe neuroscientists and cognitive scientists should proudly embrace the idea that their fields will eventually succeed in revealing a complete and satisfying mechanistic explanation of the human mind and consciousness, and that this understanding will inevitably lead to the technology of mind uploading. There is so much hard work to be done that we cannot afford to neglect the tremendous payoffs awaiting in the event we succeed.

---

**Luke**: Thanks, Kenneth!

The post [Ken Hayworth on brain emulation prospects](https://intelligence.org/2014/09/09/hayworth/) appeared first on [Machine Intelligence Research Institute](https://intelligence.org).


Friendly AI Research Help from MIRI

Earlier this year, a student told us he was writing an honors thesis on logical decisions theories such as TDT and UDT — one of MIRI’s core research areas. Our reply was “Why didn’t you tell us this earlier? When can we fly you to Berkeley to help you with it?”

So we flew Danny Hintze to Berkeley and he spent a couple days with Eliezer Yudkowsky to clarify the ideas for the thesis. Then Danny went home and wrote what is [probably the best current introduction to logical decision theories](https://intelligence.org/wp-content/uploads/2014/10/Hintze-Problem-Class-Dominance-In-Predictive-Dilemmas.pdf).

Inspired by this success, today we are launching the [Friendly AI Research Help program](http://intelligence.org/researchhelp/), which encourages students of mathematics, computer science, and formal philosophy to collaborate and consult with our researchers to help steer and inform their work.

Apply for research help [here](http://intelligence.org/researchhelp/).

[![thesishelpheadersmall](https://intelligence.org/wp-content/uploads/2014/09/thesishelpheadersmall.jpg)](https://intelligence.org/wp-content/uploads/2014/09/thesishelpheadersmall.jpg)
The post [Friendly AI Research Help from MIRI](https://intelligence.org/2014/09/08/friendly-ai-research-help-miri/) appeared first on [Machine Intelligence Research Institute](https://intelligence.org).


John Fox on AI safety

[John Fox](http://www.cossac.org/people/johnfox) is an interdisciplinary scientist with theoretical interests in AI and computer science, and an applied focus in medicine and medical software engineering. After training in experimental psychology at Durham and Cambridge Universities and post-doctoral fellowships at CMU and Cornell in the USA and UK (MRC) he joined the Imperial Cancer Research Fund (now Cancer Research UK) in 1981 as a researcher in medical AI. The group’s research was explicitly multidisciplinary and it subsequently made significant contributions in basic computer science, AI and medical informatics, and developed a number of successful technologies which have been commercialised.

In 1996 he and his team were awarded the 20th Anniversary Gold Medal of the European Federation of Medical Informatics for the development of PROforma, arguably the first formal computer language for modeling clinical decision and processes. Fox has published widely in computer science, cognitive science and biomedical engineering, and was the founding editor of the *Knowledge Engineering Review*  (Cambridge University Press). Recent publications include a research monograph *[Safe and Sound: Artificial Intelligence in Hazardous Applications](http://smile.amazon.com/Safe-Sound-Artificial-Intelligence-Applications/dp/0262062119/ref=nosim?tag=793775876-20)* (MIT Press, 2000) which deals with the use of AI in safety-critical fields such as medicine.

**Luke Muehlhauser**: You’ve spent many years studying AI safety issues, in particular in medical contexts, e.g. in your 2000 book with Subrata Das, *[Safe and Sound: Artificial Intelligence in Hazardous Applications](http://smile.amazon.com/Safe-Sound-Artificial-Intelligence-Applications/dp/0262062119/ref=nosim?tag=793775876-20)*. What kinds of AI safety challenges have you focused on in the past decade or so?

---

**John Fox**: From my first research job, as a post-doc with AI founders Allen Newell and Herb Simon at CMU, I have been interested in computational theories of high level cognition. As a cognitive scientist I have been interested in theories that subsume a range of cognitive functions, from perception and reasoning to the uses of knowledge in autonomous decision-making. After I came back to the UK in 1975 I began to combine my theoretical interests with the practical goals of designing and deploying AI systems in medicine.

Since our book was published in 2000 I have been committed to testing the ideas in it by designing and deploying many kind of clinical systems, and demonstrating that AI techniques can significantly improve quality and safety of clinical decision-making and process management. Patient safety is fundamental to clinical practice so, alongside the goals of building systems that can improve on human performance, safety and ethics have always been near the top of my research agenda.

---

**Luke Muehlhauser:** Was it straightforward to address issues like safety and ethics in practice?

---

**John Fox**: While our concepts and technologies have proved to be clinically successful we have not achieved everything we hoped for. Our attempts to ensure, for example, that practical and commercial deployments of AI technologies should explicitly honor ethical principles and carry out active safety management have not yet achieved the traction that we need to achieve. I regard this as a serious cause for concern, and unfinished business in both scientific and engineering terms.

The next generation of large-scale knowledge based systems and software agents that we are now working on will be more intelligent and will have far more autonomous capabilities than current systems. The challenges for human safety and ethical use of AI that this implies are beginning to mirror those raised by the singularity hypothesis. We have much to learn from singularity researchers, and perhaps our experience in deploying autonomous agents in human healthcare will offer opportunities to ground some of the singularity debates as well.

---

**Luke**: You write that your “attempts to ensure… [that] commercial deployments of AI technologies should… carry out active safety management” have not yet received as much traction as you would like. Could you go into more detail on that? What did you try to accomplish on this front that didn’t get adopted by others, or wasn’t implemented?

---

**John**: Having worked in medical AI from the early ‘seventies I have always been keenly aware that while AI can help to mitigate the effects of human error there is a potential downside too. AI systems could be programmed incorrectly, or their knowledge could prescribe inappropriate practices, or they could have the effect of deskilling the human professionals who have the final responsibility for their patients. Despite well-known limitations of human cognition people remain far and away the most versatile and creative problem solvers on the planet.

In the early ‘nineties I had the opportunity to set up a project whose goal was to establish a rigorous framework for the design and implementation of AI systems for safety critical applications. Medicine was our practical focus but the RED project[1](https://intelligence.org/2014/09/04/john-fox/#footnote_0_11319 "Rigorously Engineered Decisions") was aimed at the development of a general architecture for the design of autonomous agents that could be trusted to make decisions and carry out plans as reliably and safely as possible, certainly to be as competent and hence as trustworthy as human agents in comparable tasks. This is obviously a hard problem but we made [sufficient progress](http://www.tandfonline.com/doi/abs/10.1080/095281397146979) on theoretical issues and design principles that I thought there was a good chance the techniques might be applicable in medical AI and maybe even more widely.

I thought AI was like medicine, where we all take it for granted that medical equipment and drug companies have a duty of care to show that their products are effective and safe before they can be certificated for commercial use. I also assumed that AI researchers would similarly recognize that we have a “duty of care” to all those potentially affected by poor engineering or misuse in safety critical settings but this was naïve. The commercial tools that have been based on the technologies derived from AI research have to date focused on just getting and keeping customers and safety always takes a back seat.

In retrospect I should have predicted that making sure that AI products are safe is not going to capture the enthusiasm of commercial suppliers. If you compare AI apps with drugs we all know that pharmaceutical companies have to be firmly regulated to make sure they fulfill their duty of care to their customers and patients. However proving drugs are safe is expensive and also runs the risk of revealing that your new wonder-drug isn’t even as effective as you claim! It’s the same with AI.

I continue to be surprised how optimistic software developers are – they always seem to have supreme confidence that worst-case scenarios wont happen, or that if they do happen then their management is someone else’s responsibility. That kind of technical over-confidence has led to countless catastrophes in the past, and it amazes me that it persists.

There is another piece to this, which concerns the roles and responsibilities of AI researchers. How many of us take the risks of AI seriously so that it forms a part of our day-to-day theoretical musings and influences our projects? MIRI has put one worst case scenario in front of us – the possibility that our creations might one day decide to obliterate us – but so far as I can tell the majority of working AI professionals either see safety issues as irrelevant to the pursuit of interesting scientific questions or, like the wider public, that the issues are just science fiction.

I think experience in medical AI trying to articulate and cope with human risk and safety may have a couple of important lessons for the wider AI community. First we have a duty of care that professional scientists cannot responsibly ignore. Second, the AI business will probably need to be regulated, in much the same way as the pharmaceutical business is. If these propositions are correct then the AI research community would be wise to engage with and lead on discussions around safety issues if it wants to ensure that the regulatory framework that we get is to our liking!

---

**Luke**: Now you write, “That kind of technical over-confidence has led to countless catastrophes in the past…” What are some example “catastrophes” you’re thinking of?

---

**John**: Psychologists have known for years that human decision-making is flawed, even if amazingly creative sometimes, and overconfidence is an important source of error in routine settings. A large part of the motivation for applying AI in medicine comes from the knowledge that, in the words of the Institute of Medicine, “To err is human” and overconfidence is an established cause of clinical mistakes.[2](https://intelligence.org/2014/09/04/john-fox/#footnote_1_11319 "Overconfidence in major disasters:
• D. Lucas. Understanding the Human Factor in Disasters. Interdisciplinary Science Reviews. Volume 17 Issue 2 (01 June 1992), pp. 185-190.
• “Nuclear safety and security.
Psychology of overconfidence:
• Overconfidence effect.
• C. Riordan. Three Ways Overconfidence Can Make a Fool of You Forbes Leadership Forum.
Overconfidence in medicine:
• R. Hanson. Overconfidence Erases Doc Advantage. Overcoming Bias, 2007.
• E. Berner, M. Graber. Overconfidence as a Cause of Diagnostic Error in Medicine. The American Journal of Medicine. Volume 121, Issue 5, Supplement, Pages S2–S23, May 2008.
• T. Ackerman. Doctors overconfident, study finds, even in hardest cases. Houston Chronicle, 2013.
General technology example:
• J. Vetter, A. Benlian, T. Hess. Overconfidence in IT Investment Decisions: Why Knowledge can be a Boon and Bane at the same Time. ICIS 2011 Proceedings. Paper 4. December 6, 2011.")

Over-confidence and its many relatives (complacency, optimism, arrogance and the like) have a huge influence on our personal successes and failures, and our collective futures. The outcomes of the US and UK’s recent adventures around the world can be easily identified as consequences of overconfidence, and it seems to me that the polarized positions about global warming and planetary catastrophe are both expressions of overconfidence, just in opposite directions.

---

**Luke**: Looking much further out… if one day we can engineer [AGIs](http://intelligence.org/2013/08/11/what-is-agi/), do you think we are likely to figure out how to make them safe?

---

**John**: History says that making any technology safe is not an easy business. It took quite a few boiler explosions before high-pressure steam engines got their iconic centrifugal governors. Ensuring that new medical treatments are safe as well as effective is famously difficult and expensive. I think we should assume that getting to the point where an AGI manufacturer could guarantee its products are safe will be a hard road, and it is possible that guarantees are not possible in principle. We are not even clear yet what it means to be “safe”, at least not in computational terms.

It seems pretty obvious that entry level robotic products like the robots that carry out simple domestic chores or the “nursebots” that are being trialed for hospital use, have such a simple repertoire of behaviors that it should not be difficult to design their software controllers to operate safely in most conceivable circumstances. Standard safety engineering techniques like HAZOP[3](https://intelligence.org/2014/09/04/john-fox/#footnote_2_11319 "Hazard and operability study") are probably up to the job I think, and where software failures simply cannot be tolerated software engineering techniques like formal specification and model-checking are available.

There is also quite a lot of optimism around more challenging robotic applications like autonomous vehicles and medical robotics. Moustris et al.[4](https://intelligence.org/2014/09/04/john-fox/#footnote_3_11319 "Int J Med Robotics Comput Assist Surg 2011; 7: 375–39") say that autonomous surgical robots are emerging that can be used in various roles, automating important steps in complex operations like open-heart surgery for example, and they expect them to become standard in – and to revolutionize the practice of – surgery. However at this point it doesn’t seem to me that surgical robots with a significant cognitive repertoire are feasible and a human surgeon will be in the loop for the foreseeable future.

---

**Luke**: So what might artificial intelligence learn from natural intelligence?

---

As a cognitive scientist working in medicine my interests are co-extensive with those of scientists working on AGIs. Medicine is such a vast domain that practicing it safely requires the ability to deal with countless clinical scenarios and interactions and even when working in a single specialist subfield requires substantial knowledge from other subfields. So much so that it is now well known that even very experienced humans with a large clinical repertoire are subject to significant levels of error.[5](https://intelligence.org/2014/09/04/john-fox/#footnote_4_11319 "A. Ford. Domestic Robotics – Leave it to Roll-Oh, our Fun loving Retrobot. Institute for Ethics and Emerging Technologies, 2014.") An artificial intelligence that could be helpful across medicine will require great versatility, and this will require a general understanding of medical expertise and a range of cognitive capabilities like reasoning, decision-making, planning, communication, reflection, learning and so forth.

If human experts are not safe is it well possible to ensure that an AGI, however sophisticated, will be? I think that it is pretty clear that the range of techniques currently available for assuring system safety will be useful in making specialist AI systems reliable and minimizing the likelihood of errors in situations that their human designers can anticipate. However, AI systems with general intelligence will be expected to address scenarios and hazards that are beyond us to solve currently and often beyond designers even to anticipate. I am optimistic but at the moment I don’t see any convincing reason to believe that we have the techniques that would be sufficient to guarantee that a clinical super-intelligence is safe, let alone an AGI that might be deployed in many domains.

---

**Luke**: Thanks, John!

---

1. [Rigorously Engineered Decisions](http://www.cossac.org/projects/red)
2. Overconfidence in major disasters:
• D. Lucas. [*Understanding the Human Factor in Disasters.*](http://www.maneyonline.com/doi/abs/10.1179/isr.1992.17.2.185) Interdisciplinary Science Reviews. Volume 17 Issue 2 (01 June 1992), pp. 185-190.

• “[Nuclear safety and security.](http://en.wikipedia.org/wiki/Nuclear_safety_and_security)

Psychology of overconfidence:

• [Overconfidence effect.](http://en.wikipedia.org/wiki/Overconfidence_effect)

• C. Riordan. [Three Ways Overconfidence Can Make a Fool of You](http://www.forbes.com/sites/forbesleadershipforum/2013/01/08/three-ways-overconfidence-can-make-a-fool-of-you/) Forbes Leadership Forum.

Overconfidence in medicine:

• R. Hanson. [Overconfidence Erases Doc Advantage.](http://www.overcomingbias.com/2007/04/overconfidence_.html) Overcoming Bias, 2007.

• E. Berner, M. Graber. [Overconfidence as a Cause of Diagnostic Error in Medicine.](http://www.amjmed.com/article/S0002-9343(08)00040-5/abstract) The American Journal of Medicine. Volume 121, Issue 5, Supplement, Pages S2–S23, May 2008.

• T. Ackerman. [Doctors overconfident, study finds, even in hardest cases.](http://www.houstonchronicle.com/news/health/article/Doctors-overconfident-study-finds-even-in-4766096.php) Houston Chronicle, 2013.

General technology example:

• J. Vetter, A. Benlian, T. Hess. [Overconfidence in IT Investment Decisions: Why Knowledge can be a Boon and Bane at the same Time.](http://aisel.aisnet.org/icis2011/proceedings/generaltopics/4/) ICIS 2011 Proceedings. Paper 4. December 6, 2011.
3. [Hazard and operability study](http://en.wikipedia.org/wiki/Hazard_and_operability_study)
4. Int J Med Robotics Comput Assist Surg 2011; 7: 375–39
5. A. Ford. [Domestic Robotics – Leave it to Roll-Oh, our Fun loving Retrobot](http://ieet.org/index.php/IEET/more/ford20140702). Institute for Ethics and Emerging Technologies, 2014.

The post [John Fox on AI safety](https://intelligence.org/2014/09/04/john-fox/) appeared first on [Machine Intelligence Research Institute](https://intelligence.org).


![Daniel Roy portrait](http://intelligence.org/wp-content/uploads/2014/08/Roy__h692_w540.jpg) [Daniel Roy](http://danroy.org/) is an Assistant Professor of Statistics at the University of Toronto. Roy earned an S.B. and M.Eng. in Electrical Engineering and Computer Science, and a Ph.D. in Computer Science, from MIT.  His dissertation on probabilistic programming received the department’s George M Sprowls Thesis Award.  Subsequently, he held a Newton International Fellowship of the Royal Society, hosted by the Machine Learning Group at the University of Cambridge, and then held a Research Fellowship at Emmanuel College. Roy’s research focuses on theoretical questions that mix computer science, statistics, and probability.

**Luke Muehlhauser**: The abstract of [Ackerman, Freer, and Roy (2010)](http://danroy.org/papers/AckFreRoy-CompCondProb-preprint.pdf) begins:

> As inductive inference and machine learning methods in computer science see continued success, researchers are aiming to describe even more complex probabilistic models and inference algorithms. What are the limits of mechanizing probabilistic inference? We investigate the computability of conditional probability… and show that there are computable joint distributions with noncomputable conditional distributions, ruling out the prospect of general inference algorithms.
>
>

In what sense does your result (with Ackerman & Freer) rule out the prospect of general inference algorithms?

---

**Daniel Roy**: First, it’s important to highlight that when we say “probabilistic inference” we are referring to the problem of computing [conditional probabilities](http://en.wikipedia.org/wiki/Conditional_probability), while highlighting the role of conditioning in Bayesian statistical analysis.

Bayesian inference centers around so-called posterior distributions. From a subjectivist standpoint, the posterior represents one’s updated beliefs after seeing (i.e., conditioning on) the data. Mathematically, a posterior distribution is simply a conditional distribution (and every conditional distribution can be interpreted as a posterior distribution in some statistical model), and so our study of the computability of conditioning also bears on the problem of computing posterior distributions, which is arguably one of the core computational problems in Bayesian analyses.

Second, it’s important to clarify what we mean by “general inference”. In machine learning and artificial intelligence (AI), there is a long tradition of defining formal languages in which one can specify probabilistic models over a collection of variables. Defining distributions can be difficult, but these languages can make it much more straightforward.

The goal is then to design algorithms that can use these representations to support important operations, like computing conditional distributions. Bayesian networks can be thought of as such a language: You specify a distribution over a collection of variables by specifying a graph over these variables, which breaks down the entire distribution into “local” conditional distributions corresponding with each node, which are themselves often represented as tables of probabilities (at least in the case where all variables take on only a finite set of values). Together, the graph and the local conditional distributions determine a unique distribution over all the variables.

An inference algorithms that support the entire class of all finite, discrete, Bayesian networks might be called general, but as a class of distributions, those having finite, discrete Bayesian networks is a rather small one.

In this work, we are interested in the prospect of algorithms that work on very large classes of distributions. Namely, we are considering the class of samplable distributions, i.e., the class of distributions for which there exists a probabilistic program that can generate a sample using, e.g., uniformly distributed random numbers or independent coin flips as a source of randomness. The class of samplable distributions is a natural one: indeed it is equivalent to the class of computable distributions, i.e., those for which we can devise algorithms to compute lower bounds on probabilities from descriptions of open sets. The class of samplable distributions is also equivalent to the class of distributions for which we can compute expectations from descriptions of bounded continuous functions.

The class of samplable distributions is, in a sense, the richest class you might hope to deal with. The question we asked was: is there an algorithm that, given a samplable distribution on two variables X and Y, represented by a program that samples values for both variables, can compute the conditional distribution of, say, Y given X=x, for almost all values for X? When X takes values in a finite, discrete set, e.g., when X is binary valued, there is a general algorithm, although it is inefficient. But when X is continuous, e.g., when it can take on every value in the unit interval [0,1], then problems can arise. In particular, there exists a distribution on a pair of numbers in [0,1] from which one can generate perfect samples, but for which it is impossible to compute conditional probabilities for one of the variables given the other. As one might expect, the proof reduces the halting problem to that of conditioning a specially crafted distribution.

This pathological distribution rules out the possibility of a general algorithm for conditioning (equivalently, for probabilistic inference). The paper ends by giving some further conditions that, when present, allow one to devise general inference algorithms. Those familiar with computing conditional distributions for finite-dimensional statistical models will not be surprised that conditions necessary for Bayes’ theorem are one example.

---

**Luke**: In your dissertation (and perhaps elsewhere) you express a particular interest in the relevance of probabilistic programming to AI, including the original aim of AI to build machines which rival the general intelligence of a human. How would you describe the relevance of probabilistic programming to the long-term dream of AI?

---

**Daniel**: If you look at early probabilistic programming systems, they were built by AI researchers: De Raedt, Koller, McAllester, Muggleton, Pfeffer, Poole, Sato, to name a few. The Church language, which was introduced in joint work with Bonawitz, Mansinghka, Goodman, and Tenenbaum while I was a graduate student at MIT, was conceived inside a cognitive science laboratory, foremost to give us a language rich enough to express the range of models that people were inventing all around us. So, for me, there’s always been a deep connection. On the other hand, the machine learning community as a whole is somewhat allergic to AI and so the pitch to that community has more often been pragmatic: these systems may someday allow experts to conceive, prototype, and deploy much larger probabilistic systems, and at the same time, empower a much larger community of nonexperts to use probabilistic modeling techniques to understand their data. This is the basis for the DARPA [PPAML](http://ppaml.galois.com/wiki/) program, which is funding 8 or so teams to engineer scalable systems over the next 4 years.

From an AI perspective, probabilistic programs are an extremely general representation of knowledge, and one that identifies uncertainty with stochastic computation. Freer, Tenenbaum, and I recently wrote [a book chapter](http://danroy.org/papers/FreRoyTen-Turing.pdf) for the Turing centennial that uses a classical medical diagnosis example to showcase the flexibility of probabilistic programs and a general QUERY operator for performing probabilistic conditioning. Admittedly, the book chapter ignores the computational complexity of the QUERY operator, and any serious proposal towards AI cannot do this indefinitely. Understanding when we can hope to efficiently update our knowledge in light of new observations is a rich source of research questions, both applied and theoretical, spanning not only AI and machine learning, but also statistics, probability, physics, theoretical computer science, etc.

---

**Luke**: Is it fair to think of QUERY as a “toy model” that we can work with in concrete ways to gain more general insights into certain parts of the long-term AI research agenda, even though QUERY is unlikely to be directly implemented in advanced AI systems? (E.g. that’s how I think of AIXI.)

---

**Daniel**: I would hesitate to call QUERY a toy model. Conditional probability is a difficult concept to master, but, for those adept at reasoning about the execution of programs, QUERY demystifies the concept considerably. QUERY is an important conceptual model of probabilistic conditioning.

That said, the simple guess-and-check algorithm we present in our Turing article runs in time inversely proportional to the probability of the event/data on which one is conditioning. In most statistical settings, the probability of a data set decays exponentially towards 0 as a function of the number of data points, and so guess-and-check is only useful for reasoning with toy data sets in these settings. It should come as no surprise to hear that state-of-the-art probabilistic programming systems work nothing like this.

On the other hand, QUERY, whether implemented in a rudimentary fashion or not, can be used to represent and reason probabilistically about arbitrary computational processes, whether they are models of the arrival time of spam, the spread of disease through networks, or the light hitting our retinas. Computer scientists, especially those who might have had a narrow view of the purview of probability and statistics, will see a much greater overlap between these fields and their own once they understand QUERY.

To those familiar with AIXI, the difference is hopefully clear: QUERY performs probabilistic reasoning in a model given as input. AIXI, on the other hand, is itself a “universal” model that, although not computable, would likely predict (hyper)intelligent behavior, were we (counterfactually) able to perform the requisite probabilistic inferences (and feed it enough data). Hutter gives an algorithm implementing an approximation to AIXI, but its computational complexity still scales exponentially in space. AIXI is fascinating in many ways: If we ignore computational realities, we get a complete proposal for AI. On the other hand, AIXI and its approximations take maximal advantage of this computational leeway and are, therefore, ultimately unsatisfying. For me, AIXI and related ideas highlight that AI must be as much a study of the particular as it of the universal. Which potentially unverifiable, but useful, assumptions will enable us to efficiently represent, update, and act upon knowledge under uncertainty?

---

**Luke**: You write that “AI must be as much a study of the particular as it is of the universal.” Naturally, most AI scientists are working on the particular, the near term, the applied. In your view, what are some other examples of work on the universal, in AI? Schmidhuber’s Gödel machine comes to mind, and also some work that is as likely to be done in a logic or formal philosophy department as a computer science department — e.g. perhaps work on logical priors — but I’d love to hear what kinds of work you’re thinking of.

---

**Daniel**: I wouldn’t equate any two of the particular, near-term, or applied. By the word particular, I am referring to, e.g., the way that our environment affects, but is also affected by, our minds, especially through society. More concretely, both the physical spaces in which most of us spend our days and the mental concepts we regularly use to think about our daily activities are products of the human mind. But more importantly, these physical and mental spaces are necessarily ones that are easily navigated by our minds. The coevolution by which this interaction plays out is not well studied in the context of AI. And to the extent that this cycle dominates, we would expect a universal AI to be truly alien. On the other hand, exploiting the constraints of human constructs may allow us to build more effective AIs.

As for the universal, I have an interest in the way that noise can render idealized operations computable or even efficiently computable. In our work on the computability of conditioning that came up earlier in the discussion, we show that adding sufficiently smooth independent noise to a random variable allows us to perform conditioning in situations where we would not have been able to otherwise. There are examples of this idea elsewhere. For example, [Braverman, Grigo, and Rojas](http://arxiv.org/abs/1201.0488) study noise and intractability in dynamical systems. Specifically, they show that computing the invariant measure characterizing the long-term statistical behavior of dynamical systems is not possible. The road block is the computational power of the dynamical system itself. The addition of a small amount of noise to the dynamics, however, decreases the computational power of the dynamical system, and suffices to make the invariant measure computable. In a world subject to noise (or, at least, well modeled as such), it seems that many theoretical obstructions melt away.

---

**Luke**: Thanks, Daniel!

The post [Daniel Roy on probabilistic programming and AI](https://intelligence.org/2014/09/04/daniel-roy/) appeared first on [Machine Intelligence Research Institute](https://intelligence.org).


MIRI’s September Newsletter

|  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- |
|

|  |  |
| --- | --- |
|

|  |
| --- |
|
[Machine Intelligence Research Institute](http://intelligence.org)
 |

 |
|

|  |  |
| --- | --- |
|

|  |
| --- |
|
Thanks to the generosity of 100+ donors, we [successfully completed](http://intelligence.org/2014/08/15/2014-summer-matching-challenge-completed/) our 2014 summer matching challenge on August 15th, raising more than $400,000 total for [our research program](http://intelligence.org/research/). Our deepest thanks to all our supporters!

**Research updates**
* New paper: “[Exploratory engineering in artificial intelligence](http://intelligence.org/2014/08/22/new-paper-exploratory-engineering-artificial-intelligence/).”
* 2 new analyses: “[Groundwork for AGI safety engineering](http://intelligence.org/2014/08/04/groundwork-ai-safety-engineering/)” and “[Loosemore on AI safety and attractors](http://nothingismere.com/2014/08/25/loosemore-on-ai-safety-and-attractors/).”

**News updates**
* MIRI is running an online reading group for Nick Bostrom’s *Superintelligence*. Join the discussion [here](http://intelligence.org/2014/08/31/superintelligence-reading-group/)!
* MIRI participated in the 2014 Effective Altruism Summit. Slides from our talks are available [here](http://intelligence.org/2014/08/11/miris-recent-effective-altruism-talks/).

**Other updates**
* Nick Bostrom is touring the US and Europe to discuss *Superintelligence*. Dates are available on [his website](http://www.nickbostrom.com/). He visits Berkeley on September 12th; details [here](http://intelligence.org/2014/07/25/bostrom/).
* Paul Christiano and Katja Grace have launched a new website containing many analyses related to the long-term future of AI: [AI Impacts](http://www.aiimpacts.org/).
* Paul Christiano has published several additional analyses closely related to MIRI’s Friendly AI interests: “[Three impacts of machine intelligence](http://rationalaltruist.com/2014/08/23/three-impacts-of-machine-intelligence/),” “[Approval-seeking [agents]](http://ordinaryideas.wordpress.com/2014/07/21/approval-seeking/),” “[Straightforward vs. goal-oriented communication](http://ordinaryideas.wordpress.com/2014/08/23/straightforward-vs-goal-oriented-communication/),” “[Specifying a human precisely](http://ordinaryideas.wordpress.com/2014/08/24/specifying-a-human-precisely-reprise/),” “[Specifying enlightened judgment precisely](http://ordinaryideas.wordpress.com/2014/08/27/specifying-enlightened-judgment-precisely-reprise/),” “[Challenges for extrapolation](http://ordinaryideas.wordpress.com/2014/08/27/challenges-for-extrapolation/),” and “[Confronting Gödelian difficulties](http://ordinaryideas.wordpress.com/2014/08/30/confronting-godelian-difficulties-reprise/).”
* The [Hanson-Yudkowsky AI Foom Debate](http://intelligence.org/ai-foom-debate/) continues with Hanson’s post “[I Still Don’t Get Foom](http://www.overcomingbias.com/2014/07/30855.html).”

As always, please don’t hesitate to let us know if you have any questions or comments.
Best,
Luke Muehlhauser
Executive Director
 |

 |

 |
|  |

  |

The post [MIRI’s September Newsletter](https://intelligence.org/2014/09/01/september-newsletter-2/) appeared first on [Machine Intelligence Research Institute](https://intelligence.org).


Superintelligence reading group

Nick Bostrom’s eagerly awaited [*Superintelligence*](http://smile.amazon.com/Superintelligence-Dangers-Strategies-Nick-Bostrom/dp/0199678111/) comes out in the US this week. To help you get the most out of it, MIRI is running an online reading group where you can join with others to ask questions, discuss ideas, and probe the arguments more deeply.

The reading group will “meet” on a weekly post on the [LessWrong discussion forum](http://lesswrong.com/r/discussion/new/). For each ‘meeting’, we will read about half a chapter of *Superintelligence*, then come together virtually to discuss. I’ll summarize the chapter, and offer a few relevant notes, thoughts, and ideas for further investigation. (My notes will also be used as the source material for the final reading guide for the book.)

Discussion will take place in the comments. I’ll offer some questions, and invite you to bring your own, as well as thoughts, criticisms and suggestions for interesting related material. Your contributions to the reading group might also (with permission) be used in our final reading guide for the book.

We welcome both newcomers and veterans on the topic. Content will aim to be intelligible to a wide audience, and topics will range from novice to expert level. All levels of time commitment are welcome. **We especially encourage AI researchers and practitioners to participate.** Just use a pseudonym if you don’t want your questions and comments publicly linked to your identity.

We will follow [**this preliminary reading guide**](https://intelligence.org/wp-content/uploads/2014/08/Superintelligence-Readers-Guide-early-version.pdf), produced by MIRI, reading one section per week.

If you have already read the book, don’t worry! To the extent you remember what it says, your superior expertise will only be a bonus. To the extent you don’t remember what it says, now is a good time for a review! If you don’t have time to read the book, but still want to participate, you are also welcome to join in. I will provide summaries, and many things will have page numbers, in case you want to skip to the relevant parts.

If this sounds good to you, first grab a copy of *[Superintelligence](http://www.amazon.com/Superintelligence-Dangers-Strategies-Nick-Bostrom/dp/0199678111)*. You may also want to [**sign up here**](http://eepurl.com/1-S41) to be emailed when the discussion begins each week. The first virtual meeting (forum post) will go live at 6pm Pacific on **Monday, September 15th**. Following meetings will start at 6pm every Monday, so if you’d like to coordinate for quick fire discussion with others, put that into your calendar. If you prefer flexibility, come by any time! And remember that if there are any people you would especially enjoy discussing *Superintelligence* with, link them to this post!

Topics for the first week will include impressive displays of artificial intelligence, why computers play board games so well, and what a reasonable person should infer from the agricultural and industrial revolutions.

![](http://images.lesswrong.com/t3_ku5_0.png)

The post [Superintelligence reading group](https://intelligence.org/2014/08/31/superintelligence-reading-group/) appeared first on [Machine Intelligence Research Institute](https://intelligence.org).


[![Exploratory engineering](https://intelligence.org/wp-content/uploads/2014/08/Exploratory-engineering.png)](https://intelligence.org/files/ExploratoryEngineeringAI.pdf)Luke Muehlhauser and Bill Hibbard have a new paper ([PDF](https://intelligence.org/files/ExploratoryEngineeringAI.pdf)) in the September 2014 issue of *Communications of the ACM*, the world’s most-read peer-reviewed computer science publication. The title is “[Exploratory Engineering in Artificial Intelligence](http://cacm.acm.org/magazines/2014/9/177932-exploratory-engineering-in-artificial-intelligence/abstract).”

Excerpt:

> We regularly see examples of new artificial intelligence (AI) capabilities… No doubt such automation will produce tremendous economic value, but will we be able to *trust* these advanced autonomous systems with so much capability?
>
>
> Today, AI safety engineering mostly consists in a combination of formal methods and testing. Though powerful, these methods lack foresight: they can be applied only to particular extant systems. We describe a third, complementary approach that aims to predict the (potentially hazardous) properties and behaviors of broad classes of future AI agents, based on their mathematical structure (for example, reinforcement learning)… We call this approach “exploratory engineering in AI.”
>
>
> …
>
>
> In this Viewpoint, we focus on theoretical AI models inspired by Marcus Hutter’s AIXI, an optimal agent model for maximizing an environmental reward signal…
>
>
> …
>
>
> Autonomous intelligent machines have the potential for large impacts on our civilization. Exploratory engineering gives us the capacity to have some foresight into what these impacts might be, by analyzing the properties of agent designs based on their mathematical form. Exploratory engineering also enables us to identify lines of research — such as the study of Dewey’s value-learning agents — that may be important for anticipating and avoiding unwanted AI behaviors. This kind of foresight will be increasingly valuable as machine intelligence comes to play an ever-larger role in our world.
>
>

The post [New paper: “Exploratory engineering in artificial intelligence”](https://intelligence.org/2014/08/22/new-paper-exploratory-engineering-artificial-intelligence/) appeared first on [Machine Intelligence Research Institute](https://intelligence.org).


Thanks to the generosity of 100+ donors, today we successfully completed our [2014 summer matching challenge](http://intelligence.org/2014/07/21/2014-summer-matching-challenge/), raising more than $400,000 total for our [research program](http://intelligence.org/research/).

Our deepest thanks to all our supporters!

Also, Jed McCaleb’s new crypto-currency [Stellar](https://www.stellar.org/blog/introducing-stellar/) was launched during MIRI’s fundraiser, and we decided to [accept donated stellars](https://intelligence.org/donate/). These donations weren’t counted toward the matching drive, and their [market value](http://www.stellarvalue.org/) is unstable at this early stage, but as of today we’ve received 850,000+ donated stellars from 3000+ different stellar accounts. Our thanks to everyone who donated in stellar!

The post [2014 Summer Matching Challenge Completed!](https://intelligence.org/2014/08/15/2014-summer-matching-challenge-completed/) appeared first on [Machine Intelligence Research Institute](https://intelligence.org).


MIRI’s recent effective altruism talks

![EA Summit 14](https://intelligence.org/wp-content/uploads/2014/08/EA-Summit-14.png)MIRI recently participated in the 2014 Effective Altruism Retreat and [Effective Altruism Summit](http://www.effectivealtruismsummit.com/) organized by Leverage Research. We gave four talks, participated in a panel, and held “office hours” during which people could stop by and ask us questions.

The slides for our talks are available below:

* Muehlhauser, [MIRI Intro](https://intelligence.org/wp-content/uploads/2014/08/Muehlhauser-MIRI-Intro-2014.pdf)
* Yudkowsky, Caring about Many Distant People in Strange Situations [no slides]
* Muehlhauser, [Steering the Future of AI](https://intelligence.org/wp-content/uploads/2014/08/Muehlhauser-Steering-the-Future-of-AI.pdf)
* Yudkowsky, [Why We’re Doing It and What MIRI is Doing](https://intelligence.org/wp-content/uploads/2014/08/Yudkowsky-Why-were-doing-it-and-what-MIRI-is-doing.pdf)

If videos of these talks become available, we’ll link them from here as well.

See also our earlier posts [Friendly AI Research as Effective Altruism](http://intelligence.org/2013/06/05/friendly-ai-research-as-effective-altruism/) and [Why MIRI?](http://intelligence.org/2014/04/20/why-miri/)

The post [MIRI’s recent effective altruism talks](https://intelligence.org/2014/08/11/miris-recent-effective-altruism-talks/) appeared first on [Machine Intelligence Research Institute](https://intelligence.org).