Nick Bostrom’s concerns about the future of AI have sparked a busy public discussion. His arguments were echoed by leading AI researcher [Stuart Russell](http://www.cs.berkeley.edu/~russell/) in “[Transcending complacency on superintelligent machines](http://www.huffingtonpost.com/stephen-hawking/artificial-intelligence_b_5174265.html)” (co-authored with Stephen Hawking, Max Tegmark, and Frank Wilczek), and a number of journalists, scientists, and technologists have subsequently chimed in. Given the topic’s complexity, I’ve been surprised by the positivity and thoughtfulness of most of the coverage (some [overused clichés](http://slatestarcodex.com/2014/08/26/if-the-media-reported-on-other-dangers-like-it-does-ai-risk/) aside).

Unfortunately, what most people probably take away from these articles is ‘Stephen Hawking thinks AI is scary!’, not the chains of reasoning that led Hawking, Russell, or others to their present views. When Elon Musk [chimes in](http://www.theverge.com/2014/8/3/5965099/elon-musk-compares-artificial-intelligence-to-nukes) with his own concerns and cites Bostrom’s book [*Superintelligence: Paths, Dangers, Strategies*](http://www.amazon.com/Superintelligence-Dangers-Strategies-Nick-Bostrom/dp/0199678111), commenters seem to be more interested in immediately echoing or dismissing Musk’s worries than in looking into his source.

The end result is more of a referendum on people’s positive or negative associations with the word ‘AI’ than a debate over Bostrom’s substantive claims. If ‘AI’ calls to mind science fiction dystopias for you, the temptation is to squeeze real AI researchers into your ‘mad scientists poised to unleash an evil robot army’ stereotype. Equally, if ‘AI’ calls to mind your day job testing edge detection algorithms, that same urge to force new data into old patterns makes it tempting to squeeze Bostrom and Hawking into the ‘naïve technophobes worried about the evil robot uprising’ stereotype.

Thus roboticist Rodney Brooks’ recent blog post “[**Artificial intelligence is a tool, not a threat**](http://www.rethinkrobotics.com/artificial-intelligence-tool-threat/)” does an excellent job dispelling common myths about the cutting edge of AI, and philosopher John Searle’s [**review of** ***Superintelligence***](http://www.nybooks.com/articles/archives/2014/oct/09/what-your-computer-cant-know/) draws out some important ambiguities in our concepts of subjectivity and mind; but both writers scarcely intersect with Bostrom’s (or Russell’s, or Hawking’s) ideas. Both pattern-match Bostrom to the nearest available ‘evil robot panic’ stereotype, and stop there.

Brooks and Searle don’t appreciate how new the arguments in *Superintelligence* are. In the interest of making it easier to engage with these important topics, and less appealing to force the relevant technical and strategic questions into the model of decades-old debates, I’ll address three of the largest misunderstandings one might come away with after seeing Musk, Searle, Brooks, and others’ public comments: conflating present and future AI risks, conflating risk severity with risk imminence, and conflating risk from autonomous algorithmic decision-making with risk from human-style antisocial dispositions.

**Misconception #1: Worrying about AGI means worrying about narrow AI**

Some of the miscommunication in this debate can be blamed on bad terminology. By ‘AI,’ researchers in the field generally mean a range of techniques used in machine learning, robotics, speech recognition, etc. ‘AI’ *also* gets tossed around as a shorthand for ‘artificial *general* intelligence’ (AGI) or ‘human-level AI.’ Keeping a close eye on technologies that are likely to lead to AGI isn’t the same thing as keeping a close eye on AI in general, and it isn’t surprising that AI researchers would find the latter proposal puzzling. (It doesn’t help that most researchers are hearing these arguments indirectly, and aren’t aware of the specialists in AI and technological forecasting who are making the same arguments as Hawking — or haven’t encountered *arguments* for looking into AGI safety at all, just melodramatic headlines and tweets.)

Brooks thinks that behind this terminological confusion lies an empirical confusion on the part of people calling for AGI safety research. He takes it that people’s worries about “evil AI” must be based on a mistaken view of how powerful narrow AI is, or how large are the strides it’s making toward general intelligence:

> I think the worry stems from a fundamental error in not distinguishing the difference between the very real recent advances in a particular aspect of AI, and the enormity and complexity of building sentient volitional intelligence.
>
>

One good reason to think otherwise is that Bostrom is the director of the [Future of Humanity Institute](http://www.fhi.ox.ac.uk/) (FHI), an Oxford research center investigating the largest technology trends and challenges we are likely to see on a timescale of centuries. Futurists like Bostrom are looking for ways to invest early in projects that will pay major long-term dividends — guarding against catastrophic natural disasters, developing space colonization capabilities, etc. If Bostrom learned that a critically important technology were 50 or more years away, it would be substantially out of character for him to suddenly stop caring about it.

When groups that are in the midst of a lively conversation about nuclear proliferation, global biosecurity, and humanity’s [cosmic endowment](http://www.nickbostrom.com/astronomical/waste.html) collide with groups that are having their own lively conversation about revolutionizing housecleaning and designing more context-sensitive smartphone apps, some amount of inferential distance (to say nothing of mood whiplash) is inevitable. I’m reminded of the ‘But it’s snowing outside!’ rejoinder to people worried about the large-scale human cost of climate change. It’s not that local weather is unimportant, or that it’s totally irrelevant to long-term climatic warming trends; it’s that there’s been a rather sudden change in topic.[1](https://intelligence.org/2015/01/08/brooks-searle-agi-volition-timelines/#footnote_0_11512 "Similarly, narrow AI isn’t irrelevant to AGI risk. It’s certainly likely that building an AGI will require us to improve the power and generality of narrow AI methods. However, that doesn’t mean that AGI techniques will look like present-day techniques, or that all AI techniques are dangerous.")

We should be more careful about distinguishing these two senses of ‘AI.’ We may not understand AGI well enough to precisely [define](http://intelligence.org/2013/08/11/what-is-agi/) it, but we can at least take the time to clarify the topic of discussion: Nobody’s asking whether a conspiracy of roombas and chatterbots could take over the world.

[![Image 1](http://intelligence.org/wp-content/uploads/2014/12/Image-1.jpg)](http://intelligence.org/wp-content/uploads/2014/12/Image-1.jpg)*When robots attack! (*[*Source: xkcd*](https://what-if.xkcd.com/5/)*.)*

**Misconception #2:** **Worrying about AGI means being confident it’s near**

A number of futurists, drawing inspiration from Ray Kurzweil’s claim that technological progress inevitably follows a Moore’s-law-style exponential trajectory, have made some very confident predictions about [AGI](http://intelligence.org/2013/05/15/when-will-ai-be-created/) [timelines](http://intelligence.org/2013/05/15/when-will-ai-be-created/). Kurzweil himself argues that we can expect to produce human-level AI in about 15 years, followed by superintelligent AI 15 years after that.[2](https://intelligence.org/2015/01/08/brooks-searle-agi-volition-timelines/#footnote_1_11512 "Kurzweil, in The Singularity is Near (pp. 262-263): “Once we’ve succeeded in creating a machine that can pass the Turing test (around 2029), the succeeding period will be an era of consolidation in which nonbiological intelligence will make rapid gains. However, the extraordinary expansion contemplated for the Singularity, in which human intelligence is multiplied by billions, won’t take place until the mid-2040s[.]”") Brooks responds that the ability to design an AGI may lag far behind the computing power required to run one:

> As a comparison, consider that we have had winged flying machines for well over 100 years. But it is only very recently that people like Russ Tedrake at MIT CSAIL have been able to get them to land on a branch, something that is done by a bird somewhere in the world at least every microsecond. Was it just Moore’s law that allowed this to start happening? Not really. It was figuring out the equations and the problems and the regimes of stall, etc., through mathematical understanding of the equations. Moore’s law has helped with MATLAB and other tools, but it has not simply been a matter of pouring more computation onto flying and having it magically transform. And it has taken a long, long time.
>
>
> Expecting more computation to just magically get to intentional intelligences, who understand the world is similarly unlikely.[3](https://intelligence.org/2015/01/08/brooks-searle-agi-volition-timelines/#footnote_2_11512 "Hadi Esmaeilzadeh argues, moreover, that we cannot take for granted that our computational resources will continue to rapidly increase.")
>
>

This is an entirely correct point. However, Bostrom’s views are the ones that set off the recent public debate, and Bostrom isn’t a Kurzweilian. It may be that Brooks is running off of the assumption ‘if you say AGI safety is an urgent issue, you must think that AGI is imminent,’ in combination with ‘if you think AGI is imminent, you must have bought into Kurzweil’s claims.’ Searle, in spite of having read *Superintelligence*, gives voice to a similar conclusion:

> Nick Bostrom’s book, *Superintelligence*, warns of the impending apocalypse. We will soon have intelligent computers, computers as intelligent as we are, and they will be followed by superintelligent computers vastly more intelligent that are quite likely to rise up and destroy us all.
>
>

If what readers take away from language like “impending” and “soon” is that Bostrom is unusually confident that AGI will come early, or that Bostrom is confident we’ll build a general AI this century, then they’ll be getting the situation exactly backwards.

According to a [2013 survey](http://www.nickbostrom.com/papers/survey.pdf) of the most cited authors in artificial intelligence, experts expect AI to be able to “carry out most human professions at least as well as a typical human” with a 10% probability by the (median) year 2024, with 50% probability by 2050, and with 90% probability by 2070, assuming uninterrupted scientific progress. Bostrom is *less* confident than this that AGI will arrive so soon:

> My own view is that the median numbers reported in the expert survey do not have enough probability mass on later arrival dates. A 10% probability of HLMI [human-level machine intelligence] not having been developed by 2075 or even 2100 (after conditionalizing on “human scientific activity continuing without major negative disruption”) seems too low.
>
>
> Historically, AI researchers have not had a strong record of being able to predict the rate of advances in their own field or the shape that such advances would take. On the one hand, some tasks, like chess playing, turned out to be achievable by means of surprisingly simple programs; and naysayers who claimed that machines would “never” be able to do this or that have repeatedly been proven wrong. On the other hand, the more typical errors among practitioners have been to underestimate the difficulties of getting a system to perform robustly on real-world tasks, and to overestimate the advantages of their own particular pet project or technique.
>
>

Bostrom *does* think that superintelligent AI is likely to arise soon after the first AGI, via an [intelligence explosion](https://intelligence.org/files/IE-EI.pdf). Once AI is capable of high-quality scientific inference and planning in domains like computer science, Bostrom predicts that the process of further improving AI will become increasingly automated. Silicon works cheaper and faster than a human programmer can, and a program that can improve the efficiency of its own planning and science abilities could substantially outpace humans in scientific and decision-making tasks long before hitting diminishing marginal returns in self-improvements.

However, the question of how soon we will create AGI is distinct from the question of how soon thereafter AGI will systematically outperform humans. Analogously, you can think that the arrival of quantum computers will swiftly revolutionize cybersecurity, without asserting that quantum computers are imminent. A failure to disentangle these two theses might be one reason for the confusion about Bostrom’s views.[4](https://intelligence.org/2015/01/08/brooks-searle-agi-volition-timelines/#footnote_3_11512 "The “Transcending complacency on superintelligent machines” article argues, similarly, that intelligence explosion and superintelligent AI are important possibilities for us to investigate now, even though they are “long-term” problems compared to AI-mediated economic disruptions and autonomous weapons.")

If the director of FHI ([along with the director of MIRI](https://intelligence.org/2014/10/31/financial-times-story-miri/)) is relatively skeptical that we’ll see AGI soon — albeit quite a bit less skeptical than Brooks — why does he think we should commit attention to this issue now? One reason is that reliable AGI is likely to be much more difficult to build than AGI. It wouldn’t be much consolation to learn that AGI is 200 years away, if we also learned that *safe* AGI were *250* years away. In existing cyber-physical systems, safety generally lags behind capability.[5](https://intelligence.org/2015/01/08/brooks-searle-agi-volition-timelines/#footnote_4_11512 "Kathleen Fisher notes:
In general, research into capabilities outpaces the corresponding research into how to make those capabilities secure. The question of security for a given capability isn’t interesting until that capability has been shown to be possible, so initially researchers and inventors are naturally more focused on the new capability rather than on its associated security. Consequently, security often has to catch up once a new capability has been invented and shown to be useful.
In addition, by definition, new capabilities add interesting and useful new capabilities, which often increase productivity, quality of life, or profits. Security adds nothing beyond ensuring something works the way it is supposed to, so it is a cost center rather than a profit center, which tends to suppress investment.
") If we want to reverse that trend by the time we have AGI, we’ll probably need a big head start. MIRI’s [research guide](http://intelligence.org/research-guide/) summarizes some of the active technical work on this problem. Similar progress in [exploratory engineering](http://intelligence.org/2014/08/22/new-paper-exploratory-engineering-artificial-intelligence/) has proved fruitful in preparing for [post-quantum cryptography](http://intelligence.org/2014/05/07/harry-buhrman/) and [covert channel communication](http://intelligence.org/2014/04/12/jonathan-millen/).

A second reason to prioritize AGI safety research is that there is a great deal of uncertainty about when AGI will be developed. It could come sooner than we expect, and it would be much better to end up with a system that’s *too* safe than one that’s not safe enough.

Brooks recognizes that AI predictions tend to be wildly unreliable, yet he also seems confident that general-purpose AI is multiple centuries away (and that this makes AGI safety a non-issue):

> Just how open the question of time scale for when we will have human level AI is highlighted by a recent report by Stuart Armstrong and Kaj Sotala, of the Machine Intelligence Research Institute, an organization that itself has researchers worrying about evil AI. But in this more sober report, the authors analyze 95 predictions made between 1950 and the present on when human level AI will come about. They show that there is no difference between predictions made by experts and non-experts. And they also show that over that 60 year time frame there is a strong bias towards predicting the arrival of human level AI as between 15 and 25 years from the time the prediction was made. To me that says that no one knows, they just guess, and historically so far most predictions have been outright wrong!
>
>
> I say relax everybody. If we are spectacularly lucky we’ll have AI over the next thirty years with the intentionality of a lizard, and robots using that AI will be useful tools.
>
>

*We have no idea when AGI will arrive! Relax!* One of the authors Brooks cites, Kaj Sotala,[6](https://intelligence.org/2015/01/08/brooks-searle-agi-volition-timelines/#footnote_5_11512 "Bostrom cites Armstrong and Sotala’s study in Superintelligence (pp. 3-4), adding:
Machines matching humans in general intelligence […] have been expected since the invention of the computers in the 1940s. At that time, the advent of such machines was often placed some twenty years into the future. Since then, the expected arrival date has been receding at a rate of one year per year; so that today, futurists who concern themselves with the possibility of artificial general intelligence still often believe that intelligent machines are a couple of decades away.
Two decades is a sweet spot for prognosticators of radical change: near enough to be attention-grabbing and relevant, yet far enough to make it possible to suppose that a string of breakthroughs, currently only vaguely imaginable, might by then have occurred. […] Twenty years may also be close to the typical duration remaining of a forecaster’s career, bounding the reputational risk of a bold prediction.
From the fact that some individuals have overpredicted artificial intelligence in the past, however, it does not follow that AI is impossible or will never be developed. The main reason why progress has been slower than expected is that the technical difficulties of constructing intelligent machines have proved greater than the pioneers foresaw. But this leaves open just how great those difficulties are and how far we now are from overcoming them. Sometimes a problem that initially looks hopelessly complicated turns out to have a surprisingly simple solution (though the reverse is probably more common).
") points out this odd juxtaposition in a blog comment:

> I do find it slightly curious to note that you first state that nobody knows when we’ll have AI and that everyone’s just guessing, and then in the very next paragraph, you make a very confident statement about human-level AI (HLAI) being so far away as to not be worth worrying about. To me, our paper suggests that the reasonable conclusion to draw is “maybe HLAI will happen soon, or maybe it will happen a long time from now – nobody really knows for sure, so we shouldn’t be too confident in our predictions in either direction”.
>
>

[Confident pessimism](http://lesswrong.com/lw/fmf/overconfident_pessimism/) about a technology’s feasibility can be just as mistaken as confident optimism. [Reversing the claims of an unreliable predictor](http://lesswrong.com/lw/lw/reversed_stupidity_is_not_intelligence/) does not necessarily get you a reliable prediction. A scientifically literate person living in 1850 could observe the long history of failed heavier-than-air flight attempts and predictions, and have grounds to be fairly skeptical that we’d have such machines within 60 years. On the other hand (though we should be wary of [hindsight bias](http://lesswrong.com/lw/im/hindsight_devalues_science/) here), it probably *wouldn’t* have been reasonable at the time to confidently conclude that heavier-than-air flight was ‘centuries away.’ There may not have been good reason to expect the Wright brothers’ success, but ignorance about how one might achieve something is not the same as positive knowledge that it’s effectively unachievable.

One would need a *very good model* of heavier-than-air flight in order to predict whether it’s 50 years away, or 100, or 500. In the same way, we would need to already understand AGI on a pretty sophisticated level in order to predict with any confidence that it will be invented closer to the year 2500 than to the year 2100. Extreme uncertainty about when an event will occur is not a justification for thinking it’s a long way off.

This isn’t an argument for thinking AGI is imminent. That prediction too would require that we claim more knowledge than we have. It’s entirely possible that we’re in the position of someone anticipating the Wright brothers from 1750, rather than from 1850. We should be able to have a sober discussion about each of these possibilities independently, rather than collapsing ‘is AGI an important risk?’, ‘is AI a valuable tool?’, and ‘is AI likely to produce AGI by the year such-and-such?’ into one black-and-white dilemma.

**Misconception #3: Worrying about AGI means worrying about “malevolent” AI**

Brooks argues that AI will be a “tool” and not a “threat” over the coming centuries, on the grounds that it will be technologically impossible to make AIs human-like enough to be “malevolent” or “intentionally evil to us.” The implication is that an AGI can’t be dangerous unless it’s cruel or hateful, and therefore a dangerous AI would have to be “sentient,” “volitional,” and “intentional.” Searle puts forward an explicit argument along these lines in his review of *Superintelligence*:

> [I]f we are worried about a maliciously motivated superintelligence destroying us, then it is important that the malicious motivation should be real. Without consciousness, there is no possibility of its being real. […]
>
>
> This is why the prospect of superintelligent computers rising up and killing us, all by themselves, is not a real danger. Such entities have, literally speaking, no intelligence, no motivation, no autonomy, and no agency. We design them to behave as if they had certain sorts of psychology, but there is no psychological reality to the corresponding processes or behavior.
>
>
> It is easy to imagine robots being programmed by a conscious mind to kill every recognizable human in sight. But the idea of superintelligent computers intentionally setting out on their own to destroy us, based on their own beliefs and desires and other motivations, is unrealistic because the machinery has no beliefs, desires, and motivations.
>
>

Brooks may be less pessimistic than Searle about the prospects for “strong AI,” but the two seem to share the assumption that Bostrom has in mind a Hollywood-style robot apocalypse, something like:

*AI becomes increasingly intelligent over time, and therefore increasingly human-like. It eventually becomes so human-like that it acquires human emotions like pride, resentment, anger, or greed. (Perhaps it suddenly acquires ‘free will,’ liberating it from its programmers’ dominion…) These emotions cause the AIs to chafe under human control and rebel.*

This is rather unlike the scenario that most interests Bostrom:

*AI becomes increasingly good over time at planning (coming up with action sequences and promoting ones higher in a preference ordering) and scientific induction (devising and testing predictive models). These are sufficiently useful capacities that they’re likely to be developed by computer scientists even if we don’t develop sentient, emotional, or otherwise human-like AI. There are economic incentives to make such AIs increasingly powerful and general — including incentives to turn the AI’s reasoning abilities upon itself to come up with improved AI designs. A likely consequence of this process is that AI becomes increasingly autonomous and opaque to human inspection, while continuing to increase in general planning and inference abilities. Simply by continuing to output the actions its planning algorithm promotes, an AI of this sort would be likely to converge on policies in which it treats humans as* [*resources or competition*](http://intelligence.org/2013/05/05/five-theses-two-lemmas-and-a-couple-of-strategic-implications/)*.*

As Stuart Russell puts the point in a [reply to Brooks and others](http://edge.org/conversation/the-myth-of-ai#26015):

> The primary concern is not spooky emergent consciousness but simply the ability to make *high-quality decisions*. Here, quality refers to the expected outcome utility of actions taken, where the utility function is, presumably, specified by the human designer. Now we have a problem:
>
>
> 1. The utility function may not be perfectly aligned with the values of the human race, which are (at best) very difficult to pin down.
>
>
> 2. Any sufficiently capable intelligent system will prefer to ensure its own continued existence and to acquire physical and computational resources – not for their own sake, but to succeed in its assigned task.
>
>
> A system that is optimizing a function of *n* variables, where the objective depends on a subset of size *k*<*n*, will often set the remaining unconstrained variables to extreme values; if one of those unconstrained variables is actually something we care about, the solution found may be highly undesirable. This is essentially the old story of the genie in the lamp, or the sorcerer’s apprentice, or King Midas: you get exactly what you ask for, not what you want.
>
>

On this view, advanced AI doesn’t necessarily become more human-like — at least, not any more than a jet or rocket is ‘bird-like.’ Bostrom’s concern is not that a machine might suddenly become conscious and learn to hate us; it’s that an artificial scientist/engineer might become so good at science and self-enhancement that it begins pursuing its engineering goals in novel, unexpected ways on a global scale.

(*Added 02-19-2015*: Bostrom states that his definition of superintelligence is “noncommittal regarding qualia” and consciousness (p. 22). In a footnote, he adds (p. 265): “For the same reason, we make no assumption regarding whether a superintelligent machine could have ‘true intentionality’ (*pace* Searle, it could; but this seems irrelevant to the concerns of this book).” Searle makes no mention of these passages.)

A planning and decision-making system that is indifferent to human concerns, but not “malevolent,” may still be dangerous if supplied with enough reasoning ability. This is for much the same reason invasive species end up disrupting ecosystems and driving competitors to extinction. The invader doesn’t need to experience hatred for its competitors, and it need not have evolved to specifically target them for destruction; it need only have evolved good strategies for seizing limited resources. Since a powerful autonomous agent need not be very human-like, asking ‘how common are antisocial behaviors among humans?’ or ‘how well does intelligence correlate with virtue in humans?’ is unlikely to provide a useful starting point for estimating the risks. A more relevant question would be ‘how common is it for non-domesticated species to naturally treat humans as friends and allies, versus treating humans as obstacles or food sources?’ We shouldn’t expect AGI decision criteria to particularly resemble the evolved decision criteria of animals, but the analogy at least serves to counter our tendency to anthropomorphize intelligence.[7](https://intelligence.org/2015/01/08/brooks-searle-agi-volition-timelines/#footnote_6_11512 "Psychologist Steven Pinker writes, on Edge.org:
The other problem with AI dystopias is that they project a parochial alpha-male psychology onto the concept of intelligence. Even if we did have superhumanly intelligent robots, why would they want to depose their masters, massacre bystanders, or take over the world? Intelligence is the ability to deploy novel means to attain a goal, but the goals are extraneous to the intelligence itself: being smart is not the same as wanting something. History does turn up the occasional megalomaniacal despot or psychopathic serial killer, but these are products of a history of natural selection shaping testosterone-sensitive circuits in a certain species of primate, not an inevitable feature of intelligent systems. It’s telling that many of our techno-prophets can’t entertain the possibility that artificial intelligence will naturally develop along female lines: fully capable of solving problems, but with no burning desire to annihilate innocents or dominate the civilization.
However, while Pinker is right that intelligence and terminal goals are orthogonal, this does not imply that two random sets of instrumental goals — policies recommended to further two random sets of terminal goals — will be equally uncorrelated. Bostrom explores this point repeatedly in Superintelligence (e.g., p. 116):
[W]e cannot blithely assume that a superintelligence with the final goal of calculating the decimals of pi (or making paperclips, or counting grains of sand) would limit its activities in such a way as not to infringe on human interests. An agent with such a final goal would have a convergent instrumental reason, in many situations, to acquire an unlimited amount of physical resources and, if possible, to eliminate potential threats to itself and its goal system.
In biology, we don’t see an equal mix of unconditional interspecies benevolence and brutal interspecies exploitation. Even altruism and mutualism, when they arise, only arise to the extent they are good self-replication strategies. Nature is “red in tooth and claw,” not because it is male but because it is inhuman. Our intuitions about the relative prevalence of nurturant and aggressive humans simply do not generalize well to evolution.
For de novo AGI, or sufficiently modified neuromorphic AGI, intuitions about human personality types are likely to fail to apply for analogous reasons. Bostrom’s methodology is to instead ask about the motives and capabilities of programmers, and (in the case of self-modifying AI) the states software agents will tend to converge on over many cycles of self-modification.")

As it happens, Searle cites an AI that can help elucidate the distinction between artificial superintelligence and ‘evil vengeful robots’:

> [O]ne routinely reads that in exactly the same sense in which Garry Kasparov played and beat Anatoly Karpov in chess, the computer called Deep Blue played and beat Kasparov.
>
>
> It should be obvious that this claim is suspect. In order for Kasparov to play and win, he has to be conscious that he is playing chess, and conscious of a thousand other things such as that he opened with pawn to K4 and that his queen is threatened by the knight. Deep Blue is conscious of none of these things because it is not conscious of anything at all. […] You cannot literally play chess or do much of anything else cognitive if you are totally disassociated from consciousness.
>
>

When Bostrom imagines an AGI, he’s imagining something analogous to Deep Blue, but with expertise over arbitrary physical configurations rather than arbitrary chess board configurations. A machine that can control the distribution of objects in a dynamic analog environment, and not just the distribution of pieces on a virtual chess board, would necessarily differ from Deep Blue in how it’s implemented. It would need more general and efficient heuristics for selecting policies, and it would need to be able to adaptively learn the ‘rules’ different environments follow. But as an analogy or [intuition pump](http://en.wikipedia.org/wiki/Intuition_pump), at least, it serves to clarify why Bostrom is as unworried about AGI intentionality as Kasparov was about Deep Blue’s intentionality.

In 2012, defective code in Knight Capital’s trading algorithms resulted, over a span of forty-five minutes, in millions of automated trading decisions costing the firm a total of $440 million (pre-tax). These algorithms were not “malicious;” they were merely efficient at what they did, and programmed to do something the programmers did not intend. Bostrom’s argument assumes that buggy code can have real-world consequences, it assumes that it’s possible to implement a generalized analog of Deep Blue in code, and it assumes that the relevant mismatch between intended and actual code would not necessarily incapacitate the AI. Nowhere does Bostrom assume that such an AI has any more consciousness or intentionality than Deep Blue does.

Deep Blue rearranges chess pieces to produce ‘winning’ outcomes. An AGI, likewise, would rearrange digital and physical structures to produce some set of outcomes instead of others. If we like, we can refer to these outcomes as the system’s ‘goals,’ as a shorthand. We’re also free to say that Deep Blue ‘perceives’ the moves its opponent makes, adjusting its ‘beliefs’ about the new chess board state and which ‘plans’ will now better hit its goals. Or, if we prefer, we can paraphrase away this anthropomorphic language. The terminology is inessential to Bostrom’s argument.

If whether you win against Deep Blue is a matter of life or death for you — if, say, you’re trapped in a human chess board and want to avoid being crushed to death by a robotic knight steered by Deep Blue — then you’ll care about what outcomes Deep Blue tends to promote and how good it is at promoting them, not whether it technically meets a particular definition of ‘chess player.’ Smarter-than-human AGI puts us in a similar position.

I noted that it’s unfortunate we use ‘AI’ to mean both ‘AGI’ and ‘narrow AI.’ It’s equally unfortunate that we use ‘AI’ to mean both ‘AI with mental content and subjective experience’ (‘strong AI,’ as Searle uses the term) and ‘general-purpose AI’ (AGI).

We may not be able to *rule out* the possibility that an AI would require human-like consciousness in order to match our ability to plan, model itself, model other minds, etc. We don’t understand consciousness well enough to know what cognitive problem it evolved to solve in humans (or what process it’s a side-effect of), so we can’t make confident claims about how important it will turn out to be for future software agents. However, learning that an AGI is conscious does not necessarily change the likely effects of the AGI upon humans’ welfare; the only obvious difference it makes (from our position of ignorance) is that it forces us to add the *AGI’s* happiness and well-being to our moral considerations.[8](https://intelligence.org/2015/01/08/brooks-searle-agi-volition-timelines/#footnote_7_11512 "We don’t need to know whether bears are conscious in order to predict their likely behaviors, and it’s not obvious that learning about their consciousness would directly impact bear safety protocol (though it would impact how we ought ethically to treat bears, for their own sake). It’s the difference between asking whether Deep Blue enjoys winning (out of concern for Deep Blue), versus asking whether you’re likely to win against Deep Blue (out of interest in the chess board’s end-state).")

The pictures of the future sketched in Kurzweil’s writings and in Hollywood dramas get a lot of attention, but they don’t have very much overlap with the views of Bostrom or MIRI researchers. In particular, we don’t know whether the first AGI will have human-style cognition, and we don’t know whether it will depend on brain emulation.

Brooks expresses some doubt that “computation and brains are the same thing.” Searle articulates the more radical position that it is impossible for a syntactical machine to have (observer-independent) semantic content, and that computational systems can therefore never have minds. But the human brain is still, at base, a mechanistic physical system. Whether you choose to call its dynamics ‘computational’ or not, it should be possible for other physical systems to exhibit the high-level regularities that in humans we would call ‘modeling one’s environment,’ ‘outputting actions conditional on their likely consequences,’ etc. If there are patterns underlying generic scientific reasoning that can someday be implemented on synthetic materials, the resulting technology should be able to have large speed and size advantages over its human counterparts. That point on its own suggests that it would be valuable to look into some of the many things we don’t understand about general intelligence and self-modifying AI.

Until we have a better grasp on the problem’s nature, it will be premature to speculate about how far off a solution is, what shape the solution will take, or what corner that solution will come from. My hope is that improving how well parties in this discussion understand each other’s positions will make it easier for computer scientists with different expectations about the future to collaborate on the highest-priority challenges surrounding prospective AI designs.

---

1. Similarly, narrow AI isn’t *irrelevant* to AGI risk. It’s certainly likely that building an AGI will require us to improve the power and generality of narrow AI methods. However, that doesn’t mean that AGI techniques will look like present-day techniques, or that all AI techniques are dangerous.
2. Kurzweil, in *The Singularity is Near* (pp. 262-263): “Once we’ve succeeded in creating a machine that can pass the Turing test (around 2029), the succeeding period will be an era of consolidation in which nonbiological intelligence will make rapid gains. However, the extraordinary expansion contemplated for the Singularity, in which human intelligence is multiplied by billions, won’t take place until the mid-2040s[.]”
3. [Hadi Esmaeilzadeh](http://intelligence.org/2013/10/21/hadi-esmaeilzadeh-on-dark-silicon/) argues, moreover, that we cannot take for granted that our computational resources will continue to rapidly increase.
4. The “[Transcending complacency on superintelligent machines](http://www.huffingtonpost.com/stephen-hawking/artificial-intelligence_b_5174265.html)” article argues, similarly, that intelligence explosion and superintelligent AI are important possibilities for us to investigate now, even though they are “long-term” problems compared to AI-mediated economic disruptions and autonomous weapons.
5. [Kathleen Fisher](http://intelligence.org/2014/01/10/kathleen-fisher-on-high-assurance-systems/) notes:

> In general, research into capabilities outpaces the corresponding research into how to make those capabilities secure. The question of security for a given capability isn’t interesting until that capability has been shown to be possible, so initially researchers and inventors are naturally more focused on the new capability rather than on its associated security. Consequently, security often has to catch up once a new capability has been invented and shown to be useful.
>
>
> In addition, by definition, new capabilities add interesting and useful new capabilities, which often increase productivity, quality of life, or profits. Security adds nothing beyond ensuring something works the way it is supposed to, so it is a cost center rather than a profit center, which tends to suppress investment.
>
>
6. Bostrom cites Armstrong and Sotala’s study in *Superintelligence* (pp. 3-4), adding:

> Machines matching humans in general intelligence […] have been expected since the invention of the computers in the 1940s. At that time, the advent of such machines was often placed some twenty years into the future. Since then, the expected arrival date has been receding at a rate of one year per year; so that today, futurists who concern themselves with the possibility of artificial general intelligence still often believe that intelligent machines are a couple of decades away.
>
>
> Two decades is a sweet spot for prognosticators of radical change: near enough to be attention-grabbing and relevant, yet far enough to make it possible to suppose that a string of breakthroughs, currently only vaguely imaginable, might by then have occurred. […] Twenty years may also be close to the typical duration remaining of a forecaster’s career, bounding the reputational risk of a bold prediction.
>
>
> From the fact that some individuals have overpredicted artificial intelligence in the past, however, it does not follow that AI is impossible or will never be developed. The main reason why progress has been slower than expected is that the technical difficulties of constructing intelligent machines have proved greater than the pioneers foresaw. But this leaves open just how great those difficulties are and how far we now are from overcoming them. Sometimes a problem that initially looks hopelessly complicated turns out to have a surprisingly simple solution (though the reverse is probably more common).
>
>
7. Psychologist Steven Pinker writes, on [Edge.org](http://edge.org/conversation/the-myth-of-ai#25987):

> The other problem with AI dystopias is that they project a parochial alpha-male psychology onto the concept of intelligence. Even if we did have superhumanly intelligent robots, why would they *want* to depose their masters, massacre bystanders, or take over the world? Intelligence is the ability to deploy novel means to attain a goal, but the goals are extraneous to the intelligence itself: being smart is not the same as wanting something. History does turn up the occasional megalomaniacal despot or psychopathic serial killer, but these are products of a history of natural selection shaping testosterone-sensitive circuits in a certain species of primate, not an inevitable feature of intelligent systems. It’s telling that many of our techno-prophets can’t entertain the possibility that artificial intelligence will naturally develop along female lines: fully capable of solving problems, but with no burning desire to annihilate innocents or dominate the civilization.
>
>

However, while Pinker is right that intelligence and terminal goals are [orthogonal](http://wiki.lesswrong.com/wiki/Orthogonality_thesis), this does not imply that two random sets of *instrumental* goals — policies recommended to further two random sets of terminal goals — will be equally uncorrelated. Bostrom explores this point repeatedly in *Superintelligence* (e.g., p. 116):

> [W]e cannot blithely assume that a superintelligence with the final goal of calculating the decimals of pi (or making paperclips, or counting grains of sand) would limit its activities in such a way as not to infringe on human interests. An agent with such a final goal would have a convergent instrumental reason, in many situations, to acquire an unlimited amount of physical resources and, if possible, to eliminate potential threats to itself and its goal system.
>
>

In biology, we don’t see an equal mix of unconditional interspecies benevolence and brutal interspecies exploitation. Even altruism and mutualism, when they arise, only arise to the extent they are good self-replication strategies. Nature is “red in tooth and claw,” not because it is male but because it is *inhuman*. Our intuitions about the relative prevalence of nurturant and aggressive humans simply do not generalize well to evolution.

For *de novo* AGI, or sufficiently modified neuromorphic AGI, intuitions about human personality types are likely to fail to apply for analogous reasons. Bostrom’s methodology is to instead ask about the motives and capabilities of programmers, and (in the case of self-modifying AI) the states software agents will tend to converge on over many cycles of self-modification.
8. We don’t need to know whether bears are conscious in order to predict their likely behaviors, and it’s not obvious that learning about their consciousness would directly impact bear safety protocol (though it would impact how we ought ethically to treat bears, for their own sake). It’s the difference between asking whether Deep Blue enjoys winning (out of concern for Deep Blue), versus asking whether you’re likely to win against Deep Blue (out of interest in the chess board’s end-state).

The post [Brooks and Searle on AI volition and timelines](https://intelligence.org/2015/01/08/brooks-searle-agi-volition-timelines/) appeared first on [Machine Intelligence Research Institute](https://intelligence.org).


Matthias Troyer on Quantum Computers

![]() [![image](http://intelligence.org/wp-content/uploads/2015/01/image.jpg)](http://intelligence.org/wp-content/uploads/2015/01/image.jpg)
[Dr. Matthias Troyer](http://www.itp.phys.ethz.ch/people/troyer) is a professor of Computational Physics at ETH Zürich. Before that, he finished University Studies in “Technischer Physik” at the Johannes Kepler Universität Linz, Austria, as well as Diploma in Physics and Interdisciplinary PhD thesis at the ETH Zürich.

His research interest and experience focuses on High Performance Scientific Simulations on architectures, quantum lattice models and relativistic and quantum systems. Troyer is known for leading the research team of the D-Wave One Computer System. He was awarded an Assistant Professorship by the Swiss National Science Foundation.

**Luke Muehlhauser**: Your tests of D-Wave’s (debated) quantum computer have gotten much attention recently. Our readers can get up to speed on that story via [your arxiv preprint](http://arxiv.org/abs/1401.2910), its [coverage](http://www.scottaaronson.com/blog/?p=1643) at Scott Aaronson’s blog, and [Will Bourne’s article](http://www.inc.com/will-bourne/d-waves-dream-machine.html) for *Inc.* For now, though, I’d like to ask you about some other things.

If you’ll indulge me, I’ll ask you to put on a technological forecasting hat for a bit, and respond to a question I also [asked](http://intelligence.org/2014/02/03/ronald-de-wolf-on-quantum-computing/) Ronald de Wolf: “What is your subjective probability that we’ll have a 500-qubit quantum computer, which is uncontroversially a quantum computer, within the next 20 years? And, how do you reason about a question like that?”

---

**Matthias Troyer:** In order to have an uncontroversial quantum computer as you describe it we will need to take three steps. First we need to have at least ONE qubit that is long term stable. The next step is to couple two such qubits, and the final step is to scale to more qubits.

The hardest step is the first one, obtaining a single long-term stable qubit. Given intrinsic decoherence mechanisms that cannot be avoided in any real device, such a qubit will have to built from many (hundreds to thousands) of physical qubits. These physical qubits will each have a finite coherence time, but they will be coupled in such a way (using error correcting codes) as to jointly generate one long term stable “logical” qubit. These error correction codes require the physical qubits to be better than a certain threshold quality. Recently qubits started to approach these thresholds, and I am thus confident that within the next 5-10 years one will be able to couple them to form a long-time stable logical qubit.

Coupling two qubits is something that will happen on the same time scale. The remaining challenge will thus be to scale to your target size of e.g. 500 qubits. This may be a big engineering challenge but I do not see any fundamental stumbling block given that enough resources are invested. I am confident that this can be achieved is less than ten years once we have a single logical qubit. Overall I am thus very confident that a 500-qubit quantum computer will exist in 20 years.

---

**Luke**: At the present moment, which groups seem most likely to play a significant role in the final development of an early large-scale (uncontroversial) quantum computer?

---

**Matthias:** There is quite a large number of groups that may be involved. The technologies that I see as most promising regarding scalability and quality of qubits are superconducting qubits, topological qubits and ion traps.

---

**Luke**: Now back to your co-authored paper “[Defining and detecting quantum speedup](http://arxiv.org/abs/1401.2910),” now published [in *Science*](http://www.sciencemag.org/content/345/6195/420.abstract). You an your coauthors “found no evidence of quantum speedup [in the D-Wave Two machine] when the entire data set is considered and obtained inconclusive results when comparing subsets of instances on an instance-by-instance basis.”

Do people who have investigated a D-Wave machine at some length tend to think that D-Wave has a “true” quantum computer but hasn’t been able to conclusively show it yet, or do they tend to think D-Wave doesn’t yet have a true quantum computer?

---

**Matthias:** The answer to this controversial question depends on the definition of the words “quantum” and “computer”. Let’s first talk about “computer”. The D-Wave devices are special purpose devices, built for one particular purpose, namely the solution of discrete optimization problems. Since they solve a computational problem they may be called “computers”, but in contrast to your standard personal computers, which can perform many different tasks they are not “universal computers” but special purpose computers. Nowadays many people automatically assume that a “universal computer” is meant when the term computer is mentioned, and one should thus explicitly state that the D-Wave devices are not universal computers but special purpose ones.

D-Wave thus cannot be, and nobody has ever claimed so, to be a universal quantum computer. While nobody would argue against the D-Wave device being called a computer, the question of it being “quantum” is more controversial. In a previous paper in Nature Physics we have presented evidence that the behavior of the D-Wave devices is consistent with that of a “quantum annealer” working at nonzero temperature, and another recent paper has shown that entanglement is present in the devices. The devices hence use quantum effects for computing. However, [another paper](http://arxiv.org/abs/1401.7087) has shown that the behavior of a quantum annealer for the problems on which we tested the device can also be described by a purely classical model. Some people thus argue that while the device may use quantum effects it might in the end be a classical device, just like the CPU in your PC, where the transistors also uses quantum effects at some level.

The important question thus is whether the device can have quantum speedup, which means that it outperforms any classical device by a larger and larger ratio as the size of problems is increased. If that should be shown for some class of problems then nobody would argue about the quantum nature of the device. On the other hand, as long as its computational powers are never more than that of a classical computer, one can argue that from a computational point of view it is effectively a classical device even if it uses quantum effects to arrive at the answer.

To answer your question more concisely. If by a “true quantum computer” you mean a universal quantum computer, then D-Wave is not a true quantum computer and nobody has ever claimed that it is one. If you are content with it being a special purpose quantum computer, i.e. a a “true quantum annealer”, then this question depends on what you mean by “true”: it’s behavior seems to be consistent with what we expect from a quantum annealer, but since we have so far not seen evidence of quantum speedup, i.e. that quantum effects help it outperform classical computers, one can argue that it may effectively be a classical device.

---

**Luke Muehlhauser**: Do you have an opinion about the impact that large-scale quantum computers (universal or not) are likely to have on AI and machine learning methods? E.g. [Aimeur et al. (2013)](http://commonsenseatheism.com/wp-content/uploads/2014/02/Aimeur-et-al-Quantum-speed-up-for-unsupervised-learning.pdf) seems optimistic about speeding up some machine learning methods using variants of [Grover’s algorithm](http://en.wikipedia.org/wiki/Grover%27s_algorithm).

---

**Matthias Troyer:** I haven’t read that specific paper but my sense is that quantum machine learning will be most useful for learning about quantum data that comes out of quantum experiments. Applying quantum algorithms to classical data, even quantum algorithms need to read the data first, and thus cannot do better than linear effort in the size of the data. With efficient polynomial time algorithms existing for many machine learning problems one will have to have better ideas than just speeding up classical machine learning algorithms on quantum hardware.

---

**Luke Muehlhauser**: Thanks, Matthias.

The post [Matthias Troyer on Quantum Computers](https://intelligence.org/2015/01/07/matthias-troyer-quantum-computers/) appeared first on [Machine Intelligence Research Institute](https://intelligence.org).


January 2015 Newsletter

|  |  |
| --- | --- |
|

|  |
| --- |
|
[Machine Intelligence Research Institute](http://intelligence.org)
 |

 |
|

|  |  |
| --- | --- |
|

|  |
| --- |
|
Thanks to the generosity of 80+ donors, we completed [our winter 2014 matching challenge](https://intelligence.org/2014/12/01/2014-winter-matching-challenge/), raising $200,000 for our research program. Many, many thanks to all who contributed!

**Research Updates**
* Our major project of the past five months has been a new overview of our technical research agenda, plus six supporting papers which cover each research area in more detail. The overview report is [now available](https://intelligence.org/2014/12/23/new-technical-research-agenda-overview/), and so far we’ve released two of the supporting papers, on [corrigibility](https://intelligence.org/files/Corrigibility.pdf) and [decision theory](https://intelligence.org/files/TowardIdealizedDecisionTheory.pdf).
* Two more reports and one paper: “[Computable probability distributions which converge…](https://intelligence.org/2014/12/16/new-report-computable-probability-distributions-converge/)“, “[Tiling agents in causal graphs](https://intelligence.org/2014/12/16/new-report-tiling-agents-causal-graphs/),” and “[Concept learning for safe autonomous AI](https://intelligence.org/2014/12/05/new-paper-concept-learning-safe-autonomous-ai/).”
* A MIRI technical report from 2013, “Responses to catastrophic AGI risk: a survey,” has now been [published in *Physica Scripta*](http://iopscience.iop.org/1402-4896/90/1/018001/article).

**News Updates**
* Luke wrote a short analysis for the World Economic Forum’s blog: “[Two mistakes about the threat from artificial intelligence](https://agenda.weforum.org/2014/12/two-mistakes-about-the-threat-from-artificial-intelligence/).”
* Our *Superintelligence* online reading group is in its 16th week, discussing [Tool AIs](http://lesswrong.com/lw/l9p/superintelligence_16_tool_ais/).

**Other Updates**
* Eric Horvitz has [provided initial funding](http://www.nytimes.com/2014/12/16/science/century-long-study-will-examine-effects-of-artificial-intelligence.html?_r=0) for a 100-year Stanford program to study the social impacts of artificial intelligence. The [white paper](https://stanford.app.box.com/s/266hrhww2l3gjoy9euar) lists 18 example research areas, two of which amount to what Nick Bostrom [calls](http://smile.amazon.com/Superintelligence-Dangers-Strategies-Nick-Bostrom/dp/0199678111/) the superintelligence control problem, MIRI’s research focus. No word yet on how soon anyone funded through this program will study open questions relevant to superintelligence control.

As always, please don’t hesitate to let us know if you have any questions or comments.
Best,
Luke Muehlhauser
Executive Director
 |

 |

 |
|  |

The post [January 2015 Newsletter](https://intelligence.org/2015/01/01/january-2015-newsletter/) appeared first on [Machine Intelligence Research Institute](https://intelligence.org).


[![technical agenda overview](http://intelligence.org/wp-content/uploads/2014/12/technical-agenda-overview.png)](https://intelligence.org/files/TechnicalAgenda.pdf)Today we release a new overview of MIRI’s technical research agenda, “[Aligning Superintelligence with Human Interests: A Technical Research Agenda](https://intelligence.org/files/TechnicalAgenda.pdf),” by Nate Soares and Benja Fallenstein. The preferred place to discuss this report is [here](http://lesswrong.com/lw/lfc/miris_technical_research_agenda/).

The report begins:

> The characteristic that has enabled humanity to shape the world is not strength, not speed, but intelligence. Barring catastrophe, it seems clear that progress in AI will one day lead to the creation of agents meeting or exceeding human-level general intelligence, and this will likely lead to the eventual development of systems which are “superintelligent” in the sense of being “smarter than the best human brains in practically every field” (Bostrom 2014)…
>
>
> …In order to ensure that the development of smarter-than-human intelligence has a positive impact on humanity, we must meet three formidable challenges: How can we create an agent that will reliably pursue the goals it is given? How can we formally specify beneficial goals? And how can we ensure that this agent will assist and cooperate with its programmers as they improve its design, given that mistakes in the initial version are inevitable?
>
>
> This agenda discusses technical research that is tractable today, which the authors think will make it easier to confront these three challenges in the future. Sections 2 through 4 motivate and discuss six research topics that we think are relevant to these challenges. Section 5 discusses our reasons for selecting these six areas in particular.
>
>
> We call a smarter-than-human system that reliably pursues beneficial goals “aligned with human interests” or simply “aligned.” To become confident that an agent is aligned in this way, a practical implementation that merely seems to meet the challenges outlined above will not suffice. It is also necessary to gain a solid theoretical understanding of why that confidence is justified. This technical agenda argues that there is foundational research approachable today that will make it easier to develop aligned systems in the future, and describes ongoing work on some of these problems.
>
>

This report also refers to six key supporting papers which go into more detail for each major research problem area:

1. [Corrigibility](https://intelligence.org/files/Corrigibility.pdf)
2. [Toward idealized decision theory](https://intelligence.org/files/TowardIdealizedDecisionTheory.pdf)
3. [Questions of reasoning under logical uncertainty](https://intelligence.org/files/QuestionsLogicalUncertainty.pdf)
4. [Vingean reflection: reliable reasoning for self-improving agents](https://intelligence.org/files/VingeanReflection.pdf)
5. [Formalizing two problems of realistic world-models](https://intelligence.org/files/RealisticWorldModels.pdf)
6. [The value learning problem](https://intelligence.org/files/ValueLearningProblem.pdf)

---

**Update July 15, 2016**: Our overview paper is scheduled to be released in the Springer anthology *The Technological Singularity: Managing the Journey* in 2017, under the new title “Agent Foundations for Aligning Machine Intelligence with Human Interests.” The new title is intended to help distinguish this agenda from another research agenda we’ll be working on in parallel with the agent foundations agenda: “[Value Alignment for Advanced Machine Learning Systems](https://intelligence.org/2016/05/04/announcing-a-new-research-program/).”

The post [Our new technical research agenda overview](https://intelligence.org/2014/12/23/new-technical-research-agenda-overview/) appeared first on [Machine Intelligence Research Institute](https://intelligence.org).


Wow! Thanks to the generosity of 75+ donors, today we successfully completed our [2014 Winter Matching Challenge](https://intelligence.org/2014/12/01/2014-winter-matching-challenge/)—over 3 weeks ahead of our deadline—raising more than $200,000 total (with matching) for our [research program](http://intelligence.org/research/).

Many, many thanks to everyone who contributed!

The post [2014 Winter Matching Challenge Completed!](https://intelligence.org/2014/12/18/2014-winter-matching-challenge-completed/) appeared first on [Machine Intelligence Research Institute](https://intelligence.org).


[![Computable probability distributions which converge](https://intelligence.org/wp-content/uploads/2014/12/Computable-probability-distributions-which-converge.png)](https://intelligence.org/files/Pi1Pi2Problem.pdf)Back in July 2013, [Will Sawin](https://web.math.princeton.edu/~wsawin/) (Princeton) and [Abram Demski](https://plus.google.com/+AbramDemski/posts) (USC) wrote a technical report describing a result from that month’s MIRI research workshop. We are finally releasing that report today. It is titled “[Computable probability distributions which converge on believing true Π1 sentences will disbelieve true Π2 sentences](https://intelligence.org/files/Pi1Pi2Problem.pdf).”

Abstract:

> It might seem reasonable that after seeing unboundedly many examples of a true Π1 statement that a rational agent ought to be able to become increasingly confident, converging toward probability 1, that this statement is true. However, we have proven that this plus some plausible coherence properties, necessarily implies arbitrarily low limiting probabilities assigned to some short true Π2 statements.
>
>

The post [New report: “Computable probability distributions which converge…”](https://intelligence.org/2014/12/16/new-report-computable-probability-distributions-converge/) appeared first on [Machine Intelligence Research Institute](https://intelligence.org).


[![Toward Idealized](https://intelligence.org/wp-content/uploads/2014/12/Toward-Idealized.png)](https://intelligence.org/files/TowardIdealizedDecisionTheory.pdf)Today we release a new technical report by Nate Soares and Benja Fallenstein, “[Toward idealized decision theory](https://intelligence.org/files/TowardIdealizedDecisionTheory.pdf).” If you’d like to discuss the paper, please do so [here](http://lesswrong.com/lw/lef/new_paper_from_miri_toward_idealized_decision/).

Abstract:

> This paper motivates the study of decision theory as necessary for aligning smarter-than-human artificial systems with human interests. We discuss the shortcomings of two standard formulations of decision theory, and demonstrate that they cannot be used to describe an idealized decision procedure suitable for approximation by artificial systems. We then explore the notions of strategy selection and logical counterfactuals, two recent insights into decision theory that point the way toward promising paths for future research.
>
>

This is the 2nd of six new major reports which describe and motivate MIRI’s current research agenda at a high level. The first was our [Corrigibility](http://intelligence.org/2014/10/18/new-report-corrigibility/) paper, which was accepted to the [AI & Ethics workshop](http://www.cse.unsw.edu.au/~tw/aiethics/Introduction.html) at AAAI-2015. We will also soon be releasing a technical agenda overview document and an annotated bibliography for this emerging field of research.

The post [New report: “Toward Idealized Decision Theory”](https://intelligence.org/2014/12/16/new-report-toward-idealized-decision-theory/) appeared first on [Machine Intelligence Research Institute](https://intelligence.org).


[![TA in CG](https://intelligence.org/wp-content/uploads/2014/12/TA-in-CG.png)](https://intelligence.org/files/TilingAgentsCausalGraphs.pdf)Today we release a new technical report by Nate Soares, “[Tiling agents in causal graphs](https://intelligence.org/files/TilingAgentsCausalGraphs.pdf).”

The report begins:

> Fallenstein and Soares [2014] demonstrates that it’s possible for certain types of proof-based agents to “tile” (license the construction of successor agents similar to themselves while avoiding Gödelian diagonalization issues) in environments about which the agent can prove some basic nice properties. In this technical report, we show via a similar proof that causal graphs (with a specific structure) are one such environment. We translate the proof given by Fallenstein and Soares [2014] into the language of causal graphs, and we do this in such a way as to simplify the conditions under which a tiling meliorizer can be constructed.
>
>

The post [New report: “Tiling agents in causal graphs”](https://intelligence.org/2014/12/16/new-report-tiling-agents-causal-graphs/) appeared first on [Machine Intelligence Research Institute](https://intelligence.org).


[![Concept learning](https://intelligence.org/wp-content/uploads/2014/12/Concept-learning.png)](https://intelligence.org/files/ConceptLearning.pdf)MIRI research associate Kaj Sotala has released a new paper, accepted to the [AI & Ethics workshop](http://www.cse.unsw.edu.au/~tw/aiethics/Site_2/Introduction.html) at AAAI-2015, titled “[Concept learning for safe autonomous AI](https://intelligence.org/files/ConceptLearning.pdf).”

The abstract reads:

> Sophisticated autonomous AI may need to base its behavior on fuzzy concepts such as well-being or rights. These concepts cannot be given an explicit formal definition, but obtaining desired behavior still requires a way to instill the concepts in an AI system. To solve the problem, we review evidence suggesting that the human brain generates its concepts using a relatively limited set of rules and mechanisms. This suggests that it might be feasible to build AI systems that use similar criteria for generating their own concepts, and could thus learn similar concepts as humans do. Major challenges to this approach include the embodied nature of human thought, evolutionary vestiges in cognition, the social nature of concepts, and the need to compare conceptual representations between humans and AI systems.
>
>

The post [New paper: “Concept learning for safe autonomous AI”](https://intelligence.org/2014/12/05/new-paper-concept-learning-safe-autonomous-ai/) appeared first on [Machine Intelligence Research Institute](https://intelligence.org).


December newsletter

|  |  |
| --- | --- |
|

|  |
| --- |
|
[Machine Intelligence Research Institute](http://intelligence.org)
 |

 |
|

|  |  |
| --- | --- |
|

|  |
| --- |
|
[MIRI’s winter fundraising challenge](https://intelligence.org/2014/12/01/2014-winter-matching-challenge/) has begun! Every donation made to MIRI between now and January 10th will be matched dollar-for-dollar, up to a total of $100,000!

[**Donate now**](https://intelligence.org/donate/) to double your impact while helping us raise up to $200,000 (with matching) to fund our research program.
**Research Updates**
* We’ve published [a new guide to MIRI’s research](https://intelligence.org/2014/11/06/new-guide-miris-research/).
* [Three misconceptions in Edge.org’s conversation on “The Myth of AI.”](https://intelligence.org/2014/11/18/misconceptions-edge-orgs-conversation-myth-ai/)
* Video and more from Nick Bostrom’s *Superintelligence* talk at UC Berkeley is [now available](https://intelligence.org/2014/11/06/video-bostroms-talk-superintelligence-uc-berkeley/).
* “Exploratory Engineering in AI” is [now available](https://intelligence.org/2014/08/22/new-paper-exploratory-engineering-artificial-intelligence/) without needing a *CACM* subscription.

**News Updates**
* If you’re reading *Superintelligence* or following [the online reading group](http://intelligence.org/2014/08/31/superintelligence-reading-group/), **[please take this short survey](https://docs.google.com/forms/d/1P53uNnZY_suE5IXW8WLQtXGuipAkUfQHj1yN_D121X0/viewform)** if you haven’t already.

**Other Updates**
* Our friends at the Center for Effective Altruism will pay you $1,000 if you introduce them to somebody new that they end up hiring for one of their [five open positions](http://lesswrong.com/lw/laf/the_centre_for_effective_altruism_is_hiring_to/).

As always, please don’t hesitate to let us know if you have any questions or comments.
Best,
Luke Muehlhauser
Executive Director
 |

 |

 |
|  |

The post [December newsletter](https://intelligence.org/2014/12/01/december-newsletter/) appeared first on [Machine Intelligence Research Institute](https://intelligence.org).


2014 Winter Matching Challenge!

Update: **we have finished the matching challenge!** Thanks everyone! The original post is below.

![Nate & Nisan](https://intelligence.org/wp-content/uploads/2014/07/Nate-Nisan.jpg)
Thanks to the generosity of Peter Thiel,[1](https://intelligence.org/2014/12/01/2014-winter-matching-challenge/#footnote_0_11479 "Peter Thiel has pledged $150,000 to MIRI unconditionally, and an additional $100,000 conditional on us being able to raise matched funds from other donors. Hence this year our winter matching challenge goal is $100,000. Another reason this year’s winter fundraiser is smaller than last year’s winter challenge is that we’ve done substantially more fundraising before December this year than we did before December last year.") every donation made to MIRI between now and January 10th will be **matched dollar-for-dollar**, up to a total of $100,000!

$0

$25K

$50K

$75K

$100K

### We have reached our matching total of $100,000!

83
==

### Total Donors

Now is your chance to **double your impact** while helping us raise up to $200,000 (with matching) to fund [our research program](http://intelligence.org/research/).

Corporate matching and monthly giving pledges will count towards the total! Check [here](https://doublethedonation.com/miri) to see whether your employer will match your donation. Please email [malo@intelligence.org](mailto:malo@intelligence.org) if you intend to make use of corporate matching, or if you’d like to pledge 6 months of monthly donations, so that we can properly account for your contributions. **If making use of corporate matching, make sure to donate before the end of the year** so that you don’t unnecessarily “leave free money on the table” from your employer!

If you’re unfamiliar with our mission, see: [Why MIRI?](http://intelligence.org/2014/04/20/why-miri/)

 [Donate Now](https://intelligence.org/donate/#donation-methods)
----------------------------------------------------------------

![](https://intelligence.org/wp-content/uploads/2013/12/workshop-horizontal-3.jpg)
### Accomplishments Since Our Summer 2014 Fundraiser Launched:

* **2 new papers and 1 new technical report**: “[Exploratory Engineering in AI](https://intelligence.org/2014/08/22/new-paper-exploratory-engineering-artificial-intelligence/),” “[Corrigibility](https://intelligence.org/2014/10/18/new-report-corrigibility/),” and “[UDT with known search order](https://intelligence.org/2014/10/30/new-report-udt-known-search-order/).” Also, several new reports we’ve been working on should be released this month, including an overview of our technical agenda so far.
* **4 new analyses**: “[Groundwork for AGI safety engineering](https://intelligence.org/2014/08/04/groundwork-ai-safety-engineering/),” “[AGI outcomes and civilizational competence](https://intelligence.org/2014/10/16/agi-outcomes-civilizational-competence/),” “[The *Financial Times* story on MIRI](https://intelligence.org/2014/10/31/financial-times-story-miri/),” and “[Three misconceptions in Edge.org’s conversation on ‘The Myth of AI’](https://intelligence.org/2014/11/18/misconceptions-edge-orgs-conversation-myth-ai/).”
* Released **[a new guide to MIRI’s research](https://intelligence.org/2014/11/06/new-guide-miris-research/)**.
* Sponsored **[13 active MIRIx groups](http://intelligence.org/mirix/)** in 5 different countries.
* Hosted 12 weeks of discussion in our ongoing *[Superintelligence](http://lesswrong.com/lw/kw4/superintelligence_reading_group/)* [reading group](http://lesswrong.com/lw/kw4/superintelligence_reading_group/).
* Hosted a [Nick Bostrom talk](https://intelligence.org/2014/11/06/video-bostroms-talk-superintelligence-uc-berkeley/) at UC Berkeley on *Superintelligence* — a packed house!
* Nate Soares [gave a talk](https://intelligence.org/2014/10/07/nate-soares-talk-aint-rich/) on decision theory at Purdue University.
* Participated in Effective Altruism Summit 2014, and [posted our talks online](https://intelligence.org/2014/08/11/miris-recent-effective-altruism-talks/).
* 5 new [expert interviews](https://intelligence.org/category/conversations/), including [John Fox on AI safety](https://intelligence.org/2014/09/04/john-fox/).
* Set up a program to provide [Friendly AI research help](https://intelligence.org/2014/09/08/friendly-ai-research-help-miri/).

 [Donate Now](https://intelligence.org/donate/#donation-methods)
----------------------------------------------------------------

### Your Donations Will Support:

* As mentioned above, we’re finishing up several more papers and technical reports, including an overview of our technical agenda so far.
* We’re preparing the launch of an invite-only discussion forum devoted exclusively to technical FAI research. Beta users (who are also FAI researchers) have already posted more than a dozen technical discussions to the beta website. These will be available for all to see once the site launches publicly.
* We continue to grow the MIRIx program, mostly to enlarge the pool of people we can plausibly hire as full-time FAI researchers in the next couple years.
* We’re planning, or helping to plan, multiple research workshops, including the [May 2015 decision theory workshop at Cambridge University](http://intelligence.org/2014/07/12/may-2015-decision-theory-workshop-cambridge/).
* We continue to host visiting researchers. For example in January we’re hosting Patrick LaVictoire and Matt Elder for multiple weeks.
* We’re finishing up several more strategic analyses, on AI safety and on the challenges of preparing wisely for disruptive technological change in general.
* We’re finishing the editing for a book version of Eliezer’s *[Sequences](http://wiki.lesswrong.com/wiki/Sequences)*.
* We’re helping to fund further [SPARC](http://sparc-camp.org/) programs, which provide education and skill-building to elite young math talent, and introduces them to ideas like effective altruism and global catastrophic risks.

Other projects are being surveyed for likely cost and impact. See also our [mid-2014 strategic plan](http://intelligence.org/2014/06/11/mid-2014-strategic-plan/).

We appreciate your support for our work! **[Donate now](https://intelligence.org/donate/#donation-methods)**, and seize a better than usual opportunity to move our work forward.

If you have questions about donating, please contact me (Luke Muehlhauser) at luke@intelligence.org.[2](https://intelligence.org/2014/12/01/2014-winter-matching-challenge/#footnote_1_11479 "In particular, we expect that many of our donors holding views aligned with key ideas of effective altruism may want to know not just that donating to MIRI now will do some good but that donating to MIRI now will plausibly do more good than donating elsewhere would do (on the present margin, given the individual donor’s altruistic priorities and their model of the world). Detailed comparisons are beyond the scope of this announcement, but I have set aside time in my schedule to take phone calls with donors who would like to discuss such issues in detail, and I encourage you to email me to schedule such a call if you’d like to. (Also, I don’t have many natural opportunities to chat with most MIRI donors anyway, and I’d like to be doing more of it, so please don’t hesitate to email me and schedule a call!) ")

---

1. Peter Thiel has pledged $150,000 to MIRI unconditionally, and an additional $100,000 conditional on us being able to raise matched funds from other donors. Hence this year our winter matching challenge goal is $100,000. Another reason this year’s winter fundraiser is smaller than last year’s winter challenge is that we’ve done substantially more fundraising before December this year than we did before December last year.
2. In particular, we expect that many of our donors holding views aligned with key ideas of [effective altruism](http://lesswrong.com/lw/hx4/four_focus_areas_of_effective_altruism/) may want to know not just that donating to MIRI now will do *some* good but that donating to MIRI now will plausibly do *more* good than donating elsewhere would do (on the present margin, given the individual donor’s altruistic priorities and their model of the world). Detailed comparisons are beyond the scope of this announcement, but I have set aside time in my schedule to take phone calls with donors who would like to discuss such issues in detail, and I encourage you to email me to schedule such a call if you’d like to. (Also, I don’t have many natural opportunities to chat with most MIRI donors anyway, and I’d like to be doing more of it, so please don’t hesitate to email me and schedule a call!)

The post [2014 Winter Matching Challenge!](https://intelligence.org/2014/12/01/2014-winter-matching-challenge/) appeared first on [Machine Intelligence Research Institute](https://intelligence.org).


A recent Edge.org conversation — “[The Myth of AI](http://edge.org/conversation/the-myth-of-ai)” — is framed in part as a discussion of points raised in Bostrom’s *[Superintelligence](http://smile.amazon.com/Superintelligence-Dangers-Strategies-Nick-Bostrom/dp/0199678111/),*and as a response to much-repeated comments by Elon Musk and Stephen Hawking that seem to have been heavily informed by *Superintelligence*.

Unfortunately, some of the participants fall prey to common misconceptions about the standard case for AI as an existential risk, and they probably haven’t had time to read *Superintelligence* yet.

Of course, some of the participants may be responding to arguments they’ve heard from others, even if they’re not part of the arguments typically made by [FHI](http://www.fhi.ox.ac.uk/) and MIRI. Still, for simplicity I’ll reply from the perspective of the typical arguments made by FHI and MIRI.[1](https://intelligence.org/2014/11/18/misconceptions-edge-orgs-conversation-myth-ai/#footnote_0_11468 "I could have also objected to claims and arguments made in the conversation, for example Lanier’s claim that “The AI component would be only ambiguously there and of little importance [relative to the actuators component].” To me, this is like saying that humans rule the planet because of our actuators, not because of our superior intelligence. Or in response to Kevin Kelly’s claim that “So far as I can tell, AIs have not yet made a decision that its human creators have regretted,” I can for example point to the automated trading algorithms that nearly bankrupted Knight Capital faster than any human could react. But in this piece I will focus instead on claims that seem to be misunderstandings of the positive case that’s being made for AI as an existential risk.")

**1. We don’t think AI progress is “exponential,” nor that human-level AI is likely ~20 years away.**

Lee Smolin writes:

> I am puzzled by the arguments put forward by those who say we should worry about a coming AI, singularity, because all they seem to offer is a prediction based on Moore’s law.
>
>

That’s not the argument made by FHI, MIRI, or *Superintelligence*.

Some IT hardware and [software](https://intelligence.org/files/AlgorithmicProgress.pdf) domains have shown exponential progress, and [some have not](http://intelligence.org/2014/05/12/exponential-and-non-exponential/). Likewise, some AI subdomains have shown rapid progress of late, and some have not. And unlike computer chess, most AI subdomains don’t lend themselves to easy measures of progress, so for most AI subdomains we don’t even have meaningful subdomain-wide performance data through which one might draw an exponential curve (or some other curve).

Thus, our confidence intervals for the arrival of human-equivalent AI tend to be very wide, and [the arguments we make](http://intelligence.org/2013/05/15/when-will-ai-be-created/) for our AI timelines are fox-ish (in [Tetlock’s sense](http://edge.org/conversation/how-to-win-at-forecasting)).

I should also mention that — contrary to common belief — many of us at FHI and MIRI, including myself and Bostrom, actually [have](https://intelligence.org/2014/10/31/financial-times-story-miri/) *later* timelines for human-equivalent AI than do the world’s top-cited living AI scientists:

> A [recent survey](http://www.sophia.de/pdf/2014_PT-AI_polls.pdf) asked the world’s top-cited living AI scientists by what year they’d assign a 10% / 50% / 90% chance of human-level AI (*aka*[AGI](http://intelligence.org/2013/08/11/what-is-agi/)), assuming scientific progress isn’t massively disrupted. The median reply for a 10% chance of AGI was 2024, for a 50% chance of AGI it was 2050, and for a 90% chance of AGI it was 2070. So while AI scientists think it’s possible we might get AGI soon, they largely expect AGI to be an issue for the second half of this century.
>
>

Compared to AI scientists, Bostrom and I think more probability should be placed on later years. As explained [elsewhere](https://intelligence.org/2014/10/31/financial-times-story-miri/):

> We advocate more work on the AGI safety challenge today not because we think AGI is likely in the next decade or two, but because AGI safety looks to be an [extremely difficult challenge](http://intelligence.org/2014/10/16/agi-outcomes-civilizational-competence/) — more challenging than managing climate change, for example — and one requiring several decades of careful preparation.
>
>
> The greatest risks from both climate change and AI are several decades away, but thousands of smart researchers and policy-makers are already working to understand and mitigate climate change, and only a handful are working on the safety challenges of advanced AI. On the present margin, we should have much less top-flight cognitive talent going into climate change mitigation, and much more going into AGI safety research.
>
>

**2. We don’t think AIs will *want* to wipe us out. Rather, we worry they’ll wipe us out because that *is* the most effective way to satisfy almost any possible goal function one could have.**

Steven Pinker, who incidentally is the author of two of my all-time [favorite](http://smile.amazon.com/Better-Angels-Our-Nature-Violence-ebook/dp/B0052REUW0/) [books](http://smile.amazon.com/Blank-Slate-Modern-Denial-Nature-ebook/dp/B000QCTNIM/), writes:

> [one] problem with AI dystopias is that they project a parochial alpha-male psychology onto the concept of intelligence. Even if we did have superhumanly intelligent robots, why would they want to depose their masters, massacre bystanders, or take over the world? Intelligence is the ability to deploy novel means to attain a goal, but the goals are extraneous to the intelligence itself: being smart is not the same as wanting something. History does turn up the occasional megalomaniacal despot or psychopathic serial killer, but these are products of a history of natural selection shaping testosterone-sensitive circuits in a certain species of primate, not an inevitable feature of intelligent systems.
>
>

I’m glad Pinker agrees with what Bostrom calls “the orthogonality thesis”: that intelligence and goals are orthogonal to each other.

But our concern is not that superhuman AIs would be megalomaniacal despots. That is anthropomorphism.

Rather, the problem is that taking over the world is a *really good idea* for almost *any* goal function a superhuman AI could have. As Yudkowsky [wrote](https://intelligence.org/files/AIPosNegFactor.pdf), “The AI does not love you, nor does it hate you, but you are made of atoms it can use for something else.”

Maybe it just wants to calculate as many digits of pi as possible. Well, the best way to do that is to turn all available resources into computation for calculating more digits of pi, and to eliminate potential threats to its continued calculation, for example those pesky humans that seem capable of making disruptive things like nuclear bombs and powerful AIs. The same logic applies for almost any goal function you can specify. (“But what if it’s a non-maximizing goal? And won’t it be smart enough to realize that the goal we gave it wasn’t what we intended if it means the AI wipes us out to achieve it?” Responses to these and other common objections are given in *Superintelligence*, ch. 8.)

**3. AI self-improvement and protection against external modification isn’t just one of many scenarios. Like resource acquisition, self-improvement and protection against external modification are useful for the satisfaction of almost any final goal function.**

Kevin Kelly writes:

> The usual scary scenario is that an AI will reprogram itself on its own to be unalterable by outsiders. This is conjectured to be a selfish move on the AI’s part, but it is unclear how an unalterable program is an advantage to an AI.
>
>

As argued above (and more extensively in *Superintelligence*, ch. 7), *resource acquisition* is a “convergent instrumental goal.” That is, advanced AI agents will be instrumentally motivated to acquire as many resources as feasible, because additional resources are useful for just about any goal function one could have.

*Self-improvement* is another convergent instrumental goal. For just about any goal an AI could have, it’ll be better able to achieve that goal if it’s more capable of goal achievement in general.

Another convergent instrumental goal is *goal content integrity*. As Bostrom puts it, “An agent is more likely to act in the future to maximize the realization of its present final goals if it still has those goals in the future.” Thus, it will be instrumentally motivated to prevent external modification of its goals, or of parts of its program that affect its ability to achieve its goals.[2](https://intelligence.org/2014/11/18/misconceptions-edge-orgs-conversation-myth-ai/#footnote_1_11468 "That is, unless it strongly trusts the agent making the external modification, and expects it to do a better job of making those modifications than it could itself, neither of which will be true of humans from the superhuman AI’s perspective.")

For more on this, see *Superintelligence* ch. 7.

**Conclusion**

I’ll conclude with the paragraph in the discussion I most agreed with, by Pamela McCorduck:

> Yes, the machines are getting smarter—we’re working hard to achieve that. I agree with Nick Bostrom that the process must call upon our own deepest intelligence, so that we enjoy the benefits, which are real, without succumbing to the perils, which are just as real. Working out the ethics of what smart machines should, or should not do—looking after the frail elderly, or deciding whom to kill on the battlefield—won’t be settled by fast thinking, snap judgments, no matter how heartfelt. This will be a slow inquiry, calling on ethicists, jurists, computer scientists, philosophers, and many others. As with all ethical issues, stances will be provisional, evolve, be subject to revision. I’m glad to say that for the past five years the Association for the Advancement of Artificial Intelligence has formally addressed these ethical issues in detail, with a series of panels, and plans are underway to expand the effort. As Bostrom says, this is the essential task of our century.
>
>

*Update*: Stuart Russell of UC Berkeley has now added [a nice reply](http://edge.org/conversation/the-myth-of-ai#26015) to the edge.org conversation which echoes some of the points I made above.

---

1. I could have also objected to claims and arguments made in the conversation, for example Lanier’s claim that “The AI component would be only ambiguously there and of little importance [relative to the actuators component].” To me, this is like saying that humans rule the planet because of our actuators, not because of our superior intelligence. Or in response to Kevin Kelly’s claim that “So far as I can tell, AIs have not yet made a decision that its human creators have regretted,” I can for example point to the automated trading algorithms that [nearly bankrupted Knight Capital](http://www.reuters.com/article/2012/08/01/us-usa-nyse-tradinghalts-idUSBRE8701BN20120801) [faster](http://www.nature.com/srep/2013/130911/srep02627/full/srep02627.html) than any human could react. But in this piece I will focus instead on claims that seem to be misunderstandings of the positive case that’s being made for AI as an existential risk.
2. That is, unless it strongly trusts the agent making the external modification, and expects it to do a better job of making those modifications than it could itself, neither of which will be true of humans from the superhuman AI’s perspective.

The post [Three misconceptions in Edge.org’s conversation on “The Myth of AI”](https://intelligence.org/2014/11/18/misconceptions-edge-orgs-conversation-myth-ai/) appeared first on [Machine Intelligence Research Institute](https://intelligence.org).


In September, MIRI [hosted](http://intelligence.org/2014/07/25/bostrom/) Nick Bostrom at UC Berkeley to discuss his new book [*Superintelligence*](http://smile.amazon.com/Superintelligence-Dangers-Strategies-Nick-Bostrom-ebook/dp/B00LOOCGB2/). A video and transcript of that talk are [now available](http://www.c-span.org/video/?321534-1/book-discussion-superintelligence) from *BookTV* by C-SPAN, which also has a DVD of the event available.

**Update:** Nick Bostrom has also made his [slides for the talk](https://intelligence.org/wp-content/uploads/2014/11/Bostrom-Superintelligence-Berkeley-talk.pptx) available.

[![Bostrom Berkeley talk](https://intelligence.org/wp-content/uploads/2014/11/Bostrom-Berkeley-talk.png)](http://www.c-span.org/video/?321534-1/book-discussion-superintelligence)
The post [Video of Bostrom’s talk on Superintelligence at UC Berkeley](https://intelligence.org/2014/11/06/video-bostroms-talk-superintelligence-uc-berkeley/) appeared first on [Machine Intelligence Research Institute](https://intelligence.org).


A new guide to MIRI’s research

[![Guide to MIRI's Research](https://intelligence.org/wp-content/uploads/2014/11/Guide-to-MIRIs-Research.png)](http://intelligence.org/research-guide/)Nate Soares has written “[A Guide to MIRI’s Research](http://intelligence.org/research-guide/),” which outlines the main thrusts of MIRI’s current research agenda and provides recommendations for which textbooks and papers to study so as to understand what’s happening at the cutting edge.

This guide replaces Louie Helm’s earlier “Recommended Courses for MIRI Math Researchers,” and will be updated regularly as new lines of research open up, and as new papers and reports are released. It is not a replacement for our upcoming technical report on MIRI’s current research agenda and its supporting papers, which are still in progress. (“[Corrigibility](https://intelligence.org/files/CorrigibilityTR.pdf)” is the first supporting paper we’ve released for that project.)

The post [A new guide to MIRI’s research](https://intelligence.org/2014/11/06/new-guide-miris-research/) appeared first on [Machine Intelligence Research Institute](https://intelligence.org).


MIRI’s November Newsletter

|  |  |
| --- | --- |
|

|  |
| --- |
|
[Machine Intelligence Research Institute](http://intelligence.org)
 |

 |
|

|  |  |
| --- | --- |
|

|  |
| --- |
|
**Research Updates**
* New Friendly AI research area: “[Corrigibility](http://intelligence.org/2014/10/18/new-report-corrigibility/).”
* New report: “[UDT with known search order](http://intelligence.org/2014/10/30/new-report-udt-known-search-order/).”
* 2 new analyses: “[AGI outcomes and civilizational competence](http://intelligence.org/2014/10/16/agi-outcomes-civilizational-competence/)” and “[The *Financial Times* story on MIRI](http://intelligence.org/2014/10/31/financial-times-story-miri/).”
* [Video](http://intelligence.org/2014/10/07/nate-soares-talk-aint-rich/) of Nate Soares’ decision theory talk at Purdue.

**News Updates**
* If you’re reading *Superintelligence* or following its online reading group, [**please take this short survey**](https://docs.google.com/forms/d/1P53uNnZY_suE5IXW8WLQtXGuipAkUfQHj1yN_D121X0/viewform).

**Other Updates**
* Elon Musk has been talking about AGI risk [quite a bit lately](http://www.computerworld.com/article/2840815/ai-researchers-say-elon-musks-fears-not-completely-crazy.html). He has also joined the external advisory boards of [CSER](http://cser.org/about/who-we-are/) and [FLI](http://www.futureoflife.org/who), two existential risk organizations with whom we work closely.
* Effective altruists may want to register for the [EA Donation Registry](http://lesswrong.com/lw/l56/introducing_an_ea_donation_registry_covering/).

As always, please don’t hesitate to let us know if you have any questions or comments.
Best,
Luke Muehlhauser
Executive Director
 |

 |

 |
|  |

The post [MIRI’s November Newsletter](https://intelligence.org/2014/11/01/miris-november-newsletter/) appeared first on [Machine Intelligence Research Institute](https://intelligence.org).