Bill Hibbard on Ethical Artificial Intelligence

![Bill Hibbard portrait](https://intelligence.org/wp-content/uploads/2013/02/associate_hibbard.png)Bill Hibbard is an Emeritus Senior Scientist at the University of Wisconsin-Madison Space Science and Engineering Center, currently working on issues of AI safety and unintended behaviors. He has a BA in Mathematics and MS and PhD in Computer Sciences, all from the University of Wisconsin-Madison. He is the author of *[Super-Intelligent Machines](http://www.amazon.com/Super-Intelligent-Machines-International-Systems-Engineering/dp/0306473887/)*, [“Avoiding Unintended AI Behaviors,”](https://intelligence.org/files/UnintendedBehaviors.pdf) [“Decision Support for Safe AI Design,”](https://intelligence.org/files/DecisionSupport.pdf) and [“Ethical Artificial Intelligence.”](http://arxiv.org/abs/1411.1373) He is also principal author of the [Vis5D](http://en.wikipedia.org/wiki/Vis5D), [Cave5D](http://en.wikipedia.org/wiki/Cave5D), and [VisAD](http://en.wikipedia.org/wiki/VisAD) open source visualization systems.

**Luke Muehlhauser**: You recently released a self-published book, *[Ethical Artificial Intelligence](http://arxiv.org/abs/1411.1373)*, which “combines several peer reviewed papers and new material to analyze the issues of ethical artificial intelligence.” Most of the book is devoted to the kind of exploratory engineering in AI that you and I described in [a recent CACM article](https://intelligence.org/2014/08/22/new-paper-exploratory-engineering-artificial-intelligence/), such that you mathematically analyze the behavioral properties of classes of future AI agents, e.g. utility-maximizing agents.

Many AI scientists have the intuition that such early, exploratory work is very unlikely to pay off when we are so far from building an AGI, and don’t what an AGI will look like. For example, Michael Littman [wrote](http://kruel.co/2015/02/05/interview-with-michael-littman-on-ai-risks/#sthash.YrQRrluT.dpbs):

…proposing specific mechanisms for combatting this amorphous threat [of AGI] is a bit like trying to engineer airbags before we’ve thought of the idea of cars. Safety has to be addressed in context and the context we’re talking about is still absurdly speculative.

How would you defend the value of the kind of work you do in Ethical Artificial Intelligence to Littman and others who share his skepticism?

---

**Bill Hibbard**: This is a good question, Luke. The analogy with cars is useful. Unlike engineering airbags before cars are even thought of, we are already working hard to develop AI and can anticipate various types of dangers.

When cars were first imagined, engineers probably knew that they would propel human bodies at speed and that they would need to carry some concentrated energy source. They knew from accidents with horse carriages that human bodies travelling at speed are liable to injury, and they knew that concentrated energy sources are liable to fire and explosion which may injure humans. This is analogous with what we know about future AI: that to serve humans well AI will have to know a lot about individual humans and that humans will not be able to monitor every individual action by AI. These properties of future AI pose dangers just as the basic properties of cars (propelling humans and carrying energy) pose dangers.

Early car designers could have anticipated that no individual car would carry all of humanity and thus car accidents would not pose existential threats to humanity. To the extent that cars threaten human safety and health via pollution, we have time to notice these threats and address them. With AI we can anticipate possible scenarios that do threaten humanity and that may be difficult to address once the AI system is operational. For example, as described in the first chapter of my book, the Omniscience AI, with a detailed model of human society and a goal of maximizing profits, threatens to control human society. However, AI poses much greater potential benefits than cars but also much greater dangers. This justifies greater effort to anticipate the dangers of AI.

It’s also worth noting that the abstract frameworks for exploratory engineering apply to any reasonable future AI design. As the second chapter of my book describes, any set of complete and transitive preferences among outcomes can be expressed by a utility function. If preferences are incomplete then there are outcomes A and B with no preference between them, so the AI agent cannot decide. If preferences are not transitive then there are outcomes A, B, and C such that A is preferred to B, B is preferred to C, and C is preferred to A. Again, the AI agent cannot decide. Thus our exploratory engineering can assume utility maximizing agents and cover all cases in which the AI agent can decide among outcomes.

Similarly, the dangers discussed in the book are generally applicable. Any design for powerful AI should explain how it will avoid the self-delusion problem described by Ring and Orseau, the problem of corrupting the reward generator as described by Hutter, and the problem of unintended instrumental actions as described by Omohundro (he called them basic AI drives).

The threat level from AI justifies addressing AI dangers now and with significant resources. And we are developing tools that enable us to analyze dangers of AI systems before we know the specifics of their designs.

---

**Luke**: Your book mostly discusses AGIs rather than contemporary narrow AI systems. Roughly when do you expect humanity will develop something resembling the kind of AGIs you have in mind? Or, what does your probability distribution over “Years to AGI” look like?

---

**Bill**: In my 2002 book, Super-Intelligent Machines, I wrote that “machines as intelligent as humans are possible and will exist within the next century or so.” (The publisher owns the copyright for my 2002 book, preventing me from giving electronic copies to people, and charges more than $100 per print copy. This largely explains my decision to put my current book on arxiv.org.) I like to say that we will get to human-level AI during the lives of children already born and in fact I can’t help looking at children with amazement, contemplating the events they will see.

In his 2005 book, The Singularity is Near, Ray Kurzweil predicted human-level AI by 2029. He has a good track record at technology prediction and I hope he is right: I was born in 1948 so have a good chance of living until 2029. He also predicted the singularity by 2045, which must include the kind of very powerful AI systems discussed in my recent book.

Although it has nowhere near human-level intelligence, the DeepMind Atari player is a general AI system in the sense that it has no foreknowledge of Atari games other than knowing that the goal is to get a high score. The remarkable success of this system increases my confidence that we will create true AGI systems. DeepMind was purchased by Google, and all the big IT companies are energetically developing AI. It is the combination of AGI techniques and access to hundreds of millions of human users that can create the scenario of the Omniscience AI described in Chapter 1 of my book. Similarly for government surveillance agencies, which have hundreds of millions of unwitting users.

In 1983 I made a wager that a computer would beat the world Go champion by 2013, and lost. In fact, most predictions about AI have been wrong. Thus we must bring some humility to our predictions about the dates of AI milestones.

Because Ray Kurzweil’s predictions are based on quantitative extrapolation from historical trends and because of his good track record, I generally defer to his predictions. If human- level AI will exist by 2029 and very capable and dangerous AGI systems will exist by 2045, it is urgent that we understand the social effects and dangers of AI as soon as possible.

---

**Luke**: Which section(s) of your book do you think are most likely to be intriguing to computer scientists, because they’ll learn something that seems novel (to them) and plausibly significant?

---

**Bill**: Thanks Luke. There are several sections of the book that may be interesting or useful.

At the Workshop on AI and Ethics at AAAI-15 there was some confusion about the generality of utilitarian ethics, based on the connotation that a utility function is defined as a linear sum of features or similar simple expression. However, as explained in Chapter 2 and in my first answer in this interview, more complex utility functions can express any set of complete and transitive preferences among outcomes. That is, if an agent always has a most preferred outcome among any finite set of outcomes, then that agent can be expressed as a utility-maximizing agent.

Chapter 4 goes into detail on the issues of agents whose environment models are finite stochastic programs. Most of the papers in the AGI community assume that environments are modeled by programs for universal Turing machines, with no limit on their memory use. I think that much can be added to what I wrote in Chapter 4, and hope that someone will do that.

The self-modeling agents of Chapter 8 are the formal framework analog of value learners such as the DeepMind Atari player, and their use as a formal framework is novel. Self-modeling agents have useful properties, such as the capability to value agent resource increases and a way to avoid the problem of the agent utility function being inconsistent with the agent’s definition. An example of this problem is what Armstrong refers to as “motivated value selection.” More generally, it is the problem of adding any “special” actions to a utility maximizing agent, where those special actions do not maximize the utility function. In motivated value selection, the special action is the agent evolving its utility function. A utility maximizing agent may choose an action of removing the special actions from its definition, as counter-productive to maximizing its utility function. Self-modeling agents include such evolutionary special actions in the definition of their value functions, and they learn a model of their value function which they use to choose their next action. Thus there is no

inconsistency. I think these ideas should be interesting to other computer scientists.

At the FLI conference in San Juan in January 2015 there was concern about the kind of technical AI risks described in Chapters 5 – 9 of my book, and concern about technological unemployment. However, there was not much concern about the dangers associated with:

1. Large AI servers connected to the electronic companions that will be carried by large numbers of people and the

ability of the human owners of those AI servers to manipulate society, and
2. A future world in which great wealth can buy increased intelligence and superior intelligence can generate increased wealth. This positive feedback loop will result in a power law distribution of intelligence as opposed to the current normal distribution of IQs with mean = 100 and standard deviation = 15.

These issues are discussed in Chapters 1 and 10 of my book. The Global Brain researchers study the way intelligence is exhibited by the network of humans; the change in distribution of intelligence of humans and machines who are nodes of the network will have profound effects on the nature of the Global Brain. Beyond computer scientists, I think the public needs to be aware of these issues.

Finally, I’d like to expand on my previous answer, specifically that the DeepMind Atari player is an example of general AI. In Chapter 1 of my book I describe how current AI systems have environment models that are designed by human engineers, whereas future AI systems will need to learn environment models that are too complex to be designed by human engineers. The DeepMind system does not use an environment model designed by engineers. It is “model-free” but the value function that it learns is just as complex as an environment model and in fact encodes an implicit environment model. Thus the DeepMind system is the first example of a future AI system with significant functionality.

---

**Luke**: Can elaborate what you mean by saying that “the self-modeling agents of Chapter 8 are the formal framework analog of value learners such as the DeepMind Atari player”? Are you saying that the formal work you do in chapter 8 has implications even for an extant system like the DeepMind Atari player, because they are sufficiently analogous?

---

**Bill**: To elaborate on what I mean by “the self-modeling agents of Chapter 8 are the formal framework analog of value learners such as the DeepMind Atari player,” self-modeling agents and value learners both learn a function v(ha) that produces the expected value of proposed action a after interaction history h (that is, h is a sequence of observations and actions; see my book for details). For the DeepMind Atari player, v(ha) is the expected game score after action a and h is restricted to the most recent observation (i.e., a game screen snapshot). Whereas the DeepMind system must be practically computable, the self-modeling agent framework is a purely mathematical definition. This framework is finitely computable but any practical implementation would have to use approximations. The book offers a few suggestions about computing techniques, but the discussion is not very deep.

Because extant systems such as the DeepMind Atari player are not yet close to human-level intelligence, there is no implication that this system should be subject to safety constraints. It is encouraging that the folks at DeepMind and at Vicarious are concerned about AI ethics, for two reasons: 1) They are likely to apply ethical requirements to their systems as they approach human-level, and 2) They are very smart and can probably add a lot to AI safety research.

Generally, research on safe and ethical AI complicates the task of creating AI by adding requirements. My book develops a three-argument utility function expressing human values which will be very complex to compute. Similarly for other components of the definition of self-modeling agents in the book.

I think there are implications the other way around. The self-modeling framework is based on statistical learning and the success of the DeepMind Atari player, the Vicarious captcha solver, IBM’s Watson, and other practical systems that use statistical learning techniques increases our confidence that these techniques can actually work for AI capability and safety.

Some researchers suggest that safe AI should rely on logical deduction rather than statistical learning. This idea offers greater possibility of proving safety properties of AI, but so far there are no compelling demonstrations of AI systems based on logical deduction (at least, none that I am aware of). Such demonstrations would add a lot of confidence in our ability to prove safety properties of AI systems.

---

**Luke**: Your 10th chapter considers the political aspects of advanced AI. What do you think can be done now to improve our chances of solving the political challenges of AI in the future? Sam Altman of YC has [proposed](http://blog.samaltman.com/machine-intelligence-part-2) various kinds of regulation — do you agree with his general thinking? What other ideas do you have?

---

**Bill**: The central point of my 2002 book was the need for public education about and control over above-human-level AI. The current public discussion by Stephen Hawking, Bill Gates, Elon Musk, Ray Kurzweil, and others about the dangers of AI is very healthy, as it educates the public. Similarly for the Singularity Summits organized by the Singularity Institute (MIRI’s predecessor), which I thought were the best thing the Singularity Institute did.

In the US people cannot own automatic weapons, guns of greater than .50 caliber, or explosives without a license. It would be absurd to license such things but to allow unregulated development of above-human-level AI. As the public is educated about AI, I think some form of regulation will be inevitable.

However, as they say, the devil will be in the details and humans will be unable to compete with future AI on details. Complex details will be AI’s forte. So formulating effective regulation will be a political challenge. The Glass-Steagal Act of 1933, regulating banking, was 37 pages long. The Dodd-Frank bill of 2010, also to regulate banking 77 years later, was 848 pages long. An army of lawyers drafted the bill, many employed to protect the interests of groups affected by the bill. The increasing complexity of laws reflects efforts by regulated entities to lighten the burden of regulation. The stakes in regulating AI will be huge and we can expect armies of lawyers, with the aid of the AI systems being regulated, to create very complex laws.

In the second chapter of my book, I conclude that ethical rules are inevitably ambiguous and base my proposed safe AI design on human values expressed in a utility function rather than rules. Consider the current case before the US Supreme Court to interpret the meaning of the words “established by the state” in the context of the 363,086 words of the Affordable Care Act. This is a good example of the ambiguity of rules. Once AI regulations become law, armies of lawyers, aided by AI, will be engaged in debates over their interpretation and application.

The best counterbalance to armies of lawyers creating complexity on any legal issue is a public educated about the issue and engaged in protecting their own interests. Automobile safety is a good example. This will also be the case with AI regulation. And, as discussed in the introductory section of Chapter 10, there is precedent for the compassionate intentions of some wealthy and powerful people and this may serve to counterbalance their interest in creating complexity.

Privacy regulations, which affect existing large IT systems employing AI, already exist in the US and even more so in Europe. However, many IT services depend on accurate models of users’ preferences. At the recent FLI conference in San Juan, I tried to make the point that a danger from AI will be that people will want the kind of close, personal relationship with AI systems that will enable intrusion and manipulation by AI. The Omniscience AI described in Chapter 1 of my book is an example. As an astute IT lawyer said at the FLI conference, the question of whether an IT innovation will be legal depends on whether it will be popular.

This brings us back to the need for public education about AI. For people to resist being seduced by the short term benefits of close relationships with AI, they need to understand the long term consequences. I think it is not realistic to prohibit close relationships between people and AI, but perhaps the public, if it understands the issues, can demand some regulation over the goals for which those relationships are exploited.

The final section of my Chapter 10 says that AI developers and testers should recognize that they are acting as agents for the future of humanity and that their designs and test results should be transparent to the public. The FLI open letter and Google’s panel on AI ethics are encouraging signs that AI developers do recognize their role as agents for future humanity. Also, DeepMind has been transparent about the technology of their Atari player, even making source code available for non-commercial purposes.

AI developers deserve to be rewarded for their success. On the other hand, people have a right to avoid losing control over their own lives to an all-powerful AI and its wealthy human owners. The problem is to find a way to achieve both of these goals.

Among current humans, with naturally evolved brains, IQ has a normal distribution. When brains are artifacts, their intelligence is likely to have a power law distribution. This is the pattern of distributions of sizes of other artifacts such as trucks, ships, buildings, and computers. The average human will not be able to understand or ever learn the languages used by the most intelligent minds. This may mean the end of any direct voice in public policy decisions for average humans – effectively the end of democracy. But if large AI systems are maximizing utility functions that account for the values of individual humans, that may take the place of direct democracy.

Chapters 6 – 8 of my book propose mathematical definitions for an AI design that does balance the values of individual humans. Chapter 10 suggests that this design may be modified to provide different weights to the values of different people, for example to reward those who develop AI systems. I must admit that the connection between the technical chapters of my book and Chapter 10, on politics, is weak. Political issues are just difficult. For example, the future will probably have multiple AI systems with conflicting utility functions and a power law distribution of intelligence. It is difficult to predict how such a society would function and how it would affect humans, and this unpredictability is a risk. Creating a world with a single powerful AI system also poses risks, and may be difficult to achieve.

Since my first paper about future AI in 2001, I have thought that the largest risks from AI are political rather than technical. We have an ethical obligation to educate the public about the future of AI, and an educated public is an essential element of finding a good outcome from AI.

---

**Luke**: Thanks, Bill!

The post [Bill Hibbard on <em>Ethical Artificial Intelligence</em>](https://intelligence.org/2015/03/09/bill-hibbard/) appeared first on [Machine Intelligence Research Institute](https://intelligence.org).


[![Fallenstein APS talk](http://intelligence.org/wp-content/uploads/2015/03/Fallenstein-APS-talk.png)](https://intelligence.org/wp-content/uploads/2015/03/Fallenstein-Beneficial-smarter-than-human-intelligence.pdf)MIRI researcher Benja Fallenstein recently delivered an invited talk at the [March 2015](http://www.aps.org/meetings/march/index.cfm) meeting of the American Physical Society in San Antonio, Texas. Her talk was one of four in a [special session on artificial intelligence](http://meeting.aps.org/Meeting/MAR15/Session/M3).

Fallenstein’s title was “Beneficial Smarter-than-human Intelligence: the Challenges and the Path Forward.” The slides are available [here](https://intelligence.org/wp-content/uploads/2015/03/Fallenstein-Beneficial-smarter-than-human-intelligence.pdf). Abstract:

> Today, human-level machine intelligence is still in the domain of futurism, but there is every reason to expect that it will be developed eventually. A generally intelligent agent as smart or smarter than a human, and capable of improving itself further, would be a system we’d need to design for safety from the ground up: There is no reason to think that such an agent would be driven by human motivations like a lust for power; but almost any goals will be easier to meet with access to more resources, suggesting that most goals an agent might pursue, if they don’t explicitly include human welfare, would likely put its interests at odds with ours, by incentivizing it to try to acquire the physical resources currently being used by humanity. Moreover, since we might try to prevent this, such an agent would have an incentive to deceive its human operators about its true intentions, and to resist interventions to modify it to make it more aligned with humanity’s interests, making it difficult to test and debug its behavior. This suggests that in order to create a beneficial smarter-than-human agent, we will need to face three formidable challenges: How can we formally specify goals that are in fact beneficial? How can we create an agent that will reliably pursue the goals that we give it? And how can we ensure that this agent will not try to prevent us from modifying it if we find mistakes in its initial version? In order to become confident that such an agent behaves as intended, we will not only want to have a practical implementation that seems to meet these challenges, but to have a solid theoretical understanding of why it does so. In this talk, I will argue that even though human-level machine intelligence does not exist yet, there are foundational technical research questions in this area which we can and should begin to work on today. For example, probability theory provides a principled framework for representing uncertainty about the physical environment, which seems certain to be helpful to future work on beneficial smarter-than-human agents, but standard probability theory assumes omniscience about *logical* facts; no similar principled framework for representing uncertainty about the outputs of deterministic computations exists as yet, even though any smarter-than-human agent will certainly need to deal with uncertainty of this type. I will discuss this and other examples of ongoing foundational work.
>
>

Stuart Russell of UC Berkeley also gave a talk at this session, about [the long-term future of AI](http://meeting.aps.org/Meeting/MAR15/Session/M3.1).

The post [Fallenstein talk for APS March Meeting 2015](https://intelligence.org/2015/03/09/fallenstein-talk-aps-march-meeting-2015/) appeared first on [Machine Intelligence Research Institute](https://intelligence.org).


March 2015 newsletter

|  |  |
| --- | --- |
|

|  |
| --- |
|
[Machine Intelligence Research Institute](http://intelligence.org)
 |

 |
|

|  |  |
| --- | --- |
|

|  |
| --- |
|
**Research updates**
* We’ve made significant updates to three pages: [Research](https://intelligence.org/research/), [All Publications](https://intelligence.org/all-publications/), and [A Guide to MIRI’s Research](https://intelligence.org/research-guide/).
* We’ve published an [annotated bibliography](https://intelligence.org/2015/02/05/new-annotated-bibliography-miris-technical-agenda/) and [dedicated page](https://intelligence.org/technical-agenda/) for our new technical research agenda overview.
* [This new mailing list](https://intelligence.org/2015/02/03/keep-date-miris-research-via-new-mailing-list/) sends one email per new technical paper, and nothing else.
* New analyses: [Davis on AI capability and motivation](https://intelligence.org/2015/02/06/davis-ai-capability-motivation/), [How AI timelines are estimated](http://aiimpacts.org/how-ai-timelines-are-estimated/), [The slow traversal of ‘human-level’](http://aiimpacts.org/the-slow-traversal-of-human-level/), [Multipolar research questions](http://aiimpacts.org/multipolar-research-questions/), [At-least-human-level-at-human-cost AI](http://aiimpacts.org/at-least-human-level-at-human-cost-ai/), and three analyses of discontinuous technological progress (to investigate the plausibility of AI fast takeoff): [Penicillin and syphilis](http://aiimpacts.org/penicillin-and-syphilis/), [Discontinuous progress investigation](http://aiimpacts.org/discontinuous-progress-investigation/), and [What’s up with nuclear weapons?](http://aiimpacts.org/whats-up-with-nuclear-weapons/)

**News updates**
* MIRI’s [paper on value learning](https://intelligence.org/files/ValueLearningProblem.pdf) was covered in some detail [at *Nautilus*](http://nautil.us/blog/will-humans-be-able-to-control-computers-that-are-smarter-than-us).
* The *Superintelligence* reading group is in [week 24](http://lesswrong.com/lw/llp/superintelligence_24_morality_models_and_do_what/), on morality models and “do what I mean.”

**Other news**
* The Future of Life Institute has released a new [survey of research questions for robust and beneficial AI](http://futureoflife.org/static/data/documents/suggested_research.pdf).

As always, please don’t hesitate to let us know if you have any questions or comments.
Best,
Luke Muehlhauser
Executive Director
 |

 |

 |
|

|  |
| --- |
|  |

  |

The post [March 2015 newsletter](https://intelligence.org/2015/03/01/march-newsletter-2/) appeared first on [Machine Intelligence Research Institute](https://intelligence.org).


Davis on AI capability and motivation

In a [review of *Superintelligence*](https://cs.nyu.edu/davise/papers/Bostrom.pdf), NYU computer scientist Ernest Davis voices disagreement with a number of claims he attributes to Nick Bostrom: that “intelligence is a potentially infinite quantity with a well-defined, one-dimensional value,” that a superintelligent AI could “easily resist and outsmart the united efforts of eight billion people” and achieve “virtual omnipotence,” and that “though achieving intelligence is more or less easy, giving a computer an ethical point of view is really hard.”

These are all stronger than Bostrom’s actual claims. For example, Bostrom never characterizes building a generally intelligent machine as “easy.” Nor does he say that intelligence can be infinite or that it can produce “omnipotence.” Humans’ intelligence and accumulated knowledge gives us a decisive advantage over chimpanzees, even though our power is limited in important ways. An AI need not be magical or all-powerful in order to have the same kind of decisive advantage over humanity.

Still, Davis’ article is one of the more substantive critiques of MIRI’s core assumptions that I have seen, and he addresses several deep issues that directly bear on AI forecasting and strategy. I’ll sketch out a response to his points here.

**Measuring an intelligence explosion**

Davis writes that Bostrom assumes “that a large gain in intelligence would necessarily entail a correspondingly large increase in power.” This is again too strong. (Or it’s trivial, if we’re using the word “intelligence” to pick out a specific kind of power.)

Bostrom *is* interested in intelligence for its potential to solve practical problems and shape the future. If there are other kinds of intelligence, they’re presumably of less economic and strategic importance than the “cognitive superpowers” Bostrom describes in chapter 6. It is the potential power autonomous machines could exhibit that should primarily concern us from a safety standpoint, and “intelligence” seems as good a term as any for the kind of power that doesn’t depend on an agent’s physical strength or the particularities of its environment.

When it comes to Bostrom’s intelligence explosion thesis, I don’t think ‘does an increase in intelligence *always* yield a corresponding increase in power?’ gets at the heart of the issue. Consider [David Chalmers’ version of the argument](http://consc.net/papers/singularity.pdf):

[I]t is not unreasonable to hold that we can create systems with greater programming ability than our own, and that systems with greater programming ability will be able to create systems with greater programming ability in turn. It is also not unreasonable to hold that programming ability will correlate with increases in various specific reasoning abilities. If so, we should expect that absent defeaters, the reasoning abilities in question will explode.

Here, there’s no explicit appeal to “intelligence,” which is replaced with programming ability plus an arbitrarily large number of “specific reasoning abilities.” Yet if anything I find this argument more plausible than Bostrom’s formulation. For that reason, I agree with Bostrom that the one-dimensional representation of intelligence is inessential and “one could, for example, instead represent a cognitive ability profile as a hypersurface in a multidimensional space” (p. 273).[1](https://intelligence.org/2015/02/06/davis-ai-capability-motivation/#footnote_0_11582 " Still more fine-grained versions of the same argument may be possible. E.g., “programming ability” might decompose into multiple abilities, such as the ability to efficiently explore search spaces for code that meets constraints and the ability to efficiently test candidate code. ")

Moreover, the relevant question isn’t whether an increase in a self-improving AI’s general programming ability *always* yields a corresponding increase in its ability to improve its own programming ability. Nor is the question whether either of those abilities *always* correlates with the other cognitive capabilities Bostrom is interested in (“strategizing,” “social manipulation,” “hacking,” “technology research,” “economic productivity”). I’d instead say that the five core questions from Bostrom’s point of view are:

1. Is the first superintelligent AI likely to result from self-improving AI systems?
2. If so, how much of the AI’s self-improvement is likely to be driven by improvements to some cognitive capability (e.g., programming ability) that facilitates further enhancement of the capability in question?
3. Are improvements to this self-improving capability likely to accelerate, as early advances result in cascades of more rapid advances? Or will the self-improving capability repeatedly stall out, advancing in small fits and starts?
4. If self-improvement cascades are likely, are they also likely to result in improvements to other cognitive capabilities that we more directly care about?[2](https://intelligence.org/2015/02/06/davis-ai-capability-motivation/#footnote_1_11582 " For example: If an AI approaching superintelligence stumbles upon a cascade of improvements to its programming ability, will its capabilities and decision criteria also result in repeated improvements its physics modules, or its psychology modules? ") Or will those other cognitive capabilities lag far behind the capabilities that ‘explode’?
5. If an ability like ‘programming’ is likely to be self-reinforcing in an accelerating way, and is likely to foster accelerating improvements in other cognitive abilities, exactly how fast will those accelerations be? Are we talking about a gap of decades between the AI’s first self-reinforcing self-improvements and its attainment of superintelligence? Months? Hours?

Bostrom’s position on those questions — that a fast or moderate intelligence explosion is likely — at no point presupposes that “intelligence” is a well-defined scalar value one could do calculus with, except as a toy model for articulating various qualitative possibilities. When he writes out differential equations, Bostrom is careful to note that intelligence cannot be infinite, that one-dimensionality is a simplifying assumption, and that his equations are “intended for illustration only.”[3](https://intelligence.org/2015/02/06/davis-ai-capability-motivation/#footnote_2_11582 " On page 76, for example, Bostrom writes: “This particular growth trajectory has a positive singularity at t = 18 months. In reality, the assumption that recalcitrance is constant would cease to hold as the system began to approach the physical limits of information processing, if not sooner.” On page 77, Bostrom says that the point he intends to illustrate is only that if AI progress is primarily AI-driven, resultant feedback loops that do arise will have a larger accelerating effect. ")

We should expect artificial general intelligence (AGI) to specialize in some domains and neglect others. Bostrom’s own analysis assumes that a recursively self-improving AI would tend to prioritize acquiring skills like electrical engineering over skills like impressionist painting, all else being equal.[4](https://intelligence.org/2015/02/06/davis-ai-capability-motivation/#footnote_3_11582 " This is because a wide variety of final goals are best served through the acquisition or resources and the building of infrastructure, a set of objectives that are more likely to be furthered by electrical engineering skills than by painting skills. This argument is an instance of Bostrom’s instrumental convergence thesis in chapter 7. ") For that matter, present-day AI is already superhuman in some cognitive tasks (e.g., chess and mental arithmetic), yet subhuman in many others. Any attempt to quantify the ‘overall intelligence’ of Deep Blue or Google Maps will obscure some important skills and deficits of these systems.[5](https://intelligence.org/2015/02/06/davis-ai-capability-motivation/#footnote_4_11582 "While it turns out that many intelligence-related characteristics correlate with a single easily-measured number in humans (g), this still doesn’t allow us to make fine-grained predictions about individual competency levels. I also can’t think of an obvious reason to expect a number even as informative as g to arise for AGI, especially early-stage AGI. Bostrom writes (p. 93):
[S]uppose we could somehow establish that a certain future AI will have an IQ of 6,455: then what? We would have no idea of what such an AI could actually do. We would not even know that such an AI had as much general intelligence as a normal human adult–perhaps the AI would instead have a bundle of special-purpose algorithms enabling it to solve typical intelligence test questions with superhuman efficiency but not much else.
Some recent efforts have been made to develop measurements of cognitive capacity that could be applied to a wider range of information-processing systems, including artificial intelligences. Work in this direction, if it can overcome various technical difficulties, may turn out to be quite useful for some scientific purposes including AI development. For purposes of the present investigation, however, its usefulness would be limited since we would remain unenlightened about what a given superhuman performance score entails for actual ability to achieve practically important outcomes in the world. ") Still, using concepts like ‘intelligence’ or ‘power’ to express imprecise hypotheses is better than substituting precise metrics that overstate how much we currently know about general intelligence and the future of AI.[6](https://intelligence.org/2015/02/06/davis-ai-capability-motivation/#footnote_5_11582 " Imagine a Sumerian merchant living 5,000 years ago, shortly after the invention of writing, who has noticed the value of writing for storing good ideas over time, and not just market transactions. Writing could even allow one to transmit good ideas to someone you’ve never met, such as a future descendant. The merchant notices that his own successes have often involved collecting others’ good ideas, and that good ideas often open up pathways to coming up with other, even better ideas. From his armchair, he concludes that if writing becomes sufficiently popular, it will allow a quantity called society’s knowledge level to increase in an accelerating fashion; which, if the knowledge is used wisely, could result in unprecedented improvements to human life.
In retrospect we can say ‘knowledge’ would have been too coarse-grained a category to enable any precise predictions, and that there have really been multiple important breakthroughs that can be considered ‘knowledge explosions’ in different senses. Yet the extremely imprecise prediction can still give us a better sense of what to expect than we previously had. It’s a step up from the (historically common) view that civilizational knowledge only diminishes over time, the view that things will always stay the same, the view that human welfare will radically improve for reasons unrelated to knowledge build-up, etc.
The point of this analogy is not that people are good at making predictions about the distant future. Rather, my point is that hand-wavey quantities like ‘society’s knowledge level’ can be useful for making predictions, and can be based on good evidence, even if the correspondence between the quantity and the phenomenon it refers to is inexact. ")

**Superintelligence superiority**

In *Superintelligence* (pp. 59-61), Bostrom lists a variety of ways AGI may surpass humans in intelligence, owing to differences in hardware (speed and number of computational elements, internal communication speed, storage capacity, reliability, lifespan, and sensors) and software (editability, duplicability, goal coordination, memory sharing, and new modules, modalities, and algorithms). Davis grants that these may allow AGI to outperform humans, but expresses skepticism that this could give AGI a decisive advantage over humans. To paraphrase his argument:

*Elephants’ larger brains don’t make them superintelligent relative to mice; squirrels’ speed doesn’t give them a decisive strategic advantage over turtles; and we don’t know enough about what makes Einstein smarter than the village idiot to make confident predictions about how easy it is to scale up from village idiot to Einstein, or from Einstein to super-Einstein. So there’s no particular reason to expect a self-improving AGI to be able to overpower humanity.*

My response is that the capabilities Bostrom describes, like “speed superintelligence,” predict much larger gaps than the gaps we see between elephants and mice. Bostrom writes (in a footnote on p. 270):

At least a millionfold speedup compared to human brains is physically possible, as can be seen by considering the difference in speed and energy of relevant brain processes in comparison to more efficient information processing. The speed of light is more than a million times greater than that of neural transmission, synaptic spikes dissipate more than a million times more heat than is thermodynamically necessary, and current transistor frequencies are more than a million times faster than neuron spiking frequencies.

Davis objects that “all that running faster does is to save you time,” noting that a slower system could eventually perform all the feats of a faster one. But the ability to save time is exactly the kind of ability Bostrom is worried about. Even if one doubts that large improvements in collective or quality superintelligence are possible, a ‘mere’ speed advantage makes an enormous practical difference.

Imagine a small community of scientist AIs whose only advantages over human scientists stem from their hardware speed — they can interpret sensory information and assess hypotheses and policies a million times faster than a human. At that speed, an artificial agent could make ~115 years of intellectual progress in an hour, ~2700 years of progress in a day, and 250,000 years of progress in three months.

The effect of this speedup would be to telescope human history. Scientific and technological advances that would have taken us tens of thousands of years to reach can be ours by Tuesday. If we sent a *human* a thousand years into the past, equipped with all the 21st-century knowledge and technologies they wanted, they could conceivably achieve dominant levels of wealth and power in that time period. This gives us cause to worry about building machines that can rapidly accumulate millennia of experience over humans, even before we begin considering any potential advantages in memory, rationality, editability, etc.[7](https://intelligence.org/2015/02/06/davis-ai-capability-motivation/#footnote_6_11582 " Most obviously, a speed advantage can give the AI the time to design an even better AI.
None of this means that we can make specific highly confident predictions about when and how AI will achieve superintelligence. An AGI that isn’t very human-like may be slower than a human at specific tasks, or faster, in hard-to-anticipate ways. If a certain scientific breakthrough requires that one first build a massive particle accelerator, then the resources needed to build that accelerator may be a more important limiting factor than the AGI’s thinking speed. In that case, humans would have an easier time monitoring and regulating an AGI’s progress. We can’t rule out the possibility that speed superintelligence will face large unexpected obstacles, but we also shouldn’t gamble on that possibility or take it for granted. ")

At least some of the routes to superintelligence described by Bostrom look orders of magnitude larger than the cognitive advantages Einstein has over a village idiot (or than an elephant has over a mouse). We can’t rule out the possibility that we’ll run into larger-than-expected obstacles when we attempt to build AGI, but we shouldn’t gamble on that possibility. Black swans happen, but superintelligent AI is a *white* swan, and white swans happen too.

Since Bostrom’s pathways to superintelligence don’t have a lot in common with the neurological differences we can observe in mammals, there is no special reason to expect the gap between smarter-than-human AI and humans to resemble the gap between elephants and mice. Bostrom’s pathways also look difficult to biologically evolve, which means that their absence in the natural world tells us little about their feasibility.

We have even less cause to expect, then, that the gap between advanced AI and humans will resemble the gap between Einstein and a median human. If a single generation of random genetic recombination can produce an Einstein, the planning and design abilities of human (and artificial) engineers should make much greater feats a possibility.

**Delegating AI problems to the AI**

Separately, Davis makes the claim that “developing an understanding of ethics as contemporary humans understand it is actually one of the easier problems facing AI”. Again, I’ll attempt to summarize his argument:

*Morality doesn’t look particularly difficult, especially compared to, e.g., computer vision. Moreover, if we’re going to build AGI, we’re going to solve computer vision. Even if morality is as tough as vision, why assume one will be solved and not the other?*

Here my response is that you probably don’t need to solve every AI problem to build an AGI. We may be able to cheat at vision, for example, by tasking a blind AGI with solving the problem for us. But it is much easier to observe whether an algorithm is making progress on solving visual puzzles than to observe whether one is making progress on *ethical* questions, where there is much more theoretical and object-level disagreement among humans, there is less incentive for artificial agents to move from partial solutions to complete solutions, and failures don’t necessarily reduce the system’s power.

Davis notes that a socially fluent superintelligence would need to be an expert moralist:

Bostrom refers to the AI’s ‘social manipulation superpowers’. But if an AI is to be a master manipulator, it will need a good understanding of what people consider moral; if it comes across as completely amoral, it will be at a very great disadvantage in manipulating people. […] If the AI can understand human morality, it is hard to see what is the technical difficulty in getting it to follow that morality.

This is similar to Richard Loosemore’s argument against AI safety research, which I’ve [responded to in a blog post](http://nothingismere.com/2014/08/25/loosemore-on-ai-safety-and-attractors/). My objection is that an AI could come to understand human morality without thereby becoming moral, just as a human can come to understand the motivations of a stranger without thereby *acquiring* those motivations.

Since we don’t understand our preferences in enough generality or detail to translate them into code, it would be nice to be able to delegate the bulk of this task to a superintelligence. But if we program it to hand us an answer to the morality problem, how will we know whether it is being honest with us? To trust an AGI’s advice about how to make it trustworthy, we’d need to already have solved enough of the problem ourselves to make the AGI a reliable advisor.

Taking Davis’ proposal as an example, we can imagine instilling behavioral prescriptions into the AI by programming it to model Gandhi’s preferences and do what Gandhi would want it to. But if we try to implement this idea in code before building seed AI, we’re stuck with our own fallible attempts to operationalize concepts like ‘Gandhi’ and ‘preference;’ and if we try to implement it after, recruiting the AI to solve the problem, we’ll need to have already instilled some easier-to-program safeguards into it. What makes AGI safety research novel and difficult is our lack of understanding of how to initiate this bootstrapping process with any confidence.

The idea of using the deceased for value learning is interesting. Bostrom endorses the generalized version of this approach when he says that it is critical for value learning that the locus of value be “an object at a particular time” (p. 193). However, this approach may still admit of non-obvious loopholes, and it may still be too complicated for us to directly implement without recourse to an AGI. If so, it will need to be paired with solutions to the problems of [corrigibility](https://intelligence.org/2014/10/18/new-report-corrigibility/) and [stability](https://intelligence.org/2015/01/15/new-report-vingean-reflection-reliable-reasoning-self-improving-agents/) in self-modifying AI, just as “shutdown buttons” and other tripwire solutions will.

From Bostrom’s perspective, what makes advanced AI a game-changer is first and foremost its capacity to meaningfully contribute to AI research. The vision problem may be one of many areas where we can outsource sophisticated AGI problems to AGI, or to especially advanced narrow-AI algorithms. This is the idea underlying the intelligence explosion thesis, and it also underlies Bostrom’s worry that capabilities research will continue to pull ahead of safety research.

In self-improving AI scenarios, the key question is which AI breakthroughs are prerequisites for automating high-leverage computer science tasks. This holds for capabilities research, and it also holds for safety research. Even if AI safety turned out to be easier than computer vision in an absolute sense, it would still stand out as a problem that is neither a prerequisite for building a self-improving AI, nor one we can safely delegate to such an AI.

---

1. Still more fine-grained versions of the same argument may be possible. E.g., “programming ability” might decompose into multiple abilities, such as the ability to efficiently explore search spaces for code that meets constraints and the ability to efficiently test candidate code.
2. For example: If an AI approaching superintelligence stumbles upon a cascade of improvements to its programming ability, will its capabilities and decision criteria also result in repeated improvements its physics modules, or its psychology modules?
3. On page 76, for example, Bostrom writes: “This particular growth trajectory has a positive singularity at t = 18 months. In reality, the assumption that recalcitrance is constant would cease to hold as the system began to approach the physical limits of information processing, if not sooner.” On page 77, Bostrom says that the point he intends to illustrate is only that if AI progress is primarily AI-driven, resultant feedback loops that do arise will have a larger accelerating effect.
4. This is because a wide variety of final goals are best served through the acquisition or resources and the building of infrastructure, a set of objectives that are more likely to be furthered by electrical engineering skills than by painting skills. This argument is an instance of Bostrom’s instrumental convergence thesis in chapter 7.
5. While it turns out that many intelligence-related characteristics correlate with a single easily-measured number in humans (*g*), this still doesn’t allow us to make fine-grained predictions about individual competency levels. I also can’t think of an obvious reason to expect a number even as informative as *g* to arise for AGI, especially early-stage AGI. Bostrom writes (p. 93):
[S]uppose we could somehow establish that a certain future AI will have an IQ of 6,455: then what? We would have no idea of what such an AI could actually do. We would not even know that such an AI had as much general intelligence as a normal human adult–perhaps the AI would instead have a bundle of special-purpose algorithms enabling it to solve typical intelligence test questions with superhuman efficiency but not much else.

Some recent efforts have been made to develop measurements of cognitive capacity that could be applied to a wider range of information-processing systems, including artificial intelligences. Work in this direction, if it can overcome various technical difficulties, may turn out to be quite useful for some scientific purposes including AI development. For purposes of the present investigation, however, its usefulness would be limited since we would remain unenlightened about what a given superhuman performance score entails for actual ability to achieve practically important outcomes in the world.
6. Imagine a Sumerian merchant living 5,000 years ago, shortly after the invention of writing, who has noticed the value of writing for storing *good ideas* over time, and not just market transactions. Writing could even allow one to transmit good ideas to someone you’ve never met, such as a future descendant. The merchant notices that his own successes have often involved collecting others’ good ideas, and that good ideas often open up pathways to coming up with other, even better ideas. From his armchair, he concludes that if writing becomes sufficiently popular, it will allow a quantity called *society’s knowledge level* to increase in an accelerating fashion; which, if the knowledge is used wisely, could result in unprecedented improvements to human life.
In retrospect we can say ‘knowledge’ would have been too coarse-grained a category to enable any precise predictions, and that there have really been multiple important breakthroughs that can be considered ‘knowledge explosions’ in different senses. Yet the extremely imprecise prediction can still give us a better sense of what to expect than we previously had. It’s a step up from the (historically common) view that civilizational knowledge only diminishes over time, the view that things will always stay the same, the view that human welfare will radically improve for reasons unrelated to knowledge build-up, etc.

The point of this analogy is not that people are good at making predictions about the distant future. Rather, my point is that hand-wavey quantities like ‘society’s knowledge level’ *can* be useful for making predictions, and *can* be based on good evidence, even if the correspondence between the quantity and the phenomenon it refers to is inexact.
7. Most obviously, a speed advantage can give the AI the time to design an even better AI.
None of this means that we can make specific highly confident predictions about when and how AI will achieve superintelligence. An AGI that isn’t very human-like may be slower than a human at specific tasks, or faster, in hard-to-anticipate ways. If a certain scientific breakthrough requires that one first build a massive particle accelerator, then the resources needed to build that accelerator may be a more important limiting factor than the AGI’s thinking speed. In that case, humans would have an easier time monitoring and regulating an AGI’s progress. We can’t rule out the possibility that speed superintelligence will face large unexpected obstacles, but we also shouldn’t gamble on that possibility or take it for granted.

The post [Davis on AI capability and motivation](https://intelligence.org/2015/02/06/davis-ai-capability-motivation/) appeared first on [Machine Intelligence Research Institute](https://intelligence.org).


[![annotated bibliography](http://intelligence.org/wp-content/uploads/2015/02/annotated-bibliography.png)](https://intelligence.org/files/AnnotatedBibliography.pdf)Today we release a new [annotated bibliography](https://intelligence.org/files/AnnotatedBibliography.pdf) accompanying our [new technical agenda](https://intelligence.org/2014/12/23/new-technical-research-agenda-overview/), written by Nate Soares. If you’d like to discuss the paper, please do so [here](http://lesswrong.com/lw/lmn/the_value_learning_problem/).

Abstract:

> How could superintelligent systems be aligned with the interests of humanity? This annotated bibliography compiles some recent research relevant to that question, and categorizes it into six topics: (1) realistic world models; (2) idealized decision theory; (3) logical uncertainty; (4) Vingean reflection; (5) corrigibility; and (6) value learning. Within each subject area, references are organized in an order amenable to learning the topic. These are by no means the only six topics relevant to the study of alignment, but this annotated bibliography could be used by anyone who wants to understand the state of the art in one of these six particular areas of active research.
>
>

Today we’ve also released a page that collects the technical agenda and supporting reports. See our **[Technical Agenda](https://intelligence.org/technical-agenda/)** page.

The post [New annotated bibliography for MIRI’s technical agenda](https://intelligence.org/2015/02/05/new-annotated-bibliography-miris-technical-agenda/) appeared first on [Machine Intelligence Research Institute](https://intelligence.org).


As requested, we now offer email notification of new technical (math or computer science) papers and reports from MIRI. Simply subscribe to the mailing list below.

This list sends **one email per new technical paper**, and contains only the paper’s title, author(s), and abstract, plus a link to the paper.

#### Sign up to get updates on new MIRI technical results

*Get notified every time a new technical paper is published.*

*
*

×

The post [New mailing list for MIRI math/CS papers only](https://intelligence.org/2015/02/03/keep-date-miris-research-via-new-mailing-list/) appeared first on [Machine Intelligence Research Institute](https://intelligence.org).


February 2015 Newsletter

|  |  |
| --- | --- |
|

|  |
| --- |
|
[Machine Intelligence Research Institute](http://intelligence.org)
 |

 |
|

|  |  |
| --- | --- |
|

|  |
| --- |
|

**Research Updates**
* Four new reports in support of our [new technical agenda overview](https://intelligence.org/2014/12/23/new-technical-research-agenda-overview/), on [logical uncertainty](https://intelligence.org/2015/01/09/new-report-questions-reasoning-logical-uncertainty/), [Vingean reflection](https://intelligence.org/2015/01/15/new-report-vingean-reflection-reliable-reasoning-self-improving-agents/), [realistic world models](https://intelligence.org/2015/01/22/new-report-formalizing-two-problems-realistic-world-models/), and [value learning](https://intelligence.org/2015/01/29/new-report-value-learning-problem/).
* [AI Impacts site relaunched](https://intelligence.org/2015/01/11/improved-ai-impacts-website/) with new content.
* [Reply to Rodney Brooks and John Searle on AI volition and timelines](https://intelligence.org/2015/01/08/brooks-searle-agi-volition-timelines/).
* Interview: [Matthias Troyer on quantum computers](https://intelligence.org/2015/01/07/matthias-troyer-quantum-computers/).

**News Updates**
* Our *Superintelligence* online reading group is in its 20th week, discussing [the value loading problem](http://lesswrong.com/lw/llr/superintelligence_20_the_valueloading_problem/).

**Other Updates**
* Top AI scientists and many others have signed an [open letter](http://futureoflife.org/misc/open_letter) advocating more research into robust and beneficial AI. The letter cites several MIRI papers.
* Elon Musk has provided $10 million in funding for the types of research described in the open letter. The funding will be distributed in grants by the Future of Life Institute. Apply [here](http://futureoflife.org/grants/large/initial).

As always, please don’t hesitate to let us know if you have any questions or comments.
Best,
Luke Muehlhauser
Executive Director
 |

 |

 |
|

|  |  |  |  |  |
| --- | --- | --- | --- | --- |
|

|  |
| --- |
| [Facebook](https://www.facebook.com/MachineIntelligenceResearchInstitute)  |  [Twitter](https://twitter.com/MIRIBerkeley)  |  [Google+](https://plus.google.com/+IntelligenceOrg/)  |  [Forward to a friend](https://intelligence.org/feed/*|FORWARD|*) |
|  |  |
|
You’re receiving this because you subscribed to the [M](http://intelligence.org)[IRI](http://intelligence.org) newsletter.
[unsubscribe from this list](https://intelligence.org/feed/*|UNSUB|*) | [update subscription preferences](https://intelligence.org/feed/*|UPDATE_PROFILE|*)
 |

 |

 |

The post [February 2015 Newsletter](https://intelligence.org/2015/02/01/february-2015-newsletter/) appeared first on [Machine Intelligence Research Institute](https://intelligence.org).


[![Value learning](http://intelligence.org/wp-content/uploads/2015/01/Value-learning.png)](https://intelligence.org/files/ValueLearningProblem.pdf)Today we release a new technical report by Nate Soares, “[The value learning problem](https://intelligence.org/files/ValueLearningProblem.pdf).” If you’d like to discuss the paper, please do so [here](http://lesswrong.com/lw/lmn/the_value_learning_problem/).

Abstract:

> A superintelligent machine would not automatically act as intended: it will act as programmed, but the fit between human intentions and formal specification could be poor. We discuss methods by which a system could be constructed to learn what to value. We highlight open problems specific to inductive value learning (from labeled training data), and raise a number of questions about the construction of systems which model the preferences of their operators and act accordingly.
>
>

This is the last of six new major reports which describe and motivate [MIRI’s current research agenda](https://intelligence.org/2014/12/23/new-technical-research-agenda-overview/) at a high level.

**Update May 29, 2016**: A revised version of “The Value Learning Problem” (available at the [original link](https://intelligence.org/files/ValueLearningProblem.pdf)) has been accepted to the IJCAI-16 Ethics for Artificial Intelligence workshop. The original version of the paper can be found [here](https://intelligence.org/files/obsolete/ValueLearningProblem.pdf).

The post [New report: “The value learning problem”](https://intelligence.org/2015/01/29/new-report-value-learning-problem/) appeared first on [Machine Intelligence Research Institute](https://intelligence.org).


[![Formalizing two problems](http://intelligence.org/wp-content/uploads/2015/01/Formalizing-two-problems.png)](https://intelligence.org/files/RealisticWorldModels.pdf)Today we release a new technical report by Nate Soares, “[Formalizing two problems of realistic world models](https://intelligence.org/files/RealisticWorldModels.pdf).” If you’d like to discuss the paper, please do so [here](http://lesswrong.com/lw/lkk/formalizing_two_problems_of_realistic_world_models/).

Abstract:

> An intelligent agent embedded within the real world must reason about an environment which is larger than the agent, and learn how to achieve goals in that environment. We discuss attempts to formalize two problems: one of induction, where an agent must use sensory data to infer a universe which embeds (and computes) the agent, and one of interaction, where an agent must learn to achieve complex goals in the universe. We review related problems formalized by Solomonoff and Hutter, and explore challenges that arise when attempting to formalize analogous problems in a setting where the agent is embedded within the environment.
>
>

This is the 5th of six new major reports which describe and motivate [MIRI’s current research agenda](https://intelligence.org/2014/12/23/new-technical-research-agenda-overview/) at a high level.

The post [New report: “Formalizing Two Problems of Realistic World Models”](https://intelligence.org/2015/01/22/new-report-formalizing-two-problems-realistic-world-models/) appeared first on [Machine Intelligence Research Institute](https://intelligence.org).


[![Vingean reflection](http://intelligence.org/wp-content/uploads/2015/01/Vingean-reflection.png)](https://intelligence.org/files/VingeanReflection.pdf)Today we release a new technical report by Benja Fallenstein and Nate Soares, “[Vingean Reflection: Reliable Reasoning for Self-Improving Agents](https://intelligence.org/files/VingeanReflection.pdf).” If you’d like to discuss the paper, please do so [here](http://lesswrong.com/lw/ljp/vingean_reflection_reliable_reasoning_for/).

Abstract:

> Today, human-level machine intelligence is in the domain of futurism, but there is every reason to expect that it will be developed eventually. Once artificial agents become able to improve themselves further, they may far surpass human intelligence, making it vitally important to ensure that the result of an “intelligence explosion” is aligned with human interests. In this paper, we discuss one aspect of this challenge: ensuring that the initial agent’s reasoning about its future versions is reliable, even if these future versions are far more intelligent than the current reasoner. We refer to reasoning of this sort as Vingean reflection.
>
>
> A self-improving agent must reason about the behavior of its smarter successors in abstract terms, since if it could predict their actions in detail, it would already be as smart as them. This is called the Vingean principle, and we argue that theoretical work on Vingean reflection should focus on formal models that reflect this principle. However, the framework of expected utility maximization, commonly used to model rational agents, fails to do so. We review a body of work which instead investigates agents that use formal proofs to reason about their successors. While it is unlikely that real-world agents would base their behavior entirely on formal proofs, this appears to be the best currently available formal model of abstract reasoning, and work in this setting may lead to insights applicable to more realistic approaches to Vingean reflection.
>
>

This is the 4th of six new major reports which describe and motivate [MIRI’s current research agenda](https://intelligence.org/2014/12/23/new-technical-research-agenda-overview/) at a high level.

The post [New report: “Vingean Reflection: Reliable Reasoning for Self-Improving Agents”](https://intelligence.org/2015/01/15/new-report-vingean-reflection-reliable-reasoning-self-improving-agents/) appeared first on [Machine Intelligence Research Institute](https://intelligence.org).


An improved “AI Impacts” website

[![AI Impacts](http://intelligence.org/wp-content/uploads/2015/01/AI-Impacts.png)](http://aiimpacts.org/)Recently, MIRI received a targeted donation to improve the [AI Impacts website](http://aiimpacts.org/) initially created by frequent MIRI collaborator Paul Christiano and part-time MIRI researcher Katja Grace. Collaborating with Paul and Katja, we ported the old content to a more robust and navigable platform, and made some improvements to the content. You can see the result at [AIImpacts.org](http://aiimpacts.org/).

As explained in the site’s [introductory blog post](http://aiimpacts.wpengine.com/the-ai-impacts-blog/),

> AI Impacts is premised on two ideas (at least!):
>
>
> * **The details of the arrival of human-level artificial intelligence matter**Seven years to prepare is very different from seventy years to prepare. A weeklong transition is very different from a decade-long transition. Brain emulations require different preparations than do synthetic AI minds. Etc.
> * **Available data and reasoning can substantially educate our guesses about these details**
>
> We can track progress in AI subfields. We can estimate the hardware represented by the human brain. We can detect the effect of additional labor on software progress. Etc.
>
>
> Our goal is to assemble relevant evidence and considerations, and to synthesize reasonable views on questions such as when AI will surpass human-level capabilities, how rapid development will be at that point, what advance notice we might expect, and what kinds of AI are likely to reach human-level capabilities first.
>
>

The meat of the website is in its [articles](http://aiimpacts.wpengine.com/articles/). Here are two examples to start with:

* [A summary of AI surveys](http://aiimpacts.wpengine.com/a-summary-of-ai-surveys/)
* [Cases of discontinuous technological progress](http://aiimpacts.wpengine.com/cases-of-discontinuous-technological-progress/)

The post [An improved “AI Impacts” website](https://intelligence.org/2015/01/11/improved-ai-impacts-website/) appeared first on [Machine Intelligence Research Institute](https://intelligence.org).


[![Reasoning under LU](http://intelligence.org/wp-content/uploads/2015/01/Reasoning-under-LU.png)](https://intelligence.org/files/QuestionsLogicalUncertainty.pdf)Today we release a new technical report by Nate Soares and Benja Fallenstein, “[Questions of reasoning under logical uncertainty](https://intelligence.org/files/QuestionsLogicalUncertainty.pdf).” If you’d like to discuss the paper, please do so [here](http://lesswrong.com/r/lesswrong/lw/lgd/questions_of_reasoning_under_logical_uncertainty/).

Abstract:

> A logically uncertain reasoner would be able to reason as if they know both a programming language and a program, without knowing what the program outputs. Most practical reasoning involves some logical uncertainty, but no satisfactory theory of reasoning under logical uncertainty yet exists. A better theory of reasoning under logical uncertainty is needed in order to develop the tools necessary to construct highly reliable artificial reasoners. This paper introduces the topic, discusses a number of historical results, and describes a number of open problems.
>
>

This is the 3rd of six new major reports which describe and motivate [MIRI’s current research agenda](https://intelligence.org/2014/12/23/new-technical-research-agenda-overview/) at a high level.

The post [New report: “Questions of reasoning under logical uncertainty”](https://intelligence.org/2015/01/09/new-report-questions-reasoning-logical-uncertainty/) appeared first on [Machine Intelligence Research Institute](https://intelligence.org).