![rsz_11james-d-miller](https://intelligence.org/wp-content/uploads/2013/07/rsz_11james-d-miller.png)[James D. Miller](http://sophia.smith.edu/~jdmiller/resume.pdf) is an associate professor of economics at [Smith College](http://www.smith.edu/). He is the author of *[Singularity Rising](http://www.amazon.com/Singularity-Rising-Surviving-Thriving-Dangerous/dp/1936661659/ref=sr_1_1?ie=UTF8&qid=1373066969&sr=8-1&keywords=singularity+rising), [Game Theory at Work](http://www.amazon.com/Game-Theory-Work-Outmaneuver-Competition/dp/0071400206/ref=sr_1_1?ie=UTF8&qid=1373066976&sr=8-1&keywords=game+theory+at+work),* and a [principles of microeconomics textbook](http://www.amazon.com/Principles-Microeconomics-James-Miller/dp/0073402834/ref=sr_1_1?ie=UTF8&qid=1373067243&sr=8-1&keywords=james+miller+microeconomics) along with several academic articles.

He has a PhD in economics from the University of Chicago and a J.D. from Stanford Law School where he was on *Law Review*. He is a member of cryonics provider [Alcor](http://www.alcor.org/) and a research advisor to MIRI. He is currently co-writing a book on better decision making with the [Center for Applied Rationality](http://rationality.org/) and will be probably be an editor on the next edition of the *[Singularity Hypotheses](http://www.amazon.com/Singularity-Hypotheses-Scientific-Philosophical-Assessment/dp/3642325599/ref=sr_1_1?s=books&ie=UTF8&qid=1373129221&sr=1-1&keywords=singularity+hypotheses+a+scientific+and+philosophical+assessment)* book. He is a committed bio-hacker currently practicing or consuming a [paleo diet](http://www.amazon.com/Perfect-Health-Diet-Regain-Weight/dp/145169914X/ref=sr_1_1?ie=UTF8&qid=1373067533&sr=8-1&keywords=perfect+health+diet), [neurofeedback](http://www.nytimes.com/2010/10/05/health/05neurofeedback.html?pagewanted=all&_r=0), [cold thermogenesis](http://www.bulletproofexec.com/cold-thermogenesis-in-tibet-and-the-dangers-of-biohacking-made-real/), [intermittent fasting](http://www.bulletproofexec.com/bulletproof-fasting/), [brain fitness](http://www.lumosity.com/) [video games](http://brainworkshop.sourceforge.net/), [smart drugs](http://examine.com/supplements/Nootropic/), [bulletproof coffee](http://www.bulletproofexec.com/category/coffee-2/), and [rationality](http://rationality.org/workshops/) [training](http://lesswrong.com/).

**Luke Muehlhauser:** Your book chapter in *[Singularity Hypothesis](http://www.amazon.com/Singularity-Hypotheses-Scientific-Philosophical-Assessment/dp/3642325599/)* describes some unusual economic incentives facing a future business that is working to create [AGI](http://en.wikipedia.org/wiki/Strong_ai#Artificial_General_Intelligence_research). To explain your point, you make the simplifying assumption that “a firm’s attempt to build an AGI will result in one of three possible outcomes”:

* *Unsuccessful*: The firm fails to create AGI, losing value for its owners and investors.

* *Riches*: The firm creates AGI, bringing enormous wealth to its owners and investors.

* *Foom*: The firm creates AGI but this event quickly destroys the value of money, e.g. via an [intelligence explosion](https://intelligence.org/files/IE-EI.pdf) that eliminates scarcity, or creates a weird world without money, or exterminates humanity.

How does this setup allow us to see the unusual incentives facing a future business that is working to create AGI?

---

**James Miller:** A huge asteroid might hit the earth, and if it does it will destroy mankind. You should be willing to bet everything you have that the asteroid will miss our planet because either you win your bet or Armageddon renders the wager irrelevant. Similarly, if I’m going to start a company that will either make investors extremely rich or create a *Foom* that destroys the value of money, you should be willing to invest a lot in my company’s success because either the investment will pay off, or you would have done no better making any other kind of investment.

Pretend I want to create a controllable AGI, and if successful I will earn great *Riches* for my investors. At first I intend to follow a research and development path in which if I fail to achieve *Riches*, my company will be *Unsuccessful* and have no significant impact on the world. Unfortunately, I can’t convince potential investors that the probability of my achieving *Riches* is high enough to make my company worth investing in. The investors assign too large a likelihood that other potential investments would outperform my firm’s stock. But then I develop an evil alternative research and development plan under which I have the exact same probability of achieving *Riches* as before but now if I fail to create a controllable AGI, an unfriendly *Foom* will destroy humanity. Now I can truthfully tell potential investors that it’s highly unlikely any other company’s stock will outperform mine.

---

**Luke:** In the case of the asteroid, the investor (me) has no ability to affect whether asteroid hits us or not. In contrast, assuming investors want to avoid risking the destruction of money, there is something they can do about it: they can simply not invest in the project to create controllable AGI, and they can encourage others not to invest in it either. If the dangerous AGI project fails to get investment, then those potential investors need not worry that the value of money — or indeed their very lives — will be destroyed by that particular AGI project.

Why would investors fund something that has a decent chance of destroying everything?

---

**James:** Consider a mine that probably contains the extremely valuable metal mithril. Alas, digging in the mine might awaken a Balrog who would then destroy our civilization. I come to you asking for money to fund my mining expedition. When you raise concerns about the Balrog, I explain that even if you don’t fund me I’m probably going to get the money elsewhere, perhaps from one of the many people who don’t believe in Balrogs. Plus, since you are just one of many individuals I’m hoping to get to back me, even if I never make up for losing your investment, I will almost certainly still get enough funds to dig for mithril. And even if you manage to stop my mining operation, someone else is going to eventually work the mine. At the very least, mithril’s military value will eventually cause some nations to seek it. Therefore, if the mine does contain an easily-awakened Balrog we’re all dead regardless of what you do. Consequently, since you have no choice but to bear the downside risk of the mine, I explain that you might as well invest some of your money with me so you can also share in the upside benefit.

If you’re a self-interested investor then when deciding on whether to invest in an AGI-seeking company you don’t look at (a) the probability that the company will destroy mankind, but rather consider (b) the probability of mankind being destroyed if you do invest in the company minus the probability if you don’t invest in it. For most investors (b) will be zero or at least really, really small because if you don’t invest in the company someone else probably will take your place, the company probably doesn’t need your money to become operational, and if this one company doesn’t pursue AGI another almost certainly will.

---

**Luke:**  Thanks for that clarification. Now I’d like to give you an opportunity to reply to one of your critics. In response to your book chapter, Robin Hanson [wrote](http://hanson.gmu.edu/SingularityBook-8A.pdf): “[Miller’s analysis] is only as useful as the assumptions on which it is based. Miller’s chosen assumptions seem to me quite extreme, and quite unlikely.”

I asked Hanson if he could clarify (and be quoted), and he told me:

> In [this] context the assumptions were: “a single public ﬁrm developing the entire system in one go. . . . it succeeds so quickly that there is no chance for others to react – the world is remade overnight.” I would instead assume a whole AI industry, where each firm only makes a small part of the total product and competes with several other firms to make that small part. Each new innovation by each firm only adds a small fraction to the capability of that small part.
>
>

Hanson also wrote some comments about your book *Singularity Rising* [here](http://www.overcomingbias.com/2012/09/millers-singularity-rising.html).

How would you reply to Hanson, regarding the economic incentives facing a business working on AGI?

---

**James:** Hanson places a very low probability on the likelihood of an intelligence explosion. And my chapter’s analysis does become worthless if you sufficiently discount the possibility of such a *Foom*.

But in large part because of [Eliezer Yudkowsky](https://intelligence.org/files/IEM.pdf), I do think that there is a significant chance of their being an intelligence explosion in which an AGI within a matter of days goes from human-level intelligence to something orders of magnitude smarter. I described in *[Singularity Rising](http://www.singularityrising.com/)* how this might happen:

> Somehow, a human-level AI is created that wishes to improve its cognitive powers. The AI examines its own source code, looking for ways to augment itself. After finding a potential improvement, the original AI makes a copy of itself with the enhancement added. The original AI then tests the new AI to see if the new AI is smarter than the original one. If the new AI is indeed better, it replaces the current AI. This new AI then proceeds in turn to enhance its code. Since the new AI is smarter than the old one, it finds improvements that the old AI couldn’t. This process continues to repeat itself.
>
>

Intelligence is a reflective superpower, able to turn on itself to decipher its own workings. I think that there probably exists some intelligence threshold for an AGI, whose intelligence is based on easily alterable and testable computer code, at which it could *Foom*.

Hanson, however, is correct that an intelligence explosion would represent a sharp break with all past technological developments and so an [outside view](http://wiki.lesswrong.com/wiki/Outside_view) of technology supports his position that my chosen assumptions are “quite extreme, and quite unlikely.”

---

**Luke:** I think Yudkowsky would reply that there are *many* outside views, and some of them suggest a rapid intelligence explosion while others do not. For example in [Intelligence Explosion Microeconomics](https://intelligence.org/files/IEM.pdf) he argues that an outside view with respect to brain size and hominid evolution suggests that an AGI would lead to intelligence explosion. My own preferred solution to “[reference class tennis](http://lesswrong.com/lw/gvk/induction_or_the_rules_and_etiquette_of_reference/)” is to take each outside view as a model for how the phenomena works, acknowledge our model uncertainty, and then use something like [model combination](http://lesswrong.com/lw/hzu/model_combination_and_adjustment/) to figure out what our posterior probability distribution should be.

Okay, last question. What are you working on now?

---

**James:** I’m co-authoring a book on better decision making with the  [Center for Applied Rationality](http://rationality.org). Although the book is still in the early stages, it might be based on short, original fictional stories. I’m also probably going to be an editor on the next edition of the *[Singularity Hypotheses](http://www.amazon.com/Singularity-Hypotheses-Scientific-Philosophical-Assessment/dp/3642325599/ref=sr_1_1?s=books&ie=UTF8&qid=1373129221&sr=1-1&keywords=singularity+hypotheses+a+scientific+and+philosophical+assessment)* book.

---

**Luke:** Thanks, James!

The post [James Miller on Unusual Incentives Facing AGI Companies](https://intelligence.org/2013/07/12/james-miller-interview/) appeared first on [Machine Intelligence Research Institute](https://intelligence.org).


|  |  |
| --- | --- |
|

|  |
| --- |
|

 |

 |
|

|  |  |  |  |
| --- | --- | --- | --- |
|

|  |  |  |
| --- | --- | --- |
|

|  |  |
| --- | --- |
|

|  |
| --- |
|
Greetings from the Executive Director
Dear friends,
Another busy month! Since our last newsletter, we’ve published 3 new papers and 2 new “analysis” blog posts, we’ve significantly improved our website (especially the [Research](http://intelligence.org/research/) page), we’ve [relocated](http://intelligence.org/2013/07/08/miri-has-moved/) to downtown Berkeley, and we’ve launched [our summer 2013 matching fundraiser](http://intelligence.org/2013/07/08/2013-summer-matching-challenge/)!
MIRI also recently presented at the [Effective Altruism Summit](http://www.effectivealtruismsummit.com/), a gathering of 60+ effective altruists in Oakland, CA. As philosopher Peter Singer explained in his [TED talk](http://www.ted.com/talks/peter_singer_the_why_and_how_of_effective_altruism.html), effective altruism “combines both the heart and the head.” The heart motivates us to be empathic and altruistic toward others, while the head can “make sure that what [we] do is effective and well-directed,” so that altruists can do not just *some* good but *as much good as possible*.
As I explain in [Friendly AI Research as Effective Altruism](http://intelligence.org/2013/06/05/friendly-ai-research-as-effective-altruism/), MIRI was founded in 2000 on the premise that creating Friendly AI might be a particularly efficient way to do as much good as possible. Effective altruists focus on a variety of other causes, too, such as poverty reduction. As I say in [Four Focus Areas of Effective Altruism](http://lesswrong.com/lw/hx4/four_focus_areas_of_effective_altruism/), I think it’s important for effective altruists to cooperate and collaborate, despite their differences of opinion about which focus areas are optimal. The world needs more effective altruists, of all kinds.
MIRI engages in direct efforts — e.g. Friendly AI research — to improve the odds that machine superintelligence has a positive rather than a negative impact. But indirect efforts — such as spreading rationality and effective altruism — are also likely to play a role, for they will influence the context in which powerful AIs are built. That’s part of why we created [CFAR](http://rationality.org/).
If you think this work is important, I hope you’ll [donate now](http://intelligence.org/donate/) to support our work. MIRI is *entirely* supported by private funders like *you*. And if you donate before August 15th, your contribution will be matched by one of the generous backers of [our current fundraising drive](http://intelligence.org/2013/07/08/2013-summer-matching-challenge/).
Thank you,
Luke Muehlhauser
Executive Director

Our Summer 2013 Matching Fundraiser
Thanks to the generosity of several major donors, every donation to MIRI made from now until August 15th, 2013 will be matched dollar-for-dollar, up to a total of $200,000!
Now is your chance to double your impact while helping us raise up to $400,000 (with matching) to fund [our research program](http://intelligence.org/research/).
Early this year we made a transition from movement-building to research, and we’ve hit the ground running with six major new research papers, six new strategic analyses on our blog, and much more. Give now to support our ongoing work on [the future’s most important problem](http://intelligence.org/2013/06/05/friendly-ai-research-as-effective-altruism/).
**Accomplishments in 2013 so far**
* [Changed our name](http://intelligence.org/blog/2013/01/30/we-are-now-the-machine-intelligence-research-institute-miri/) to MIRI and launched our new website at intelligence.org.
* Released six new research papers: [Definability of Truth in Probabilistic Logic](http://lesswrong.com/lw/h1k/reflection_in_probabilistic_set_theory/), [Intelligence Explosion Microeconomics](https://intelligence.org/files/IEM.pdf), [Tiling Agents for Self-Modifying AI](https://intelligence.org/files/TilingAgents.pdf), [Robust Cooperation in the Prisoner’s Dilemma](https://intelligence.org/files/RobustCooperation.pdf), [A Comparison of Decision Algorithms on Newcomblike Problems](https://intelligence.org/files/Comparison.pdf), and [Responses to Catastrophic AGI Risk: A Survey](http://intelligence.org/2013/07/07/responses-to-c%E2%80%A6-risk-a-survey/%20%E2%80%8E).
* Held our [2nd research workshop](http://intelligence.org/2013/03/07/upcoming-miri-research-workshops/). (Our [3rd workshop](http://intelligence.org/2013/06/07/miris-july-2013-workshop/) is currently ongoing.)
* Published six new analyses to our blog: [The Lean Nonprofit](http://intelligence.org/2013/04/04/the-lean-nonprofit/), [AGI Impact Experts and Friendly AI Experts](http://intelligence.org/2013/05/01/agi-impacts-experts-and-friendly-ai-experts/), [Five Theses…](http://intelligence.org/2013/05/05/five-theses-two-lemmas-and-a-couple-of-strategic-implications/), When Will AI Be Created?, [Friendly AI Research as Effective Altruism](http://intelligence.org/2013/06/05/friendly-ai-research-as-effective-altruism/), and [What is Intelligence?](http://intelligence.org/2013/06/19/what-is-intelligence-2/)
* Published the [Facing the Intelligence Explosion](http://intelligenceexplosion.com/ebook/) ebook.
* Published several other substantial articles: [Recommended Courses for MIRI Researchers](http://intelligence.org/courses/), [Decision Theory FAQ](http://lesswrong.com/lw/gu1/decision_theory_faq/), [A brief history of ethically concerned scientists](http://lesswrong.com/lw/gln/a_brief_history_of_ethically_concerned_scientists/), [Bayesian Adjustment Does Not Defeat Existential Risk Charity](http://lesswrong.com/lw/gzq/bayesian_adjustment_does_not_defeat_existential/), and others.
* And of course much more.

**Future Plans You Can Help Support**
* We will host many more research workshops, including [one in September](http://intelligence.org/2013/07/07/miris-september-2013-workshop/%20%E2%80%8E), and one in December (with [John Baez](http://math.ucr.edu/home/baez/) attending, among others).
* Eliezer will continue to publish about open problems in Friendly AI. (Here is [#1](http://lesswrong.com/lw/hbd/new_report_intelligence_explosion_microeconomics/) and [#2](http://lesswrong.com/lw/hmt/tiling_agents_for_selfmodifying_ai_opfai_2/).)
* We will continue to publish strategic analyses, mostly via our blog.
* We will publish nicely-edited ebooks (Kindle, iBooks, and PDF) for more of our materials, to make them more accessible: [The Sequences, 2006-2009](http://wiki.lesswrong.com/wiki/Sequences) and [The Hanson-Yudkowsky AI Foom Debate](http://wiki.lesswrong.com/wiki/The_Hanson-Yudkowsky_AI-Foom_Debate).
* We will continue to set up the infrastructure (e.g. [new offices](http://intelligence.org/2013/07/06/miri-has-moved/), researcher endowments) required to host a productive Friendly AI research team, and (over several years) recruit enough top-level math talent to launch it.

(Other projects are still being surveyed for likely cost and strategic impact.)
We appreciate your support for our high-impact work! Donate now, and seize a better than usual chance to move our work forward. If you have questions about donating, please contact Louie Helm at (510) 717-1477 or louie@intelligence.org.

New Research Page, Three New Publications
 Our new [Research](http://intelligence.org/research/) page has launched!
Our previous research page was a simple list of articles, but the new page describes the purpose of our research, explains four categories of research to which we contribute, and highlights the papers we think are most important to read.
We’ve also released three new research articles.
[Tiling Agents for Self-Modifying AI, and the Löbian Obstacle](https://intelligence.org/files/TilingAgents.pdf) (discuss it [here](http://lesswrong.com/lw/hmt/tiling_agents_for_selfmodifying_ai_opfai_2/)), by Yudkowsky and Herreshoff, explains one of the key open problems in MIRI’s research agenda:
We model self-modification in AI by introducing “tiling” agents whose decision systems will approve the construction of highly similar agents, creating a repeating pattern (including similarity of the offspring’s goals). Constructing a formalism in the most straightforward way produces a Gödelian difficulty, the “Löbian obstacle.” By technical methods we demonstrates the possibility of avoiding this obstacle, but the underlying puzzles of rational coherence are thus only partially addressed. We extend the formalism to partially unknown deterministic environments, and show a very crude extension to probabilistic environments and expected utility; but the problem of finding a fundamental decision criterion for self-modifying probabilistic agents remains open.
[Robust Cooperation in the Prisoner’s Dilemma: Program Equilibrium via Provability Logic](https://intelligence.org/files/RobustCooperation.pdf) (discuss it [here](http://lesswrong.com/lw/hmw/robust_cooperation_in_the_prisoners_dilemma/)), by LaVictoire et al., explains some progress in program equilibrium made by MIRI research associate Patrick LaVictoire and several others during MIRI’s April 2013 workshop:
Rational agents defect on the one-shot prisoner’s dilemma even though mutual cooperation would yield higher utility for both agents. Moshe Tennenholtz showed that if each program is allowed to pass its playing strategy to all other players, some programs can then cooperate on the one-shot prisoner’s dilemma. Program equilibria is Tennenholtz’s term for Nash equilibria in a context where programs can pass their playing strategies to the other players. One weakness of this approach so far has been that any two programs which make different choices cannot “recognize” each other for mutual cooperation, even if they are functionally identical. In this paper, provability logic is used to enable a more flexible and secure form of mutual cooperation.
[Responses to Catastrophic AGI Risk: A Survey](https://intelligence.org/files/ResponsesAGIRisk.pdf) (discuss it [here](http://lesswrong.com/r/discussion/lw/hxi/responses_to_catastrophic_agi_risk_a_survey/)), by Sotala and Yampolskiy, is a summary of the extant literature (250+ references) on AGI risk, and can serve either as a guide for researchers or as an introduction for the uninitiated.
Many researchers have argued that humanity will create artificial general intelligence (AGI) within the next twenty to one hundred years. It has been suggested that AGI may pose a catastrophic risk to humanity. After summarizing the arguments for why AGI may pose such a risk, we survey the field’s proposed responses to AGI risk. We consider societal proposals, proposals for external constraints on AGI behaviors, and proposals for creating AGIs that are safe due to their internal design.

Two New Analyses
MIRI publishes some of its most substantive research to its blog, under the [Analysis](http://intelligence.org/category/analysis/) category. For example, [When Will AI Be Created?](http://intelligence.org/2013/05/15/when-will-ai-be-created/) is the product of 20+ hours of work, and has 14 footnotes and 40+ scholarly references (all of them linked to PDFs).
Last month, we published two new analyses.
[Friendly AI Research as Effective Altruism](http://intelligence.org/2013/06/05/friendly-ai-research-as-effective-altruism/) presents a bare-bones version of an argument that Friendly AI research is a particularly efficient way to purchase expected value, so that the argument can be elaborated and critiqued by MIRI and others.
[What is Intelligence?](http://intelligence.org/2013/06/19/what-is-intelligence-2/) argues that imprecise working definitions can be useful, an explains the particular imprecise working definition for *intelligence* that we tend to use at MIRI: efficient cross-domain optimization. A future post will discuss some potentially useful working definitions for “artificial general intelligence.”

Grant Writer Needed
MIRI would like to hire someone to write grant applications, both for our research efforts and for STEM education. If you have experience with either, please [**apply here**](https://docs.google.com/forms/d/1YEkv-CYfgIBkxALyi70mT1pG5UvuGWv_-E_dTM-oQ3c/viewform).
The pay will depend on skill and experience, and is negotiable.

Featured Volunteer
Oliver Habryka helps out by proofreading MIRI’s paper, and would be able to contribute to our research at some point, perhaps on the subject of “lessons for ethics from machine ethics.” Independent of his direct contributions to MIRI’s work, Oliver has also lectured on topics related to MIRI’s work at his high school, and has also taught a class on rationality, where he inspired participation by using a “leveling up” reward system. Oliver is currently studying the foundations of mathematics and hopes one day to direct his career goals in such a way that his contributions to our mission increase over time.
 |

 |

 |

 |

 |
|  |

The post [MIRI’s July Newsletter: Fundraiser and New Papers](https://intelligence.org/2013/07/11/july-newsletter/) appeared first on [Machine Intelligence Research Institute](https://intelligence.org).


2013 Summer Matching Challenge!

Thanks to the generosity of several major donors,† every donation to the Machine Intelligence Research Institute made from now until August 15th, 2013 will be matched dollar-for-dollar, up to a total of $200,000!

**[Donate Now!](https://intelligence.org/donate/#donation-methods)**

$0

$50,000

$100,000

$150,000

$200,000

### We have reached our goal of $200,000!

Now is your chance to **double your impact** while helping us raise up to $400,000 (with matching) to fund [our research program](https://intelligence.org/research/).

---

Early this year we made a transition from movement-building to research, and we’ve *hit the ground running* with six major new research papers, six new strategic analyses on our blog, and much more. Give now to support our ongoing work on [the future’s most important problem](http://intelligence.org/2013/06/05/friendly-ai-research-as-effective-altruism/).

### Accomplishments in 2013 so far

* [Changed our name](https://intelligence.org/feed/2013/01/30/we-are-now-the-machine-intelligence-research-institute-miri/) to MIRI and launched our new website at intelligence.org.
* Released **six new research papers**: [Definability of Truth in Probabilistic Logic](http://lesswrong.com/lw/h1k/reflection_in_probabilistic_set_theory/), [Intelligence Explosion Microeconomics](https://intelligence.org/files/IEM.pdf',100])), [Tiling Agents for Self-Modifying AI](https://intelligence.org/files/TilingAgents.pdf), [Robust Cooperation in the Prisoner’s Dilemma](https://intelligence.org/files/RobustCooperation.pdf), [A Comparison of Decision Algorithms on Newcomblike Problems](https://intelligence.org/files/Comparison.pdf), and [Responses to Catastrophic AGI Risk: A Survey](https://intelligence.org/2013/07/07/responses-to-c…-risk-a-survey/ ).
* Held our [2nd research workshop](https://intelligence.org/2013/03/07/upcoming-miri-research-workshops/). (Our [3rd workshop](https://intelligence.org/2013/06/07/miris-july-2013-workshop/) is currently ongoing.)
* Published **six new analyses** to our blog: [The Lean Nonprofit](https://intelligence.org/2013/04/04/the-lean-nonprofit/), [AGI Impact Experts and Friendly AI Experts](https://intelligence.org/2013/05/01/agi-impacts-experts-and-friendly-ai-experts/), [Five Theses…](https://intelligence.org/2013/05/05/five-theses-two-lemmas-and-a-couple-of-strategic-implications/), When Will AI Be Created?, [Friendly AI Research as Effective Altruism](https://intelligence.org/2013/06/05/friendly-ai-research-as-effective-altruism/), and [What is Intelligence?](https://intelligence.org/2013/06/19/what-is-intelligence-2/)
* Published the *[Facing the Intelligence Explosion](http://intelligenceexplosion.com/ebook/)*ebook.
* Published several other substantial articles: [Recommended Courses for MIRI Researchers](https://intelligence.org/courses/), [Decision Theory FAQ](http://lesswrong.com/lw/gu1/decision_theory_faq/), [A brief history of ethically concerned scientists](http://lesswrong.com/lw/gln/a_brief_history_of_ethically_concerned_scientists/), [Bayesian Adjustment Does Not Defeat Existential Risk Charity](http://lesswrong.com/lw/gzq/bayesian_adjustment_does_not_defeat_existential/), and others.
* And of course *much* more.

### Future Plans You Can Help Support

* We will host many more research workshops, including [one in September](https://intelligence.org/2013/07/07/miris-september-2013-workshop/ ), and one in December (with [John Baez](http://math.ucr.edu/home/baez/) attending, among others).
* Eliezer will continue to publish about open problems in Friendly AI. (Here is [#1](http://lesswrong.com/lw/hbd/new_report_intelligence_explosion_microeconomics/) and [#2](http://lesswrong.com/lw/hmt/tiling_agents_for_selfmodifying_ai_opfai_2/).)
* We will continue to publish strategic analyses, mostly via our blog.
* We will publish nicely-edited ebooks (Kindle, iBooks, and PDF) for more of our materials, to make them more accessible: *[The Sequences, 2006-2009](http://wiki.lesswrong.com/wiki/Sequences)*and *[The Hanson-Yudkowsky AI Foom Debate](http://wiki.lesswrong.com/wiki/The_Hanson-Yudkowsky_AI-Foom_Debate)*.
* We will continue to set up the infrastructure (e.g. [new offices](https://intelligence.org/2013/07/06/miri-has-moved/), researcher endowments) required to host a productive Friendly AI research team, and (over several years) recruit enough top-level math talent to launch it.

(Other projects are still being surveyed for likely cost and strategic impact.)

We appreciate your support for our high-impact work! Donate now, and seize a better than usual chance to move our work forward. If you have questions about donating, please contact Louie Helm at (510) 717-1477 or louie@intelligence.org.

† $200,000 of total matching funds has been provided by Jaan Tallinn, Loren Merritt, Rick Schwall, and Alexei Andreev.

The post [2013 Summer Matching Challenge!](https://intelligence.org/2013/07/08/2013-summer-matching-challenge/) appeared first on [Machine Intelligence Research Institute](https://intelligence.org).


MIRI Has Moved!

For the past several months, MIRI and its child organization [CFAR](http://rationality.org/) have been working from a much-too-small office on the outskirts of Berkeley. At the end of June, MIRI and CFAR took over the 3rd floor of [2030 Addison St.](https://www.google.com/maps/preview#!q=2030+Addison+St%2C+Berkeley) in downtown Berkeley, which has sufficient space for both organizations.

Our new office is 0.5 blocks from the Downtown Berkeley BART exit at Shattuck & Addison, and 2 blocks from the UC Berkeley campus. Here’s a photo of the campus from our roof:

![view of campus from roof (500px)](http://intelligence.org/wp-content/uploads/2013/07/view-of-campus-from-roof-500px.jpg)
The proximity to UC Berkeley will make it easier for MIRI to network with Berkeley’s professors and students. Conveniently, UC Berkeley is ranked [5th in the world](http://www.usnews.com/education/worlds-best-universities-rankings/best-universities-mathematics) in mathematics, and [1st in the world](http://grad-schools.usnews.rankingsandreviews.com/best-graduate-schools/top-science-schools/logic-rankings) in mathematical logic.

Sharing an office with CFAR carries many benefits for both organizations:

1. CFAR and MIRI can “flex” into each other’s space for short periods as needed, for example when MIRI is holding a week-long [research workshop](https://intelligence.org/get-involved/#workshop).
2. We can share resources (printers, etc.).
3. Both organizations can benefit from interaction between our two communities.

Getting the new office was a team effort, but the person *most* responsible for this success was MIRI Deputy Director Louie Helm.

Note that MIRI isn’t yet able to accommodate “drop in” visitors, as we keep irregular hours throughout the week. So if you’d like to visit, please [contact us](https://intelligence.org/team/) first.

We retain 2721 Shattuck Ave. #1023 as an alternate mailing address.

The post [MIRI Has Moved!](https://intelligence.org/2013/07/08/miri-has-moved/) appeared first on [Machine Intelligence Research Institute](https://intelligence.org).


MIRI’s September 2013 Workshop

![Paul at April workshop](http://intelligence.org/wp-content/uploads/2013/07/Paul-at-April-workshop.jpg)
From September 7-13, MIRI will host its **4th Workshop on Logic, Probability, and Reflection**. The focus of this workshop will be the foundations of [decision theory](http://lesswrong.com/lw/gu1/decision_theory_faq/).

Participants confirmed so far include:

* [Paul Christiano](http://rationalaltruist.com/) (UC Berkeley)
* [Wei Dai](http://www.weidai.com/) (independent)
* [Gary Drescher](http://en.wikipedia.org/wiki/Gary_Drescher) (independent)
* [Kenny Easwaran](http://www.kennyeaswaran.org/) (USC)
* [Cameron Freer](http://math.mit.edu/~freer/) (MIT)
* [Patrick LaVictoire](http://www.math.wisc.edu/~patlavic/) (Quixey)
* [Ilya Shpitser](http://www.southampton.ac.uk/maths/about/staff/is1d12.page) (U Southampton)
* [Vladimir Slepnev](http://lesswrong.com/user/cousin_it/submitted/) (Google)
* [Nisan Stiennon](http://lesswrong.com/user/Nisan/submitted/) (Stanford)
* [Andreas Stuhlmüller](http://stuhlmueller.org/) (MIT & Stanford)
* [Eliezer Yudkowsky](http://yudkowsky.net/) (MIRI)

If you have a strong mathematics background and might like to attend this workshop, it’s not too late to [apply](http://intelligence.org/get-involved/#workshop)! And even if *this* workshop doesn’t fit your schedule, please **do apply**, so that we can notify you of other workshops (long before they are announced publicly).

The post [MIRI’s September 2013 Workshop](https://intelligence.org/2013/07/08/miris-september-2013-workshop/) appeared first on [Machine Intelligence Research Institute](https://intelligence.org).


MIRI is self-publishing another technical report that was too lengthy (60 pages) for publication in a journal: **[Responses to Catastrophic AGI Risk: A Survey](https://intelligence.org/files/ResponsesAGIRisk.pdf)**.

The report, co-authored by past MIRI researcher [Kaj Sotala](http://kajsotala.fi/) and University of Louisville’s [Roman Yampolskiy](http://louisville.edu/speed/computer/people/faculty/yampolskiy), is a summary of the extant literature (250+ references) on AGI risk, and can serve either as a guide for researchers or as an introduction for the uninitiated.

Here is the abstract:

> Many researchers have argued that humanity will create artificial general intelligence (AGI) within the next twenty to one hundred years. It has been suggested that AGI may pose a catastrophic risk to humanity. After summarizing the arguments for why AGI may pose such a risk, we survey the field’s proposed responses to AGI risk. We consider societal proposals, proposals for external constraints on AGI behaviors, and proposals for creating AGIs that are safe due to their internal design.
>
>

The preferred discussion page for the paper is [here](http://lesswrong.com/r/discussion/lw/hxi/responses_to_catastrophic_agi_risk_a_survey/).

**Update:** This report has now been published in *Physica Scripta*, available [here](http://iopscience.iop.org/1402-4896/90/1/018001/article).

The post [Responses to Catastrophic AGI Risk: A Survey](https://intelligence.org/2013/07/08/responses-to-catastrophic-agi-risk-a-survey/) appeared first on [Machine Intelligence Research Institute](https://intelligence.org).


What is Intelligence?

When asked their opinions about “human-level artificial intelligence” — *aka* “artificial general intelligence” (AGI)[1](https://intelligence.org/2013/06/19/what-is-intelligence-2/#footnote_0_10275 "I use the HLAI and AGI interchangeably, but lately I’ve been using AGI almost exclusively, because I’ve learned that many people in the AI community react negatively to any mention of “human-level” AI but have no objection to the concept of narrow vs. general intelligence. See also Ben Goertzel’s comments here.") — many experts understandably reply that these terms haven’t yet been precisely defined, and it’s hard to talk about something that hasn’t been defined.[2](https://intelligence.org/2013/06/19/what-is-intelligence-2/#footnote_1_10275 "Asked when he thought HLAI would be created, Pat Hayes (a past president of AAAI) replied: “I do not consider this question to be answerable, as I do not accept this (common) notion of ‘human-level intelligence’ as meaningful.” Asked the same question, AI scientist William Uther replied: “You ask a lot about ‘human level AGI’. I do not think this term is well defined,” while AI scientist Alan Bundy replied: “I don’t think the concept of ‘human-level machine intelligence’ is well formed.”") In this post, I want to briefly outline an imprecise but useful “working definition” for *intelligence* we tend to use at MIRI. In a future post I will write about some useful working definitions for *artificial general intelligence*.

### Imprecise definitions can be useful

Precise definitions are important, but I concur with Bertrand Russell that

> [You cannot] start with anything precise. You have to achieve such precision… as you go along.
>
>

Physicist [Milan Ćirković](http://ieet.org/index.php/IEET/bio/cirkovic/) agrees, and [gives](http://www.amazon.com/Astrobiological-Landscape-Start-Publishing-ebook/dp/B008CDSB30/) an example:

> The formalization of knowledge — which includes giving precise definitions — usually comes at the end of the original research in a given field, not at the very beginning. A particularly illuminating example is the concept of *number*, which was properly defined in the modern sense only after the development of axiomatic set theory in the… twentieth century.[3](https://intelligence.org/2013/06/19/what-is-intelligence-2/#footnote_2_10275 "Sawyer (1943) gives another example: “Mathematicians first used the sign √-1, without in the least knowing what it could mean, because it shortened work and led to correct results. People naturally tried to find out why this happened and what √-1 really meant. After two hundreds years they succeeded.” Dennett (2013) makes a related comment: “Define your terms, sir! No, I won’t. That would be premature… My [approach] is an instance of nibbling on a tough problem instead of trying to eat (and digest) the whole thing from the outset… In Elbow Room, I compared my method to the sculptor’s method of roughing out the form in a block of marble, approaching the final surfaces cautiously, modestly, working by successive approximation.”")
>
>

For a more AI-relevant example, consider the concept of a “self-driving car,” which has been given a variety of vague definitions [since the 1930s](http://books.google.com/books?id=7OEDAAAAMBAJ&lpg=PA210&dq=automatic%20car&pg=PA210#v=onepage&q&f=false). Would a car [guided by a buried cable](http://books.google.com/books?id=xiUDAAAAMBAJ&lpg=PA75&dq=car%20drives%20itself&pg=PA75#v=onepage&q&f=false) qualify? What about a [modified 1955 Studebaker](http://books.google.com/books?id=jd8DAAAAMBAJ&lpg=PA128&dq=automatic%20car&pg=PA128#v=onepage&q&f=false) that could use sound waves to detect obstacles and automatically engage the brakes if necessary, but could only steer “on its own” if each turn was preprogrammed? Does that count as a “self-driving car”?

What about the “[VaMoRs](http://commonsenseatheism.com/wp-content/uploads/2013/05/Dickmanns-et-al-The-seeing-passenger-car-VaMoRs-P.pdf)” of the 1980s that could avoid obstacles and steer around turns using computer vision, but weren’t advanced enough to be ready for public roads? How about the 1995 [Navlab](http://en.wikipedia.org/wiki/Navlab) car that drove across the USA and was fully autonomous for 98.2% of the trip, or the robotic cars which finished the 132-mile off-road course of the [2005 DARPA Grand Challenge](http://is.gd/CDvT8V), supplied only with the GPS coordinates of the route? What about the winning cars of the [2007 DARPA Grand Challenge](http://is.gd/kOjor6), which finished an urban race while obeying all traffic laws and avoiding collisions with other cars? Does [Google’s driverless car](http://www.forbes.com/sites/joannmuller/2013/03/21/no-hands-no-feet-my-unnerving-ride-in-googles-driverless-car/) qualify, given that it has logged more than 500,000 autonomous miles without a single accident under computer control, but still struggles with difficult merges and snow-covered roads?[4](https://intelligence.org/2013/06/19/what-is-intelligence-2/#footnote_3_10275 "With self-driving cars, researchers did use many precise external performance measures (e.g. accident rates, speed, portion of the time they could run unassisted, frequency of getting stuck) to evaluate progress, as well as internal performance metrics (speed of search, bounded loss guarantees, etc.). Researchers could see that these bits of progress were in the right direction, even if their relative contribution long-term was unclear. And so it is with AI in general. AI researchers use many precise external and internal performance measures to evaluate progress, but it is difficult to know the relative contribution of these bits of progress toward the final goal of AGI.")

Our lack of a precise definition for “self-driving car” doesn’t seem to have hindered progress on self-driving cars very much.[5](https://intelligence.org/2013/06/19/what-is-intelligence-2/#footnote_4_10275 "Heck, we’ve had pornography for millennia and still haven’t been able to define it precisely. Encyclopedia entries for “pornography” often simply quote Justice Potter Stewart: “I shall not today attempt further to define the kinds of material I understand to be [pornography]… but I know it when I see it.”") And I’m glad we didn’t wait to seriously discuss self-driving cars until we had a precise definition for the term.

Similarly, I don’t think we should wait for a precise definition of AGI before discussing the topic seriously. On the other hand, the term is useless if it carries *no* information. So let’s work our way toward a stipulative, operational definition for AGI. We’ll start by developing an operational definition for *intelligence*.

### A definition for “intelligence”

[Legg and Hutter (2007)](http://arxiv.org/pdf/0706.3639.pdf) found that definitions of intelligence converge toward the idea that “Intelligence measures an agent’s ability to achieve goals in a wide range of environments.” Let’s call this the “optimization power” concept of intelligence, because it measures an agent’s power to optimize the world according to its preferences.

I think this is a productive approach to the issue, since it identifies intelligence with externally measurable *performance* rather than with the details of *how* that performance might be achieved (e.g. via consciousness, [brute force calculation](http://www.hutter1.net/publ/uaigentle.pdf), “complexity,” or something else). Moreover, it’s usually *performance* we care about: we tend to care most about whether an AI will perform well enough to replace human workers, or whether it will perform well enough improve its own abilities without human assistance, not whether it has some particular internal feature.[6](https://intelligence.org/2013/06/19/what-is-intelligence-2/#footnote_5_10275 "We might care about whether machines are conscious in addition to being intelligent, but we already have a convenient term for that: consciousness. In particular, we might care about machine consciousness because the slow, plodding invention of AI may involve the creation and destruction of millions of partially-conscious near-AIs that are switched on, suffer for a while, and are then switched off — all while being unable to signal to us that they are suffering. This is especially likely if we remain unclear about the nature of consciousness for several more decades, and thus have no principled way (e.g. via nonperson predicates) to create intelligent machines that we know are not conscious (and are thus incapable of suffering). One of the first people to make this point clearly was Metzinger (2003), p. 621: “What would you say if someone came along and said, ‘Hey, we want to genetically engineer mentally retarded human infants! For reasons of scientific progress we need infants with certain cognitive and emotional deficits in order to study their postnatal psychological development—we urgently need some funding for this important and innovative kind of research!’ You would certainly think this was not only an absurd and appalling but also a dangerous idea. It would hopefully not pass any ethics committee in the democratic world. However, what today’s ethics committees don’t see is how the first machines satisfying a minimally sufficient set of constraints for conscious experience could be just like such mentally retarded infants. They would suffer from all kinds of functional and representational deficits too. But they would now also subjectively experience those deficits. In addition, they would have no political lobby—no representatives in any ethics committee.” Metzinger repeats the point in Metzinger (2010), starting on page 194.")

Furthermore, the concept of optimization power allows us to compare the intelligence of different kinds of agents. As Albus ([1991](ftp://calhau.dca.fee.unicamp.br/pub/docs/ia005/Albus-outline.pdf)) said, “A useful definition of intelligence… should include both biological and machine embodiments, and these should span an intellectual range from that of an insect to that of an Einstein, from that of a thermostat to that of the most sophisticated computer system that could ever be built.”

I’d like to add one more consideration, though. What if two agents have roughly equal ability to optimize the world according to their preferences, but the second agent requires far more resources to do so? These agents have the same optimization power, but the first one seems to be optimizing more intelligently. So perhaps we could use “intelligence” to mean “optimization power divided by resources used” — what Yudkowsky called [efficient cross-domain optimization](http://lesswrong.com/lw/vb/efficient_crossdomain_optimization/).[7](https://intelligence.org/2013/06/19/what-is-intelligence-2/#footnote_6_10275 "Admittedly, this is still pretty vague. One step toward precision would be to propose a definition of intelligence as optimization power for some canonical distribution of possible preferences, over some canonical distribution of environments, with a penalty for resource use. The canonical preferences and canonical environments could be weighted toward preferences and environments relevant to our concerns: we care more about whether AIs can do science than whether they can paint abstract art, and we care more about whether they can achieve their goals in our solar system than whether they can achieve their goals inside a black hole. Also see Goertzel (2010)‘s “efficient pragmatic general intelligence.”")

Other definitions[8](https://intelligence.org/2013/06/19/what-is-intelligence-2/#footnote_7_10275 "Hibbard (2011); Legg & Veness (2011); Wang (2008); Schaul et al. (2011); Dowe & Hernandez-Orallo (2012); Goertzel (2010); Adams et al. (2011).") have their merits, too. But at MIRI we find the concept of “efficient cross-domain optimization” sufficiently useful that it serves as our (still imprecise!) working definition for intelligence.

In a future post (**Edit**: [here](https://intelligence.org/2013/08/11/what-is-agi/)), I’ll discuss some useful working definitions for *artificial general intelligence*.

---

1. I use the HLAI and AGI interchangeably, but lately I’ve been using AGI almost exclusively, because I’ve learned that many people in the AI community react negatively to any mention of “human-level” AI but have no objection to the concept of narrow vs. general intelligence. See also Ben Goertzel’s comments [here](http://wp.goertzel.org/?p=173).
2. Asked when he thought HLAI would be created, Pat Hayes (a past president of [AAAI](http://en.wikipedia.org/wiki/Association_for_the_Advancement_of_Artificial_Intelligence)) [replied](http://lesswrong.com/r/discussion/lw/999/qa_with_experts_on_risks_from_ai_1/): “I do not consider this question to be answerable, as I do not accept this (common) notion of ‘human-level intelligence’ as meaningful.” Asked the same question, AI scientist [William Uther](http://www.cse.unsw.edu.au/~willu/) [replied](http://lesswrong.com/r/discussion/lw/9cm/qa_with_experts_on_risks_from_ai_3/): “You ask a lot about ‘human level AGI’. I do not think this term is well defined,” while AI scientist [Alan Bundy](http://homepages.inf.ed.ac.uk/bundy/) [replied](http://lesswrong.com/r/discussion/lw/9cm/qa_with_experts_on_risks_from_ai_3/): “I don’t think the concept of ‘human-level machine intelligence’ is well formed.”
3. [Sawyer (1943)](http://www.amazon.com/Mathematicians-Delight-Dover-Science-Books/dp/0486462404/) gives another example: “Mathematicians first used the sign √-1, without in the least knowing what it could mean, because it shortened work and led to correct results. People naturally tried to find out why this happened and what √-1 really meant. After two hundreds years they succeeded.” [Dennett (2013)](http://www.amazon.com/Intuition-Pumps-Other-Tools-Thinking/dp/0393082067/) makes a related comment: “*Define your terms, sir!* No, I won’t. That would be premature… My [approach] is an instance of *nibbling* on a tough problem instead of trying to eat (and digest) the whole thing from the outset… In *Elbow Room*, I compared my method to the sculptor’s method of roughing out the form in a block of marble, approaching the final surfaces cautiously, modestly, working by successive approximation.”
4. With self-driving cars, researchers did use many precise external performance measures (e.g. accident rates, speed, portion of the time they could run unassisted, frequency of getting stuck) to evaluate progress, as well as internal performance metrics (speed of search, bounded loss guarantees, etc.). Researchers could see that these bits of progress were in the right direction, even if their relative contribution long-term was unclear. And so it is with AI in general. AI researchers use many precise external and internal performance measures to evaluate progress, but it is difficult to know the relative contribution of these bits of progress toward the final goal of AGI.
5. Heck, we’ve had pornography for millennia and *still* haven’t been able to define it precisely. Encyclopedia entries for “pornography” [often](http://books.google.com/books?id=xOJvVJv2YlAC&pg=PA636&lpg=PA636&source=bl&ots=N3PI1PbmK1&sig=SIhfXbej_Z_HRDAxxnGZWftTZhg&hl=en&sa=X&ei=9o-NUfTEAqafiALa34CoDQ&ved=0CC8Q6AEwADgK#v=onepage&q&f=false) [simply](http://books.google.com/books?id=TN-qpt7kAK4C&pg=PA336&lpg=PA336&source=bl&ots=CspWxfT67z&sig=_ymQ5x3lvNnMF0DsaSZkJzsSaqM&hl=en&sa=X&ei=9o-NUfTEAqafiALa34CoDQ&ved=0CDMQ6AEwAjgK#v=onepage&q&f=false) quote Justice Potter Stewart: “I shall not today attempt further to define the kinds of material I understand to be [pornography]… but I know it when I see it.”
6. We might care about whether machines are conscious in addition to being intelligent, but we already have a convenient term for that: *consciousness*. In particular, we might care about machine consciousness because the slow, plodding invention of AI may involve the creation and destruction of millions of partially-conscious near-AIs that are switched on, suffer for a while, and are then switched off — all while being unable to signal to us that they are suffering. This is especially likely if we remain unclear about the nature of consciousness for several more decades, and thus have no principled way (e.g. via [nonperson predicates](http://lesswrong.com/lw/x4/nonperson_predicates/)) to create intelligent machines that we *know* are not conscious (and are thus incapable of suffering). One of the first people to make this point clearly was [Metzinger (2003)](http://www.amazon.com/Being-No-One-Self-Model-Subjectivity/dp/0262633086/), p. 621: “What would you say if someone came along and said, ‘Hey, we want to genetically engineer mentally retarded human infants! For reasons of scientific progress we need infants with certain cognitive and emotional deficits in order to study their postnatal psychological development—we urgently need some funding for this important and innovative kind of research!’ You would certainly think this was not only an absurd and appalling but also a dangerous idea. It would hopefully not pass any ethics committee in the democratic world. However, what today’s ethics committees don’t see is how the first machines satisfying a minimally sufficient set of constraints for conscious experience could be just like such mentally retarded infants. They would suffer from all kinds of functional and representational deficits too. But they would now also subjectively experience those deficits. In addition, they would have no political lobby—no representatives in any ethics committee.” Metzinger repeats the point in [Metzinger (2010)](http://www.amazon.com/Ego-Tunnel-Science-Mind-Myth/dp/0465020690/), starting on page 194.
7. Admittedly, this is still pretty vague. One step toward precision would be to propose a definition of intelligence as optimization power for some canonical distribution of possible preferences, over some canonical distribution of environments, with a penalty for resource use. The canonical preferences and canonical environments could be weighted toward preferences and environments relevant to our concerns: we care more about whether AIs can do science than whether they can paint abstract art, and we care more about whether they can achieve their goals in our solar system than whether they can achieve their goals inside a black hole. Also see [Goertzel (2010)](http://agi-conf.org/2010/wp-content/uploads/2009/06/paper_14.pdf)‘s “efficient pragmatic general intelligence.”
8. [Hibbard (2011)](http://www.ssec.wisc.edu/~billh/g/hibbard_agi11a.pdf); [Legg & Veness (2011)](http://arxiv.org/pdf/1109.5951.pdf); [Wang (2008)](http://www.cis.temple.edu/~wangp/Publication/AI_Definitions.pdf); [Schaul et al. (2011)](http://arxiv.org/pdf/1109.1314.pdf); [Dowe & Hernandez-Orallo (2012)](http://www.csse.monash.edu.au/~dld/Publications/2012/Dowe%2BHernandez-Orallo_2012_IQ_tests_are_not_for_machines_comma_yet_IN_PRESS.pdf); [Goertzel (2010)](http://agi-conf.org/2010/wp-content/uploads/2009/06/paper_14.pdf); [Adams et al. (2011)](http://www.cse.buffalo.edu/faculty/shapiro/Papers/hlai.pdf).

The post [What is Intelligence?](https://intelligence.org/2013/06/19/what-is-intelligence-2/) appeared first on [Machine Intelligence Research Institute](https://intelligence.org).


MIRI’s July 2013 Workshop

[![Mihaly at April workshop](http://intelligence.org/wp-content/uploads/2013/06/Mihaly-at-April-workshop.jpg)](http://intelligence.org/get-involved/#workshop)
From July 8-14, MIRI will host its **3rd Workshop on Logic, Probability, and Reflection**. The focus of this workshop will be the [Löbian obstacle to self-modifying systems](http://lesswrong.com/lw/hmt/tiling_agents_for_selfmodifying_ai_opfai_2/).

Participants confirmed so far include:

* [Andrew Critch](http://acritch.com/) (just finished his math PhD at UC Berkeley, now working at [CFAR](http://rationality.org/))
* [Abram Demski](https://plus.google.com/111568410659864255951) (USC)
* [Benja Fallenstein](http://lesswrong.com/user/Benja/submitted/) (Bristol U)
* [Marcello Herreshoff](http://www.linkedin.com/pub/marcello-herreshoff/0/8b4/51a) (Google)
* Jonathan Lee (Cambridge)
* [Will Sawin](http://www.ctpost.com/news/article/Wisdom-beyond-his-years-1390299.php) (Princeton)
* [Qioachu Yuan](http://math.berkeley.edu/~qchu/) (UC Berkeley)
* [Eliezer Yudkowsky](http://yudkowsky.net/) (MIRI)

If you have a strong mathematics background and might like to attend this workshop, it’s not too late to [apply](http://intelligence.org/get-involved/#workshop)! And even if *this* workshop doesn’t fit your schedule, please **do apply**, so that we can notify you of other workshops (long before they are announced publicly).

Information on past workshops:

* Our **1st Workshop** (Nov. 11-18, 2012; 4 participants) resulted in Christiano’s [probabilistic logic](http://lesswrong.com/lw/h1k/reflection_in_probabilistic_logic/), an attack on the [Löbian obstacle for self-modifying systems](http://lesswrong.com/lw/hmt/tiling_agents_for_selfmodifying_ai_opfai_2/).
* Our **2nd Workshop** (Apr. 3-24, 2013; 12 participants coming in and out) resulted in (1) some as-yet unpublished progress on Christiano’s probabilistic logic, (2) some progress on program equilibrium recorded in [LaVictoire et al. (2013)](https://intelligence.org/files/RobustCooperation.pdf), and some progress on the Löbian obstacle resulting in [Yudkowsky & Herreschoff (2013)](https://intelligence.org/files/TilingAgents.pdf).

The post [MIRI’s July 2013 Workshop](https://intelligence.org/2013/06/07/miris-july-2013-workshop/) appeared first on [Machine Intelligence Research Institute](https://intelligence.org).


New Research Page and Two New Articles

[![research page](http://intelligence.org/wp-content/uploads/2013/06/research-page.png)](https://intelligence.org/research/)
Our new [Research](http://intelligence.org/research/) page has launched!

Our previous research page was a simple list of articles, but the new page describes the purpose of our research, explains four categories of research to which we contribute, and highlights the papers we think are most important to read.

We’ve also released drafts of two new research articles.

[Tiling Agents for Self-Modifying AI, and the Löbian Obstacle](https://intelligence.org/files/TilingAgents.pdf) (discuss it [here](http://lesswrong.com/lw/hmt/tiling_agents_for_selfmodifying_ai_opfai_2/)), by Yudkowsky and Herreshoff, explains one of the key open problems in MIRI’s research agenda:

> We model self-modification in AI by introducing “tiling” agents whose decision systems will approve the construction of highly similar agents, creating a repeating pattern (including similarity of the offspring’s goals). Constructing a formalism in the most straightforward way produces a Gödelian difficulty, the “Löbian obstacle.” By technical methods we demonstrates the possibility of avoiding this obstacle, but the underlying puzzles of rational coherence are thus only partially addressed. We extend the formalism to partially unknown deterministic environments, and show a very crude extension to probabilistic environments and expected utility; but the problem of finding a fundamental decision criterion for self-modifying probabilistic agents remains open.
>
>

[Robust Cooperation in the Prisoner’s Dilemma: Program Equilibrium via Provability Logic](https://intelligence.org/files/RobustCooperation.pdf) (discuss it [here](http://lesswrong.com/lw/hmw/robust_cooperation_in_the_prisoners_dilemma/)), by LaVictoire et al., explains some progress in program equilibrium made by MIRI research associate Patrick LaVictoire and several others during MIRI’s April 2013 workshop:

> Rational agents defect on the one-shot prisoner’s dilemma even though mutual cooperation would yield higher utility for both agents. Moshe Tennenholtz showed that if each program is allowed to pass its playing strategy to all other players, some programs can then cooperate on the one-shot prisoner’s dilemma. Program equilibria is Tennenholtz’s term for Nash equilibria in a context where programs can pass their playing strategies to the other players.
>
>
> One weakness of this approach so far has been that any two programs which make different choices cannot “recognize” each other for mutual cooperation, even if they are functionally identical. In this paper, provability logic is used to enable a more flexible and secure form of mutual cooperation.
>
>

Participants of MIRI’s April workshop also made progress on [Christiano’s probabilistic logic](http://lesswrong.com/lw/h1k/reflection_in_probabilistic_logic/) (an attack on the Löbian obstacle), but that work is not yet ready to be released.

We’ve also revamped the [Get Involved](http://intelligence.org/get-involved/) page, which now includes an [application form](https://intelligence.org/get-involved/#workshop) for forthcoming workshops. If you *might* like to work with MIRI on some of its open research problems sometime in the next 18 months, [please apply](https://intelligence.org/get-involved/#workshop)! Likewise, if you know someone who might enjoy attending such a workshop, please encourage *them* to apply.

The post [New Research Page and Two New Articles](https://intelligence.org/2013/06/06/new-research-page-and-two-new-articles/) appeared first on [Machine Intelligence Research Institute](https://intelligence.org).


MIRI was founded in 2000 on the premise that creating[1](https://intelligence.org/2013/06/05/friendly-ai-research-as-effective-altruism/#footnote_0_10240 "In this post, I talk about the value of humanity in general creating Friendly AI, though MIRI co-founder Eliezer Yudkowsky usually talks about MIRI in particular — or at least, a functional equivalent — creating Friendly AI. This is because I am not as confident as Yudkowsky that it is best for MIRI to attempt to build Friendly AI. When updating MIRI’s bylaws in early 2013, Yudkowsky and I came to a compromise on the language of MIRI’s mission statement, which now reads: “[MIRI] exists to ensure that the creation of smarter-than-human intelligence has a positive impact. Thus, the charitable purpose of [MIRI] is to: (a) perform research relevant to ensuring that smarter-than-human intelligence has a positive impact; (b) raise awareness of this important issue; (c) advise researchers, leaders and laypeople around the world; and (d) as necessary, implement a smarter-than-human intelligence with humane, stable goals” (emphasis added). My own hope is that it will not be necessary for MIRI (or a functional equivalent) to attempt to build Friendly AI itself. But of course I must remain open to the possibility that this will be the wisest course of action as the first creation of AI draws nearer. There is also the question of capability: few people think that a non-profit research organization has much chance of being the first to build AI. I worry, however, that the world’s elites will not find it fashionable to take this problem seriously until the creation of AI is only a few decades away, at which time it will be especially difficult to develop the mathematics of Friendly AI in time, and humanity will be forced to take a gamble on its very survival with powerful AIs we have little reason to trust.") Friendly AI might be a particularly efficient way to do as much good as possible.

Some developments since then include:

* The field of “[effective altruism](http://www.ted.com/talks/peter_singer_the_why_and_how_of_effective_altruism.html)” — trying not just to do good but to do *as much good as possible*[2](https://intelligence.org/2013/06/05/friendly-ai-research-as-effective-altruism/#footnote_1_10240 "One might think of effective altruism as a straightforward application of decision theory to the subject of philanthropy. Philanthropic agents of all kinds (individuals, groups, foundations, etc.) ask themselves: “How can we choose philanthropic acts (e.g. donations) which (in expectation) will do as much good as possible, given what we care about?” The consensus recommendation for all kinds of choices under uncertainty, including philanthropic choices, is to maximize expected utility (Chater & Oaksford 2012; Peterson 2004; Stein 1996; Schmidt 1998:19). Different philanthropic agents value different things, but decision theory suggests that each of them can get the most of what they want if they each maximize their expected utility. Choices which maximize expected utility are in this sense “optimal,” and thus another term for effective altruism is “optimal philanthropy.” Note that effective altruism in this sense is not too dissimilar from earlier approaches to philanthropy, including high-impact philanthropy (making “the biggest difference possible, given the amount of capital invested“), strategic philanthropy, effective philanthropy, and wise philanthropy. Note also that effective altruism does not say that a philanthropic agent should specify complete utility and probability functions over outcomes and then compute the philanthropic act with the highest expected utility — that is impractical for bounded agents. We must keep in mind the distinction between normative, descriptive, and prescriptive models of decision-making (Baron 2007): “normative models tell us how to evaluate… decisions in terms of their departure from an ideal standard. Descriptive models specify what people in a particular culture actually do and how they deviate from the normative models. Prescriptive models are designs or inventions, whose purpose is to bring the results of actual thinking into closer conformity to the normative model.” The prescriptive question — about what bounded philanthropic agents should do to maximize expected utility with their philanthropic choices — tends to be extremely complicated, and is the subject of most of the research performed by the effective altruism community.") — has seen more publicity and better research than ever before, in particular through the work of [GiveWell](http://www.givewell.org/), the [Center for Effective Altruism](http://centreforeffectivealtruism.org/), the philosopher [Peter Singer](http://en.wikipedia.org/wiki/Peter_Singer), and the community at [Less Wrong](http://lesswrong.com/).[3](https://intelligence.org/2013/06/05/friendly-ai-research-as-effective-altruism/#footnote_2_10240 "See, for example: Efficient Charity, Efficient Charity: Do Unto Others, Politics as Charity, Heuristics and Biases in Charity, Public Choice and the Altruist’s Burden, On Charities and Linear Utility, Optimal Philanthropy for Human Beings, Purchase Fuzzies and Utilons Separately, Money: The Unit of Caring, Optimizing Fuzzies and Utilons: The Altruism Chip Jar, Efficient Philanthropy: Local vs. Global Approaches, The Effectiveness of Developing World Aid, Against Cryonics & For Cost-Effective Charity, Bayesian Adjustment Does Not Defeat Existential Risk Charity, How to Save the World, and What is Optimal Philanthropy?")
* In his recent [PhD dissertation](https://sites.google.com/site/nbeckstead/research/Beckstead%2C%20Nick--On%20the%20Overwhelming%20Importance%20of%20Shaping%20the%20Far%20Future.pdf?attredirects=0&d=1), [Nick Beckstead](https://sites.google.com/site/nbeckstead/) has clarified the assumptions behind the claim that shaping the far future (e.g. via Friendly AI) is overwhelmingly important.
* Due to research performed by MIRI, the [Future of Humanity Institute](http://www.fhi.ox.ac.uk/) (FHI), and others, our strategic situation with regard to machine superintelligence is more clearly understood, and FHI’s [Nick Bostrom](http://nickbostrom.com/) has organized much of this work in a [forthcoming book](http://lesswrong.com/lw/bd6/ai_risk_opportunity_a_timeline_of_early_ideas_and/91zl).[4](https://intelligence.org/2013/06/05/friendly-ai-research-as-effective-altruism/#footnote_3_10240 "I believe Beckstead and Bostrom have done the research community an enormous service in creating a framework, a shared language, for discussing trajectory changes, existential risks, and machine superintelligence. When discussing these topics with my colleagues, it has often been the case that the first hour of conversation is spent merely trying to understand what the other person is saying — how they are using the terms and concepts they employ. Beckstead’s and Bostrom’s recent work should enable clearer and more efficient communication between researchers, and therefore greater research productivity. Though I am not aware of any controlled, experimental studies on the effect of shared language on research productivity, a shared language is widely considered to be of great benefit for any field of research, and I shall provide a few examples of this claim which appear in print. Fuzzi et al. (2006): “The use of inconsistent terms can easily lead to misunderstandings and confusion in the communication between specialists from different [disciplines] of atmospheric and climate research, and may thus potentially inhibit scientiﬁc progress.” Hinkel (2008): “Technical languages enable their users, e.g. members of a scientiﬁc discipline, to communicate eﬃciently about a domain of interest.” Madin et al. (2007): “terminological ambiguity slows scientiﬁc progress, leads to redundant research efforts, and ultimately impedes advances towards a uniﬁed foundation for ecological science.”")
* MIRI’s Eliezer Yudkowsky has [begun](http://lesswrong.com/lw/hmt/tiling_agents_for_selfmodifying_ai_opfai_2/) to describe in more detail which open research problems constitute “Friendly AI research,” in his view.

Given these developments, we are in a better position than ever before to assess the value of Friendly AI research as effective altruism.

Still, this is a difficult question. It is challenging enough to evaluate the cost-effectiveness of [anti-malaria nets](http://blog.givewell.org/2012/10/18/revisiting-the-case-for-insecticide-treated-nets-itns/) or [direct cash transfers](http://www.givewell.org/international/top-charities/give-directly). Evaluating the cost-effectiveness of attempts to shape the far future (e.g. via Friendly AI) is even more difficult than that. Hence, **this short post sketches an argument that can be given in favor of Friendly AI research as effective altruism, to enable future discussion**, and is **not intended as a thorough analysis.**

### An argument for Friendly AI research as effective altruism

[Beckstead (2013)](https://sites.google.com/site/nbeckstead/research/Beckstead%2C%20Nick--On%20the%20Overwhelming%20Importance%20of%20Shaping%20the%20Far%20Future.pdf?attredirects=0&d=1) argues[5](https://intelligence.org/2013/06/05/friendly-ai-research-as-effective-altruism/#footnote_4_10240 "In addition to Beckstead’s thesis, see also A Proposed Adjustment to the Astronomical Waste Argument.") for the following thesis:

> From a global perspective, what matters most (in expectation) is that we do what is best (in expectation) for the general trajectory along which our descendants develop over the coming millions, billions, and trillions of years.
>
>

Why think this? Astronomical facts suggest that humanity (including “post-humanity”) could survive for billions or trillions of years ([Adams 2008](http://books.google.com/books?id=X5jdMyJKNL4C&pg=PT77&lpg=PT77#v=onepage&q&f=false)), and could thus produce enormous amounts of good.[6](https://intelligence.org/2013/06/05/friendly-ai-research-as-effective-altruism/#footnote_5_10240 "Beckstead doesn’t mention this, but I would like to point out that moral realism is not required for Beckstead’s arguments to go through. In fact, I generally accept Beckstead’s arguments even though most philosophers would not consider me a moral realist, though to some degree that is a semantic debate (Muehlhauser 2011; Joyce 2012). If you’re a moral realist and you believe your intuitive moral judgments are data about what is morally true, then Beckstead’s arguments (if successful) have something to say about what is morally true, and about what you should do if you want to act in morally good ways. If you’re a moral anti-realist but you think your intuitive judgments are data about what you value — or about what you would value if you had more time to think about your values and how to resolve the contradictions among them — then Beckstead’s arguments (if successful) have something to say about what you value, and about what you should do if you want to help achieve what you value.") But the value produced by our future depends on our *development trajectory*. If humanity destroys itself with powerful technologies in the 21st century, then nearly all that future value is lost. And if we survive but develop along a trajectory dominated by conflict and poor decisions, then the future could be much less good than if our trajectory is dominated by altruism and wisdom. Moreover, some of our actions today can have “ripple effects”[7](https://intelligence.org/2013/06/05/friendly-ai-research-as-effective-altruism/#footnote_6_10240 "Karnofsky calls these “flow-through effects.”") which determine the trajectory of human development, because many outcomes are [path-dependent](http://en.wikipedia.org/wiki/Path_dependence). Hence, actions which directly or indirectly precipitate particular trajectory changes (e.g. mitigating existential risks) can have vastly more value (in expectation) than actions with merely proximate benefits (e.g. saving the lives of 20 wild animals). Beckstead calls this the “rough future-shaping argument.”

If we accept the normative assumptions lurking behind this argument (e.g. [risk neutrality](http://en.wikipedia.org/wiki/Risk_neutral); see Beckstead’s dissertation), then the far future is enormously valuable (if it goes at least as well on average as the past century), and existential risk reduction is much more important than producing proximate benefits (e.g. global health, poverty reduction) or speeding up development (which could in fact increase existential risks, and even if it doesn’t, has lower expected value than existential risk reduction).

However, Beckstead’s conclusion is not necessarily that existential risk reduction should be our global priority, because

> there may be other ways to have a large, persistent effect on the far future without reducing existential risk… Some persistent changes in values and social norms could make the future [some fraction] better or worse… Sure, succeeding in preventing an existential catastrophe would be better than making a smaller trajectory change, but creating a small positive trajectory change may be significantly easier.
>
>

Instead, Beckstead’s arguments suggest that “what matters most for shaping the far future is producing positive trajectory changes and avoiding negative ones.” Existential risk reduction is one important kind of positive trajectory change that could turn out to be the intervention with the highest expected value.

One important clarification is in order. It could turn out to be that working toward proximate benefits or development acceleration does more good than “direct” efforts for trajectory change, if working toward proximate benefits or development acceleration turns out to have major ripple effects which produce important trajectory change. For example, perhaps an “ordinary altruistic effort” like solving India’s [iodine deficiency problem](http://www.dnaindia.com/india/1593586/report-71-million-hit-by-iodine-deficiency-in-india) would cause there to be thousands of “extra” world-class elite thinkers two generations from now, which could increase humanity’s chances of intelligently navigating the crucial 21st century and spreading to the stars. (I don’t think this is likely; I suggest it merely for illustration.)

For the sake of argument, suppose you agree with Beckstead’s core thesis that “what matters most (in expectation) is that we do what is best (in expectation) for the general trajectory along which our descendants develop.” Suppose you also think, as I do, that machine superintelligence is probably inevitable.[8](https://intelligence.org/2013/06/05/friendly-ai-research-as-effective-altruism/#footnote_7_10240 "See Bostrom (forthcoming) for an extended argument. Perhaps the most likely defeater for machine superintelligence is that global catastrophe may halt scientific progress before human-level AI is created.")

In that case, you might think that Friendly AI research is a uniquely foreseeable and impactful way to shape the far future in an enormously positive way, [because](http://lesswrong.com/lw/hjb/a_proposed_adjustment_to_the_astronomical_waste/91zq) “our effects on the far future must almost entirely pass through our effects on the development of machine superintelligence.” All other developing trends might be overridden by the overwhelming effectiveness of machine superintelligence — and specifically, by the values that were (explicitly or implicitly, directly or indirectly) written into the machine superintelligence(s).

If that’s right, our situation is a bit like sending an interstellar probe to [colonize distant solar systems](http://commonsenseatheism.com/wp-content/uploads/2013/05/Armstrong-Sandberg-Eternity-in-six-hours-intergalactic-spreading-of-intelligent-life-and-sharpening-the-Fermi-paradox.pdf) before they recede beyond the [cosmological horizon](http://en.wikipedia.org/wiki/Observable_universe#Particle_horizon) and can thus never be reached from Earth again due to [the expansion of the universe](https://en.wikipedia.org/wiki/Metric_expansion_of_space). Anything on Earth that doesn’t affect the content of the probe will have no impact on those solar systems. (See also [this comment](http://lesswrong.com/lw/hjb/a_proposed_adjustment_to_the_astronomical_waste/921r).)

### Potential defeaters

The rough argument above — in favor of Friendly AI research as an efficient form of effective altruism — deserves to be “fleshed out” in more detail.[9](https://intelligence.org/2013/06/05/friendly-ai-research-as-effective-altruism/#footnote_8_10240 "Beckstead, in personal communication, suggested (but didn’t necessarily endorse) the following formalization of the rough argument sketched in the main text of the blog post: “(1) To a first approximation, the future of humanity is all that matters. (2) To a much greater extent than anything else, the future of humanity is highly sensitive to how machine intelligence unfolds. (3) Therefore, there is a very strong presumption in favor of working on any project which makes machine intelligence unfold in a better way. (4) FAI research is the most promising route to making machine intelligence unfold in a better way. (5) Therefore, there is a very strong presumption in favor of doing FAI research.” Beckstead (2013) examines the case for (1). Bostrom (forthcoming), in large part, examines the case for (2). Premise (3) informally follows from (1) and (2), and the conclusion (5) informally follows from (3) and (4). Premise (4) appears to me to be the most dubious part of the argument, and the least explored in the extant literature.")

Potential defeaters should also be examined:

* Perhaps we ought to reject one or more of the normative assumptions behind Beckstead’s rough future-shaping argument.
* Perhaps it’s not true that “our effects on the far future must almost entirely pass through our effects on the development of machine superintelligence.”
* Perhaps Friendly AI research is not (today) a particularly efficient way to positively affect the development of machine superintelligence. Competing interventions may include: (1) [AI risk strategy research](http://lesswrong.com/lw/bd6/ai_risk_opportunity_a_timeline_of_early_ideas_and/91zl), (2) improving [technological forecasting](http://intelligence.org/2013/05/15/when-will-ai-be-created/), (3) [improving science in general](http://www.vannevargroup.org/), (4) [improving and spreading effective altruism and rationality](http://rationalaltruist.com/2013/06/03/my-outlook/), and (5) many others.

In future blog posts, members of the effective altruist community (including myself) will expand on the original argument and examine potential defeaters.

####

#### Acknowledgements

My thanks to those who provided feedback on this post: Carl Shulman, Nick Beckstead, Jonah Sinick, and Eliezer Yudkowsky.

---

1. In this post, I talk about the value of *humanity in general* creating Friendly AI, though MIRI co-founder Eliezer Yudkowsky usually talks about *MIRI in particular* — or at least, a functional equivalent — creating Friendly AI. This is because I am not as confident as Yudkowsky that it is best for MIRI to attempt to build Friendly AI. When updating MIRI’s bylaws in early 2013, Yudkowsky and I came to a compromise on the language of MIRI’s mission statement, which now reads: “[MIRI] exists to ensure that the creation of smarter-than-human intelligence has a positive impact. Thus, the charitable purpose of [MIRI] is to: (a) perform research relevant to ensuring that smarter-than-human intelligence has a positive impact; (b) raise awareness of this important issue; (c) advise researchers, leaders and laypeople around the world; and (d) *as necessary*, implement a smarter-than-human intelligence with humane, stable goals” (emphasis added). My own hope is that it will not be necessary for MIRI (or a functional equivalent) to attempt to build Friendly AI itself. But of course I must remain open to the possibility that this will be the wisest course of action as the first creation of AI [draws nearer](http://intelligence.org/2013/05/15/when-will-ai-be-created/). There is also the question of capability: few people think that a non-profit research organization has much chance of being the first to build AI. I worry, however, that the world’s elites will not find it fashionable to take this problem seriously until the creation of AI is only a few decades away, at which time it will be especially difficult to develop the mathematics of Friendly AI in time, and humanity will be forced to take a gamble on its very survival with powerful AIs we have little reason to trust.
2. One might think of effective altruism as a straightforward application of [decision theory](http://lesswrong.com/lw/gu1/decision_theory_faq/) to the subject of philanthropy. Philanthropic agents of all kinds (individuals, groups, foundations, etc.) ask themselves: “How can we choose philanthropic acts (e.g. donations) which (in expectation) will do as much good as possible, given what we care about?” The consensus recommendation for *all* kinds of choices under uncertainty, including philanthropic choices, is to maximize expected utility ([Chater & Oaksford 2012](http://books.google.com/books?id=S1-K4AT3zXYC&lpg=PA11&ots=wjavib87qF&dq=normative%20systems%3A%20logic%2C%20probability%2C%20and%20rational%20choice&lr&pg=PA11#v=onepage&q&f=false); [Peterson 2004](http://commonsenseatheism.com/wp-content/uploads/2012/05/Peterson-From-Outcomes-to-Acts-a-non-standard-axiomatization-of-the-expected-utility-principle.pdf); [Stein 1996](http://www.amazon.com/Without-Good-Reason-Rationality-Philosophy/dp/0198235747/); [Schmidt 1998](http://www.amazon.com/Axiomatic-Utility-Theory-under-Risk/dp/3540643192):19). Different philanthropic agents value different things, but decision theory suggests that each of them can get the most of what they want if they each maximize their expected utility. Choices which maximize expected utility are in this sense “optimal,” and thus another term for effective altruism is “[optimal philanthropy](http://lesswrong.com/lw/da4/what_is_optimal_philanthropy/).” Note that effective altruism in this sense is not too dissimilar from earlier approaches to philanthropy, including [high-impact philanthropy](http://en.wikipedia.org/wiki/High_impact_philanthropy) (making “[the biggest difference possible, given the amount of capital invested](http://www.impact.upenn.edu/faq/)“), [strategic philanthropy](http://www.amphilsoc.org/sites/default/files/490202.pdf), [effective philanthropy](http://www.hudson.org/files/pdf_upload/Stanley_Katz_APS_Proceedings_Piece_June_2005.pdf), and [wise philanthropy](http://wisephilanthropy.com/). Note also that effective altruism does not say that a philanthropic agent should specify complete utility and probability functions over outcomes and then compute the philanthropic act with the highest expected utility — that is impractical for bounded agents. We must keep in mind the distinction between normative, descriptive, and prescriptive models of decision-making (Baron 2007): “normative models tell us how to evaluate… decisions in terms of their departure from an ideal standard. Descriptive models specify what people in a particular culture actually do and how they deviate from the normative models. Prescriptive models are designs or inventions, whose purpose is to bring the results of actual thinking into closer conformity to the normative model.” The *prescriptive* question — about what bounded philanthropic agents should do to maximize expected utility with their philanthropic choices — tends to be extremely complicated, and is the subject of most of the research performed by the effective altruism community.
3. See, for example: [Efficient Charity](http://lesswrong.com/lw/37f/efficient_charity/), [Efficient Charity: Do Unto Others](http://lesswrong.com/lw/3gj/efficient_charity_do_unto_others/), [Politics as Charity](http://lesswrong.com/lw/2qq/politics_as_charity/), [Heuristics and Biases in Charity](http://lesswrong.com/lw/aid/heuristics_and_biases_in_charity/), [Public Choice and the Altruist’s Burden](http://lesswrong.com/lw/2hv/public_choice_and_the_altruists_burden/), [On Charities and Linear Utility](http://lesswrong.com/lw/44c/on_charities_and_linear_utility/), [Optimal Philanthropy for Human Beings](http://lesswrong.com/lw/6py/optimal_philanthropy_for_human_beings/), [Purchase Fuzzies and Utilons Separately](http://lesswrong.com/lw/6z/purchase_fuzzies_and_utilons_separately/), [Money: The Unit of Caring](http://lesswrong.com/lw/65/money_the_unit_of_caring/), [Optimizing Fuzzies and Utilons: The Altruism Chip Jar](http://lesswrong.com/lw/3kl/optimizing_fuzzies_and_utilons_the_altruism_chip/), [Efficient Philanthropy: Local vs. Global Approaches](http://lesswrong.com/lw/684/efficient_philanthropy_local_vs_global_approaches/), [The Effectiveness of Developing World Aid](http://lesswrong.com/lw/2pq/the_effectiveness_of_developing_world_aid/), [Against Cryonics & For Cost-Effective Charity](http://lesswrong.com/lw/2kh/against_cryonics_for_costeffective_charity/), [Bayesian Adjustment Does Not Defeat Existential Risk Charity](http://lesswrong.com/lw/gzq/bayesian_adjustment_does_not_defeat_existential/), [How to Save the World](http://lesswrong.com/lw/373/how_to_save_the_world/), and [What is Optimal Philanthropy?](http://lesswrong.com/lw/da4/what_is_optimal_philanthropy/)
4. I believe Beckstead and Bostrom have done the research community an enormous service in creating a *framework*, a *shared language*, for discussing trajectory changes, existential risks, and machine superintelligence. When discussing these topics with my colleagues, it has often been the case that the first hour of conversation is spent merely trying to understand what the other person is saying — how they are using the terms and concepts they employ. Beckstead’s and Bostrom’s recent work should enable clearer and more efficient communication between researchers, and therefore greater research productivity. Though I am not aware of any controlled, experimental studies on the effect of shared language on research productivity, a shared language is widely considered to be of great benefit for any field of research, and I shall provide a few examples of this claim which appear in print. [Fuzzi et al. (2006)](http://atmos-chem-phys.net/6/2017/2006/acp-6-2017-2006.pdf): “The use of inconsistent terms can easily lead to misunderstandings and confusion in the communication between specialists from different [disciplines] of atmospheric and climate research, and may thus potentially inhibit scientiﬁc progress.” [Hinkel (2008)](http://www.pik-potsdam.de/research/transdisciplinary-concepts-and-methods/projects/project-archive/favaia/pubs/hinkel-knowledge-integration.pdf): “Technical languages enable their users, e.g. members of a scientiﬁc discipline, to communicate eﬃciently about a domain of interest.” [Madin et al. (2007)](http://www.cs.cofc.edu/~bowring/classes/csis%20633/readings/madin-etal-tree-2008.pdf): “terminological ambiguity slows scientiﬁc progress, leads to redundant research efforts, and ultimately impedes advances towards a uniﬁed foundation for ecological science.”
5. In addition to Beckstead’s thesis, see also [A Proposed Adjustment to the Astronomical Waste Argument](http://lesswrong.com/lw/hjb/a_proposed_adjustment_to_the_astronomical_waste/).
6. Beckstead doesn’t mention this, but I would like to point out that moral realism is not required for Beckstead’s arguments to go through. In fact, I generally accept Beckstead’s arguments even though most philosophers would not consider me a moral realist, though to some degree that is a semantic debate ([Muehlhauser 2011](http://lesswrong.com/lw/5u2/pluralistic_moral_reductionism/); [Joyce 2012](http://www.victoria.ac.nz/staff/richard_joyce/acrobat/joyce_metaethical.pluralism.pdf)). If you’re a moral realist and you believe your intuitive moral judgments are data about what is morally true, then Beckstead’s arguments (if successful) have something to say about what is morally true, and about what you should do if you want to act in morally good ways. If you’re a moral anti-realist but you think your intuitive judgments are data about what you value — or about what you would value if you had more time to think about your values and how to resolve the contradictions among them — then Beckstead’s arguments (if successful) have something to say about what you value, and about what you should do if you want to help achieve what you value.
7. Karnofsky calls these “[flow-through effects](http://blog.givewell.org/2013/05/15/flow-through-effects/).”
8. See [Bostrom (forthcoming)](http://lesswrong.com/lw/bd6/ai_risk_opportunity_a_timeline_of_early_ideas_and/91zl) for an extended argument. Perhaps the most likely defeater for machine superintelligence is that global catastrophe may halt scientific progress before human-level AI is created.
9. Beckstead, in personal communication, suggested (but didn’t necessarily endorse) the following formalization of the rough argument sketched in the main text of the blog post: “(1) To a first approximation, the future of humanity is all that matters. (2) To a much greater extent than anything else, the future of humanity is highly sensitive to how machine intelligence unfolds. (3) Therefore, there is a very strong presumption in favor of working on any project which makes machine intelligence unfold in a better way. (4) FAI research is the most promising route to making machine intelligence unfold in a better way. (5) Therefore, there is a very strong presumption in favor of doing FAI research.” [Beckstead (2013)](https://sites.google.com/site/nbeckstead/research/Beckstead%2C%20Nick--On%20the%20Overwhelming%20Importance%20of%20Shaping%20the%20Far%20Future.pdf?attredirects=0&d=1) examines the case for (1). [Bostrom (forthcoming)](http://lesswrong.com/lw/bd6/ai_risk_opportunity_a_timeline_of_early_ideas_and/91zl), in large part, examines the case for (2). Premise (3) informally follows from (1) and (2), and the conclusion (5) informally follows from (3) and (4). Premise (4) appears to me to be the most dubious part of the argument, and the least explored in the extant literature.

The post [Friendly AI Research as Effective Altruism](https://intelligence.org/2013/06/05/friendly-ai-research-as-effective-altruism/) appeared first on [Machine Intelligence Research Institute](https://intelligence.org).