2012 Summer Singularity Challenge

Thanks to the generosity of several major donors,† every donation to the Machine Intelligence Research Institute made now until July 31, 2012 will be matched dollar-for-dollar, up to a total of $150,000!

**[Donate Now!](https://intelligence.org/donate/)**

$0

$37.5K

$75K

$112.5K

$150K

Now is your chance to **double your impact** while helping us raise up to $300,000 to help fund [our research program](https://intelligence.org/research/) and stage the upcoming [Singularity Summit](https://intelligence.org/singularitysummit/)… which you can [register for now](http://singularitysummit.com/)!

**Note**: If you prefer to support rationality training, you are welcome to *earmark your donations* for “CFAR” ([Center for Applied Rationality](http://www.appliedrationality.org/)). Donations earmarked for CFAR will *only* be used for CFAR, and donations *not* earmarked for CFAR will *only* be used for Singularity research and outreach.

[![](http://miri.wpengine.com/wp-content/uploads/2012/06/towardapositivesingularity.jpg "towardapositivesingularity")](https://intelligence.org/donate/)
Since we published our [strategic plan](https://intelligence.org/files/strategicplan20112.pdf) in August 2011, we have [achieved most of the near-term goals outlined therein](http://lesswrong.com/r/discussion/lw/dm9/revisiting_sis_2011_strategic_plan_how_are_we/#summary). Here are just a *few* examples:

* We outlined the [open research problems](http://lukeprog.com/SaveTheWorld.html) related to our work (Section 1.1).
* We recruited several more [research associates](https://intelligence.org/research-associates/) and about a dozen [remote researchers](http://lesswrong.com/lw/bke/the_singularity_institute_still_needs_remote/) (Section 1.2e).
* We held our annual [Singularity Summit](http://singularitysummit.com/) and gained corporate sponsors for it (Section 2.1).
* We made progress in decision theory ([example](http://lesswrong.com/lw/8wc/a_model_of_udt_with_a_halting_oracle)) via LessWrong.com and our research associates (Section 2.2b).
* We published [How to Run a Successful Less Wrong Meetup Group](http://lesswrong.com/lw/crs/how_to_run_a_successful_less_wrong_meetup/) (Section 2.2d).
* We released pre-prints of several forthcoming research articles, including [How Hard is Artificial Intelligence?](http://www.nickbostrom.com/aievolution.pdf), [Intelligence Explosion: Evidence and Import](https://intelligence.org/files/IE-EI.pdf), and [The Singularity and Machine Ethics](https://intelligence.org/files/SaME.pdf) (Section 2.3).
* We [redesigned](https://intelligence.org/blog/2012/06/18/welcome-to-the-new-singularity-org/) our primary website (Section 2.6).
* We acquired $40,000/month in free Google Adwords advertising, to drive traffic to websites operated by the Machine Intelligence Research Institute (Section 2.6c).
* We began publishing [monthly progress reports](https://intelligence.org/blog/category/monthly-progress/) (Section 2.9b).
* We built up the Center for Applied Rationality such that it should be able to spin off from the Machine Intelligence Research Institute later this year (Section 3.1).
* We created a [transparency section](https://intelligence.org/transparency/) on our website, where visitors can find our IRS 990 forms, and also several standard organizational policies, e.g. a conflict of interest policy, non-discrimination policy, etc (Section 3.2).

In the coming year, the **Machine Intelligence Research Institute plans to do the following**:

* **Hold our annual [Singularity Summit](http://singularitysummit.com/)**, this year in San Francisco!
* **Spin off the [Center for Applied Rationality](http://www.appliedrationality.org/)** as a separate organization focused on rationality training, so that the Machine Intelligence Research Institute can be focused more exclusively on Singularity research and outreach.
* **Publish additional [research](https://intelligence.org/research/)** on AI risk and Friendly AI.
* **Eliezer will write an “Open Problems in Friendly AI” sequence** for *Less Wrong*. (For news on his rationality books, see [here](http://lesswrong.com/lw/d06/intellectual_insularity_and_productivity/6swt).)
* **Finish *[Facing the Singularity](http://facingthesingularity.com/)*** and publish ebook versions of *Facing the Singularity* and *[The Sequences, 2006-2009](http://wiki.lesswrong.com/wiki/Sequences)*.
* And much more! For details on what we might do with additional funding, see [How to Purchase AI Risk Reduction](http://lesswrong.com/lw/cs6/how_to_purchase_ai_risk_reduction/).

If you’re planning to earmark your donation to CFAR (Center for Applied Rationality), here’s a preview of **what CFAR plans to do in the next year**:

* **Develop additional lessons** teaching the most important and useful parts of rationality. CFAR has already developed and tested *over 18 hours of lessons* so far, including classes on how to evaluate evidence using Bayesianism, how to make more accurate predictions, how to be more efficient using economics, how to use thought experiments to better understand your own motivations, and much more.
* **Run immersive rationality retreats** to teach from our curriculum and to connect aspiring rationalists with each other. CFAR ran pilot retreats in May and June. Participants in the May retreat called it “transformative” and “astonishing,” and the average response on the survey question, “Are you glad you came? (1-10)” was a 9.4. (We don’t have the June data yet, but people were similarly enthusiastic about that one.)
* **Run SPARC, a camp on the advanced math of rationality** for mathematically gifted high school students. CFAR has a stellar first-year class for SPARC 2012; most students admitted to the program placed in the top 50 on the USA Math Olympiad (or performed equivalently in a similar contest).
* **Collect longitudinal data on the effects of rationality training**, to improve our curriculum and to generate promising hypotheses to test and publish, in collaboration with other researchers. CFAR has already launched a one-year randomized controlled study tracking reasoning ability and various metrics of life success, using participants in our June minicamp and a control group.
* **Develop apps and games about rationality**, with the dual goals of (a) helping aspiring rationalists practice essential skills, and (b) making rationality fun and intriguing to a much wider audience. CFAR has two apps in beta testing: one training players to update their own beliefs the right amount after hearing other people’s beliefs, and another training players to calibrate their level of confidence in their own beliefs. CFAR is working with a developer on several more games training people to avoid cognitive biases.
* And more!

We appreciate your support for our high-impact work! **[Donate now](https://intelligence.org/donate/)**, and seize a better than usual chance to move our work forward. Credit card transactions are securely processed using either PayPal or Google Checkout. If you have questions about donating, please contact Louie Helm at (510) 717-1477 or louie@intelligence.org.

† $150,000 of total matching funds has been provided by Jaan Tallinn, Tomer Kagan, Alexei Andreev, and Brandon Reinhart.

The post [2012 Summer Singularity Challenge](https://intelligence.org/2012/07/03/summer-challenge/) appeared first on [Machine Intelligence Research Institute](https://intelligence.org).


Past progress reports: [April 2012](http://intelligence.org/blog/2012/05/08/singularity-institute-progress-report-april-2012/), [March 2012](http://intelligence.org/blog/2012/04/06/singularity-institute-progress-report-march-2012/), [February 2012](http://intelligence.org/blog/2012/03/03/singularity-institute-progress-report-february-2012/), [January 2012](http://intelligence.org/blog/2012/02/05/singularity-institute-progress-report-january-2012/), [December 2011](http://intelligence.org/blog/2012/01/16/singularity-institute-progress-report-december-2011/).

Here’s what the Machine Intelligence Research Institute did in May 2012:

* **How to Purchase AI Risk Reduction**: Luke wrote [a series of posts](http://lesswrong.com/r/discussion/lw/cs6/how_to_purchase_ai_risk_reduction/) on how to purchase AI risk reduction, with cost estimates for many specific projects. Some projects are currently in place at SI; others can be launched if we are able to raise sufficient funding.
* **Research articles**: Luke continued to work with about a dozen collaborators on several developing research articles, including “Responses to Catastrophic AGI Risk,” mentioned [here](http://lesswrong.com/lw/cr6/building_the_ai_risk_research_community/).
* **Other writings**: Kaj Sotala, with help from Luke and many others, published *[How to Run a Successful Less Wrong Meetup Group](http://lesswrong.com/lw/crs/how_to_run_a_successful_less_wrong_meetup/)*. Carl published several articles: (1) [Utilitarianism, contractualism, and self-sacrifice](http://reflectivedisequilibrium.blogspot.com/2012/05/utilitarianism-contractualism-and-self.html), (2) [Philosophers vs. economists on discounting](http://reflectivedisequilibrium.blogspot.com/2012/05/philosophers-vs-economists-on.html), (3) [Economic growth: more costly disasters, better prevention](http://reflectivedisequilibrium.blogspot.com/2012/05/economic-growth-more-costly-disasters.html), and (4) [What to eat during impact winter?](http://reflectivedisequilibrium.blogspot.com/2012/05/what-to-eat-during-impact-winter.html) Eliezer wrote [Avoid Motivated Cognition](http://lesswrong.com/lw/bnk/sotw_avoid_motivated_cognition/). Luke posted part 2 of his [dialogue with Ben Goertzel](https://intelligence.org/feed/?paged=79) about AGI.
* **Ongoing long-term projects**: Amy continued to work on Singularity Summit 2012. Michael continued to work on the Machine Intelligence Research Institute’s new primary website, new annual report, and new newsletter design. Louie and SI’s new executive assistant Ioven Fables are hard at work on organizational development and transparency (some of which will be apparent when the new website launches).
* **Center for Applied Rationality (CFAR)**: The CFAR team continued to make progress toward spinning off this rationality-centric organization, in keeping with [SI’s strategic plan](https://intelligence.org/files/strategicplan2011.pdf). We also held the first summer minicamp, which surpassed our expectations and was very positively received. (More details on this will be compiled later.)
* **Meetings with advisors, supporters, and potential researchers:** As usual, various SI staff met or spoke with dozens of advisors, supporters, and collaborators about how to build the existential risk community, how to mitigate AI risk, how to improve the Machine Intelligence Research Institute’s effectiveness, and other topics.
* And of course much more than is listed here!

Finally, we’d like to recognize our **most active volunteers**in May 2012: Matthew Fallshaw, Gerard McCusker, Frank Adamek, David Althaus, Tim Oertel, and Casey Pfluger. Thanks everyone! (And, our apologies if we forgot to name you!)

The post [Machine Intelligence Research Institute Progress Report, May 2012](https://intelligence.org/2012/06/16/singularity-institute-progress-report-may-2012/) appeared first on [Machine Intelligence Research Institute](https://intelligence.org).


Past progress reports: [March 2012](http://intelligence.org/blog/2012/04/06/singularity-institute-progress-report-march-2012/), [February 2012](http://intelligence.org/blog/2012/03/03/singularity-institute-progress-report-february-2012/), [January 2012](http://intelligence.org/blog/2012/02/05/singularity-institute-progress-report-january-2012/), [December 2011](http://intelligence.org/blog/2012/01/16/singularity-institute-progress-report-december-2011/).

Here’s what the Machine Intelligence Research Institute did in April 2012:

* **SPARC**: Several MIRI staff members are working in collaboration with SI research associate Paul Christiano and a few others to develop a rationality camp for high school students with exceptional mathematical ability (SPARC). This is related to our efforts to spin off a new rationality-focused organization, and it is also a major step forward in our efforts to locate elite young math talent that may be useful in our research efforts.
* **Research articles**: Luke published [AI Risk Bibliography 2012](http://intelligence.org/upload/AI%20Risk%20Bibliography%202012.pdf). He is currently developing nearly a dozen other papers with a variety of co-authors. New SI research associate [Kaj Sotala](http://www.xuenay.net/) has two papers forthcoming in the *International Journal of Machine Consciousness*: [Advantages of Artificial Intelligences, Uploads, and Digital Minds](http://www.xuenay.net/Papers/DigitalAdvantages.pdf) and [Coalescing Minds: Brain Uploading-Related Group Mind Scenarios](http://www.xuenay.net/Papers/CoalescingMinds.pdf).
* **Other articles**: Luke published [a dialogue](http://lesswrong.com/r/discussion/lw/bxr/muehlhauserwang_dialogue/) with AGI researcher Pei Wang and several more posts in the [AI Risk and Opportunity](http://lesswrong.com/r/discussion/lw/ajm/ai_risk_and_opportunity_a_strategic_analysis/) series. Luke also worked with Kaj Sotala to develop an instructional booklet for Less Wrong meetup group organizers, which is nearly complete.
* **Ongoing long-term projects**: Amy continued to work on Singularity Summit 2012. Michael launched the new [Singularity Summit website](http://singularitysummit.com/), continued to work on the Machine Intelligence Research Institute’s new primary website, new annual report, and new newsletter design. Luke uploaded several more volunteer-prepared translations of *[Facing the Singularity](http://facingthesingularity.com/)*. Luke also continued to build the Machine Intelligence Research Institute’s set of remote collaborators, who are hard at work converting the Machine Intelligence Research Institute’s research articles to a new template, hunting down predictions of AI, writing literature summaries on heuristics and biases, and more.
* **Center for Applied Rationality (CFAR)**: “Rationality Group” now has a final name: the Center for Applied Rationality (CFAR). The CFAR team has been hard at work preparing for the upcoming [rationality minicamps](http://lesswrong.com/lw/b98/minicamps_on_rationality_and_awesomeness_may_1113/), as well as continuing to develop the overall strategy for the emerging organization.
* **Meetings with advisors, supporters, and potential researchers:** As usual, various SI staff met or spoke with dozens of advisors, supporters, and collaborators about how to build the existential risk community, how to mitigate AI risk, how to improve the Machine Intelligence Research Institute’s effectiveness, and other topics. Quixey co-founder and CEO Liron Shapira was added as an advisor.
* And of course much more than is listed here!

Finally, we’d like to recognize our **most active volunteers**in April 2012: Matthew Fallshaw, Gerard McCusker, Frank Adamek, David Althaus, Tim Oertel, Casey Pfluger, Paul Gentemann, and John Maxwell. Thanks everyone! (And, our apologies if we forgot to name you!)

The post [Machine Intelligence Research Institute Progress Report, April 2012](https://intelligence.org/2012/05/08/singularity-institute-progress-report-april-2012/) appeared first on [Machine Intelligence Research Institute](https://intelligence.org).


Past progress reports: [February 2012](http://intelligence.org/blog/2012/03/03/singularity-institute-progress-report-february-2012/), [January 2012](http://intelligence.org/blog/2012/02/05/singularity-institute-progress-report-january-2012/), [December 2011](http://intelligence.org/blog/2012/01/16/singularity-institute-progress-report-december-2011/).

Fun fact of the day: [The Machine Intelligence Research Institute’s research fellows and research associates have more peer-reviewed publications forthcoming in 2012 than they had published in all past years combined](http://lesswrong.com/r/discussion/lw/axr/three_new_papers_on_ai_risk/627o).

Here’s what the Machine Intelligence Research Institute did in March 2012:

* **Research articles**: Luke and Anna released an updated draft of [Intelligence Explosion: Evidence and Import](http://commonsenseatheism.com/wp-content/uploads/2012/02/Muehlhauser-Salamon-Intelligence-Explosion-Evidence-and-Import.pdf), and Luke and Louie released an updated draft of [The Singularity and Machine Ethics](http://commonsenseatheism.com/wp-content/uploads/2011/11/Muehlhauser-Helm-The-Singularity-and-Machine-Ethics-draft.pdf). Luke submitted an article (co-authored with Nick Bostrom) to *Communications of the ACM* — an article on Friendly AI. Machine Intelligence Research Institute research associate Joshua Fox released two forthcoming articles co-authored with (past Machine Intelligence Research Institute Visiting Fellow) Roman Yampolskiy: [Safety Engineering for Artificial General Intelligence](http://joshuafox.com/professional/media/YampolskiyFox__SafetyEngineeringforAGI.pdf) and [Artificial General Intelligence and the Human Mental Model](http://joshuafox.com/professional/media/YampolskiyFox__AGIAndTheHumanModel.pdf).
* **Other articles**: Luke published [The AI Problem, with Solutions](http://facingthesingularity.com/2012/ai-the-problem-with-solutions/), [How to Fix Science](http://lesswrong.com/lw/ajj/how_to_fix_science/), [Muehlhauser-Goertzel Dialogue Part 1](http://lesswrong.com/r/discussion/lw/aw7/muehlhausergoertzel_dialogue_part_1/), a [list of journals that may publish articles on AI risk](http://tinyurl.com/AI-risk-journals), and the first three posts in his series [AI Risk and Opportunity: A Strategic Analysis](http://lesswrong.com/r/discussion/lw/ajm/ai_risk_and_opportunity_a_strategic_analysis/). The Machine Intelligence Research Institute paid (past Visiting Fellow) [Kaj Sotala](http://www.xuenay.net/) to write most of a new instructional booklet for Less Wrong meetup group organizers, which should be published in the next month or two. Eliezer continued work on his new Bayes’ Theorem tutorial and other writing projects. Carl published [Using degrees of freedom to change the past for fun and profit](http://lesswrong.com/lw/a9f/using_degrees_of_freedom_to_change_the_past_for/) and [Are pain and pleasure equally energy efficient?](http://reflectivedisequilibrium.blogspot.com/2012/03/are-pain-and-pleasure-equally-energy.html)
* **Ongoing long-term projects**: Amy continued to work on Singularity Summit 2012. Michael continued to work on the Machine Intelligence Research Institute’s new primary website, new Summit website, new annual report, and new newsletter design. Louie continued to improve our accounting processes and also handled several legal and tax issues. Luke uploaded several more volunteer-prepared translations of *[Facing the Singularity](http://facingthesingularity.com/)*. Luke also continued to build the Machine Intelligence Research Institute’s set of remote collaborators, who are hard at work converting the Machine Intelligence Research Institute’s research articles to a new template, hunting down predictions of AI, writing literature summaries on heuristics and biases, and more.
* **Rationality Group**: Per our [strategic plan](https://intelligence.org/files/strategicplan2011.pdf), we will launch this new “Rationality Group” organization soon, so that the Machine Intelligence Research Institute can focus on its efforts on activities related to AI risk. In March, Rationality Group (led by Anna) contracted with [Julia Galef](http://measureofdoubt.com/about/) and Michael Smith to work toward launching the organization. Eliezer continued to help Rationality Group develop and test its lessons. Rationality Group has begun offering **prizes** for suggesting exercises for developing rationality skills, starting with the skills of “[Be Specific](http://lesswrong.com/lw/bc3/sotw_be_specific/)” and “[Check Consequentialism](http://lesswrong.com/lw/b4f/sotw_check_consequentialism/).” Rationality Group has also announced three [Minicamps on Rationality and Awesomeness](http://lesswrong.com/lw/b98/minicamps_on_rationality_and_awesomeness_may_1113/), for May 11-13, June 22-24, and July 21-28. [Apply now](http://api.viglink.com/api/click?format=go&key=9f37ca02a1e3cbd4f3d0a3618a39fbca&loc=http%3A%2F%2Flesswrong.com%2Flw%2Fb98%2Fminicamps_on_rationality_and_awesomeness_may_1113%2F&v=1&libid=1333494789559&out=https%3A%2F%2Fdocs.google.com%2Fspreadsheet%2Fviewform%3Fformkey%3DdEctaFJONTk1UVdfdE9sSEpiQTFLLWc6MA&ref=http%3A%2F%2Flesswrong.com%2Fpromoted%2F&title=Minicamps%20on%20Rationality%20and%20Awesomeness%3A%20May%2011-13%2C%20June%2022-24%2C%20and%20July%2021-28%20-%20Less%20Wrong&txt=%3Cstrong%3EApply%20now.%3C%2Fstrong%3E&jsonp=vglnk_jsonp_13334948542221).
* **Meetings with advisors, supporters, and potential researchers:** As usual, various SI staff met or spoke with dozens of advisors, supporters, and collaborators about how to build the existential risk community, how to mitigate AI risk, how to improve the Machine Intelligence Research Institute’s effectiveness, and other topics. This included a two-week visit by [Nick Beckstead](https://sites.google.com/site/nbeckstead/), who worked with us on AI risk reduction strategy.
* And of course much more than is listed here!

Finally, we’d like to recognize our **most active volunteers**in March 2012: Matthew Fallshaw, Gerard McCusker, Frank Adamek, and David Althaus. Thanks everyone! (And, our apologies if we forgot to name you!)

The post [Machine Intelligence Research Institute Progress Report, March 2012](https://intelligence.org/2012/04/06/singularity-institute-progress-report-march-2012/) appeared first on [Machine Intelligence Research Institute](https://intelligence.org).


Past progress reports: [January 2012](http://intelligence.org/blog/2012/02/05/singularity-institute-progress-report-january-2012/), [December 2011](http://intelligence.org/blog/2012/01/16/singularity-institute-progress-report-december-2011/).

Here’s what the Machine Intelligence Research Institute did in February 2012:

* **Winter fundraiser completed**: Thanks to the generous contributions of our supporters, our latest winter fundraiser was a success, raising [much more](http://intelligence.org/blog/2012/02/20/2011-2012-winter-fundraiser-completed/) than our target of $100,000!
* **Research articles**: Luke and Anna published the [Singularity Summit 2011 Workshop Report](http://intelligence.org/upload/Singularity%20Summit%202011%20Workshop%20Report.pdf) and released a draft of their article [Intelligence Explosion: Evidence and Import](http://commonsenseatheism.com/wp-content/uploads/2012/02/Muehlhauser-Salamon-Intelligence-Explosion-Evidence-and-Import.pdf), forthcoming in Springer’s *The Singularity Hypothesis*. Luke also worked on an article forthcoming in *Communications of the ACM*.
* **Other articles**: Luke published a continuously updated list of [Forthcoming and desired articles on AI risk](http://tinyurl.com/ai-risk-research). For Less Wrong, Carl published [Feed the Spinoff Heuristic](http://lesswrong.com/lw/9xs/feed_the_spinoff_heuristic/), and Luke published [My Algorithm for Beating Procrastination](http://lesswrong.com/lw/9wr/my_algorithm_for_beating_procrastination/), [A brief tutorial on preferences in AI](http://lesswrong.com/r/discussion/lw/a73/a_brief_tutorial_on_preferences_in_ai/), and [Get Curious](http://lesswrong.com/lw/aa7/get_curious/). Carl also published 4 articles on ethical careers for the [80,000 Hours blog](http://80000hours.org/blog) (later posts will discuss optimal philanthropy and existential risks): [How hard is it to become the Prime Minister of the United Kingdom?](http://80000hours.org/blog/22-how-hard-is-it-to-become-prime-minister-of-the-united-kingdom), [Entrepreneurship: a game of poker, not roulette](http://80000hours.org/blog/23-entrepreneurship-a-game-of-poker-not-roulette), [Software engineering: Britain vs. Silicon Valley](http://80000hours.org/blog/26-software-engineering-britain-vs-silicon-valley), and [5 ways to be misled by salary rankings](http://80000hours.org/blog/24-5-ways-to-be-misled-by-salary-rankings).
* **Ongoing long-term projects**: Amy continued to work on Singularity Summit 2012. Michael continued to work on the Machine Intelligence Research Institute’s new website, and uploaded all past Singularity Summit videos to YouTube. Louie continued to improve our accounting processes and also handled several legal and tax issues. Luke uploaded several volunteer-prepared translations of *[Facing the Singularity](http://facingthesingularity.com/)*, and also a [podcast](http://lesswrong.com/r/discussion/lw/aey/facing_the_singularity_podcast/) for this online mini-book.
* **Grant awarded**: The Machine Intelligence Research Institute awarded philosopher [Rachael Briggs](http://www.rachaelbriggs.net/Rachael_Briggs/Home.html) a $20,000 grant to write a paper on Eliezer Yudkowsky’s [timeless decision theory](http://intelligence.org/upload/TDT-v01o.pdf). Two of Rachael’s papers — [Distorted Reflection](http://pgrim.org/pa2010reading/briggsdistorted.pdf) and [Decision-Theoretic Paradoxes as Voting Paradoxes](http://commonsenseatheism.com/wp-content/uploads/2011/09/Briggs-Decision-theoretic-paradoxes-as-voting-paradoxes.pdf) — have previously been selected as among the 10 best philosophy papers of the year by [The Philosopher’s Annual](http://www.philosophersannual.org/).
* **Rationality Group**: Anna and Eliezer continued to lead the development of a new rationality education organization, temporarily called “Rationality Group.” Per our [strategic plan](https://intelligence.org/files/strategicplan2011.pdf), we will launch this new organization soon, so that the Machine Intelligence Research Institute can focus on its efforts on activities related to AI risk. In February our Rationality Group team worked on curriculum development with several potential long-term hires, developed several rationality lessons which they tested (weekly) on small groups and iterated in response to feedback, spoke to advisors about how to build the organization and gather fundraising, and much more. The team also produced one example rationality lesson on sunk costs, including [a presentation and exercise booklets](http://lesswrong.com/lw/9hb/position_design_and_write_rationality_curriculum/). Note that Rationality Group is **currently hiring** curriculum developers, a remote executive assistant, and others, so [apply here](https://docs.google.com/spreadsheet/viewform?hl=en_US&formkey=dElRYS1hY3N4a3JIRV90R0lmdE9MN0E6MQ) if you’re interested!
* **Meetings with advisors, supporters, and potential researchers:** As usual, various SI staff met or spoke with dozens of advisors, supporters, and collaborators about how to build the existential risk community, how to mitigate AI risk, how to improve the Machine Intelligence Research Institute’s effectiveness, and other topics. We also met with several potential researchers to gauge their interest and abilities. Carl spent two weeks in Oxford visiting the [Future of Humanity Institute](http://www.fhi.ox.ac.uk/) and working with the researchers there.
* **Outsourcing**: On Louie’s (sound) advice, the Machine Intelligence Research Institute is undergoing a labor transition such that most of the work we do (in hours) will eventually be performed not by our core staff but by (mostly remote) hourly contractors and volunteers, for example [remote researchers](http://lesswrong.com/r/discussion/lw/9t8/the_singularity_institute_needs_remote/), remote [LaTeX workers](http://lesswrong.com/r/discussion/lw/a6m/si_wants_to_hire_a_remote_latex_guru/), remote editors, and [remote assistants](http://lesswrong.com/r/discussion/lw/aco/the_singularity_institute_is_hiring_virtual/). This shift provides numerous benefits, including (1) involving the broader community more directly in our work, (2) providing jobs for aspiring rationalists, and (3) freeing up our core staff to do the things that, due to accumulated rare expertise, only they can do.
* And of course much more than is listed here!

Finally, we’d like to recognize our **most active volunteers**in February 2012: Brian Rabkin, Cameron Taylor, Mitchell Owen, Gerard McCusker, Alex Richard, Andrew Homan, Vincent Vu, Gabriel Sztorc, Paul Gentemann, John Maxwell, and David Althaus. Thanks everyone! (And, our apologies if we forgot to name you!)

The post [Machine Intelligence Research Institute Progress Report, February 2012](https://intelligence.org/2012/03/03/singularity-institute-progress-report-february-2012/) appeared first on [Machine Intelligence Research Institute](https://intelligence.org).


2011-2012 Winter Fundraiser Completed

Thanks to our dedicated supporters, we met our goal for our 2011-2012 Winter Fundraiser. Thank you!

The fundraiser ran for 56 days, from December 27, 2011 to February 20, 2012.

We exceeded our $100K goal, raising a total of $143,048.84 from 101 individual donors.

Every donation that the Machine Intelligence Research Institute receives is powerful support for our [mission](https://intelligence.org/files/strategicplan2011.pdf) — ensuring that the creation of smarter-than-human intelligence ([superintelligence](http://www.nickbostrom.com/ethics/ai.html)) benefits human society. We welcome donors [contacting us](mailto:admin@intelligence.org) to learn more about our pursuit of this mission and our continued expansion.

Keep your eye on this blog for regular progress reports from our executive director.

The post [2011-2012 Winter Fundraiser Completed](https://intelligence.org/2012/02/20/2011-2012-winter-fundraiser-completed/) appeared first on [Machine Intelligence Research Institute](https://intelligence.org).


Past progress report: [December 2011](http://intelligence.org/blog/2012/01/16/singularity-institute-progress-report-december-2011/).

Here’s what the Machine Intelligence Research Institute did in January 2012:

* **Winter fundraiser**: We continued raising funds in January, but we still have about $30,000 left to go in our winter fundraiser before the deadline of February 20th. Please support our recent efforts toward greater transparency, efficiency, and productivity by [donating now](http://intelligence.org/donate)!
* **Strategic discussions**: In January we held a long and ongoing series of discussions concerning Machine Intelligence Research Institute strategy. Which scenarios are the most probable “desirable” futures for humanity, which ones can our species influence most significantly, and which ones should the Machine Intelligence Research Institute work to influence? Which tactical moves should the Machine Intelligence Research Institute make right now? How can our efforts best create synergies with other organizations focused on existential risks? These are complex questions, and in January, Machine Intelligence Research Institute staff members spent dozens of hours sharing their own evidence and arguments. (At one point, we also [called upon](http://lesswrong.com/lw/9l6/mathematicians_mathletes_the_singularity/) the expertise of more than a dozen elite mathematicians in our circle.) These discussions continue today, and our opinions on strategy appear to be more unified than they were at the beginning of the month. But there is more evidence to gather and more strategic analysis to be done.
* **Ongoing long-term projects**: Amy continued her preparations for Singularity Summit 2012. Michael Anissimov and others continued work on the Machine Intelligence Research Institute’s new website, which will feature loads of new content and a cleaner design. As part of our transparency efforts, Luke gave a [second Q&A](http://lesswrong.com/lw/980/singularity_institute_executive_director_qa_2/) about the Machine Intelligence Research Institute, an interview at [80,000 Hours](http://80000hours.org/blog/17-high-impact-interview-1-existential-risk-research-at-siai), and another interview at [Singularity 1 on 1](http://www.singularityweblog.com/luke-muehlhauser-on-singularity-1-on-1-superhuman-ai-is-coming-this-century/). Louie continued to work on improving our book-keeping and accounting practices. Anissimov finished thanking all donors who gave during 2011. (If you donated in 2011 and were *not* thanked, please contact michael@singularity.org!)
* **Articles**: Luke and Anna continued writing “Intelligence Explosion: Evidence and Import,” and Carl continued working with [Stuart Armstrong](http://www.fhi.ox.ac.uk/our_staff/research/stuart_armstrong) of FHI on “Arms Races and Intelligence Explosions.” Luke began adding non-English translations at *[Facing the Singularity](http://facingthesingularity.com/)*, and published [No God to Save Us](http://facingthesingularity.com/2012/no-god-to-save-us/) and [Value is Complex and Fragile](http://facingthesingularity.com/2012/value-is-complex-and-fragile/) there. Carl, with co-author Nick Bostrom, submitted a final version of “[How Hard is Artificial Intelligence?](http://www.nickbostrom.com/aievolution.pdf)” to the *Journal of Consciousness Studies*. For *Less Wrong*, Luke published [What Curiosity Looks Like](http://lesswrong.com/lw/96j/what_curiosity_looks_like/), [Can the Chain Still Hold You?](http://lesswrong.com/lw/99t/can_the_chain_still_hold_you/), [Leveling Up in Rationality](http://lesswrong.com/r/lesswrong/lw/9fy/leveling_up_in_rationality_a_personal_journey/), and [The Human’s Hidden Utility Function (Maybe)](http://lesswrong.com/lw/9jh/the_humans_hidden_utility_function_maybe/); Anna published [Urges vs. Goals](http://lesswrong.com/lw/8q8/urges_vs_goals_the_analogy_to_anticipation_and/). Eliezer continued work on his new Bayes Tutorial. Luke and Anna wrote a report on the workshops that followed Singularity Summit 2011, which should be published soon.
* **Rationality Group**: Anna continued to lead the development of a new rationality education organization, temporarily called “Rationality Group.” Per our [strategic plan](https://intelligence.org/files/strategicplan2011.pdf), we will launch this new organization soon, so that the Machine Intelligence Research Institute can focus on its efforts on activities related to AI risk. In January we made one trial-hire for the new organization, and [reached out to dozens](http://lesswrong.com/lw/9hb/position_design_and_write_rationality_curriculum/) of other potential team members. We also [published](http://lesswrong.com/lw/9hb/position_design_and_write_rationality_curriculum/) a draft of one rationality lesson as a sample (PowerPoint slides + booklet PDFs).
* **New team members**: [Kevin Fischer](http://angel.co/kevin-fischer) of GK International joined our board of directors. We also added several new [research associates](http://intelligence.org/aboutus/researchassociates): [Paul Christiano](http://ordinaryideas.wordpress.com/), [Tyrrell McAllister](http://www.linkedin.com/pub/tyrrell-mcallister/9/439/71), [János Kramar](http://www.linkedin.com/pub/janos-kramar/25/41b/72b), and Mihaly Barasz (at Google Switzerland). Luke hired an executive assistant, Denise Simard. Michael Vassar officially left his role as President to work for his new company, [Personalized Medicine](http://www.medicineispersonal.com/).
* **Meetings with advisors, supporters, and potential researchers:** As usual, various SI staff met or spoke with dozens of advisors, supporters, and collaborators about how to build the existential risk community, how to mitigate AI risk, how to improve the Machine Intelligence Research Institute’s effectiveness, and other topics. We also met with several potential researchers to gauge their interest and abilities.
* **Relaunched the Visiting Fellows program**: In January we relaunched our Visiting Fellows program. Instead of hosting many visiting fellows at once, we will now host only 1-2 fellows at a time, for a limited duration unique to each visiting fellow. Our visiting fellow for the last week of January was Princeton philosophy undergraduate [Jake Nebel](http://princeton.academia.edu/JakeNebel). If you’re interested, please apply to our Visiting Fellows program [here](http://intelligence.org/aboutus/opportunities/visiting-fellow).
* **Much more**: Launched a redesign of [HPMoR.com](http://hpmor.com/), continued work in the optimal philanthropy movement, continued work on our first annual report, and much more.

Finally, we’d like to recognize our **most active volunteers** in January 2012: Mitchell Owen, Brian Rabkin, Huon Wilson, David Althaus, Florent Berthet, Sergio Terrero, “Lightwave,” Emile Kroeger, and Giles Edkins. Thanks everyone! (And, our apologies if we forgot to name you!)

The post [Machine Intelligence Research Institute Progress Report, January 2012](https://intelligence.org/2012/02/05/singularity-institute-progress-report-january-2012/) appeared first on [Machine Intelligence Research Institute](https://intelligence.org).


“I think the Machine Intelligence Research Institute has some very smart people working on the most important mission on Earth, but… what exactly are they *doing* these days? I’m in the dark.”

There’s a good reason I hear this comment so often. We haven’t done a good job of communicating our progress to our supporters.

Since being appointed Executive Director of the Machine Intelligence Research Institute (SI) in November, I’ve been working to change that. I gave [two](http://lesswrong.com/r/discussion/lw/8s6/video_qa_with_singularity_institute_executive/) [Q&As](http://lesswrong.com/lw/980/singularity_institute_executive_director_qa_2/) about SI and explained our research program with a [list of open problems in AI risk research](http://lukeprog.com/SaveTheWorld.html). Now, I’d like to introduce our latest effort in transparency: **monthly progress reports**.

We begin with last month: December 2011. What did we do in December 2011?

(From this point on I’ll refer to myself as “Luke,” for clarity.)

* **Winter fundraiser**. We launched our winter fundraiser and have been contacting our supporters. The fundraiser has raised over $40k so far, though we still have $60k to go! (So, please [donate](http://intelligence.org/donate)!)
* **Singularity Summit 2012**. Our chief operating officer, Amy Willey, worked all month on preparations for Singularity Summit 2012, with much help from Luke. As a result we have now chosen a team of professionals with which we will take the Summit to “the next level,” and we’ve already confirmed several major speakers: Ray Kurzweil, Steven Pinker, Tyler Cowen, Temple Grandin, Peter Norvig, Robin Hanson, Peter Thiel, Melanie Mitchell, Vernor Vinge, and Carl Zimmer. We have also opened negotiations with many other speakers. This is a big improvement over our preparations for Singularity Summit 2011, which effectively began in May 2011, leaving us little time to capture certain speakers and develop certain kinds of media coverage. This much progress at such an early stage, in addition to a larger budget and greater professional assistance, will allow Singularity Summit 2012 to be a major leap forward for the event. Amy has also been developing arrangements for a *possible* European Singularity Summit in 2012.
* **Rationality Org**. As explained in our [strategic plan](https://intelligence.org/wp-content/uploads/strategicplan2011.pdf), we recognize the branding confusion produced by focusing on both AI risk research *and* rationality education, so we are preparing to spin off a separate rationality education organization so that the Machine Intelligence Research Institute can focus on AI risk research. Internally, we are calling the rationality education organization “Rationality Org.” Anna and Eliezer, with some help from Luke, did a lot of work developing plans for the future Rationality Org. We spent even *more* time developing the core rationality lessons, testing versions of them on different groups of people, and iterating the content. We expect the Rationality Org to launch late this year or early next year, and we expect it to not only [raise the sanity waterline](http://lesswrong.com/lw/1e/raising_the_sanity_waterline/) but also bring significant funding toward existential risk reduction.
* **New website design**. Our media director, Michael Anissimov, with much help from Luke, worked out the strategy and design of SI’s new website and worked with a designer to iterate the design several times. The designer is now programming the site.
* **New donor database**. In December, our Director of Development, Louie Helm, finished setting up our new donor database, including the custom code for automatically importing data from Paypal, Google Checkout, etc. This database gives us a much better view of who our supporters are, and allows us to more effectively thank them for their support. Anissimov wrote personal thank-you notes to hundreds of past donors.
* **Research articles**. Luke and Anna made continued progress on their overview article “Intelligence Explosion: Evidence and Import.” Carl continued work with [FHI](http://www.fhi.ox.ac.uk/)‘s Stuart Armstrong on their article “Arms Races and Intelligence Explosions,” and continued work with Nick Bostrom on their article “How Hard is Artificial Intelligence? Evolutionary Arguments and Selection Effects.”
* **Other articles**. Luke wrote a few articles for Less Wrong: [Hack Away at the Edges](http://lesswrong.com/lw/8ns/hack_away_at_the_edges/), [Why study the cognitive science of concepts](http://lesswrong.com/r/discussion/lw/8oy/why_study_the_cognitive_science_of_concepts/), and [So You Want to Save the World](http://lesswrong.com/lw/91c/so_you_want_to_save_the_world/). Eliezer made lots of progress on his new Bayes Theorem tutorial, including (outsourced) illustrations and much audience testing.
* **Eliezer’s book**. Eliezer finished the book proposal for his first book (already mostly written), *The Science of Changing Your Mind*. We have begun looking for good agents to represent the book.
* **Facing the Singularity**. Luke continued to develop his online book *Facing the Singularity*, a layman’s introduction to the Singularity, its consequences, and what we can do about it. The chapters he wrote in December 2011 were: [The Crazy Robot’s Rebellion](http://facingthesingularity.com/2011/the-crazy-robots-rebellion/), [Not Built to Think About AI](http://facingthesingularity.com/2011/not-built-to-think-about-ai/), [Playing Taboo with “Intelligence”](http://facingthesingularity.com/2011/playing-taboo-with-intelligence/), [Superstition in Retreat](http://facingthesingularity.com/2011/superstition-in-retreat/), [Plenty of Room Above Us](http://facingthesingularity.com/2011/plenty-of-room-above-us/), and [Don’t Flinch Away](http://facingthesingularity.com/2011/dont-flinch-away/).
* **Additional transparency efforts**. Anissimov and Luke began work on the design and content for an annual report. They also shot and produced [Luke’s video Q&A #1](http://lesswrong.com/r/discussion/lw/8s6/video_qa_with_singularity_institute_executive/).
* **Optimal philanthropy**. The optimal philanthropy movement (e.g. [Giving What We Can](http://www.givingwhatwecan.org/)) is growing exponentially. Carl and Anna did much collaboration and research with other members of the movement. Partly due to their work, the optimal philanthropy movement has great awareness of the case for [existential risk reduction as optimal philanthropy](http://www.existential-risk.org/concept.pdf), which should bring significant funding for existential risk reduction work in the coming years.
* **Meetings with advisors, supporters, and potential researchers**. During December 2011, various SI staff met or spoke with dozens of advisors, supporters, and collaborators about how to build the existential risk community, along with other topics. We also met with several potential researchers to gauge their interest and abilities.
* **Google Adwords upgrade**. For months, Louie and others have been tweaking the ads we get from $10k/month of Google Adwords donated to us by Google. By December 2011, our ads were so successful that we qualified for an upgrade, and are now receiving $40k/month of free advertising via Google Adwords.
* **Better financial management**. In December 2011 we began to train our new treasurer, long-time donor and friend of SI, Jesse Liptrap. This means that someone outside the organization is keeping a close watch on our finances. We also began work on improving our book-keeping and accounting practices, which will allow better budgeting, forecasting, and resource management.
* **Unpublished research**. As with most research institutes, most of our research does not end up in a published paper for 1-3 years, if ever, even though it informs our views on many things. Unpublished research in December 2011 included research on population ethics, brain-computer interfaces, optimal philanthropy, [technological forecasting](http://lesswrong.com/lw/9ao/longterm_technological_forecasting/), nuclear extinction risks, AI architectures, anthropics, decision theories, rationality training, Oracle AI, science productivity,  and more. SI’s research associates contributed to some of this research, including the Less Wrong discussion post [A model of UDT with a halting oracle](http://lesswrong.com/r/discussion/lw/8wc/decision_theory_with_halting_oracles/).
* **New board member**. [Quixey](http://www.quixey.com/) co-founder and CEO, [Tomer Kagan](http://www.forbes.com/pictures/lmf45kde/tomer-kagan-co-founder-and-ceo-quixey-28/), was added to SI’s board of directors. Tomer is a good friend and brings a wealth of business and management experience to our team.
* **Much more**. Of course, we worked on dozens of other, smaller projects. These include: updates to [IntelligenceExplosion.com](http://intelligenceexplosion.com/); development of contacts for Rationality Org; the organization of regular SI staff dinners, to promote coordination and friendship; speaking with donors at Peter Thiel’s “Fast Forward” party; development of a database of helpful volunteers and assistants; implementing [Olark](http://www.olark.com/) on our [donate page](http://intelligence.org/donate/); meetings with reporters from various media organizations; uploading old videos to Vimeo and YouTube; fixing errors and outdated content on our website; finishing our 2010 990 and sent it to Brandon Reinhart to add to his [financial examination](http://lesswrong.com/lw/5il/siai_an_examination/) of the Machine Intelligence Research Institute, preparing a [new template for SI research publications](http://commonsenseatheism.com/wp-content/uploads/2012/01/Dewey-Learning-what-to-value.pdf) (courtesy of research associate Daniel Dewey); and much more.

The post [Machine Intelligence Research Institute Progress Report, December 2011](https://intelligence.org/2012/01/16/singularity-institute-progress-report-december-2011/) appeared first on [Machine Intelligence Research Institute](https://intelligence.org).


**Machine Intelligence Research Institute Activities**

Bugmaster asks:

> …what does the SIAI actually do? You don’t submit your work to rigorous scrutiny by your peers in the field… you either aren’t doing any AGI research, or are keeping it so secret that no one knows about it… and you aren’t developing any practical applications of AI, either… So, what is it that you are actually working on, other than growing the SIAI itself ?
>
>

It’s a good question, and my own biggest concern right now. Donors would like to know: Where is the visible return on investment? How can I see that I’m buying existential risk reduction when I donate to the Machine Intelligence Research Institute?

SI has a problem, here, because it has done so much invisible work lately. Our researchers have done a ton of work that hasn’t been written up and published yet; Eliezer has been writing his rationality books that aren’t yet published; Anna and Eliezer have been developing a new rationality curriculum for the future “Rationality Org” that will be spun off from the Machine Intelligence Research Institute; Carl has been doing a lot of mostly invisible work in the optimal philanthropy community; and so on. I believe this is all valuable x-risk-reducing work, but of course not all of our supporters are willing to just take our word for it that we’re doing valuable work. Our supporters want to see tangible results, and all they see is the Singularity Summit, a few papers a year, some web pages and Less Wrong posts, and a couple rationality training camps. That’s good, but not good enough!

I agree with this concern, which is why I’m focused on doing things that happen to be both x-risk-reducing and visible.

First, we’ve been working on visible “meta” work that makes the Machine Intelligence Research Institute more transparent and effective in general: a strategic plan, a donor database (“visible” to donors in the form of thank-yous), a new website (forthcoming), and an annual report (forthcoming).

Second, we’re pushing to publish more research results this year. We have three chapters forthcoming in *The Singularity Hypothesis*, one chapter forthcoming in *The Cambridge Handbook of Artificial Intelligence*, one forthcoming article on the difficulty of AI, and several other articles and working papers we’re planning to publish in 2012. I’ve also begun writing the first comprehensive [outline of open problems in Singularity research](http://lukeprog.com/SaveTheWorld.html), so that interested researchers from around the world can participate in solving the world’s most important problems.

Third, there is visible rationality work forthcoming. One of Eliezer’s books is now being shopped to agents and publishers, and we’re field-testing different versions of rationality curriculum material for use in Less Wrong meetups and classes.

Fourth, we’re expanding the Singularity Summit brand, an important platform for spreading the memes of x-risk reduction and AI safety.

So my answer is to the question is: “Yes, visible return on investment has been a problem lately due to our choice of projects. Even before I was made Executive Director, it was one of my top concerns to help correct that situation, and this is still the case today.”

**What if?**

XiXiDu asks:

> What would SI do if it became apparent that AGI is at most 10 years away?
>
>

This would be a serious problem because by default, AGI will be extremely destructive, and we don’t yet know how to make AGI not be destructive.

What would we do if we thought AGI was at most 10 years away?

This depends on whether it’s apparent to a wider public that AGI is at most 10 years away, or a conclusion based only on a nonpublic analysis.

If it becomes apparent to a wide variety of folks that AGI is close, then it should be much easier to get people and support for Friendly AI work, so a big intensification of effort would be a good move. If the analysis that AGI is 10 years away leads to hundreds of well-staffed and well-funded AGI research programs and a rich public literature, then trying to outrace the rest with a Friendly AI project becomes much harder. After an intensified Friendly AI effort, one could try to build up knowledge in Friendly AI theory and practice that could be applied (somewhat less effectively) to systems not designed from the ground up for Friendliness. This knowledge could then be distributed widely to increase the odds of a project pulling through, calling in real Friendliness experts, etc. But in general, a widespread belief that AGI is only 10 years away would be a much hairier situation than the one we’re in now.

But if the basis for thinking AI was 10 years away was nonpublic (but nonetheless persuasive to supporters who have lots of resources), then it could be used to differentially attract support to a Friendly AI project, hopefully without provoking dozens of AGI teams to intensify their efforts. So if we had a convincing case that AGI was only 10 years away, we might not publicize this but would instead make the case to individual supporters that we needed to immediately intensify our efforts toward a theory of Friendly AI in a way that only much greater funding can allow.

**Budget**

MileyCyrus asks:

> What kind of budget would be required to solve the friendly AI problem?
>
>

Large research projects always come with large uncertainties concerning how difficult they will be, especially ones that require fundamental breakthroughs in mathematics and philosophy like Friendly AI does.

Even a small, 10-person team of top-level Friendly AI researchers taking academic-level salaries for a decade would require tens of millions of dollars. And even getting to the point where you can raise that kind of money requires a slow “ramping up” of researcher recruitment and output. We need enough money to attract the kinds of mathematicians who are also being recruited by hedge funds, Google, and the NSA, and have a funded “chair” for each of them such that they can be prepared to dedicate their careers to the problem. That part alone requires tens of millions of dollars for just a few researchers.

Other efforts like the Summit, Less Wrong, outreach work, and early publications cost money, and they work toward having the community and infrastructure required to start funding chairs for top-level mathematicians to be career Friendly AI researchers. This kind of work costs between $500,000 and $3 million per year, with more money per year of course producing more progress.

**Predictions**

Wix asks:

> How much do members’ predictions of when the singularity will happen differ within the Machine Intelligence Research Institute?
>
>

I asked some Machine Intelligence Research Institute staff members to answer a slightly different question, one pulled from the Future of Humanity Institute’s 2011 machine intelligence survey:

> Assuming no global catastrophe halts progress, by what year would you assign a 10% / 50% / 90% chance of the development of human-level machine intelligence? Feel free to answer ‘never’ if you believe such a milestone will never be reached.
>
>

In short, the survey participants’ median estimates (excepting 5 outliers) for 10% / 50% / 90% were:

> 2028 / 2050 / 2150
>
>

Here are five of the Machine Intelligence Research Institute’s staff members’ responses, names unattached, for the years by which they would assign a 10% / 50% / 90% chance of HLAI creation, conditioning on no global catastrophe halting scientific progress:

2025 / 2073 / 2168

2030 / 2060 / 2200

2027 / 2055 / 2160

2025 / 2045 / 2100

2040 / 2080 / 2200

Those are all the answers I had time to prepare in this round; I hope they are helpful!

The post [Q&A #2 with Luke Muehlhauser, Machine Intelligence Research Institute Executive Director](https://intelligence.org/2012/01/12/qa-2-with-luke-muehlhauser-singularity-institute-executive-director/) appeared first on [Machine Intelligence Research Institute](https://intelligence.org).


Help us raise $100,000 that will go towards funding a website redesign, [new research papers](http://intelligence.org/summary/), and staging a bigger and better Singularity Summit 2012 in San Francisco. This fundraiser ends January 20th, so please contribute now to help us achieve our goal!

[Donate now!](http://intelligence.org/donate)

$0

$25K

$50K

$75K

$100K

*This is your last chance to make a tax-deductible donation in 2011.*

### ARTIFICIAL INTELLIGENCE MORE RELEVANT THAN EVER

Recent books like *[Machine Ethics](http://www.amazon.com/Machine-Ethics-Michael-Anderson/dp/0521112354/)* from Cambridge University Press and *[Robot Ethics](http://www.amazon.com/Robot-Ethics-Implications-Intelligent-Autonomous/dp/0262016664/)* from MIT Press, along with the U.S. military-funded research that resulted in *[Governing Lethal Behavior in Autonomous Robots](http://www.amazon.com/Governing-Lethal-Behavior-Autonomous-Robots/dp/1420085948/)* show that the world is waking up to the challenges of building safe and ethical AI. But these projects focus on limited AI applications and fail to address the most important concern: how to ensure that *smarter-than-human* AI benefits humanity. The Machine Intelligence Research Institute has been working on that problem longer than anybody, a full decade before the Singularity landed on the [cover of *TIME* magazine](http://www.time.com/time/magazine/article/0,9171,2048299,00.html).

### ACCOMPLISHMENTS IN 2011

2011 was our biggest year yet. Since the year began, we have:

* Held our annual [Singularity Summit](http://www.singularitysummit.com/) in New York City, with more than 900 in attendance. Speakers included inventor and futurist Ray Kurzweil, economist Tyler Cowen, PayPal co-founder Peter Thiel, *Skeptic* publisher Michael Shermer, Mathematica and WolframAlpha creator Stephen Wolfram, neuroscientist Christof Koch, MIT physicist Max Tegmark, and famed *Jeopardy!* contestant Ken Jennings.
* Held a smaller Singularity Summit in Salt Lake City.
* Held a one-week Rationality Minicamp and a ten-week Rationality Boot Camp.
* Created the Research Associates program, which currently has 7 researchers coordinating with Singularity Institute.
* Published our [Intelligence Explosion FAQ](http://intelligence.org/ie-faq), [IntelligenceExplosion.com](http://intelligenceexplosion.com/), and [Friendly-AI.com](http://friendly-ai.com/).
* Wrote three chapters for Springer’s upcoming volume *[The Singularity Hypothesis](http://singularityhypothesis.blogspot.com/)*, along with four other research papers.
* Began work on a new, clearer website design with lots of new content, which should go live Q1 2012.
* Began [outlining open problems in Singularity research](http://lukeprog.com/SaveTheWorld.html) to help outside collaborators better understand our research priorities.

### FUTURE PLANS YOU CAN HELP SUPPORT

In the coming year, we plan to do the following:

* Hold our annual Singularity Summit, in San Francisco this year.
* Improve organizational transparency by creating a simpler, easier-to-use website that includes Machine Intelligence Research Institute planning and policy documents.
* Publish a document of open research problems in Singularity Research, to clarify the research space and encourage other researchers to contribute to our mission.
* Add additional skilled researchers to our Research Associates program.
* Publish a well-researched document making the case for existential risk reduction as optimal philanthropy.
* Diversify our funding sources by applying for targeted grants and advertising our affinity credit card program.

We appreciate your support for our high-impact work.

The post [2011 Machine Intelligence Research Institute Winter Fundraiser](https://intelligence.org/2011/12/27/2011-singularity-institute-winter-fundraiser/) appeared first on [Machine Intelligence Research Institute](https://intelligence.org).


![](https://intelligence.org/files/LukeFace.png)
#### Section One: Background and Core Ideas

Q1. [What is your personal background?](https://intelligence.org/feed/?paged=81#WhatIsYour)

Q2. [Why should we care about artificial intelligence?](https://intelligence.org/feed/?paged=81#WhyShouldWe)

Q3. [Why do you think smarter-than-human artificial intelligence is possible?](https://intelligence.org/feed/?paged=81#WhyDoYou)

Q4. [The mission of the Machine Intelligence Research Institute is to “to ensure that the creation of smarter-than- human intelligence benefits society.” How is your research contributing to that mission?](https://intelligence.org/feed/?paged=81#TheMissionOf)

Q5. [How does MIRI’s approach to making friendly AI differ from the concept of Asimov laws?](https://intelligence.org/feed/?paged=81#HowDoesSIAIs)

Q6. [Why is it necessary to make an AI that “€œwants the same things we want”?](https://intelligence.org/feed/?paged=81#WhyIsIt)

Q7. [If dangerous AI were to develop, why couldn’€™t we just “€œpull the plug”€?](https://intelligence.org/feed/?paged=81#IfDangerousAI)

Q8. [Why are you and the Machine Intelligence Research Institute focused on artificial intelligence instead of human intelligence enhancement or whole brain emulation?](https://intelligence.org/feed/?paged=81#WhyAreYou)

#### Section Two: Research Area Questions

Q9. [What research areas do you specifically investigate to develop “Friendliness content” for artificial intelligence?](https://intelligence.org/feed/?paged=81#WhatResearchAreas)

Q10. [Why are you and the Machine Intelligence Research Institute focused on artificial intelligence instead of human intelligence enhancement or whole brain emulation?](https://intelligence.org/feed/?paged=81#WhyAreYou)

Q11. [What is value extraction and how is it relevant to Friendly AI theory?](https://intelligence.org/feed/?paged=81#WhatIsValue)

Q12. [What is the psychology of concepts and how is it relevant to Friendly AI theory?](https://intelligence.org/feed/?paged=81#WhatIsThe)

Q13. [What is game theory and how is it relevant to Friendly AI theory?](https://intelligence.org/feed/?paged=81#WhatIsGame)

Q14. [What is metaethics and how is it relevant to Friendly AI theory?](https://intelligence.org/feed/?paged=81#WhatIsMetaethics)

Q15. [What is normativity and how is it relevant to Friendly AI theory?](https://intelligence.org/feed/?paged=81#WhatIsNormativity)

Q16. [What is machine ethics and how is it relevant to Friendly AI theory?](https://intelligence.org/feed/?paged=81#WhatIsMachine)

Q16. [What are some of those open problems in Friendly AI theory?](https://intelligence.org/feed/?paged=81#WhatAreSome)

Q17. [In late 2010 the Machine Intelligence Research Institute published “€œTimeless Decision Theory.” What is timeless decision theory and how is it relevant to Friendly AI theory?](https://intelligence.org/feed/?paged=81#InLate2010)

Q18. [What is reflective decision theory and how is it relevant to Friendly AI?](https://intelligence.org/feed/?paged=81#WhatIsReflective)

Q19. [Can you give a concrete example of how your research has made progress towards a solution on one or more open problems in Friendly AI?](https://intelligence.org/feed/?paged=81#CanYouGive)

Q20. [Why hasn’€™t Machine Intelligence Research Institute produced any concrete artificial intelligence code?](https://intelligence.org/feed/?paged=81#WhyHasntSingularity)

#### Section Three: Less Wrong and Rationality

Q21. [You originally came to Machine Intelligence Research Institute’s attention when you gained over 10,000 karma points very quickly on Less Wrong. For those who aren’€™t familiar with it, can you tell us what Less Wrong is and what its relationship is to the Machine Intelligence Research Institute?](https://intelligence.org/feed/?paged=81#YouOriginallyCame)

Q22. [What originally got you interested in rationality?](https://intelligence.org/feed/?paged=81#WhatOriginallyGot)

Q23. [You recently were a instructor at Rationality MiniCamp in Berkeley during the summer. Can you tell us a little about the Rationality camps, what people do there, what you taught, and so on?](https://intelligence.org/feed/?paged=81#YouRecentlyWere)

Q24. [Besides being a website, Less Wrong groups meet up around the world. If I were interested, where would I be able to get involved in one of those meetups, and what goes on at these meetups?](https://intelligence.org/feed/?paged=81#BesidesBeingA)

Q25. [The recently released Strategic Plan mentions intentions to “€œspin off rationality training to another organization so that the Machine Intelligence Research Institute can focus on Friendly AI research.” Can you tell us something about that?](https://intelligence.org/feed/?paged=81#TheRecentlyReleased)

#### Section Four: Machine Intelligence Research Institute Operations

Q26. [You and Louie Helm were hired on September 1st. How were the two of you hired?](https://intelligence.org/feed/?paged=81#YouAndLouie)

Q27. [What does Louie Helm do as Director of Development?](https://intelligence.org/feed/?paged=81#WhatDoesLouie)

Q28. [The Machine Intelligence Research Institute just raised $250,000 in our Summer Challenge Grant. What will those funds be spent on?](https://intelligence.org/feed/?paged=81#SingularityInstituteJust)

Q29: [Carl Shulman was hired not long before you and Louie. What is his role in the organization?](https://intelligence.org/feed/?paged=81#CarlShulmanWas)

Q30. [What sort of new researchers is the Machine Intelligence Research Institute looking for?](https://intelligence.org/feed/?paged=81#WhatSortOf)

**Michael Anissimov:** You were recently hired as a [full-time employee](http://intelligence.org/team#luke) by the Machine Intelligence Research Institute. What is your personal background?

**Luke Muehlhauser:** I studied psychology in university but quickly found that I learn better and faster as an autodidact. Since then, I’€™ve consumed many fields of science and philosophy, one at a time, as they were relevant to my interests. I’€™ve written [dozens of articles](http://lukeprog.com/writings.html) for [my blog](http://commonsenseatheism.com/) and for [Less Wrong](http://lesswrong.com/), and I host a [podcast](http://commonsenseatheism.com/?p=1911) on which I interview leading philosophers and scientists about their work. I also have an interest in the mathematics and cognitive science of [human rationality](http://lesswrong.com/r/lesswrong/lw/7e5/the_cognitive_science_of_rationality/), because I want the research I do to be arriving at plausibly correct answers, not just answers that make me feel good.

**Michael:** Why should we care about artificial intelligence?

**Luke:** Artificial intelligence is becoming a more powerful technology every year. We now have robots that do [original scientific research](http://commonsenseatheism.com/wp-content/uploads/2011/02/King-Rise-of-the-Robo-Scientists.pdf), and the U.S. military is developing systems for [autonomous battlefield robots](http://www.amazon.com/Governing-Lethal-Behavior-Autonomous-Robots/dp/1420085948/) that make their own decisions. Artificial intelligence will become even more important when it passes human levels of intelligence, at which points it will be able to do many things we care about better than we can — things like curing cancer and preventing disasters.

**Michael:** Why do you think smarter-than-human artificial intelligence is possible?

**Luke:** The first reason is scientific. Human intelligence is a product of information processing in a brain made of meat. But meat is not an ideal platform for intelligence; it’€™s just the first one that evolution happened to produce. Information processing on a faster, more durable, and more flexible platform like silicon should be able to surpass the abilities of an intelligence running on meat if we can figure out which information processing algorithms are required for intelligence –€” either by looking more closely at which algorithms the brain is using or by gaining new insights in mathematics.

The second reason is historical. Machines have already passed human ability in hundreds of particular tasks: [playing chess](http://commonsenseatheism.com/wp-content/uploads/2011/08/Campbell-Deep-Blue.pdf) or [Jeopardy](http://www.nytimes.com/2011/02/17/science/17jeopardy-watson.html?_r=3&ref=homepage&src=me&pagewanted=all), searching through large databanks, and in a recent advance, [reading road signs](http://yann.lecun.com/exdb/publis/pdf/sermanet-ijcnn-11.pdf). There is little reason to suspect this trend will stop, unless scientific progress in general stops.

**Michael:** The [mission](http://intelligence.org/summary/) of the Machine Intelligence Research Institute is to “to ensure that the creation of smarter-than-human intelligence benefits society.” How is your research contributing to that mission?

**Luke:** A smarter-than-human machine intelligence that benefits (rather than harms) society is called “[Friendly AI](http://intelligence.org/ie-faq#FriendlyAI).” My primary research focus is what we call the problem of “friendliness content.” What does it look like for an AI to be “friendly” or to “benefit” society? We all have ideas about what a good world looks like and what a bad world looks like, but when thinking about that in the context of AI you must be very precise, because an AI will only do exactly what it is programmed to do.

If we can figure out how to specify exactly what it would mean for an AI to be “friendly,” then the creation of Friendly AI could be the best thing that ever happened. An advanced artificial intelligence could do science better and faster than we can, and thereby cure cancer, cure diseases, allow for human immortality, prevent disasters, solve the problems of climate change, and allow us to extend our civilization to other planets. A Friendly AI could also discover better economic and political systems that improve conditions for everyone.

**Michael:** How does SIAI’s approach to making friendly AI differ from the concept of Asimov’s laws?

**Luke:** Asimov’s [Three Laws of Robotics](http://en.wikipedia.org/wiki/Three_Laws_of_Robotics) for governing robot behavior are [widely](http://commonsenseatheism.com/wp-content/uploads/2011/09/Anderson-Asimovs-three-laws-of-robotics-and-machine-metaethics.pdf) [considered](http://www.amazon.com/Moral-Machines-Teaching-Robots-Right/dp/0199737975/) to be inadequate for making sure that intelligent machines bring no harm to humans. In fact, Asimov used his stories to illustrate many of the ways in which those laws could lead to unintended consequences. SIAI’s approach is very different in that we don’t think that constraints on AI behavior will work in the long run. We need advanced AI to want the same things we want. If the AI wants something different than what we want, it will eventually find a way around whatever constraints we put on it, due to its vastly superior intelligence. But if we can make an AI want the same things we want, then it will be much more effective than we can be at bringing about the kind of world that we want – curing cancer and inventing immortality and so on.

**Michael:** Why is it necessary to make an AI that “wants the same things we want”€?

**Luke:** A powerful AI that wants something different than we do could be dangerous. For example, suppose the AI’€™s goal system is programmed to maximize pleasure. That sounds good at first, but if you tell a super-powerful AI to “maximize pleasure,” it might do something like (1) convert most of Earth’s resources into computing machinery, killing all humans in the process, so that it can (2) tile the solar system with as many small digital minds as possible, and (3) have those digital minds run a continuous cycle of the single most pleasurable experience possible. But of course, that’s not what we want! We don’t just value pleasure, we also value things like novelty and exploration. So we need to be very careful when we tell an AI precisely what it means to be “friendly.”

We must be careful not to anthropomorphize. A machine intelligence won’€™t necessarily have our “€˜common sense,”™ or our values, or even be sentient. When AI researchers talk about [machine intelligence](http://www.vetta.org/documents/Machine_Super_Intelligence.pdf), they only mean to talk about a machine that is good at achieving its goals — whatever they are –€” in a wide variety of environments. So if you tell an AI to maximize pleasure, it will do exactly that. It’s not going to stop halfway through and “€˜realize”€™ –€” like a human might –€” that maximizing pleasure isn’€™t what was intended, and that it should do something else.

**Michael:** If dangerous AI were to develop, why couldn’€™t we just “pull the plug”€?

**Luke:** We might not know that an AI was dangerous until it was too late. An AI with a certain level of intelligence is going to realize that in order to achieve its goals it needs to avoid being turned off, and so it would hide both the level of its own intelligence and its dangerousness.

But the bigger problem is that if some AI development team has already developed an AI that is intelligent enough to be dangerous, then other teams are only a few months or years behind them. You can’€™t just unplug every dangerous AI that comes along until the end of time. We’€™ll need to develop a Friendly AI that can ensure safety much better than humans can.

**Michael:** Why are you and Machine Intelligence Research Institute focused on artificial intelligence instead of human intelligence enhancement or whole brain emulation?

**Luke:** Human intelligence enhancement is important, and may be needed to solve some of the harder problems of Friendly AI. Whole brain emulation is a particularly revolutionary kind of human intelligence enhancement that, if invented, could allow us to upload human minds into computers, run them at speeds much faster than is possible with neurons, make backup copies, and allow immortality.

Many researchers think that artificial intelligence will arrive before whole brain emulation does, but predicting the timelines of future technology can be difficult. We are very interested in whole brain emulation, and in fact that was the subject of a presentation our researcher [Anna Salamon](http://annasalamon.com/) gave at a recent AI conference. One reason for us to focus on AI for the moment is that there are dozens of open problems in Friendly AI theory that we can make progress on right now without needing the vast computational resources required to make progress in whole brain emulation.

### Section Two: Research Area Questions

**Michael:** What research areas do you specifically investigate to develop “Friendliness content” for artificial intelligence?

**Luke:** One relevant area of research is [cognitive neuroscience](http://en.wikipedia.org/wiki/Cognitive_neuroscience), especially the subfields [neuroeconomics](http://en.wikipedia.org/wiki/Neuroeconomics) and [affective neuroscience](http://en.wikipedia.org/wiki/Affective_neuroscience).

Worlds are “good” or “bad” to us because of our values, and our values are stored in the brain’s neural networks. For decades, we’ve had to infer human values by observing human behavior because the brain has been a “black box” to us. But that can only take us so far because the environments in which we act are highly complex, and that makes it difficult to infer human values merely from behavior. Recently, new technologies like [fMRI](http://en.wikipedia.org/wiki/Functional_magnetic_resonance_imaging) and [TMS](http://en.wikipedia.org/wiki/Transcranial_magnetic_stimulation) and [optogenetics](http://en.wikipedia.org/wiki/Optogenetics) have allowed us to look into the black box and watch what the brain does. In fact, we’ve located the specific neurons that seem to encode the brain’s expected subjective value for the possible actions we are considering at a given moment. We’ve also learned a lot about the specific algorithms used by the brain to update how much we value certain things – in fact, they turned out to be a type of algorithm first discovered in computer science, called temporal difference reinforcement learning.

A second relevant area of research is [choice modeling](http://en.wikipedia.org/wiki/Choice_modelling) and [preference elicitation](http://en.wikipedia.org/wiki/Preference_elicitation). Economists use a variety of techniques, for example [Willingness to Pay](http://en.wikipedia.org/wiki/Willingness_to_pay) measures, to infer human preferences from human behavior. AI researchers also do this, usually for the purposes of designing a piece of software called a [decision support system](http://en.wikipedia.org/wiki/Decision_support_system). The human brain doesn’t seem to encode a coherent preference set, so we’€™ll need to use choice modeling and preference elicitation techniques to extract a coherent preference set from whatever it is that human brains actually do.

Other fields relevant to friendliness content theory include value extrapolation, the psychology of concepts, game theory, metaethics, normativity, and machine ethics.

**Michael:** What is value extrapolation and how is it relevant to Friendly AI theory?

**Luke:** Most philosophers talk about “[ideal preference theories](http://commonsenseatheism.com/wp-content/uploads/2011/09/Zimmerman-Why-Richard-Brandt-does-not-need-cognitive-psychotherapy.pdf),” but I prefer to call them “€˜value extrapolation algorithms.”€™ If we want to develop Friendly AI, we may not want to just scan human values from our brains and give those same values to an AI. I want to eat salty foods all day, but I kind of wish I didn’t want that, and I certainly don’t want an AI to feed me salty foods all day. Moreover, I would probably change my desires if I knew more and was more rational. I might learn things that would change what I want. And it’s unlikely that the human species has reached the end of moral development. So we don’t want to fix things in place by programming an AI with our current values. We want an AI to extrapolate our values so that it cares about what we would want if we knew more, were more rational, were more morally developed, and so on.

**Michael:** What is the psychology of concepts and how is it relevant to Friendly AI theory?

**Luke:** Some researchers think that part of the solution to the friendliness content problem will come from examining our intuitive concept of “€˜ought”€™ or “€˜good,”€™ and using this to inform our picture of what we think a good world would be like, and thus what the goal system of a super-powerful machine should be aimed toward. Philosophers have been examining our intuitive concepts of “€˜ought”€™ or “€˜good”€™ for centuries and made little progress, but perhaps new tools in psychology and neuroscience can help us do this conceptual analysis better than philosophers could from their armchairs.

On the other hand, psychological experiments have been undermining our classical theories about what concepts are, leading [some](http://www.amazon.com/Doing-without-Concepts-Edouard-Machery/dp/0199837562/) to go so far as to conclude that concepts do not exist in any useful sense. The results of that research program in psychology and philosophy could have profound implications for any approach to friendliness content that depends on an examination of our intuitive concepts of “€˜ought”€™ or “€˜good.”€™

**Michael:** What is game theory and how is it relevant to Friendly AI theory?

**Luke:** Game theory is a highly developed field of mathematics concerned with particular scenarios (“€˜games”€™) where an agent’s success depends on the choices of others. Its models and discoveries have been applied to business, economics, political science, biology, computer science, and philosophy.

Game theory is relevant to friendliness content because many of our values result from our need to make decisions in scenarios where our success depends on the choices of others. It may also be relevant to value extrapolation algorithms, as the extrapolation process is likely to change the ways in which our values and decisions interact with the values and decisions of others.

**Michael:** What is metaethics and how is it relevant to Friendly AI theory?

**Luke:** Philosophers often divide the field of ethics into three levels. [Applied ethics](http://www.iep.utm.edu/ethics/#H3) is the study of particular moral questions: How should we treat animals? Is lying ever acceptable? What responsibilities do corporations have concerning the environment? [Normative ethics](http://www.iep.utm.edu/ethics/#H2) considers the principles by which we make judgments in applied ethics. Do we make one judgment over another based on which action produces the most good? Or should we be following a list of rules and respecting certain rights? Perhaps we should advocate what we would all agree to behind a veil of ignorance that kept us from knowing what our lot in life will be?

[Metaethics](http://www.iep.utm.edu/ethics/#H1) goes one level deeper. What do terms like “€˜good”€™ and “€˜right”€™ even mean? Do moral facts exist, or is it all relative? Is there such a thing as moral progress? These questions are relevant to friendliness content because presumably, if moral facts exist, we would want an AI to respect them. Even if moral facts do not exist, our moral attitudes are part of what we value, and that is relevant to friendliness content theory.

**Michael:** What is normativity and how is it relevant to Friendly AI theory?

**Luke:** [Normativity](http://www.philosophyetc.net/2004/12/normativity.html) is about norms, and there are many kinds. Prudential norms concern what we ought to do to achieve our goals. Epistemic norms concern how we ought to pursue knowledge. Doxastic norms concern what we ought to believe. Moral norms concern how we ought to behave ethically. And so on.

A classic concern of normativity is the “[is-ought gap](http://en.wikipedia.org/wiki/Is%E2%80%93ought_problem).”™ Supposedly, you cannot reason from an “is”€™ statement to an “€˜ought”€™ statement. It doesn’t logically follow from “€œThe man in front of me is suffering” that “œI ought to help him.”€ Actually, it’s trivial to [bridge the is-ought gap](http://www.philosophyetc.net/2004/04/bridging-isought-gap.html) when it comes to prudential norms. “If you want Y, then you ought to do X,” is just another way of saying “Doing X will increase your chances of attaining Y.” The first sentence contains an “ought”€™ claim, but the second sentence reduces it away into a purely descriptive sentence about the natural world.

Some philosophers think that the “€˜is-ought gap”€™ can be bridged in the same way for epistemic and moral norms. Perhaps “you ought to believe X” just means “If you want true beliefs, then you ought to believe X,”€ which in turn can be reduced into the purely descriptive statement “€œBelieving X will increase your proportion of true beliefs.”€

But is there any other kind of normativity? Are there “€˜categorical”€™ oughts that do not depend on an “€œIf you want X” clause? Naturalists tend to deny this possibility, but perhaps categorical epistemic or moral oughts can be derived from the mathematics of game theory and decision theory, as naturalist Gary Drescher suggests in [*Good and Real*](http://www.amazon.com/Good-Real-Demystifying-Paradoxes-Bradford/dp/0262042339/). If so, it may be wise to make sure they are included in friendliness content theory, so that an AI can respect them.

**Michael:** What is machine ethics and how is it relevant to Friendly AI theory?

**Luke:** [Machine ethics](http://www.amazon.com/Machine-Ethics-Michael-Anderson/dp/0521112354/) is one of several names for the field that studies two major questions: (1) How can we get machines to behave ethically, and (2) which types of machines can be considered genuine moral agents (in the sense of having rights or moral worth like a human might)? Most of the work in the field so far is relevant only to “€˜narrow AI”€™ machines that are not nearly as intelligent as humans are, but two directions of research that may be useful for Friendly AI are [mechanized deontic logic](http://commonsenseatheism.com/wp-content/uploads/2011/02/Arkoudas-Toward-ethical-robots-via-mechanized-deontic-logic.pdf) and [computational metaethics](http://commonsenseatheism.com/wp-content/uploads/2011/03/Lokhorst-Computational-meta-ethics-toward-the-meta-ethical-robot.pdf).

Unfortunately, our understanding of Friendly AI –€” and of not-yet-invented AI technologies in general –€” is so primitive that we’re not even sure which fields will turn out to matter. It seems like cognitive neuroscience, preference elicitation, value extrapolation, game theory, and several other fields are relevant to Friendly AI theory, but it might turn out that as we come to understand Friendly AI better, we’€™ll learn that some research avenues are not relevant. But the only way we can learn that is to continue to make incremental progress in the areas of research that seem to be relevant.

**Michael:** What are some of those open problems in Friendly AI theory?

**Luke:** If we think just about about the issue of Friendliness content, some of the open questions are: How does the brain choose which few possible actions it will encode expected subjective value for? How does it combine absolute value and probability estimates to make those expected subjective value computations? Where is absolute value stored, and how is it encoded? How can we extract a coherent utility function or preference set from this activity in human brains? Which algorithms should we use to extrapolate these preferences, and why? When extrapolated, will the values of two different humans converge? Will the values of all humans converge? Would the values of all sentient beings converge? Will the details of human cognitive neuroscience matter much, or will such details get “˜washed out”€™ by the higher-level mathematical structure of value systems and game theory? How can these extrapolated values be implemented in the goal system of an AI?

Friendliness content is only one area of open problems in Friendly AI theory. There are many other questions. How can an agent make optimal decisions when it is capable of directly editing its own source code, including the source code of the decision mechanism? How can we get an AI to maintain a consistent utility function throughout updates to its ontology? How do we make an AI with preferences about the external world instead of its reward signal? How can we generalize the theory of machine induction –€” called [Solomonoff induction](http://www.vetta.org/documents/disSol.pdf) –€” so that it can use higher-order logics and reason correctly about observation selection effects? How can we approximate such ideal processes such that they are computable?

That’s a start, anyway. ![🙂](https://s.w.org/images/core/emoji/14.0.0/72x72/1f642.png)

**Michael:** In late 2010 the Machine Intelligence Research Institute published [“Timeless Decision Theory.”€](http://intelligence.org/upload/TDT-v01o.pdf) What is timeless decision theory and how is it relevant to Friendly AI theory?

**Luke:** Decision theory is the study of how to make optimal decisions. We value different things differently, and we are uncertain about which actions will bring about what we value. One of the problems not handled well by traditional decision theories like [Evidential decision theory](http://en.wikipedia.org/wiki/Evidential_decision_theory) (EDT) and [Causal decision theory](http://plato.stanford.edu/entries/decision-causal/) (CDT) is the problem of logical uncertainty –€” our uncertainty about mathematical and logical facts, for example what the nth decimal of pi is. One way to think about Timeless decision theory (TDT) is that it’s a step toward a decision theory that can handle logical uncertainty.

For an AI to be safe, its decision mechanism will have to be somewhat clear and mathematically testable for stability and safety. That probably means it will need to make decisions with decision theory, rather than through a relatively opaque neural nets mechanism. So we need to solve some fundamental problems in decision theory first, and logical uncertainty is one of the remaining fundamental problems in decision theory.

**Michael:** What is reflective decision theory and why is it necessary to Friendly AI?

**Luke:** Traditional decision theories cannot handle agents that can modify their own source code, including the source code for their decision mechanism. A reflective decision theory is one that can handle such a strongly self-modifying agent. Because an advanced AI will be intelligent enough to modify its own source code, we need to develop a reflective decision theory that will allow us to ensure that the AI will remain Friendly throughout the self-modification and self-improvement process.

**Michael:** Can you give a concrete example of how your research has made progress towards a solution on one or more open problems in Friendly AI?

**Luke:** I’ve only just begun working with the Machine Intelligence Research Institute, and making progress on open problems in Friendly AI theory is only one of the many things I do. My first contribution to friendliness content theory was to [summarize](http://lesswrong.com/lw/71x/a_crash_course_in_the_neuroscience_of_human/) some very recent advances in neuroeconomics that are relevant to the study of human values. I did that because other researchers in the field were not yet familiar with that material, and I think much of the work in friendliness content theory can be done collaboratively by a broad community of researchers if we are all well-informed.

These results from neuroeconomics appear to be relevant to friendliness content theory, though only time will tell. For example, we’ve learned that expected utility for human actions is encoded cardinally (not ordinally) in the brain, and thus avoids a limiting result from economics called [Arrow’s impossibility theorem](http://en.wikipedia.org/wiki/Arrow%27s_impossibility_theorem).

**Michael:** Why hasn’€™t the Machine Intelligence Research Institute produced any concrete artificial intelligence code?

**Luke:** This is a common confusion. Most of the open problems in Friendly AI theory are in math and philosophy, not in computer programming. Sometimes programmers approach us, offering to work on Friendly AI theory, and we reply: “€œWhat we need are mathematicians. Are you brilliant at math?”

As it turns out, the heroes who can save the world are not those with incredible strength or the power of flight. They are mathematicians.

### Section Three: Less Wrong and Rationality

**Michael:** You originally came to the Machine Intelligence Research Institute’€™s attention when you gained over 10,000 karma points very quickly on [Less Wrong](http://www.lesswrong.com/). For those who aren’€™t familiar with it, can you tell us what Less Wrong is and what its relationship is to the Machine Intelligence Research Institute?

**Luke:** Less Wrong is a group blog and community devoted to the study of rationality: How to get truer beliefs and make better decisions. the Machine Intelligence Research Institute’€™s co-founder Eliezer Yudkowsky originally wrote hundreds of articles about rationality for another blog, [Overcoming Bias](http://www.overcomingbias.com/), because he wanted to build a community of people that could think clearly about difficult problems like Friendly AI. Those articles were then used as the seed content for a new website, Less Wrong. I discovered Less Wrong because of my interest in rationality, and eventually started writing articles for the site –€” many of which became very popular.

**Michael:** What originally got you interested in rationality?

**Luke:** I was raised an enthusiastic evangelical Christian, and had a dramatic crisis of faith when I [learned](http://lesswrong.com/lw/7dy/a_rationalists_tale/) a few things about the historical Jesus, science, and philosophy. I was disturbed by how confidently I had believed something that was so thoroughly wrong, and I no longer trusted my intuitions. I wanted to avoid being so wrong again, so I studied the phenomena that allow human brains to be so mistaken — things like [confirmation bias](http://en.wikipedia.org/wiki/Confirmation_bias) and the [affect heuristic](http://lesswrong.com/lw/lg/the_affect_heuristic/). I also gained an interest in the mathematics of correct thinking, like [Bayesian updating](http://yudkowsky.net/rational/bayes) and [decision theory](http://wiki.lesswrong.com/wiki/Decision_theory).

**Michael:** You recently were a instructor at Rationality MiniCamp in Berkeley during the summer. Can you tell us a little about the MiniCamp, what people did there, what you taught, and so on?

**Luke:** [Anna](http://annasalamon.com/) and I put together the minicamp, a one-week camp full of classes and activities about rationality, social effectiveness, and existential risks. Over 20 people stayed in a large house in Berkeley, where we held the classes. Some of them came from as far away as Sweden and England.

Minicamp was a blast, mostly because the people were so great! We are still in contact, still learning and growing.

We taught things like how to update our beliefs using probability theory, how to use the principle of fungibility to better fulfill our goals, and how to use body language and fashion to improve some parts of our lives that math-heads sometimes neglect! We also taught classes on [optimal philanthropy](http://lesswrong.com/lw/3gj/efficient_charity_do_unto_others/) (how to get the most bang for your philanthropic buck) and [existential risks](http://www.existential-risk.org/) (risks that could cause human extinction).

**Michael:** Besides being a website, Less Wrong groups meet up around the world. If I were interested, where would I be able to get involved in one of those meetups, and what goes on at these meetups?

**Luke:** Because of [how sensitive humans are to context](http://lesswrong.com/lw/52g/the_good_news_of_situationist_psychology/), surrounding yourself with other people who are learning rationality and trying to improve themselves is one of the most powerful ways to improve yourself.

The easiest way to find a Less Wrong meetup near you is probably to check for the most recent [front-page](http://lesswrong.com/promoted/) post on Less Wrong with the title that begins “€œWeekly LW Meetups…”€ That post will list all the Less Wrong meetups happening that week.

Each Less Wrong meetup has different people and different activities. You can contact the meetup organizer for the meetup nearest you for more information.

**Michael:** The recently released [Strategic Plan](http://intelligene.org/blog/2011/08/26/singularity-institute-strategic-plan-2011/) mentions intentions to “€œspin off rationality training to another organization so that the Machine Intelligence Research Institute can focus on Friendly AI research.”€ Can you tell us something about that?

**Luke:** We believe that building a large community of rationality enthusiasts is crucial to the success of our mission. The Less Wrong rationality community has been an indispensable source of human and financial capital for the Machine Intelligence Research Institute. However, we understand that it’s confusing to be an organization devoted to two such apparently different fields: advanced artificial intelligence and human rationality. That’s why we are working toward launching a new organization devoted to rationality training. The Machine Intelligence Research Institute, then, will be more solely devoted to the safety of advanced artificial intelligence.

### Section Four: Machine Intelligence Research Institute Operations

**Michael:** You and Louie Helm were hired on September 1st. How were the two of you hired?

**Luke:** The Machine Intelligence Research Institute doesn’€™t hire someone unless they do quite a bit of volunteer work first. I first came to the Machine Intelligence Research Institute as a visiting fellow. During the next few months I co-organized and taught at the Rationality Minicamp, taught classes for the longer Rationality Boot Camp, wrote dozens of articles on metaethics and rationality for Less Wrong, wrote the [Intelligence Explosion FAQ](http://intelligence.org/ie-faq) and [IntelligenceExplosion.com](http://intelligenceexplosion.com/), led the writing of a strategic plan for the organization, and did many smaller tasks.

Louie Helm arrived in Berkeley not long after I did. As a past visiting fellow, Louie was the one who had suggested I apply to the visiting fellows program. Louie did some teaching for Rationality Boot Camp, helped me write the strategic plan, developed a donor database so that our contact with donors is more consistent, optimized the Machine Intelligence Research Institute’s finances, did lots of fundraising, and much more.

We both produced lots of value for the organization over those months as volunteers, so the Board hired us –€” me as a research fellow and Louie as Director of Development.

**Michael:** What does Louie Helm do as Director of Development?

**Luke:** We’€™re a small team, so we all do more than our title says, and Louie is no exception. Louie raises funds, communicates with donors, applies for grants, and so on. But he also launched the Research Associates program, coordinates the [Volunteer Network](http://www.singularityvolunteers.org/), helps organize the [Singularity Summit](http://www.singularitysummit.com/), seeks out potential new researchers, and more.

**Michael:** The Machine Intelligence Research Institute just raised $250,000 in our Summer Challenge Grant. What will those funds be spent on?

**Luke:** We were very pleased by the results of the summer challenge grant. No single person gave more than $25,000, so the grant succeeded because so many different people gave. More than 40 people gave $1,000 or more, which shows a high degree of trust from our core supporters.

It costs $368,000 annually to support our lean family of eight full-time staff members, four of whom are research fellows: Eliezer Yudkowsky, Anna Salamon, Carl Shulman, and myself. The money will also be used to run the [2011 Singularity Summit](http://www.singularitysummit.com/), though we expect that event to be cash-positive this year. We plan to redesign the singinst.org website so that it is easier to navigate and provides greater organizational transparency. And with enough funds after the Summit, we hope to hire additional researchers.

**Michael:** Carl Shulman was hired not long before you and Louie. What is his role in the organization?

**Luke:** Carl also did quite a lot of work for the Machine Intelligence Research Institute before being hired. He has written several papers and given a few talks, many of which you can read from our [publications](http://intelligence.org/research/) page. He continues to work on a variety of research projects, and collaborates closely with researchers at Oxfordâ€™s [Future of Humanity Institute](http://www.fhi.ox.ac.uk/).

**Michael:** What sort of new researchers is the Machine Intelligence Research Institute looking for?

**Luke:** Mathematicians, mostly. If you’€™re a brilliant math student and want to live and work in the Bay Area where you’ll be surrounded by smart, influential, altruistic people, please [apply here](http://intelligence.org/aboutus/opportunities/research-fellow).

The post [Interview with New MIRI Research Fellow Luke Muehlhauser](https://intelligence.org/2011/09/15/interview-with-new-singularity-institute-research-fellow-luke-muehlhuaser-september-2011/) appeared first on [Machine Intelligence Research Institute](https://intelligence.org).


2011 Summer Matching Challenge Success!

Thanks to the effort of our donors, the 2011 Summer Singularity Challenge has been met! All $125,000 contributed will be matched dollar for dollar by our matching backers, raising a total of $250,000 to fund the Machine Intelligence Research Institute’€™s operations. We reached our goal two days early, near midnight of August 29th.

On behalf of our staff, volunteers, and entire community, I want to personally thank everyone who donated. Your dollars make the difference.

Here’€™s to a better future for the human species.

The post [2011 Summer Matching Challenge Success!](https://intelligence.org/2011/09/01/2011-summer-matching-challenge-success/) appeared first on [Machine Intelligence Research Institute](https://intelligence.org).