November 2019 Newsletter

I'm happy to announce that Nate Soares and Ben Levinstein's “[Cheating Death in Damascus](https://intelligence.org/2017/03/18/new-paper-cheating-death-in-damascus/)” has been accepted for publication in *The Journal of Philosophy* (previously voted the [second-highest-quality](https://leiterreports.typepad.com/blog/2013/07/top-philosophy-journals-without-regard-to-area.html) journal in philosophy).
In other news, MIRI researcher Buck Shlegeris has written over 12,000 words on a variety of MIRI-relevant topics [in an EA Forum AMA](https://forum.effectivealtruism.org/posts/tDk57GhrdK54TWzPY/i-m-buck-shlegeris-i-do-research-and-outreach-at-miri-ama). (Example topics: [advice for software engineers](https://forum.effectivealtruism.org/posts/tDk57GhrdK54TWzPY/i-m-buck-shlegeris-i-do-research-and-outreach-at-miri-ama#X3EJqeWRX326HsQbK); [what alignment plans tend to look like](https://forum.effectivealtruism.org/posts/tDk57GhrdK54TWzPY/i-m-buck-shlegeris-i-do-research-and-outreach-at-miri-ama#RoXASZqvcYM6cqDSn); and [decision theory](https://forum.effectivealtruism.org/posts/tDk57GhrdK54TWzPY/i-m-buck-shlegeris-i-do-research-and-outreach-at-miri-ama#byH8abnt5RnPMunts).)

#### Other updates

* Abram Demski's [The Parable of Predict-O-Matic](https://www.lesswrong.com/posts/SwcyMEgLyd4C3Dern) is a great read: the predictor/optimizer issues it covers are deep, but I expect a fairly wide range of readers to enjoy it and get something out of it.
* Evan Hubinger's [Gradient Hacking](https://www.lesswrong.com/posts/uXH4r6MmKPedk8rMA) describes an important failure mode that hadn't previously been articulated.
* Vanessa Kosoy's [LessWrong shortform](https://www.lesswrong.com/posts/dPmmuaz9szk26BkmD#comments) has recently discussed some especially interesting topics related to her learning-theoretic agenda.
* Stuart Armstrong's [All I Know Is Goodhart](https://www.lesswrong.com/posts/uL74oQv5PsnotGzt7) constitutes nice conceptual progress on expected value maximizers that are aware of [Goodhart's law](https://www.lesswrong.com/posts/EbFABnst8LsidYs5Y/goodhart-taxonomy) and trying to avoid it.
* Reddy, Dragan, and Levine's paper on [modeling human intent](https://papers.nips.cc/paper/7419-where-do-you-think-youre-going-inferring-beliefs-about-dynamics-from-behavior.pdf) cites (of all things) [*Harry Potter and the Methods of Rationality*](http://www.hpmor.com/) as inspiration.

#### News and links

* [Artificial Intelligence Research Needs Responsible Publication Norms](https://www.lawfareblog.com/artificial-intelligence-research-needs-responsible-publication-norms): Crootof provides a good review of the issue on *Lawfare*.
* Stuart Russell's new book is out: *[Human Compatible: Artificial Intelligence and the Problem of Control](https://www.amazon.com/Human-Compatible-Artificial-Intelligence-Problem-ebook/dp/B07N5J5FTS)*([excerpt](https://spectrum.ieee.org/computing/software/many-experts-say-we-shouldnt-worry-about-superintelligent-ai-theyre-wrong)). Rohin Shah's [review](https://www.lesswrong.com/posts/nd692YfFGfZDh9Mwz/an-69-stuart-russell-s-new-book-on-why-we-need-to-replace) does an excellent job of contextualizing Russell's views within the larger AI safety ecosystem, and Rohin highlights the quote:

> The task is, fortunately, not the following: given a machine that possesses a high degree of intelligence, work out how to control it. If that were the task, we would be toast. A machine viewed as a black box, a fait accompli, might as well have arrived from outer space. And our chances of controlling a superintelligent entity from outer space are roughly zero. Similar arguments apply to methods of creating AI systems that guarantee we won’t understand how they work; these methods include whole-brain emulation — creating souped-up electronic copies of human brains — as well as methods based on simulated evolution of programs. I won’t say more about these proposals because they are so obviously a bad idea.
>
>
* Jacob Steinhardt releases an [AI Alignment Research Overview](https://www.lesswrong.com/posts/7GEviErBXcjJsbSeD/ai-alignment-research-overview-by-jacob-steinhardt).
* Patrick LaVictoire's [AlphaStar: Impressive for RL Progress, Not for AGI Progress](https://www.lesswrong.com/posts/SvhzEQkwFGNTy6CsN/alphastar-impressive-for-rl-progress-not-for-agi-progress) raises some important questions about how capable today's state-of-the-art systems are.

The post [November 2019 Newsletter](https://intelligence.org/2019/11/25/november-2019-newsletter/) appeared first on [Machine Intelligence Research Institute](https://intelligence.org).


October 2019 Newsletter

#### Updates

* Ben Pace summarizes [a second round of AI Alignment Writing Day posts](https://www.lesswrong.com/posts/DG7asvufKgaqEknKd/ai-alignment-writing-day-roundup-2).
* [The Zettelkasten Method](https://www.lesswrong.com/posts/NfdHG6oHBJ8Qxc26s): MIRI researcher Abram Demski describes a note-taking system that's had a large positive effect on his research productivity.
* Will MacAskill writes a detailed [critique of functional decision theory](https://www.lesswrong.com/posts/ySLYSsNeFL5CoAQzN/a-critique-of-functional-decision-theory); Abram Demski ([1](https://www.lesswrong.com/posts/ySLYSsNeFL5CoAQzN/a-critique-of-functional-decision-theory#y8zRwcpNeu2ZhM3yE), [2](https://www.lesswrong.com/posts/ySLYSsNeFL5CoAQzN/a-critique-of-functional-decision-theory#X4CLhPpf9zecdGxAq)) and [Matthew Graves](https://www.lesswrong.com/posts/ySLYSsNeFL5CoAQzN/a-critique-of-functional-decision-theory#mR9vHrQ7JhykiFswg) respond in the comments.

#### News and links

* Recent AI alignment posts: Evan Hubinger asks “[Are minimal circuits deceptive?](https://www.lesswrong.com/posts/fM5ZWGDbnjb7ThNKJ)”, Paul Christiano describes [the strategy-stealing assumption](https://www.lesswrong.com/posts/nRAMpjnb6Z4Qv3imF), and Wei Dai lists his [resolved confusions about Iterated Distillation and Amplification](https://www.lesswrong.com/posts/FdfzFcRvqLf4k5eoQ). See also Rohin Shah's [comparison](https://www.lesswrong.com/posts/cYduioQNeHALQAMre/what-are-the-differences-between-all-the-iterative-recursive#oecNhys3wpsrYKnBp) of recursive approaches to AI alignment.
* Also on LessWrong: A [Debate on Instrumental Convergence Between LeCun, Russell, Bengio, Zador, and More](https://www.lesswrong.com/posts/WxW6Gc6f2z3mzmqKs/debate-on-instrumental-convergence-between-lecun-russell).
* FHI's Ben Garfinkel and Allan Dafoe argue that conflicts between nations tend to exhibit “[offensive-then-defensive scaling](https://www.tandfonline.com/doi/abs/10.1080/01402390.2019.1631810)”.
* OpenAI releases a [follow-up report](https://openai.com/blog/gpt-2-6-month-follow-up/) on GPT-2, noting that several groups “have explicitly adopted similar staged release approaches” to OpenAI.
* NVIDIA Applied Deep Learning Research has trained a model that appears to [essentially replicate GPT-2](https://arxiv.org/abs/1909.08053), with 5.6x as many parameters, slightly better WikiText perplexity, and slightly worse LAMBADA accuracy. The group has elected to share their training and evaluation code, but not the model weights.
* OpenAI [fine-tunes GPT-2](https://openai.com/blog/fine-tuning-gpt-2/) for text continuation and summarization tasks that incorporate human feedback, noting, “Our motivation is to move safety techniques closer to the general task of ‘machines talking to humans,’ which we believe is key to extracting information about human values.”

The post [October 2019 Newsletter](https://intelligence.org/2019/10/25/october-2019-newsletter/) appeared first on [Machine Intelligence Research Institute](https://intelligence.org).


September 2019 Newsletter

#### Updates

* We ran a very successful MIRI Summer Fellows Program, which included a day where participants [publicly wrote up their thoughts](http://alignmentforum.org/s/YuTinYEzsyHmPoocw) on various AI safety topics. See Ben Pace’s first post in a series of [roundups](https://www.alignmentforum.org/posts/ZYGjDpGQaHvg8HLfw/ai-alignment-writing-day-roundup-1).
* A few highlights from the writing day: Adele Lopez's [Optimization Provenance](https://www.lesswrong.com/posts/Zj2PgP5A8vY2G3gYw/optimization-provenance); Daniel Kokotajlo's [Soft Takeoff Can Still Lead to Decisive Strategic Advantage](http://lesswrong.com/posts/PKy8NuNPknenkDY74/soft-takeoff-can-still-lead-to-decisive-strategic-advantage) and [The "Commitment Races" Problem](http://lesswrong.com/posts/brXr7PJ2W4Na2EW2q/the-commitment-races-problem); Evan Hubinger's [Towards a Mechanistic Understanding of Corrigibility](https://www.lesswrong.com/posts/BKM8uQS6QdJPZLqCr/towards-a-mechanistic-understanding-of-corrigibility); and John Wentworth's [Markets are Universal for Logical Induction](https://www.lesswrong.com/posts/WmNeCipNwg9CmGy3T/markets-are-universal-for-logical-induction) and [Embedded Agency via Abstraction](https://www.lesswrong.com/posts/hLFD6qSN9MmQxKjG5/embedded-agency-via-abstraction).
* New posts from MIRI staff and interns: Abram Demski's [Troll Bridge](https://www.lesswrong.com/posts/hpAbfXtqYC2BrpeiC/troll-bridge-5); Matthew Graves' [View on Factored Cognition](https://www.lesswrong.com/posts/J7Rnt8aJPH7MALkmq/vaniver-s-view-on-factored-cognition); Daniel Filan's [Verification and Transparency](https://www.lesswrong.com/posts/n3YRDJYCnQcDAw29G/verification-and-transparency); and Scott Garrabrant's [Intentional Bucket Errors](https://www.lesswrong.com/posts/ZCfwKjQhAiDogorj8/intentional-bucket-errors) and [Does Agent-like Behavior Imply Agent-like Architecture?](https://www.lesswrong.com/posts/osxNg6yBCJ4ur9hpi/does-agent-like-behavior-imply-agent-like-architecture)
* See also [a forum discussion](https://www.lesswrong.com/posts/aPwNaiSLjYP4XXZQW/ai-alignment-open-thread-august-2019#3TKtFmF3ZcQFgQNsA) on "proof-level guarantees" in AI safety.

#### News and links

* From Ben Cottier and Rohin Shah: [Clarifying Some Key Hypotheses in AI Alignment](https://www.lesswrong.com/posts/mJ5oNYnkYrd4sD5uE/clarifying-some-key-hypotheses-in-ai-alignment)
* [Classifying Specification Problems as Variants of Goodhart's Law](https://www.alignmentforum.org/posts/yXPT4nr4as7JvxLQa/classifying-specification-problems-as-variants-of-goodhart-s): Victoria Krakovna and Ramana Kumar relate DeepMind's SRA taxonomy to [mesa-optimizers](https://intelligence.org/learned-optimization/), [selection and control](https://www.alignmentforum.org/posts/ZDZmopKquzHYPRNxq/selection-vs-control), and Scott Garrabrant's Goodhart taxonomy. Also new from DeepMind: Ramana, Tom Everitt, and Marcus Hutter's [Designing Agent Incentives to Avoid Reward Tampering](https://medium.com/@deepmindsafetyresearch/designing-agent-incentives-to-avoid-reward-tampering-4380c1bb6cd).
* From OpenAI: [Testing Robustness Against Unforeseen Adversaries](https://openai.com/blog/testing-robustness/). 80,000 Hours also recently [interviewed](https://80000hours.org/podcast/episodes/paul-christiano-a-message-for-the-future/) OpenAI's Paul Christiano, with some additional material [on decision theory](https://www.lesswrong.com/posts/n6wajkE3Tpfn6sd5j/christiano-decision-theory-excerpt).
* From AI Impacts: [Evidence Against Current Methods Leading to Human-Level AI](https://aiimpacts.org/evidence-against-current-methods-leading-to-human-level-artificial-intelligence/) and [Ernie Davis on the Landscape of AI Risks](https://aiimpacts.org/ernie-davis-on-the-landscape-of-ai-risks/)
* From Wei Dai: [Problems in AI Alignment That Philosophers Could Potentially Contribute To](https://www.lesswrong.com/posts/rASeoR7iZ9Fokzh7L/problems-in-ai-alignment-that-philosophers-could-potentially)
* Richard Möhn has put together a calendar of [upcoming AI alignment events](https://www.lesswrong.com/posts/h8gypTEKcwqGsjjFT/predicted-ai-alignment-event-meeting-calendar).
* The Berkeley Existential Risk Initiative is seeking an [Operations Manager](http://existence.org/jobs/operations-lead).

The post [September 2019 Newsletter](https://intelligence.org/2019/09/30/september-2019-newsletter/) appeared first on [Machine Intelligence Research Institute](https://intelligence.org).


August 2019 Newsletter

#### Updates

* MIRI research associate Stuart Armstrong is offering $1000 for [good questions to ask an Oracle AI](https://www.lesswrong.com/posts/cSzaxcmeYW6z7cgtc/contest-usd1-000-for-good-questions-to-ask-to-an-oracle-ai).
* Recent AI safety posts from Stuart: [Indifference: Multiple Changes, Multiple Agents](https://www.alignmentforum.org/posts/XkuRKqXKAaMySbXCN/indifference-multiple-changes-multiple-agents); [Intertheoretic Utility Comparison: Examples](https://www.alignmentforum.org/posts/5bd75cc58225bf06703753ef/intertheoretic-utility-comparison-examples); [Normalising Utility as Willingness to Pay](https://www.lesswrong.com/posts/qudmaMyRuQk2pHxtj/normalising-utility-as-willingness-to-pay); and [Partial Preferences Revisited](https://www.alignmentforum.org/posts/BydQtwfN97pFwEWtW/toy-model-piece-1-partial-preferences-revisited).
* MIRI researcher Buck Shlegeris has put together a quick and informal [AI safety reading list](https://docs.google.com/document/d/1LvmP6OOcGSRsy1jAWC3Gg5plbvHwH642QUjddL-KKh0/edit).
* [There's No Fire Alarm for AGI](https://intelligence.org/2017/10/13/fire-alarm/) reports on a researcher's January 2017 prediction that “in the next two years, we will not get 80, 90%” on [Winograd schemas](https://en.wikipedia.org/wiki/Winograd_Schema_Challenge), an NLP test. Although this prediction was correct, researchers at [Microsoft](https://arxiv.org/abs/1901.11504), [Carnegie Mellon and Google Brain](https://arxiv.org/abs/1906.08237), and [Facebook](https://arxiv.org/abs/1907.11692) have now (2.5 years later) achieved Winograd scores of [89.0 and 90.4](https://gluebenchmark.com/leaderboard/).
* Ortega et al.'s “[Meta-Learning of Sequential Strategies](https://arxiv.org/abs/1905.03030)” includes a discussion of mesa-optimization, independent of Hubinger et al.'s “[Risks from Learned Optimization in Advanced Machine Learning Systems](https://intelligence.org/learned-optimization/),” under the heading of “spontaneous meta-learning.”

#### News and links

* Wei Dai outlines [forum participation as a research strategy](https://www.lesswrong.com/posts/rBkZvbGDQZhEymReM/forum-participation-as-a-research-strategy).
* On a related note, the posts on the AI Alignment Forum this month were very good — I'll spotlight them all this time around. Dai wrote on [the purposes of decision theory research](https://www.lesswrong.com/posts/JSjagTDGdz2y6nNE3/on-the-purposes-of-decision-theory-research); Shah on [learning biases and rewards simultaneously](https://www.alignmentforum.org/posts/xxnPxELC4jLKaFKqG/learning-biases-and-rewards-simultaneously); Kovarik on [AI safety debate and its applications](https://www.lesswrong.com/posts/5Kv2qNfRyXXihNrx2/ai-safety-debate-and-its-applications#comments); Steiner on [the Armstrong agenda](https://www.lesswrong.com/posts/GHNokcgERpLJwJnLW/some-comments-on-stuart-armstrong-s-research-agenda-v0-9) and [the intentional stance](https://www.lesswrong.com/posts/NvqGmLBCtvQxfMs9m/the-artificial-intentional-stance); Trazzi on [manipulative AI](https://www.lesswrong.com/posts/EpdXLNXyL4EYLFwF8/an-increasingly-manipulative-newsfeed); Cohen on [IRL](https://www.alignmentforum.org/posts/kahBLu32sZAuAZbER/irl-in-general-environments) and [imitation](https://www.alignmentforum.org/posts/LTFaD96D9kWuTibWr/just-imitate-humans); and Manheim on optimizing and Goodhart effects ([1](https://www.lesswrong.com/posts/2neeoZ7idRbZf4eNC/re-introducing-selection-vs-control-for-optimization), [2](https://www.lesswrong.com/posts/BEMvcaeixt3uEqyBk/what-does-optimization-mean-again-optimizing-and-goodhart), [3](https://www.lesswrong.com/posts/zdeYiQgwYRs2bEmCK/applying-overoptimization-to-selection-vs-control-optimizing)).
* Jade Leung discusses [AI governance](https://futureoflife.org/2019/07/22/on-the-governance-of-ai-with-jade-leung/) on the AI Alignment Podcast.
* CMU and Facebook researchers' [Pluribus](https://www.technologyreview.com/s/613943/facebooks-new-poker-playing-ai-could-wreck-the-online-poker-industryso-its-not-being/) program beats human poker professionals — using only [$144](https://science.sciencemag.org/content/early/2019/07/10/science.aay2400) in compute. The developers also [choose not to release the code](https://www.lesswrong.com/posts/6qtq6KDvj86DXqfp6/let-s-read-superhuman-ai-for-multiplayer-poker): “Because poker is played commercially, the risk associated with releasing the code outweighs the benefits.”
* [Microsoft invests $1 billion in OpenAI](https://openai.com/blog/microsoft/). From Microsoft's [press release](https://news.microsoft.com/2019/07/22/openai-forms-exclusive-computing-partnership-with-microsoft-to-build-new-azure-ai-supercomputing-technologies/): “Through this partnership, the companies will accelerate breakthroughs in AI and power OpenAI’s efforts to create artificial general intelligence (AGI).” OpenAI has also released a paper on “[The Role of Cooperation in Responsible AI Development](https://openai.com/blog/cooperation-on-safety/).”
* Ought has a new preferred [introduction to their work](https://ought.org/presentations/delegating-cognitive-work-2019-06). See also Paul Christiano's [Ought: Why it Matters and Ways to Help](https://www.alignmentforum.org/posts/cpewqG3MjnKJpCr7E/ought-why-it-matters-and-ways-to-help).
* FHI has [11 open research positions](https://www.fhi.ox.ac.uk/researcher-positions/); applications are due by Aug. 16. You can also apply to CSER's [AGI risk research associate position](https://www.jobs.cam.ac.uk/job/22457/) through Aug. 26.

The post [August 2019 Newsletter](https://intelligence.org/2019/08/06/august-2019-newsletter/) appeared first on [Machine Intelligence Research Institute](https://intelligence.org).


July 2019 Newsletter

Hubinger et al.'s “[Risks from Learned Optimization in Advanced Machine Learning Systems](https://intelligence.org/learned-optimization/)”, one of our new core resources on the alignment problem, is now available on [arXiv](https://arxiv.org/abs/1906.01820), the [AI Alignment Forum](https://www.alignmentforum.org/s/r9tYkB2a8Fp4DN8yB), and [LessWrong](https://www.lesswrong.com/s/r9tYkB2a8Fp4DN8yB).
In other news, we received an Ethereum donation worth $230,910 from Vitalik Buterin — the inventor and co-founder of Ethereum, and now our third-largest all-time supporter!

Also worth highlighting, from the Open Philanthropy Project's Claire Zabel and Luke Muehlhauser: [there's a pressing need for security professionals in AI safety and biosecurity](https://forum.effectivealtruism.org/posts/ZJiCfwTy5dC4CoxqA/information-security-careers-for-gcr-reduction).

> It’s more likely than not that within 10 years, there will be dozens of GCR-focused roles in information security, and some organizations are already looking for candidates that fit their needs (and would hire them now, if they found them).
>
>
> It’s plausible that some people focused on high-impact careers (as many effective altruists are) would be well-suited to helping meet this need by gaining infosec expertise and experience and then moving into work at the relevant organizations.
>
>

#### Other updates

* [Mesa Optimization: What It Is, And Why We Should Care](https://www.lesswrong.com/posts/XWPJfgBymBbL3jdFd/an-58-mesa-optimization-what-it-is-and-why-we-should-care) — Rohin Shah's consistently excellent Alignment Newsletter discusses “Risks from Learned Optimization…” and other recent AI safety work.
* MIRI Research Associate Stuart Armstrong releases his [Research Agenda v0.9: Synthesising a Human's Preferences into a Utility Function](https://www.alignmentforum.org/posts/CSEdLLEkap2pubjof/research-agenda-v0-9-synthesising-a-human-s-preferences-intoerences-into).
* OpenAI and MIRI staff [help talk Munich student Connor Leahy out of releasing](https://medium.com/@NPCollapse/the-hacker-learns-to-trust-62f3c1490f51) an attempted replication of OpenAI's [GPT-2](https://openai.com/blog/better-language-models/) model. ([LessWrong discussion.](https://www.lesswrong.com/posts/36fxiKdEqswkedHyG/the-hacker-learns-to-trust-1)) Although Leahy's replication attempt wasn't successful, write-ups like his suggest that OpenAI's careful discussion surrounding GPT-2 is continuing to prompt good reassessments of publishing norms within ML. Quoting from Leahy's postmortem:

> Sometime in the future we will have reached a point where the consequences of our research are beyond what we can discover in a one-week evaluation cycle. And given my recent experiences with GPT2, we might already be there. The more complex and powerful our technology becomes, the more time we should be willing to spend in evaluating its consequences. And if we have doubts about safety, we should default to caution.
>
>
>  We tend to live in an ever accelerating world. Both the industrial and academic R&D cycles have grown only faster over the decades. Everyone wants “the next big thing” as fast as possible. And with the way our culture is now, it can be hard to resist the pressures to adapt to this accelerating pace. Your career can depend on being the first to publish a result, as can your market share.
>
>
>  We as a community and society need to combat this trend, and create a healthy cultural environment that allows researchers to *take their time*. They shouldn’t have to fear repercussions or ridicule for delaying release. Postponing a release because of added evaluation should be the norm rather than the exception. We need to make it commonly accepted that we as a community respect others’ safety concerns and don’t penalize them for having such concerns, *even if they ultimately turn out to be wrong*. If we don’t do this, it will be a race to the bottom in terms of safety precautions.
>
>
* From Abram Demski: [Selection vs. Control](https://www.alignmentforum.org/posts/ZDZmopKquzHYPRNxq/selection-vs-control); [Does Bayes Beat Goodhart?](https://www.lesswrong.com/posts/YJq6R9Wgk5Atjx54D/does-bayes-beat-goodhart); and [Conceptual Problems with Updateless Decision Theory and Policy Selection](https://www.alignmentforum.org/posts/9sYzoRnmqmxZm4Whf/conceptual-problems-with-udt-and-policy-selection)
* *Vox*'s *Future Perfect* Podcast [interviews Jaan Tallinn](https://www.vox.com/future-perfect/2019/6/26/18629806/artificial-intelligence-human-extinction-podcast-jaan-tallinn) and discusses MIRI's role in originating and propagating AI safety memes.
* *The AI Does Not Hate You*, journalist Tom Chivers' well-researched book about the rationality community and AI risk, [is out in the UK](https://www.amazon.co.uk/Does-Not-Hate-You-Rationalists/dp/1474608779).

#### News and links

* Other recent AI safety write-ups: David Krueger's [Let's Talk About “Convergent Rationality”](https://www.alignmentforum.org/posts/pLZ3bdeng4u5W8Yft/let-s-talk-about-convergent-rationality-1); Paul Christiano's [Aligning a Toy Model of Optimization](https://www.lesswrong.com/posts/H5gXpFtg93qDMZ6Xn/aligning-a-toy-model-of-optimization); and Owain Evans, William Saunders, and Andreas Stuhlmüller's [Machine Learning Projects on Iterated Distillation and Amplification](https://www.alignmentforum.org/posts/Y9xD78kufNsF7wL6f/machine-learning-projects-on-ida)
* From DeepMind: Vishal Maini puts together an [AI reading list](https://medium.com/machine-learning-for-humans/ai-reading-list-c4753afd97a), Victoria Krakovna [recaps the ICLR Safe ML workshop](https://vkrakovna.wordpress.com/2019/06/18/iclr-safe-ml-workshop-report/), and Pushmeet Kohli [discusses AI safety on the 80,000 Hours Podcast](https://80000hours.org/podcast/episodes/pushmeet-kohli-deepmind-safety-research/).
* The EA Foundation is awarding grants for “efforts to reduce risks of astronomical suffering (s-risks) from advanced artificial intelligence”; [apply by Aug. 11](https://forum.effectivealtruism.org/posts/BmBpZjNJbZibLsYnu/first-application-round-of-the-eaf-fund).
* Additionally, if you're a young AI safety researcher (with a PhD) based at a European university or nonprofit, you may want to apply for [~$60,000 in funding](https://www.bosch-ai.com/young-researcher-award/) from the Bosch Center for AI.

The post [July 2019 Newsletter](https://intelligence.org/2019/07/19/july-2019-newsletter/) appeared first on [Machine Intelligence Research Institute](https://intelligence.org).


[![Risks from Learned Optimization in Advanced Machine Learning Systems](https://intelligence.org/wp-content/uploads/2019/06/inneroptimization.png)](https://arxiv.org/abs/1906.01820)Evan Hubinger, Chris van Merwijk, Vladimir Mikulik, Joar Skalse, and Scott Garrabrant have a new paper out: “**[Risks from learned optimization in advanced machine learning systems](https://arxiv.org/abs/1906.01820)**.”

The paper’s abstract:

> We analyze the type of learned optimization that occurs when a learned model (such as a neural network) is itself an optimizer—a situation we refer to as *mesa-optimization*, a neologism we introduce in this paper.
>
>
> We believe that the possibility of mesa-optimization raises two important questions for the safety and transparency of advanced machine learning systems. First, under what circumstances will learned models be optimizers, including when they should not be? Second, when a learned model is an optimizer, what will its objective be—how will it differ from the loss function it was trained under—and how can it be aligned? In this paper, we provide an in-depth analysis of these two primary questions and provide an overview of topics for future research.
>
>

The critical distinction presented in the paper is between what an AI system is optimized to do (its *base objective*) and what it actually ends up optimizing for (its *mesa-objective*), if it optimizes for anything at all. The authors are interested in when ML models will end up optimizing for something, as well as how the objective an ML model ends up optimizing for compares to the objective it was selected to achieve.

The distinction between the objective a system is selected to achieve and the objective it actually optimizes for isn’t new. Eliezer Yudkowsky has previously raised similar concerns in his discussion of [optimization daemons](https://arbital.com/p/daemons/), and Paul Christiano has discussed such concerns in “[What failure looks like](https://www.alignmentforum.org/posts/HBxe6wdjxK239zajf/more-realistic-tales-of-doom).”

The paper’s contents have also been released this week as a sequence on the [AI Alignment Forum](https://www.alignmentforum.org/s/r9tYkB2a8Fp4DN8yB), cross-posted to [LessWrong](https://www.lesswrong.com/s/r9tYkB2a8Fp4DN8yB). As the authors note there:

> We believe that this sequence presents the most thorough analysis of these questions that has been conducted to date. In particular, we plan to present not only an introduction to the basic concerns surrounding mesa-optimizers, but also an analysis of the particular aspects of an AI system that we believe are likely to make the problems related to mesa-optimization relatively easier or harder to solve. By providing a framework for understanding the degree to which different AI systems are likely to be robust to misaligned mesa-optimization, we hope to start a discussion about the best ways of structuring machine learning systems to solve these problems.
>
>
> Furthermore, in the fourth post we will provide what we think is the most detailed analysis yet of a problem we refer as *deceptive alignment* which we posit may present one of the largest—though not necessarily insurmountable—current obstacles to producing safe advanced machine learning systems using techniques similar to modern machine learning.
>
>

#### Sign up to get updates on new MIRI technical results

*Get notified every time a new technical paper is published.*

*
*

×

The post [New paper: “Risks from learned optimization”](https://intelligence.org/2019/06/07/new-paper-learned-optimization/) appeared first on [Machine Intelligence Research Institute](https://intelligence.org).


June 2019 Newsletter

Evan Hubinger, Chris van Merwijk, Vladimir Mikulik, Joar Skalse, and Scott Garrabrant have released the first two (of five) posts on “[mesa-optimization](https://www.alignmentforum.org/s/r9tYkB2a8Fp4DN8yB)”:

> The goal of this sequence is to analyze the type of learned optimization that occurs when a learned model (such as a neural network) is itself an optimizer—a situation we refer to as *mesa-optimization*.
>
>
> We believe that the possibility of mesa-optimization raises two important questions for the safety and transparency of advanced machine learning systems. First, under what circumstances will learned models be optimizers, including when they should not be? Second, when a learned model is an optimizer, what will its objective be—how will it differ from the loss function it was trained under—and how can it be aligned?
>
>

The sequence begins with [Risks from Learned Optimization: Introduction](https://www.alignmentforum.org/s/r9tYkB2a8Fp4DN8yB/p/FkgsxrGf3QxhfLWHG) and continues with [Conditions for Mesa-Optimization](https://alignmentforum.org/s/r9tYkB2a8Fp4DN8yB/p/q2rCMHNXazALgQpGH). ([LessWrong mirror.](https://www.lesswrong.com/s/r9tYkB2a8Fp4DN8yB/p/FkgsxrGf3QxhfLWHG))

#### Other updates

* New research posts: [Nash Equilibria Can Be Arbitrarily Bad](https://www.lesswrong.com/posts/jz5QoizH8HkQwWZ9Q/nash-equilibriums-can-be-arbitrarily-bad); [Self-Confirming Predictions Can Be Arbitrarily Bad](https://www.lesswrong.com/posts/KoEY9CjrKe93ErYhd/self-confirming-predictions-can-be-arbitrarily-bad); [And the AI Would Have Got Away With It Too, If…](https://www.lesswrong.com/posts/92J4zJHkqmXTduxzY/and-the-ai-would-have-got-away-with-it-too-if); [Uncertainty Versus Fuzziness Versus Extrapolation Desiderata](https://www.alignmentforum.org/posts/QJwnPRBBvgaeFeiLR/uncertainty-versus-fuzziness-versus-extrapolation-desiderata)
* We've released our [annual review for 2018](https://intelligence.org/2019/05/31/2018-in-review/).
* Applications are open for [two AI safety events](https://forum.effectivealtruism.org/posts/QmfchinDKEcHG5uz5/two-ai-safety-events-at-ea-hotel-in-august) at the EA Hotel in Blackpool, England: the Learning-By-Doing AI Safety Workshop (Aug. 16-19), and the Technical AI Safety Unconference (Aug. 22-25).
* [A discussion of takeoff speed](https://www.lesswrong.com/posts/PzAnWgqvfESgQEvdg/any-rebuttals-of-christiano-and-ai-impacts-on-takeoff-speeds), including some very incomplete and high-level MIRI comments.

#### News and links

* Other recent AI safety posts: Tom Sittler's [A Shift in Arguments for AI Risk](https://fragile-credences.github.io/prioritising-ai/) and Wei Dai's [“UDT2” and “against UD+ASSA”](https://www.alignmentforum.org/posts/zd2DrbHApWypJD2Rz/udt2-and-against-ud-assa).
* Talks from the SafeML ICLR workshop are now [available online](https://sites.google.com/view/safeml-iclr2019/schedule).
* [From OpenAI](https://openai.com/blog/better-language-models/#update): “We’re implementing two mechanisms to responsibly publish GPT-2 and hopefully future releases: staged release and partnership-based sharing.”
* FHI's Jade Leung [argues](https://forum.effectivealtruism.org/posts/fniRhiPYw8b6FETsn/jade-leung-why-companies-should-be-leading-on-ai-governance) that “states are ill-equipped to lead at the formative stages of an AI governance regime,” and that “private AI labs are best-placed to lead on AI governance”.

The post [June 2019 Newsletter](https://intelligence.org/2019/06/01/june-2019-newsletter/) appeared first on [Machine Intelligence Research Institute](https://intelligence.org).


2018 in review

Our primary focus at MIRI in 2018 was twofold: research—as always!—and growth.

Thanks to the [incredible support](https://intelligence.org/2018/11/28/2017-in-review/#4) we received from donors the previous year, in 2018 we were able to aggressively pursue the plans detailed in our [2017 fundraiser post](https://intelligence.org/2017/12/01/miris-2017-fundraiser/). The most notable goal we set was to “grow big and grow fast,” as [our new research directions](https://intelligence.org/2018/11/22/2018-update-our-new-research-directions/#section1) benefit a lot more from a larger team, and require skills that are a lot easier to hire for. To that end, we set a target of adding 10 new research staff by the end of 2019.

2018 therefore saw us accelerate the work we started in 2017, investing more in recruitment and shoring up the foundations needed for our ongoing growth. Since our 2017 fundraiser post, we’re up 3 new research staff, including noted Haskell developer [Edward Kmett](https://intelligence.org/2018/11/28/miris-newest-recruit-edward-kmett/). I now think that we’re most likely to hit 6–8 hires by the end of 2019, though hitting 9–10 still seems quite possible to me, as we are still engaging with many promising candidates, and continue to meet more.

Overall, 2018 was a great year for MIRI. Our research continued apace, and our recruitment efforts increasingly paid out dividends.

Below I’ll elaborate on our:

* [research progress and outputs](https://intelligence.org/feed/?paged=12#2018-research),
* [research program support activities](https://intelligence.org/feed/?paged=12#2018-research-program-support), including more details on our recruitment efforts,
* [outreach related activities](https://intelligence.org/feed/?paged=12#2018-outreach-and-exposition), and
* [fundraising and spending](https://intelligence.org/feed/?paged=12#2018-finances).

### 2018 Research

Our [2018 update](https://intelligence.org/2018/11/22/2018-update-our-new-research-directions/) discussed the new research directions we’re pursuing, and the nondisclosure-by-default policy we’ve adopted for our research overall. As described in the post, these new directions aim at deconfusion (similar to our traditional research programs, which we continue to pursue), and include the themes of “seeking entirely new low-level foundations for optimization,” “endeavoring to figure out parts of cognition that can be very transparent as cognition,” and “experimenting with some [relatively deep] alignment problems,” and require building software systems and infrastructure.

In 2018, our progress on these new directions and the supporting infrastructure was steady and significant, in line with our high expectations, albeit proceeding significantly slower than we’d like, due in part to the usual difficulties associated with software development. On the whole, our excitement about these new directions is high, and we remain very eager to expand the team to accelerate our progress.

In parallel, Agent Foundations work continued to be a priority at MIRI. Our biggest publication on this front was “[Embedded Agency](https://www.alignmentforum.org/posts/i3BTagvt3HbPMx6PN/embedded-agency-full-text-version),” co-written by MIRI researchers Scott Garabrant and Abram Demski. “Embedded Agency” reframes our Agent Foundations research agenda as different angles of attack on a single central difficulty: we don’t know how to characterize good reasoning and decision-making for agents embedded in their environment.

Below are notable technical results and analyses we released in each research category last year.[1](https://intelligence.org/2019/05/31/2018-in-review/#footnote_0_19235 "Our summaries of our more significant results below largely come from our 2018 fundraiser post.") These are accompanied by [predictions made last year](https://intelligence.org/2018/03/31/2018-research-plans/) by Scott Garrabrant, the research lead for MIRI’s Agent Foundations work, and Scott’s assessment of the progress our published work represents against those predictions. The research categories below are explained in detail in “[Embedded Agency](https://www.lesswrong.com/s/Rm6oQRJJmhGCcLvxh/p/i3BTagvt3HbPMx6PN).”

The actual share of MIRI’s research that was non-public in 2018 ended up being larger than Scott expected when he registered his predictions. The list below is best thought of as a collection of interesting (though not groundbreaking) results and analyses that demonstrate the flavor of some of the directions we explored in our research last year. As such, these assessments don’t represent our model of our overall progress, and aren’t intended to be a good proxy for that question. Given the difficulty of predicting what we’ll disclose for our 2019 public-facing results, we won’t register new predictions this year.

#### Decision theory

* Predicted progress: 3 (modest)
* Actual progress: 2 (weak-to-modest)

Scott sees our largest public decision theory result of 2018 as [Prisoners’ Dilemma with Costs to Modeling](https://www.lesswrong.com/posts/XjMkPyaPYTf7LrKiT/prisoners-dilemma-with-costs-to-modeling), a modified version of [open-source prisoners’ dilemmas](https://intelligence.org/2016/03/31/new-paper-on-bounded-lob/) in which agents must pay resources in order to model each other.

Other significant write-ups include:

* [Logical Inductors Converge to Correlated Equilibria (Kinda)](https://www.alignmentforum.org/posts/5bd75cc58225bf0670375569/logical-inductors-converge-to-correlated-equilibria-kinda): A game-theoretic analysis of logical inductors.
* New results in [Asymptotic Decision Theory](https://www.alignmentforum.org/posts/yXCvYqTZCsfN7WRrg/asymptotic-decision-theory-improved-writeup) and [When EDT=CDT, ADT Does Well](https://www.alignmentforum.org/posts/pgJbaXvYWBx3Mrg5T/when-edt-cdt-adt-does-well) represent incremental progress on understanding what’s possible with respect to learning the right [counterfactuals](https://intelligence.org/2018/10/31/embedded-decisions/).

Additional decision theory research posts from 2018:

* From Alex Appel, a MIRI contractor and summer intern: (a) [Distributed Cooperation](https://agentfoundations.org/item?id=1777); (b) [Cooperative Oracles](https://www.alignmentforum.org/posts/SgkaXQn3xqJkGQ2D8/cooperative-oracles); (c) [When EDT=CDT, ADT Does Well](https://www.alignmentforum.org/posts/pgJbaXvYWBx3Mrg5T/when-edt-cdt-adt-does-well); (d) [Conditional Oracle EDT Equilibria in Games](https://www.alignmentforum.org/posts/4MLpRxz7ZoX8YXSY3/coedt-equilibria-in-games)
* From Abram Demski: (a) [In Logical Time, All Games are Iterated Games](https://www.alignmentforum.org/posts/dKAJqBDZRMMsaaYo5/in-logical-time-all-games-are-iterated-games); (b) [A Rationality Condition for CDT Is That It Equal EDT (Part 1)](https://www.alignmentforum.org/posts/XW6Qi2LitMDb2MF8c/a-rationality-condition-for-cdt-is-that-it-equal-edt-part-1); (c) [A Rationality Condition for CDT Is That It Equal EDT (Part 2)](https://www.alignmentforum.org/posts/tpWfDLZy2tk97MJ3F/a-rationality-condition-for-cdt-is-that-it-equal-edt-part-2)
* From Scott Garrabrant: (a) [Knowledge is Freedom](https://www.lesswrong.com/posts/b3Bt9Cz4hEtR26ANX/knowledge-is-freedom); (b) [Counterfactual Mugging Poker Game](https://www.lesswrong.com/posts/g3PwPgcdcWiP33pYn/counterfactual-mugging-poker-game); (c) [(A → B) → A](https://www.alignmentforum.org/posts/qhsELHzAHFebRJE59/a-greater-than-b-greater-than-a)
* From Alex Mennen, a MIRI summer intern: [When Wishful Thinking Works](https://www.alignmentforum.org/posts/KbCHcb8yyjAMFAAPJ/when-wishful-thinking-works)

#### Embedded world-models

* Predicted progress: 3 (modest)
* Actual progress: 1 (limited)

Some of our relatively significant results related to embedded world-models included:

* Sam Eisenstat’s [untrollable prior](https://agentfoundations.org/item?id=1750), explained [in illustrated form](https://www.lesswrong.com/posts/CvKnhXTu9BPcdKE4W/an-untrollable-mathematician-illustrated) by Abram Demski, shows that there is a Bayesian solution to one of the basic problems which motivated the development of [non-Bayesian](https://www.lesswrong.com/posts/tKwJQbo6SfWF2ifKh/toward-a-new-technical-explanation-of-technical-explanation) logical uncertainty tools (culminating in [logical induction](https://intelligence.org/2016/09/12/new-paper-logical-induction/)). This informs our picture of what’s possible, and may lead to further progress in the direction of [Bayesian logical uncertainty](https://intelligence.org/2018/11/02/embedded-models/).
* Sam Eisenstat and Tsvi Benson-Tilsen’s formulation of Bayesian logical induction. This framework, which has yet to be written up, forces logical induction into a Bayesian framework by constructing a Bayesian prior which trusts the beliefs of a logical inductor (which must supply those beliefs to the Bayesian regularly).

Sam and Tsvi’s work can be viewed as evidence that “true” Bayesian logical induction is possible. However, it can also be viewed as a demonstration that we have to be careful what we mean by “Bayesian”—the solution is arguably cheating, and it isn’t clear that you get any new desirable properties by doing things this way.

Scott assigns the untrollable prior result a *2* (weak-to-modest progress) rather than a *1* (limited progress), but is counting this among our [2017 results](https://intelligence.org/2018/11/28/2017-in-review/#1), since it was written up in 2018 but produced in 2017.

Other recent work in this category includes:

* From Alex Appel: (a) [Resource-Limited Reflective Oracles](https://agentfoundations.org/item?id=1793); (b) [Bounded Oracle Induction](https://www.alignmentforum.org/posts/MgLeAWSeLbzx8mkZ2/bounded-oracle-induction)
* From Abram Demski: (a) [Toward a New Technical Explanation of Technical Explanation](https://www.lesswrong.com/posts/tKwJQbo6SfWF2ifKh/toward-a-new-technical-explanation-of-technical-explanation); (b) [Probability is Real, and Value is Complex](https://www.alignmentforum.org/posts/oheKfWA7SsvpK7SGp/probability-is-real-and-value-is-complex)

#### Robust delegation

* Predicted progress: 2 (weak-to-modest)
* Actual progress: 1 (limited)

Our most significant 2018 public result in this category is perhaps Sam Eisenstat’s [logical inductor tiling result](https://www.alignmentforum.org/posts/5bd75cc58225bf067037556d/logical-inductor-tiling-and-why-it-s-hard), which solves a version of the [tiling problem](https://intelligence.org/files/TilingAgentsDraft.pdf) for logically uncertain agents.[2](https://intelligence.org/2019/05/31/2018-in-review/#footnote_1_19235 "Not to be confused with Nate Soares’ forthcoming tiling agents paper.")

Other posts on robust delegation:

* From Stuart Armstrong (MIRI Research Associate): (a) [Standard ML Oracles vs. Counterfactual Ones](https://www.alignmentforum.org/posts/hJaJw6LK39zpyCKW6/standard-ml-oracles-vs-counterfactual-ones); (b) “[Occam’s Razor is Insufficient to Infer the Preferences of Irrational Agents](https://www.fhi.ox.ac.uk/occam/)“
* From Abram Demski: [Stable Pointers to Value II: Environmental Goals](https://agentfoundations.org/item?id=1762)
* From Scott Garrabrant: [Optimization Amplifies](https://www.lesswrong.com/posts/zEvqFtT4AtTztfYC4/optimization-amplifies)
* From Vanessa Kosoy (MIRI Research Associate): (a) [Quantilal Control for Finite Markov Decision Processes](https://agentfoundations.org/item?id=1785); (b) [Computing An Exact Quantilal Policy](https://agentfoundations.org/item?id=1794)
* From Alex Mennen: [Safely and Usefully Spectating on AIs Optimizing Over Toy Worlds](https://www.alignmentforum.org/posts/ikN9qQEkrFuPtYd6Y/safely-and-usefully-spectating-on-ais-optimizing-over-toy)

#### Subsystem alignment

* Predicted progress: 2 (weak-to-modest)
* Actual progress: 2

We achieved greater clarity on subsystem alignment in 2018, largely reflected in Evan Hubinger, Chris van Merwijk, Vladimir Mikulik, Joar Skalse,[3](https://intelligence.org/2019/05/31/2018-in-review/#footnote_2_19235 "Evan was a MIRI research intern, while Chris, Vladimir, and Joar are external collaborators.") and Scott Garrabrant’s forthcoming paper, “Risks from Learned Optimization in Advanced Machine Learning Systems.”[4](https://intelligence.org/2019/05/31/2018-in-review/#footnote_3_19235 "This paper was previously cited in “Embedded Agency” under the working title “The Inner Alignment Problem.”") This paper is currently being rolled out [on the AI Alignment Forum](https://www.alignmentforum.org/posts/FkgsxrGf3QxhfLWHG/risks-from-learned-optimization-introduction), as a sequence on “[Mesa-Optimization](https://www.alignmentforum.org/s/r9tYkB2a8Fp4DN8yB).”[5](https://intelligence.org/2019/05/31/2018-in-review/#footnote_4_19235 "The full PDF version of the paper will be released in conjuction with the last post of the sequence.")

Scott Garrabrant’s [Robustness to Scale](https://www.lesswrong.com/posts/bBdfbWfWxHN9Chjcq/robustness-to-scale) also discusses issues in subsystem alignment (“robustness to relative scale”), alongside other issues in AI alignment.

#### Other

* Predicted progress: 2 (weak-to-modest)
* Actual progress: 2

Some of the 2018 publications we expect to be most useful cut across all of the above categories:

* “[Embedded Agency](https://intelligence.org/embedded-agency/),” Scott and Abram’s new introduction to all of the above research directions.
* [Fixed Point Exercises](https://www.lesswrong.com/posts/mojJ6Hpri8rfzY78b/fixed-point-exercises), a set of exercises created by Scott to introduce people to the core ideas and tools in agent foundations research.

Here, other noteworthy posts include:

* From Scott Garrabrant: (a) [Sources of Intuitions and Data on AGI](https://www.lesswrong.com/posts/BibDWWeo37pzuZCmL/sources-of-intuitions-and-data-on-agi); (b) [History of the Development of Logical Induction](https://www.alignmentforum.org/posts/iBBK4j6RWC7znEiDv/history-of-the-development-of-logical-induction)

### 2018 Research Program Support

We added three new research staff to the team in 2018: Ben Weinstein-Raun, James Payor, and [Edward Kmett](https://intelligence.org/2018/11/28/miris-newest-recruit-edward-kmett/).

We invested a large share of our capacity into growing the research team in 2018, and generally into activities aimed at increasing the amount of alignment research in the world, including:

* **Running eight** [**AI Risk for Computer Scientist**](https://intelligence.org/ai-risk-for-computer-scientists/) **(AIRCS) workshops**. This is an ongoing all-expenses-paid workshop series for computer scientists and programmers who want to get started thinking about or working on AI alignment. At these workshops, we introduce AI risk and related concepts, share some CFAR-style rationality content, and introduce participants to the work done by MIRI and other safety research teams. Our overall aim is to cause good discussions to happen, improve participants’ ability to make progress on whether and how to contribute, and in the process work out whether they may be interested in joining MIRI or other alignment groups. Of 2018 workshop participants, we saw one join MIRI full-time, four take on internships with us, and on the order of ten with good prospects of joining MIRI within a year, in addition to several who have since joined other safety-related organizations.
* **Running a 2.5-week** [**AI Summer Fellows Program**](https://www.rationality.org/workshops/apply-msfp) **(AISFP) with CFAR**.[6](https://intelligence.org/2019/05/31/2018-in-review/#footnote_5_19235 "As noted in our summer updates:
We had a large and extremely strong pool of applicants, with over 170 applications for 30 slots (versus 50 applications for 20 slots in 2017). The program this year was more mathematically flavored than in 2017, and concluded with a flurry of new analyses by participants. On the whole, the program seems to have been more successful at digging into AI alignment problems than in previous years, as well as more successful at seeding ongoing collaborations between participants, and between participants and MIRI staff.
The program ended with a very active blogathon, with write-ups including: Dependent Type Theory and Zero-Shot Reasoning; Conceptual Problems with Utility Functions (and follow-up); Complete Class: Consequentialist Foundations; and Agents That Learn From Human Behavior Can’t Learn Human Values That Humans Haven’t Learned Yet.") Additionally, MIRI researcher Tsvi Benson-Tilsen and MIRI summer intern Alex Zhu ran a mid-year AI safety retreat for MIT students and alumni.
* **Running a 10-week research internship program over the summer**, reviewed in our [summer updates](https://intelligence.org/2018/09/01/summer-miri-updates/). Interns also participated in AISFP and in a joint [research workshop](https://intelligence.org/workshops/#july-2018) with interns from the [Center for Human-Compatible AI](https://humancompatible.ai/). Additionally, we hosted three more research interns later in the year. We are hopeful that at least one of them will join the team in 2019.
* **Making grants to two individuals as part of our AI Safety Retraining Program**. In 2018 we received [$150k in restricted funding](https://intelligence.org/2018/09/01/summer-miri-updates/#2) from the Open Philanthropy Project, “to provide stipends and guidance to a few highly technically skilled individuals. The goal of the program is to free up 3–6 months of time for strong candidates to spend on retraining, so that they can potentially transition to full-time work on AI alignment.” We issued grants to two people in 2018, including [Carroll Wainwright](https://www.partnershiponai.org/team/carroll-wainwright/) who went on to become a Research Scientist at Partnership on AI.

In addition to the above, in 2018 we:

* **Hired additional operations staff** to ensure we have the required operational capacity to support our continued growth.
* **Moved into new larger office space**.

### 2018 Outreach and Exposition

On the outreach, coordination, and exposition front, we:

* **Released a new edition of** [***Rationality: From AI to Zombies***](https://intelligence.org/rationality-ai-zombies/), beginning with volumes one and two, featuring a number of [updates to the text](https://intelligence.org/2018/12/15/announcing-new-raz/) and an official print edition. We also made Stuart Armstrong’s 2014 book on AI risk, *Smarter Than Us: The Rise of Machine Intelligence*, available on the web for free at [smarterthan.us](https://smarterthan.us/).
* **Released** [**2018 Update: Our New Research Directions**](https://intelligence.org/2018-update-our-new-research-directions), a lengthy discussion of our research, our nondisclosure-by-default policies, and the case for computer scientists and software engineers to apply to join our team.
* **Produced other expository writing**: [Two Clarifications About “Strategic Background”](https://www.lesswrong.com/posts/hL9ennoEfJXMj7r2D/two-clarifications-about-strategic-background); [Challenges to Paul Christiano’s Capability Amplification Proposal](https://intelligence.org/2018/05/19/challenges-to-christianos-capability-amplification-proposal/) (discussion on [LessWrong](https://www.lesswrong.com/posts/S7csET9CgBtpi7sCh/challenges-to-christiano-s-capability-amplification-proposal), including [follow-up conversations](https://www.lesswrong.com/posts/Djs38EWYZG8o7JMWY/paul-s-research-agenda-faq#79jM2ecef73zupPR4)); [Comment on Decision Theory](https://www.lesswrong.com/posts/uKbxi2EJ3KBNRDGpL/comment-on-decision-theory); [The Rocket Alignment Problem](https://intelligence.org/2018/10/03/rocket-alignment/) ([LessWrong link](https://www.lesswrong.com/posts/Gg9a4y8reWKtLe3Tn/the-rocket-alignment-problem)).
* **Received press coverage** in [*Axios*](https://www.axios.com/newsletters/axios-future-5a9e9f63-fd4f-4514-9570-ce9dc4c14348.html?chunk=2#story2), [*Forbes*](https://www.forbes.com/sites/rogerhuang/2018/11/20/wetrust-spring-will-match-your-cryptocurrency-donations-through-giving-tuesday/#1fda09b02461), [*Gizmodo*](https://gizmodo.com/how-we-can-prepare-now-for-catastrophically-dangerous-a-1830388719), and *Vox* ([1](https://www.vox.com/future-perfect/2018/12/21/18126576/ai-artificial-intelligence-machine-learning-safety-alignment), [2](https://www.vox.com/future-perfect/2018/10/22/17991736/jeff-bezos-elon-musk-colonizing-mars-moon-space-blue-origin-spacex)), and were interviewed in [*Nautilus*](http://nautil.us/issue/58/self/scary-ai-is-more-fantasia-than-terminator) and on [Sam Harris’ podcast](https://intelligence.org/2018/02/28/sam-harris-and-eliezer-yudkowsky/).
* **Spoke at** [**Effective Altruism Global**](https://sf.eaglobal.org/2018) **in San Francisco and at the** [**Human-Aligned AI Summer School**](http://humanaligned.ai/index.html) **in Prague**.
* **Presented on** [**logical induction**](https://intelligence.org/2016/09/12/new-paper-logical-induction/) **at the joint** [**Applied Theory Workshop**](https://research.chicagobooth.edu/appliedtheory/workshop-on-applied-theory) **/** [**Workshop in Economic Theory**](https://economics.uchicago.edu/content/workshop-economic-theory-joint-applied-theory-workshop).
* **Released a paper, “**[**Categorizing Variants of Goodhart’s Law**](https://intelligence.org/2018/03/27/categorizing-goodhart/)**,”** based on Scott Garrabrant’s 2017 “[Goodhart Taxonomy](https://www.lesswrong.com/posts/EbFABnst8LsidYs5Y/goodhart-taxonomy).” We also reprinted Nate Soares’ “[The Value Learning Problem](https://intelligence.org/files/ValueLearningProblem.pdf)” and Nick Bostrom and Eliezer Yudkowsky’s “[The Ethics of Artificial Intelligence](https://intelligence.org/files/EthicsofAI.pdf)” in [*Artificial Intelligence Safety and Security*](https://www.crcpress.com/Artificial-Intelligence-Safety-and-Security/Yampolskiy/p/book/9780815369820).
* **Several MIRI researchers also received recognition from the AI Alignment Prize**, including Scott Garrabrant receiving first place and second place in the [first round](https://www.lesswrong.com/posts/4WbNGQMvuFtY3So7s/announcement-ai-alignment-prize-winners-and-next-round) and [second round](https://www.lesswrong.com/posts/SSEyiHaACSYDHcYZz/announcement-ai-alignment-prize-round-2-winners-and-next), respectively, MIRI Research Associate Vanessa Kosoy winning first prize in the [third round](https://www.lesswrong.com/posts/juBRTuE3TLti5yB35/announcement-ai-alignment-prize-round-3-winners-and-next), and Scott and Abram Demski tying Alex Turner for first place in the [fourth round](https://www.lesswrong.com/posts/nDHbgjdddG5EN6ocg/announcement-ai-alignment-prize-round-4-winners).
* MIRI senior staff also participated in AI research and strategy events and conversations throughout the year.

### 2018 Finances

#### Fundraising

2018 was another strong year for MIRI’s fundraising. While the total raised of just over **$5.1M** was a 12% drop from the amount raised in 2017, the graph below shows that our strong growth trend continued—with 2017, as I surmised [in last year’s review](https://intelligence.org/2018/11/28/2017-in-review/), looking like an outlier year driven by the large influx of cryptocurrency contributions during a market high in December 2017.[7](https://intelligence.org/2019/05/31/2018-in-review/#footnote_6_19235 "Note that amounts in this section may vary slightly from our audited financial statements, due to small differences between how we tracked donations internally, and how we are required to report them in our financial statements.")

(In this chart and those that follow, “Unlapsed” indicates contributions from past supporters who did not donate in the previous year.)

Highlights include:

* **$1.02M, our largest ever single donation by an individual**, from “Anonymous Ethereum Investor #2,” based in Canada, made through Rethink Charity Forward’s recently established [tax-advantaged fund for Canadian MIRI supporters](https://rcforward.org/miri/).[8](https://intelligence.org/2019/05/31/2018-in-review/#footnote_7_19235 "A big thanks to Colm for all the work he’s put into setting this up; have a look at our Tax-Advantaged Donations page for more information.")
* **$1.4M in grants from the Open Philanthropy Project**, $1.25M in [general support](https://www.openphilanthropy.org/focus/global-catastrophic-risks/potential-risks-advanced-artificial-intelligence/machine-intelligence-research-institute-general-support-2017) and $150k for our [AI Safety Retraining Program](https://www.openphilanthropy.org/focus/global-catastrophic-risks/potential-risks-advanced-artificial-intelligence/machine-intelligence-research-institute-ai-safety-retraining-program).
* **$951k during our [annual fundraiser](https://intelligence.org/2019/02/11/our-2018-fundraiser-review/)**, driven in large part by MIRI supporters’ participation in multiple matching campaigns during the fundraiser, including [WeTrust Spring’s Ethereum-matching campaign](https://blog.wetrust.io/conclusion-of-the-first-lr-experiment-709b018b5f83), Facebook’s [Giving Tuesday event](https://donations.fb.com/giving-tuesday/), and in partnership with [Raising for Effective Giving](https://reg-charity.org/) (REG), professional poker players’ [Double Up Drive](http://www.doubleupdrive.com/).
* **$529k from 2 grants recommended by the [EA Funds Long-Term Future Fund](https://app.effectivealtruism.org/funds/far-future)**.
* **$115K from Poker Stars, also through REG**.

In 2018, we received contributions from 637 unique contributors, 16% less than in 2017. This drop was largely driven by a 27% reduction in the number of new donors, partly offset by the continuing trend of steady growth in the number of returning donors[9](https://intelligence.org/2019/05/31/2018-in-review/#footnote_8_19235 "2014 is anomalously high on this graph due to the community’s active participation in our memorable SVGives campaign."):

Donations of cryptocurrency were down in 2018 both in absolute terms (-$1.2M in value) and as a percentage of total contributions (23% compared to 42% in 2017). It’s plausible that if cryptocurrency values continue to rebound in 2019, we may see this trend reversed.

In 2017, donations received from matching initiatives dramatically increased with almost a five-fold increase over the previous year. In 2018, our inclusion in two different REG-administered matching challenges, a significantly increased engagement among MIRI supporters with Facebook’s Giving Tuesday, and MIRI’s winning success in WeTrust’s Spring campaign, offset a small decrease in corporate match dollars to improve slightly on 2017’s matching total. The following graph represents the matching amounts received over the last 5 years:

#### Spending

In our [2017 fundraiser post](https://intelligence.org/2017/12/01/miris-2017-fundraiser/#2), I projected that we’d spend ~$2.8M in 2018. Towards the end of last year, I [revised](https://intelligence.org/2018/11/26/miris-2018-fundraiser/#2) our estimate:

> Following the amazing show of support we received from donors last year (and continuing into 2018), we had significantly more funds than we anticipated, and we found more ways to usefully spend it than we expected. In particular, we’ve been able to translate the “bonus” support we received in 2017 into broadening the scope of our recruiting efforts. As a consequence, our 2018 spending, which will come in at around $3.5M, actually matches the point estimate I gave in 2017 for our 2019 budget, rather than my prediction for 2018—a large step up from what I predicted, and an even larger step from last year’s [2017] budget of $2.1M.
>
>

The post goes on to give an overview of the ways in which we put this “bonus” support to good use. These included, in descending order by cost:

* **Investing significantly more in recruiting-related activities**, including our AIRCS workshop series; and scaling up the number of interns we hosted, with an increased willingness to pay higher wages to attract promising candidates to come intern/trial with us.
* **Filtering less on price relative to fit when choosing new office space to accommodate our growth**, and spending more on renovations, than we otherwise would have been able to, in order to create a more focused working environment for research staff.
* **Raising salaries for some existing staff**, who were being paid well below market rates.

With concrete numbers now in hand, I’ll go into more detail below on how we put those additional funds to work.

Total spending came in just over **$3.75M**. The chart below compares our actual spending in 2018 with our projections, and with our spending in 2017.[10](https://intelligence.org/2019/05/31/2018-in-review/#footnote_9_19235 "Note that these numbers will differ slightly compared to our forthcoming audited financial statements for 2018, due to subtleties of how certain types of expenses are tracked. For example, in the financial statements, renovation costs are considered to be a fixed asset that depreciates over time, and as such, won’t show up as an expense.")

At a high level, as expected, personnel costs in 2018 continued to account for the majority of our spending—though represented a smaller share of total spending than in 2017, due to increased spending on recruitment-related activities along with one-time costs related to securing and renovating our new office space.

Our spending on recruitment-related activities is captured in the *program activities* category. The major ways we put additional funds to use, which account for the increase over my projections, break down as follows:

* **~$170k on internships**: We hosted nine research interns for an average of ~2.5 months each. We were able to offer more competitive wages for internships, allowing us to recruit interns (especially those with an engineering focus) that we otherwise would have had a much harder time attracting, given the other opportunities they had available to them. We are actively interested in hiring three of these interns, and have made formal offers to two of them. I’m hopeful that we’ll have added at least one of them to the team by the end of this year.
* **$54k on AI Safety Retraining Program grants**, described [above](https://intelligence.org/feed/?paged=12#2018-research-program-support).
* The bulk of the rest of the additional funds we spent in this category went towards funding our ongoing series of [AI Risk for Computer Scientists](https://intelligence.org/ai-risk-for-computer-scientists/) workshops, described [above](https://intelligence.org/feed/?paged=12#2018-research-program-support).

Expenses related to our new office space are accounted for in the *cost of doing business* category. The surplus spending in this category resulted from:

* **~$300k for securing, renovating, and filling out our new office space**. Finding a suitable new space to accommodate our growth in Berkeley ended up being much more challenging and time-consuming than we expected.[11](https://intelligence.org/2019/05/31/2018-in-review/#footnote_10_19235 "The number of options available in the relevant time frame were very limited, and most did not meet many of our requirements. Of the available spaces, the option that offered the best combination of size, layout, and location, was looking for a tenant starting November 1st 2018, while we weren’t able to move until early January 2019. Additionally, the space was configured with a very open layout that wouldn’t have met our needs, but that many other prospective tenants found desirable, such that we’d have to cover renovation costs.") We made use of additional funds to secure our preferred space ahead of when we were prepared to move, and to renovate the space to meet our needs, whereas if we’d been operating with the budget I originally projected, we would have almost certainly ended up in a much worse space.
* The remainder of the spending beyond my projection in this category comes from higher-than-expected legal costs to secure visas for staff, and slightly higher-than-projected spending across many other subcategories.

---

---

1. Our summaries of our more significant results below largely come from our [2018 fundraiser post](https://intelligence.org/2018/11/26/miris-2018-fundraiser/).
2. Not to be confused with Nate Soares’ forthcoming tiling agents paper.
3. Evan was a MIRI research intern, while Chris, Vladimir, and Joar are external collaborators.
4. This paper was previously cited in “[Embedded Agency](https://intelligence.org/embedded-agency/)” under the working title “The Inner Alignment Problem.”
5. The full PDF version of the paper will be released in conjuction with the last post of the sequence.
6. As noted in our [summer updates](https://intelligence.org/2018/09/01/summer-miri-updates/):

> We had a large and extremely strong pool of applicants, with over 170 applications for 30 slots (versus 50 applications for 20 slots in 2017). The program this year was more mathematically flavored than in 2017, and concluded with a flurry of new analyses by participants. On the whole, the program seems to have been more successful at digging into AI alignment problems than in previous years, as well as more successful at seeding ongoing collaborations between participants, and between participants and MIRI staff.
>
>

The program ended with a very active blogathon, with write-ups including: [Dependent Type Theory and Zero-Shot Reasoning](https://www.alignmentforum.org/posts/Xfw2d5horPunP2MSK/dependent-type-theory-and-zero-shot-reasoning); [Conceptual Problems with Utility Functions](https://www.alignmentforum.org/posts/Nx4DsTpMaoTiTp4RQ/conceptual-problems-with-utility-functions) (and [follow-up](https://www.alignmentforum.org/posts/QmeguSp4Pm7gecJCz/conceptual-problems-with-utility-functions-second-attempt-at)); [Complete Class: Consequentialist Foundations](https://www.alignmentforum.org/posts/sZuw6SGfmZHvcAAEP/complete-class-consequentialist-foundations); and [Agents That Learn From Human Behavior Can’t Learn Human Values That Humans Haven’t Learned Yet](https://www.alignmentforum.org/posts/DfewqowdzDdCD7S9y/agents-that-learn-from-human-behavior-can-t-learn-human).
7. Note that amounts in this section may vary slightly from our audited financial statements, due to small differences between how we tracked donations internally, and how we are required to report them in our financial statements.
8. A big thanks to Colm for all the work he’s put into setting this up; have a look at our [Tax-Advantaged Donations](https://intelligence.org/donate/tax-advantaged-donations/) page for more information.
9. 2014 is anomalously high on this graph due to the community’s active participation in [our memorable SVGives campaign](https://intelligence.org/2014/05/06/liveblogging-the-svgives-fundraiser/).
10. Note that these numbers will differ slightly compared to our forthcoming audited financial statements for 2018, due to subtleties of how certain types of expenses are tracked. For example, in the financial statements, renovation costs are considered to be a fixed asset that depreciates over time, and as such, won’t show up as an expense.
11. The number of options available in the relevant time frame were very limited, and most did not meet many of our requirements. Of the available spaces, the option that offered the best combination of size, layout, and location, was looking for a tenant starting November 1st 2018, while we weren’t able to move until early January 2019. Additionally, the space was configured with a very open layout that wouldn’t have met our needs, but that many other prospective tenants found desirable, such that we’d have to cover renovation costs.

The post [2018 in review](https://intelligence.org/2019/05/31/2018-in-review/) appeared first on [Machine Intelligence Research Institute](https://intelligence.org).


May 2019 Newsletter

#### Updates

* A new paper from MIRI researcher Vanessa Kosoy, presented at the ICLR SafeML workshop this week: "[Delegative Reinforcement Learning: Learning to Avoid Traps with a Little Help](https://intelligence.org/2019/04/24/delegative-reinforcement-learning/)."
* New research posts: [Learning "Known" Information When the Information is Not Actually Known](https://www.alignmentforum.org/posts/TrudRcZxEjA2RNCxh/learning-known-information-when-the-information-is-not); [Defeating Goodhart and the "Closest Unblocked Strategy" Problem](https://www.lesswrong.com/posts/PADPJ3xac5ogjEGwA/defeating-goodhart-and-the-closest-unblocked-strategy); [Reinforcement Learning with Imperceptible Rewards](https://www.lesswrong.com/posts/aAzApjEpdYwAxnsAS/reinforcement-learning-with-imperceptible-rewards)
* The Long-Term Future Fund has announced twenty-three new grant recommendations, and provided in-depth explanations of the grants. These include [a $50,000 grant to MIRI](https://forum.effectivealtruism.org/posts/CJJDwgyqT4gXktq6g/long-term-future-fund-april-2019-grant-decisions), and grants to CFAR and Ought. LTFF is also recommending grants to several individuals with AI alignment research proposals whose work MIRI staff will be helping assess.
* We attended the [Global Governance of AI Roundtable](https://ggar.worldgovernmentsummit.org/) at the World Government Summit in Dubai.

#### News and links

* Rohin Shah reflects on [the first year of the Alignment Newsletter](https://www.lesswrong.com/posts/3onCb5ph3ywLQZMX2/alignment-newsletter-one-year-retrospective).
* Some good recent AI alignment discussion: Alex Turner asks for [the best reasons for pessimism about impact measures;](https://www.lesswrong.com/posts/kCY9dYGLoThC3aG7w/best-reasons-for-pessimism-about-impact-of-impact-measures) Henrik Åslund and Ryan Carey discuss [corrigibility as constrained optimization](https://www.alignmentforum.org/posts/cGLgs3t9md7v7cCm4/corrigibility-as-constrained-optimisation); Wei Dai asks about [low-cost AGI coordination](https://www.alignmentforum.org/posts/gYaKZeBbSL4y2RLP3/strategic-implications-of-ais-ability-to-coordinate-at-low); and Chris Leong asks, "[Would solving counterfactuals solve anthropics?](https://www.lesswrong.com/posts/nTDQ9eNrSNoYrZzJw/would-solving-logical-counterfactuals-solve-anthropics)"
* From DeepMind: [Towards Robust and Verified AI: Specification Testing, Robust Training, and Formal Verification](https://medium.com/@deepmindsafetyresearch/towards-robust-and-verified-ai-specification-testing-robust-training-and-formal-verification-69bd1bc48bda).
* Ilya Sutskever and Greg Brockman discuss OpenAI's new status as a "[hybrid of a for-profit and nonprofit](https://www.vox.com/future-perfect/2019/4/17/18301070/openai-greg-brockman-ilya-sutskever)".
* [Misconceptions about China and AI](http://rationallyspeakingpodcast.org/show/rs-231-helen-toner-on-misconceptions-about-china-and-artific.html): Julia Galef interviews Helen Toner. ([Excerpts.](https://www.lesswrong.com/posts/cHwCBTwWiTdsqXNyn/helen-toner-on-china-cset-and-ai))

The post [May 2019 Newsletter](https://intelligence.org/2019/05/10/may-2019-newsletter/) appeared first on [Machine Intelligence Research Institute](https://intelligence.org).


[![Delegative Reinforcement Learning](https://intelligence.org/wp-content/uploads/2019/04/SafeML2019paper.png)](https://drive.google.com/uc?export=download&id=1xa7UpGGODl6mszNWkA4XQGPyeopsNuWu)MIRI Research Associate Vanessa Kosoy has written a new paper, “[Delegative reinforcement learning: Learning to avoid traps with a little help](https://drive.google.com/uc?export=download&id=1xa7UpGGODl6mszNWkA4XQGPyeopsNuWu).” Kosoy will be presenting the paper at the ICLR 2019 [SafeML workshop](https://sites.google.com/view/safeml-iclr2019) in two weeks. The abstract reads:

> Most known regret bounds for reinforcement learning are either episodic or assume an environment without traps. We derive a regret bound without making either assumption, by allowing the algorithm to occasionally delegate an action to an external advisor. We thus arrive at a setting of active one-shot model-based reinforcement learning that we call DRL (delegative reinforcement learning.)
>
>
> The algorithm we construct in order to demonstrate the regret bound is a variant of Posterior Sampling Reinforcement Learning supplemented by a subroutine that decides which actions should be delegated. The algorithm is not anytime, since the parameters must be adjusted according to the target time discount. Currently, our analysis is limited to Markov decision processes with finite numbers of hypotheses, states and actions.
>
>

The goal of Kosoy’s work on DRL is to put us on a path toward having a deep understanding of learning systems with human-in-the-loop and formal performance guarantees, including safety guarantees. DRL tries to move us in this direction by providing models in which such performance guarantees can be derived.

While these models still make many unrealistic simplifying assumptions, Kosoy views DRL as already capturing some of the most essential features of the problem—and she has a fairly ambitious vision of how this framework might be further developed.

Kosoy previously described DRL in the post [Delegative Reinforcement Learning with a Merely Sane Advisor](https://www.alignmentforum.org/posts/5bd75cc58225bf06703754d5/delegative-reinforcement-learning-with-a-merely-sane-advisor). One feature of DRL Kosoy described here but omitted from the paper (for space reasons) is DRL’s application to *corruption*. Given certain assumptions, DRL ensures that a formal agent will never have its reward or advice channel tampered with (corrupted). As a special case, the agent’s own advisor cannot cause the agent to enter a corrupt state. Similarly, the general protection from traps described in “Delegative reinforcement learning” also protects the agent from harmful self-modifications.

Another set of DRL results that didn’t make it into the paper is [Catastrophe Mitigation Using DRL](https://www.alignmentforum.org/posts/5bd75cc58225bf0670375510/catastrophe-mitigation-using-drl). In this variant, a DRL agent can mitigate catastrophes that the advisor would not be able to mitigate on its own—something that isn’t supported by the more strict assumptions about the advisor in standard DRL.

#### Sign up to get updates on new MIRI technical results

*Get notified every time a new technical paper is published.*

*
*

×

The post [New paper: “Delegative reinforcement learning”](https://intelligence.org/2019/04/24/delegative-reinforcement-learning/) appeared first on [Machine Intelligence Research Institute](https://intelligence.org).

Consider environments that have “traps”: states that permanently curtail the long-term value that an agent can achieve. A world without humans could be one such trap. Traps could also happen after any irreversible action, if the new state is not as useful for achieving high rewards as the old state.

In such an environment, an RL algorithm could simply take no actions, in which case it incurs regret that is linear in the number of timesteps so far. (Regret is the difference between the expected reward under the optimal policy and the policy actually executed, so if the average reward per timestep of the optimal policy is 2 and doing nothing is always reward 0, then the regret will be ~2T where T is the number of timesteps, so regret is linear in the number of timesteps.) Can we find an RL algorithm that will _guarantee_ regret sublinear in the number of timesteps, regardless of the environment?

Unsurprisingly, this is impossible, since during exploration the RL agent could fall into a trap, which leads to linear regret. However, let's suppose that we could delegate to an advisor who knows the environment: what must be true about the advisor for us to do better? Clearly, the advisor must be able to always avoid traps (otherwise the same problem occurs). However, this is not enough: getting sublinear regret also requires us to explore enough to eventually find the optimal policy. So, the advisor must have at least some small probability of being optimal, which the agent can then learn from. This paper proves that with these assumptions there does exist an algorithm that is guaranteed to get sublinear regret.

It's interesting to see what kinds of assumptions are necessary in order to get AI systems that can avoid catastrophically bad outcomes, and the notion of "traps" seems like a good way to formalize this. I worry about there being a Cartesian boundary between the agent and the environment, though perhaps even here as long as the advisor is aware of problems caused by such a boundary, they can be modeled as traps and thus avoided.

Of course, if we want the advisor to be a human, both of the assumptions are unrealistic, but I believe Vanessa's plan is to make the assumptions more realistic in order to see what assumptions are actually necessary.

One thing I wonder about is whether the focus on traps is necessary. With the presence of traps in the theoretical model, one of the main challenges is in preventing the agent from falling into a trap due to ignorance. However, it seems extremely unlikely that an AI system manages to take some irreversible catastrophic action _by accident_ -- I'm much more worried about the case where the AI system is adversarially optimizing against us and intentionally takes an irreversible catastrophic action.


April 2019 Newsletter

#### Updates

* New research posts: [Simplified Preferences Needed, Simplified Preferences Sufficient](https://www.alignmentforum.org/posts/sEqu6jMgnHG2fvaoQ/partial-preferences-needed-partial-preferences-sufficient); [Smoothmin and Personal Identity](https://www.alignmentforum.org/posts/MxLK2fvEuijAYgsc2/smoothmin-and-personal-identity); [Example Population Ethics: Ordered Discounted Utility](https://www.alignmentforum.org/posts/Ee29dFnPhaeRmYdMy/example-population-ethics-ordered-discounted-utility); [A Theory of Human Values](https://www.alignmentforum.org/posts/qezBTig6p6p5xtL6G/a-theory-of-human-values); [A Concrete Proposal for Adversarial IDA](https://www.alignmentforum.org/posts/jYvm4mmjvGHcPXtGL/a-concrete-proposal-for-adversarial-ida)
* MIRI has received a set of [new grants](https://intelligence.org/2019/04/01/new-grants-open-phil-beri/) from the Open Philanthropy Project and the Berkeley Existential Risk Initiative.

#### News and links

* From the DeepMind safety team and Alex Turner: [Designing Agent Incentives to Avoid Side Effects](https://medium.com/@deepmindsafetyresearch/designing-agent-incentives-to-avoid-side-effects-e1ac80ea6107).
* From Wei Dai: [Three Ways That "Sufficiently Optimized Agents Appear Coherent" Can Be False](https://www.alignmentforum.org/posts/4K52SS7fm9mp5rMdX/three-ways-that-sufficiently-optimized-agents-appear); [What's Wrong With These Analogies for Understanding Informed Oversight and IDA?](https://www.alignmentforum.org/posts/LigbvLH9yKR5Zhd6y/what-s-wrong-with-these-analogies-for-understanding-informed); and [The Main Sources of AI Risk?](https://www.lesswrong.com/posts/WXvt8bxYnwBYpy9oT/the-main-sources-of-ai-risk)
* Other recent write-ups: Issa Rice's [Comparison of Decision Theories](https://www.alignmentforum.org/posts/QPhY8Nb7gtT5wvoPH/comparison-of-decision-theories-with-a-focus-on-logical); Paul Christiano's [More Realistic Tales of Doom](https://www.lesswrong.com/posts/HBxe6wdjxK239zajf/more-realistic-tales-of-doom); and Linda Linsefors' [The Game Theory of Blackmail](https://www.alignmentforum.org/posts/wm2rdS3sDY9M5kpWb/the-game-theory-of-blackmail).
* OpenAI's Geoffrey Irving describes AI safety via debate [on FLI's AI Alignment Podcast](https://futureoflife.org/2019/03/06/ai-alignment-through-debate-with-geoffrey-irving).
* A webcomic's take on AI x-risk concepts: [*Seed*](https://www.webtoons.com/en/sf/seed/prologue/viewer?title_no=1480&episode_no=1).

The post [April 2019 Newsletter](https://intelligence.org/2019/04/21/april-2019-newsletter/) appeared first on [Machine Intelligence Research Institute](https://intelligence.org).


I’m happy to announce that MIRI has received two major new grants:

* A two-year grant totaling $2,112,500 from the [Open Philanthropy Project](https://www.openphilanthropy.org/about/who-we-are).
* A $600,000 grant from the [Berkeley Existential Risk Initiative](http://existence.org/organization-grants/).

The Open Philanthropy Project’s [grant](https://www.openphilanthropy.org/focus/global-catastrophic-risks/potential-risks-advanced-artificial-intelligence/machine-intelligence-research-institute-general-support-2019) was awarded as part of the first round of grants recommended by their new [committee for effective altruism support](https://www.openphilanthropy.org/committee-effective-altruism-support):

> We are experimenting with a new approach to setting grant sizes for a number of our largest grantees in the [effective altruism](https://www.openphilanthropy.org/focus/other-areas#EffectiveAltruism) community, including those who work on long-termist causes. Rather than have a single Program Officer make a recommendation, we have created a small committee, comprised of Open Philanthropy staff and trusted outside advisors who are knowledgeable about the relevant organizations. […] We average the committee members’ votes to arrive at final numbers for our grants.
>
>

The Open Philanthropy Project’s grant is separate from the three-year [$3.75 million grant](https://intelligence.org/2017/11/08/major-grant-open-phil/) they awarded us in 2017, the third $1.25 million disbursement of which is still scheduled for later this year. This new grant increases the Open Philanthropy Project’s total support for MIRI from $1.4 million[1](https://intelligence.org/2019/04/01/new-grants-open-phil-beri/#footnote_0_19133 "The $1.4 million counts the Open Philanthropy Project’s $1.25 million disbursement in 2018, as well as a $150,000 AI Safety Retraining Program grant to MIRI.") in 2018 to ~$2.31 million in 2019, but doesn’t reflect any decision about how much total funding MIRI might receive from Open Phil in 2020 (beyond the fact that it will be at least ~$1.06 million).

Going forward, the Open Philanthropy Project currently plans to determine the size of any potential future grants to MIRI using the above committee structure.

We’re very grateful for this increase in support from BERI and the Open Philanthropy Project—both organizations that already numbered among our largest funders of the past few years. We expect these grants to play an important role in our decision-making as we continue to grow our research team in the ways described in our 2018 [strategy update](https://intelligence.org/2018/11/22/2018-update-our-new-research-directions/) and [fundraiser](https://intelligence.org/2018/11/26/miris-2018-fundraiser/) posts.

---

1. The $1.4 million counts the Open Philanthropy Project’s $1.25 million disbursement in 2018, as well as a $150,000 [AI Safety Retraining Program grant](https://www.openphilanthropy.org/focus/global-catastrophic-risks/potential-risks-advanced-artificial-intelligence/machine-intelligence-research-institute-ai-safety-retraining-program) to MIRI.

The post [New grants from the Open Philanthropy Project and BERI](https://intelligence.org/2019/04/01/new-grants-open-phil-beri/) appeared first on [Machine Intelligence Research Institute](https://intelligence.org).


March 2019 Newsletter

Want to be in the reference class “people who solve the AI alignment problem”?
 We now have [**a guide on how to get started**](https://www.lesswrong.com/posts/PqMT9zGrNsGJNfiFR/alignment-research-field-guide), based on our experience of what tends to make research groups successful. (Also on the [AI Alignment Forum](https://www.alignmentforum.org/posts/PqMT9zGrNsGJNfiFR/alignment-research-field-guide).)

#### Other updates

* Demski and Garrabrant’s introduction to MIRI’s agent foundations research, “[Embedded Agency](https://www.lesswrong.com/s/Rm6oQRJJmhGCcLvxh/p/i3BTagvt3HbPMx6PN),” is now available (in lightly edited form) [as an arXiv paper](https://arxiv.org/abs/1902.09469).
* New research posts: [How Does Gradient Descent Interact with Goodhart?](https://www.lesswrong.com/posts/pcomQ4Fwi7FnfBZBR/how-does-gradient-descent-interact-with-goodhart); [“Normative Assumptions” Need Not Be Complex](https://www.alignmentforum.org/posts/QzsCrzGd4zkNwk9cd/normative-assumptions-need-not-be-complex); [How the MtG Color Wheel Explains AI Safety](https://www.lesswrong.com/posts/9CKBtxWtjvminNTmC/how-the-mtg-color-wheel-explains-ai-safety); [Pavlov Generalizes](https://www.alignmentforum.org/posts/XTgkhjNTEi97WHMi6/pavlov-generalizes)
* [Several MIRIx groups](https://intelligence.org/2019/03/09/a-new-field-guide-for-mirix/) are expanding and are looking for new members to join.
* Our [summer fellows program](https://intelligence.org/2019/03/10/applications-are-open-for-msfp/) is accepting applications through March 31.
* LessWrong’s web edition of *Rationality: From AI to Zombies* at [lesswrong.com/rationality](https://www.lesswrong.com/rationality) is now fully updated to reflect the print edition of *Map and Territory* and *How to Actually Change Your Mind*, the first two books. ([Announcement here.](https://www.lesswrong.com/posts/c7EjFK8yTBwdHewrg/new-versions-of-posts-in-map-and-territory-and-how-to))

#### News and links

* OpenAI’s [GPT-2 model](https://openai.com/blog/better-language-models/) shows meaningful progress on a wide variety of language tasks. OpenAI adds:

> Due to concerns about large language models being used to generate deceptive, biased, or abusive language at scale, we are only releasing a much smaller version of GPT-2 along with sampling code. We are not releasing the dataset, training code, or GPT-2 model weights. […] We believe our release strategy limits the initial set of organizations who may choose to [open source our results], and gives the AI community more time to have a discussion about the implications of such systems.
>
>
* *The Verge* [discusses](https://www.theverge.com/2019/2/21/18234500/ai-ethics-debate-researchers-harmful-programs-openai) OpenAI’s language model concerns along with [MIRI’s disclosure policies](https://intelligence.org/2018/11/22/2018-update-our-new-research-directions) for our own research. See other discussion by [Jeremy Howard](https://www.fast.ai/2019/02/15/openai-gp2/), [John Seymour](https://twitter.com/_delta_zero/status/1097314384731226113), and [Ryan Lowe](https://towardsdatascience.com/openais-gpt-2-the-model-the-hype-and-the-controversy-1109f4bfd5e8).
* AI Impacts summarizes [evidence on good forecasting practices from the Good Judgment Project](https://aiimpacts.org/evidence-on-good-forecasting-practices-from-the-good-judgment-project/).
* Recent AI alignment ideas and discussion: Carey on [quantilization](https://www.alignmentforum.org/posts/Rs6vZCrnQFWQ4p37P/when-to-use-quantilization); Filan on [impact regularization methods](https://www.alignmentforum.org/posts/wzPzPmAsG3BwrBrwy/test-cases-for-impact-regularisation-methods); Saunders’ [HCH Is Not Just Mechanical Turk](https://www.alignmentforum.org/posts/4JuKoFguzuMrNn6Qr/hch-is-not-just-mechanical-turk) and [RL in the Iterated Amplification Framework](https://www.alignmentforum.org/posts/fq7Ehb2oWwXtZic8S/reinforcement-learning-in-the-iterat); Dai on philosophical difficulty ([1](https://www.alignmentforum.org/posts/w6d7XBCegc96kz4n3/the-argument-from-philosophical-difficulty), [2](https://www.alignmentforum.org/posts/EByDsY9S3EDhhfFzC/some-thoughts-on-metaphilosophy)); Hubinger on [ascription universality](https://www.alignmentforum.org/posts/R5Euq7gZgobJi5S25/nuances-with-ascription-universality); and Everitt’s [Understanding Agent Incentives with Causal Influence Diagrams](https://medium.com/@deepmindsafetyresearch/understanding-agent-incentives-with-causal-influence-diagrams-7262c2512486).
* The Open Philanthropy Project announces their largest grant to date: $55 million to launch the [Center for Security and Emerging Technology](https://www.openphilanthropy.org/giving/grants/georgetown-university-center-security-and-emerging-technology), a Washington, D.C. think tank with an early focus on “the intersection of security and artificial intelligence”. See also CSET’s many [jobpostings](https://cset.georgetown.edu/careers/).

The post [March 2019 Newsletter](https://intelligence.org/2019/03/14/march-2019-newsletter/) appeared first on [Machine Intelligence Research Institute](https://intelligence.org).


CFAR and MIRI are running our fifth annual MIRI Summer Fellows Program (MSFP) in the San Francisco Bay Area from August 9 to August 24, 2019.
MSFP is an extended retreat for mathematicians and programmers with a serious interest in making technical progress on the problem of AI alignment. It includes an overview of CFAR’s applied rationality content, a breadth-first grounding in the MIRI perspective on AI safety, and multiple days of actual hands-on research with participants and MIRI staff attempting to make inroads on open questions.

#### Program Description

The intent of the program is to boost participants, as far as possible, in four overlapping areas:

> **Doing rationality inside a human brain**: understanding, with as much fidelity as possible, what phenomena and processes drive and influence human thinking and reasoning, so that we can account for our own biases and blindspots, better recruit and use the various functions of our brains, and, in general, be less likely to trick ourselves, gloss over our confusions, or fail to act in alignment with our endorsed values.
>
>
> **Epistemic rationality**, especially the subset of skills around [deconfusion](https://intelligence.org/2018/11/22/2018-update-our-new-research-directions/#section2). Building the skill of noticing where the dots don’t actually connect; answering the question “why do we think we know what we think we know?”, particularly when it comes to predictions and assertions around the future development of artificial intelligence.
>
>
> **Grounding in the current research landscape surrounding AI**: being aware of the primary disagreements among leaders in the field, and the arguments for various perspectives and claims. Understanding the current open questions, and why different ones seem more pressing or real under different assumptions. Being able to follow the reasoning behind various alignment schemes/theories/proposed interventions, and being able to evaluate those interventions with careful reasoning and mature (or at least more-mature-than-before) intuitions.
>
>
> **Generative research skill**: the ability to make real and relevant progress on questions related to the field of AI alignment without losing track of one’s own metacognition. The parallel processes of using one’s mental tools, critiquing and improving one’s mental tools, and making one’s own progress or deconfusion available to others through talks, papers, and models. Anything and everything involved in being the sort of thinker who can locate a good question, sniff out promising threads, and collaborate effectively with others and with the broader research ecosystem.
>
>

Food and lodging are provided free of charge at CFAR’s workshop venue in Bodega Bay, California. Participants must be able to remain onsite, largely undistracted for the duration of the program (e.g. no major appointments in other cities, no large looming academic or professional deadlines just after the program).

If you have any questions or comments, please send an email to **[colm@intelligence.org](https://intelligence.org/feed/colm@intelligence.org)**, or, if you suspect others would also benefit from hearing the answer, post them here.

**Update April 23, 2019:** Applications closed on March 31, 2019 and finalists are being contacted by a MIRI staff member for 1–2 Skype interviews. Admissions decisions — yes, no, waitlist — will go out no later than April 30th.

The post [Applications are open for the MIRI Summer Fellows Program!](https://intelligence.org/2019/03/10/applications-are-open-for-msfp/) appeared first on [Machine Intelligence Research Institute](https://intelligence.org).


A new field guide for MIRIx

We’ve just released a **[field guide](https://www.lesswrong.com/posts/PqMT9zGrNsGJNfiFR/alignment-research-field-guide)** for MIRIx groups, and for other people who want to get involved in [AI alignment](https://intelligence.org/2017/04/12/ensuring/) research.

MIRIx is a program where MIRI helps cover basic expenses for outside groups that want to work on open problems in AI safety. You can start your own group or find information on existing meet-ups at **[intelligence.org/mirix](https://intelligence.org/mirix/)**.

Several MIRIx groups have recently been ramping up their activity, including:

* **UC Irvine**: Daniel Hermann is starting a MIRIx group in Irvine, California. [Contact him](mailto:daherrma@uci.edu) if you’d like to be involved.
* **Seattle**: MIRIxSeattle is a small group that’s in the process of restarting and increasing its activities. Contact [Pasha Kamyshev](mailto:dapash@gmail.com) if you’re interested.
* **Vancouver**: [Andrew McKnight](mailto:thedonk@gmail.com) and [Evan Gaensbauer](https://intelligence.org/feed/egbauer92@gmail.com) are looking for more people who’d like to join MIRIxVancouver events.

The new alignment field guide is intended to provide tips and background models to MIRIx groups, based on our experience of what tends to make a research group succeed or fail.

The guide begins:

---

### Preamble I: Decision Theory

Hello! You may notice that you are reading a document.

This fact comes with certain implications. For instance, why are you reading this? Will you finish it? What decisions will you come to as a result? What will you do next?

Notice that, whatever you end up doing, it’s likely that there are dozens or even hundreds of other people, quite similar to you and in quite similar positions, who will follow reasoning which strongly resembles yours, and make choices which correspondingly match.

Given that, it’s our recommendation that you make your next few decisions by asking the question “What policy, if followed by all agents similar to me, would result in the most good, and what does that policy suggest in my particular case?” It’s less of a question of trying to decide for all agents sufficiently-similar-to-you (which might cause you to make the wrong choice out of guilt or pressure) and more something like “if I *were* in charge of all agents in my reference class, how would I treat instances of that class with *my specific characteristics*?”

If that kind of thinking leads you to read further, great. If it leads you to set up a MIRIx chapter, even better. In the meantime, we will proceed as if the only people reading this document are those who justifiably expect to find it reasonably useful.

### Preamble II: Surface Area

Imagine that you have been tasked with moving a cube of solid iron that is one meter on a side. Given that such a cube weighs ~16000 pounds, and that an average human can lift ~100 pounds, a naïve estimation tells you that you can solve this problem with ~150 willing friends.

But of course, a meter cube can fit at most something like 10 people around it. It doesn’t *matter* if you have the theoretical power to move the cube if you can’t bring that power to bear in an effective manner. The problem is constrained by its *surface area*.

MIRIx chapters are one of the best ways to increase the surface area of people thinking about and working on the technical problem of AI alignment. And just as it would be a bad idea to decree “the 10 people who happen to currently be closest to the metal cube are the only ones allowed to think about how to think about this problem”, we don’t want MIRI to become the bottleneck or authority on what kinds of thinking can and should be done in the realm of [embedded agency](https://intelligence.org/2018/10/29/embedded-agents/) and other relevant fields of research.

The hope is that you and others like you will help actually solve the problem, not just follow directions or read what’s already been written. This document is designed to support people who are interested in doing real groundbreaking research themselves.

[(Read more)](https://www.lesswrong.com/posts/PqMT9zGrNsGJNfiFR/alignment-research-field-guide)

The post [A new field guide for MIRIx](https://intelligence.org/2019/03/09/a-new-field-guide-for-mirix/) appeared first on [Machine Intelligence Research Institute](https://intelligence.org).


February 2019 Newsletter

#### Updates

* Ramana Kumar and Scott Garrabrant argue that the AGI safety community should begin prioritizing “[approaches that work well in the absence of human models](https://intelligence.org/2019/02/22/thoughts-on-human-models/)”:

> [T]o the extent that human modelling is a good idea, it is important to do it very well; to the extent that it is a bad idea, it is best to not do it at all. Thus, whether or not to do human modelling at all is a configuration bit that should probably be set early when conceiving of an approach to building safe AGI.
>
>
* New research forum posts: [Conditional Oracle EDT Equilibria in Games](https://www.alignmentforum.org/posts/4MLpRxz7ZoX8YXSY3/coedt-equilibria-in-games); [Non-Consequentialist Cooperation?](https://www.alignmentforum.org/posts/F9vcbEMKW48j4Z6h9/non-consequentialist-cooperation); [When is CDT Dutch-Bookable?](https://www.lesswrong.com/posts/TJT2oBMGaZTE7f2z2/when-is-cdt-dutch-bookable); [CDT=EDT=UDT](https://www.alignmentforum.org/posts/WkPf6XCzfJLCm2pbK/cdt-edt-udt)
* The [MIRI Summer Fellows Program](https://www.lesswrong.com/events/xFGQdgJndLcthgWoE/miri-summer-fellows-program) is accepting applications through the end of March! MSFP is a free two-week August retreat co-run by MIRI and CFAR, intended to bring people up to speed on problems related to [embedded agency](https://www.lesswrong.com/s/Rm6oQRJJmhGCcLvxh/p/i3BTagvt3HbPMx6PN) and AI alignment, train research-relevant skills and habits, and investigate open problems in the field.
* MIRI’s Head of Growth, Colm Ó Riain, [reviews](https://intelligence.org/2019/02/11/our-2018-fundraiser-review/) how our 2018 fundraiser went.
* [From Eliezer Yudkowsky](https://twitter.com/ESYudkowsky/status/1085225949900099585): “Along with adversarial resistance and transparency, what I’d term ‘[conservatism](https://intelligence.org/2017/02/28/using-machine-learning/)’, or trying to keep everything as interpolation rather than extrapolation, is one of the few areas modern ML can explore that I see as having potential to carry over directly to serious AGI safety.”

#### News and links

* Eric Drexler has released his book-length AI safety proposal: [Reframing Superintelligence: Comprehensive AI Services as General Intelligence](https://www.fhi.ox.ac.uk/reframing/). See discussion by [Peter McCluskey](https://www.lesswrong.com/posts/bXYtDfMTNbjCXFQPh/drexler-on-ai-risk), [Richard Ngo](https://www.lesswrong.com/posts/HvNAmkXPTSoA4dvzv/comments-on-cais), and [Rohin Shah](https://www.lesswrong.com/posts/x3fNwSe5aWZb5yXEG/reframing-superintelligence-comprehensive-ai-services-as).
* Other recent AI alignment posts include Andreas Stuhlmüller’s [Factored Cognition](https://www.alignmentforum.org/posts/DFkGStzvj3jgXibFG/factored-cognition) and Alex Turner’s [Penalizing Impact via Attainable Utility Preservation](https://www.alignmentforum.org/posts/mDTded2Dn7BKRBEPX/penalizing-impact-via-attainable-utility-preservation), and a host of new write-ups by [Stuart Armstrong](https://www.alignmentforum.org/users/stuart_armstrong).

The post [February 2019 Newsletter](https://intelligence.org/2019/02/25/february-2019-newsletter/) appeared first on [Machine Intelligence Research Institute](https://intelligence.org).