2020 Updates and Strategy

MIRI’s 2020 has been a year of experimentation and adjustment. In response to the COVID-19 pandemic, we largely moved our operations to more rural areas in March, and shifted to a greater emphasis on remote work. We took the opportunity to try new work set-ups and approaches to research, and have been largely happy with the results.

At the same time, 2020 saw limited progress in the research MIRI’s leadership had previously been most excited about: the new [research directions](https://intelligence.org/2018/11/22/2018-update-our-new-research-directions/) we started in 2017. Given our slow progress to date, we are considering a number of possible changes to our strategy, and MIRI’s research leadership is shifting much of their focus toward searching for more promising paths.

Last year, [I projected](https://intelligence.org/2019/12/02/miris-2019-fundraiser/) that our 2020 budget would be $6.4M–$7.4M, with a point estimate of $6.8M. I now expect that our 2020 spending will be slightly above $7.4M. The increase in spending above my point estimate largely comes from expenses we incurred relocating staff and taking precautions in response to the COVID-19 pandemic.

Our budget for 2021 is fairly uncertain, given that we are more likely than usual to see high-level shifts in our strategy in the coming year. My current estimate is that our spending will fall somewhere between $6M and $7.5M, which I expect to roughly break down as follows:

I’m also happy to announce that the Survival and Flourishing Fund (SFF) [has awarded MIRI $563,000](https://survivalandflourishing.fund/sff-2020-h2-recommendations) to support our research going forward, on top of support they provided [earlier this year](https://intelligence.org/2020/05/29/may-2020-newsletter/).

Given that our research program is in a transitional period, and given the strong support we have already received this year—$4.38M [from Open Philanthropy](https://intelligence.org/2020/04/27/miris-largest-grant-to-date/), $903k from SFF, and ~$1.1M from other contributors (thank you all!)—we aren’t holding a formal fundraiser this winter. Donations are still welcome and appreciated during this transition; but we’ll wait to make our case to donors when our plans are more solid. For now, see our [donate](http://intelligence.org/donate) page if you are interested in supporting our research.

Below, I’ll go into more detail on how our 2020 has gone, and on our plans for the future.

### 2017-Initiated Research Directions and Research Plans

In 2017, we [introduced](https://intelligence.org/2017/04/30/2017-updates-and-strategy/) a new set of research directions, which we described and motivated more in “[2018 Update: Our New Research Directions](https://intelligence.org/2018/11/22/2018-update-our-new-research-directions/).” We wrote that we were “seeking entirely new low-level foundations for optimization,” “endeavoring to figure out parts of cognition that can be very transparent as cognition,” and “experimenting with some specific alignment problems.” In December 2019, we noted that we felt we were making “steady progress” on this research, but were [disappointed with the concrete results we’d had to date](https://intelligence.org/2019/12/02/miris-2019-fundraiser/#2).

After pushing more on these lines of research, MIRI senior staff have become more pessimistic about this approach. MIRI executive director and senior researcher Nate Soares writes:

> The non-public-facing research I (Nate) was most excited about had a flavor of attempting to develop new pragmatically-feasible foundations for alignable AI, that did not rely on routing through gradient-descent-style machine learning foundations. We had various reasons to hope this could work, despite the obvious difficulties.
>
>
> That project has, at this point, largely failed, in the sense that neither Eliezer nor I have sufficient hope in it for us to continue focusing our main efforts there. I’m uncertain whether it failed due to implementation failures on our part, due to the inherent difficulty of the domain, or due to flaws in the underlying theory.
>
>
> Part of the reason we lost hope is a sense that we were moving too slowly, given our sense of how far off AGI may be and our sense of the difficulty of the alignment problem. The field of AI alignment is working under a deadline, such that if work is going sufficiently slowly, we’re better off [giving up](https://www.lesswrong.com/posts/wCqfCLs8z5Qw4GbKS/the-importance-of-saying-oops) and pivoting to new projects that have a real chance of resulting in the first AGI systems being built on alignable foundations.
>
>
> We are currently in a state of regrouping, weighing our options, and searching for plans that we believe may yet have a shot at working.
>
>
> Looking at the field as a whole, MIRI’s research leadership remains quite pessimistic about most alignment proposals that we have seen put forward so far. That is, our update toward being more pessimistic about our recent research directions hasn’t reduced our pessimism about the field of alternatives, and the next directions we undertake are not likely to resemble the directions that are popular outside of MIRI today.
>
>

MIRI sees the need for a change of course with respect to these projects. At the same time, many (including Nate) still have some hope in the theory underlying this research, and have hope that the projects may be rescued in some way, such as by discovering and correcting failures in how we approached this research. But time spent on rescue efforts trades off against finding better and more promising alignment plans.

So we’re making several changes affecting staff previously focused on this work. Some are departing MIRI for different work, as we shift direction away from lines they were particularly suited for. Some are seeking to rescue the 2017-initiated lines of research. Some are pivoting to different experiments and exploration.

We are uncertain about what long-term plans we’ll decide on, and are in the process of generating new possible strategies. Some (non-mutually-exclusive) possibilities include:

* We may become a home to diverse research approaches aimed at developing a new path to alignment. Given our increased uncertainty about the best angle of attack, it may turn out to be valuable to house a more diverse portfolio of projects, with some level of intercommunication and cross-pollination between approaches.
* We may commit to an entirely new approach after a period of exploration, if we can identify one that we believe has a real chance of ensuring positive outcomes from AGI.
* We may carry forward theories and insights from our 2017-initiated research directions into future plans, in a different form.

### Research Write-Ups

Although our 2017-initiated research directions have been our largest focus over the last few years, we’ve been running many other research programs in parallel with it.

The bulk of this work is [nondisclosed-by-default](https://intelligence.org/2018/11/22/2018-update-our-new-research-directions/#section3) as well, but it includes work we’ve written up publicly. (Note that as a rule, this public-facing work is unrepresentative of our research as a whole.)

From our perspective, our most interesting public work this year is Scott Garrabrant’s Cartesian frames model and Vanessa Kosoy’s work on infra-Bayesianism.

[**Cartesian frames**](https://www.lesswrong.com/posts/BSpdshJWGAW6TuNzZ/introduction-to-cartesian-frames) is a new framework for thinking about agency, intended as a successor to the [cybernetic agent model](https://www.lesswrong.com/posts/BSpdshJWGAW6TuNzZ/introduction-to-cartesian-frames#6_1__Cybernetic_Agent_Model). Whereas the cybernetic agent model assumes as basic an agent and environment persisting across time with a defined and stable I/O channel, Cartesian frames treat these features as more derived and dependent on how one conceptually carves up physical situations.

The Cartesian Frames sequence focuses especially on finding derived, approximation-friendly versions of the notion of “subagent” (previously discussed in “[Embedded Agency](https://www.lesswrong.com/posts/i3BTagvt3HbPMx6PN/embedded-agency-full-text-version#5__Subsystem_alignment)”) and temporal sequence (a source of [decision-theoretic problems](https://www.lesswrong.com/posts/i3BTagvt3HbPMx6PN/embedded-agency-full-text-version#2__Decision_theory) in cases where agents can base their decisions on predictions or proofs about their own actions). The sequence’s final post discusses these and other potential [directions for future work](https://www.lesswrong.com/s/2A7rrZ4ySx6R8mfoT/p/JTzLjARpevuNpGPZm) for the field to build on.

In general, MIRI’s researchers are quite interested in new conceptual frameworks like these, as research progress can often be bottlenecked on our using the wrong lenses for thinking about problems, or on our lack of a simple formalism for putting intuitions to the test.

Meanwhile, Vanessa Kosoy and Alex Appel’s [**infra-Bayesianism**](https://www.lesswrong.com/posts/zB4f7QqKhBHa5b37a/introduction-to-the-infra-bayesianism-sequence) is a novel framework for modeling reasoning in cases where the reasoner’s hypothesis space may not include the true environment.

This framework is interesting primarily because it seems applicable to such a wide variety of problems: non-realizability, decision theory, anthropics, embedded agency, reflection, and the synthesis of induction/probability with deduction/logic. Vanessa describes infra-Bayesianism as “opening the way towards applying learning theory to many problems which previously seemed incompatible with it.”

2020 also saw a [large update](https://www.lesswrong.com/posts/9vYg8MyLL4cMMaPQJ/updates-and-additions-to-embedded-agency) to Scott and Abram’s “[Embedded Agency](https://intelligence.org/2018/10/29/embedded-agents/),” with some discussions clarified and several new subsections added. Additionally, a revised version of Vanessa’s “[Optimal Polynomial-Time Estimators: A Bayesian Notion of Approximation Algorithm](https://arxiv.org/abs/1608.04112),” co-authored with Alex Appel, was published in the *Journal of Applied Logics*.

To give a picture of some of the other research areas we’ve been pushing on, we asked some MIRI researchers and research associates to pick out highlights from their work over the past year, with comments on their selections.

Abram Demski highlights the following write-ups:

* “[An Orthodox Case Against Utility Functions](https://www.lesswrong.com/posts/A8iGaZ3uHNNGgJeaD/an-orthodox-case-against-utility-functions)” — “Although in some sense this is a small technical point, it is indicative of a shift in perspective in some recent agent foundations research which I think is quite important.”
* “[Radical Probabilism](https://www.lesswrong.com/posts/xJyY5QkQvNJpZLJRo/radical-probabilism)” — “Again, although one could see this as merely an explanation of the older [logical induction](https://intelligence.org/2016/09/12/new-paper-logical-induction/) result, I think it points at an important shift in perspective.”
* “[Learning Normativity: A Research Agenda](https://www.lesswrong.com/posts/2JGu9yxiJkoGdQR4s/learning-normativity-a-research-agenda)” — “In a sense, this research agenda clarifies the shift in perspective which the above two posts were communicating, although I haven’t tied everything together yet.
* “[Dutch-Booking CDT: Revised Argument](https://www.lesswrong.com/posts/X7k23zk9aBjjpgLd3/dutch-booking-cdt-revised-argument)” — “To my eye, this is a large-ish decision theory result.”

Evan Hubinger summarizes his public research from the past year:

* “[An Overview of 11 Proposals for Building Safe Advanced AI](https://arxiv.org/abs/2012.07532)” — “Probably my biggest project this year, this paper is my attempt at a unified explanation of the current major leading prosaic alignment proposals. The paper includes an exploration of each proposal’s pros and cons from the perspective of outer alignment, inner alignment, training competitiveness, and performance competitiveness.”
* “I started mentoring Adam Shimi and Mark Xu this year, helping them start spinning up work in AI safety. Concrete things that came out of this include Adam Shimi’s ‘[Universality Unwrapped](https://www.alignmentforum.org/posts/farherQcqFQXqRcvv/universality-unwrapped)’ and Mark Xu’s ‘[Does SGD Produce Deceptive Alignment?](https://www.alignmentforum.org/posts/ocWqg2Pf2br4jMmKA/does-sgd-produce-deceptive-alignment)’”
* “I spent a lot of time this year thinking about [AI safety via debate](https://openai.com/blog/debate/), which resulted in two new alternative debate proposals: ‘[AI Safety via Market Making](https://www.alignmentforum.org/posts/YWwzccGbcHMJMpT45/ai-safety-via-market-making)’ and ‘[Synthesizing Amplification and Debate](https://www.alignmentforum.org/posts/dJSD5RK6Qoidb3QY5/synthesizing-amplification-and-debate).’”
* “I spent some time looking at different alignment proposals from a computational complexity standpoint, resulting in ‘[Alignment Proposals and Complexity Classes](https://www.alignmentforum.org/posts/N64THGX7XNCqRtvPG/alignment-proposals-and-complexity-classes)’ and ‘[Weak HCH Accesses EXP](https://www.alignmentforum.org/posts/CtGH3yEoo4mY2taxe/weak-hch-accesses-exp).’
* “‘[Outer Alignment and Imitative Amplification](https://www.alignmentforum.org/posts/33EKjmAdKFn3pbKPJ/outer-alignment-and-imitative-amplification)’ makes the case for why imitative amplification is outer aligned; ‘[Learning the Prior and Generalization](https://www.alignmentforum.org/posts/YhQr36yGkhe6x8Fyn/learning-the-prior-and-generalization)’ provides my perspective on Paul’s new ‘learning the prior’ approach; and ‘[Clarifying Inner Alignment Terminology](https://www.alignmentforum.org/posts/SzecSPYxqRa5GCaSF/clarifying-inner-alignment-terminology)’ revisits terminology from ‘[Risks from Learned Optimization](https://arxiv.org/abs/1906.01820).’”

Earlier this year, Buck Shlegeris ([link](https://futureoflife.org/2020/04/15/an-overview-of-technical-ai-alignment-in-2018-and-2019-with-buck-shlegeris-and-rohin-shah/)) and Evan Hubinger ([link](https://futureoflife.org/2020/07/01/evan-hubinger-on-inner-alignment-outer-alignment-and-proposals-for-building-safe-advanced-ai/)) also appeared on the Future of Life Institute’s AI Alignment Podcast. Buck also gave a talk at Stanford: “[My Personal Cruxes for Working on AI Safety](https://forum.effectivealtruism.org/posts/Ayu5im98u8FeMWoBZ/my-personal-cruxes-for-working-on-ai-safety).”

Lastly, Future of Humanity Institute researcher and MIRI research associate Stuart Armstrong summarizes his own research highlights:

* “[Pitfalls of Learning a Reward Function Online](https://www.ijcai.org/Proceedings/2020/221),” working with DeepMind’s Jan Leike, Laurent Orseau, and Shane Legg — “This shows how agents can manipulate a “learning” process, the conditions that make that learning actually uninfluenceable, and some methods for turning influenceable learning processes into uninfluenceable ones.”
* “[Model Splintering](https://www.lesswrong.com/posts/k54rgSg7GcjtXnMHX/model-splintering-moving-from-one-imperfect-model-to-another-1)” — “Here I argue that a lot of AI safety problems can be reduced to the same problem: that of dealing with what happens when you move out of distribution from the training data. I argue that a principled way of dealing with these “model splinterings” is necessary to get safe AI, and sketch out some examples.”
* “[Syntax, Semantics, and Symbol Grounding, Simplified](https://www.lesswrong.com/posts/joPoxBpZjLNx8MKaF/syntax-semantics-and-symbol-grounding-simplified)” — “Here I argue that symbol grounding is a practical, necessary thing, not an abstract philosophical concept.”

### Process Improvements and Plans

Given the unusual circumstances brought on by the COVID-19 pandemic, in 2020 MIRI decided to run various experiments to see if we could improve our researchers’ productivity while our Berkeley office was unavailable. In the process, a sizable subset of our research team has found good modifications to our work environment that we aim to maintain and expand on.

Many of our research staff who spent this year in live-work quarantine groups in relatively rural areas in response to the COVID-19 pandemic have found surprisingly large benefits from living in a quieter, lower-density area together with a number of other researchers. Coordination and research have felt faster at a meta-level, with shorter feedback cycles, more efforts on more cruxy experiments, and more resulting pivots. Our biggest such pivot has been away from our 2017-initiated research directions, as described above.

Separately, MIRI staff have been weighing the costs and benefits of possibly moving somewhere outside the Bay Area for several years—taking into account the housing crisis and other governance failures, advantages and disadvantages of the local culture, tail risks of things taking a turn for the worse in the future, and other [factors](https://www.lesswrong.com/posts/FghubkDy6Dp6mnxk7/the-rationalist-community-s-location-problem?commentId=MPM6gaXJiCnGgEmDJ).

Partly as a result of these considerations, and partly because it’s easier to move when many of us have already relocated this year due to COVID-19, MIRI is considering relocating away from Berkeley. As we weigh the options, a particularly large factor in our considerations is whether our researchers expect the location, living situation, and work setup to feel good and comfortable, as we generally expect this to result in improved research progress. Increasingly, this factor is pointing us towards moving someplace new.

Many at MIRI have noticed in the past that there are certain social settings, such as small effective altruism or alignment research retreats, that seem to spark an unusually high density of unusually productive conversations. Much of the energy and vibrancy in such retreats presumably stems from their novelty and their time-boxed nature. However, we suspect this isn’t the only reason these events tend to be dense and productive, and we believe that we may be able to create a space that has some of these features every day.

This year, a number of our researchers have indeed felt that our new work set-up during the pandemic has a lot of this quality. We’re therefore very eager to see if we can modify MIRI as a workplace so as to keep this feature around, or further augment it.

Our year, then, has been characterized by some significant shifts in our thinking about research practices and which research directions are most promising.

Although we’ve been disappointed by our level of recent concrete progress toward understanding how to align AGI-grade optimization, we plan to continue capitalizing on MIRI’s strong pool of talent and accumulated thinking about alignment as we look for new and better paths forward. We’ll provide more updates about our new strategy as our plans solidify.

The post [2020 Updates and Strategy](https://intelligence.org/2020/12/21/2020-updates-and-strategy/) appeared first on [Machine Intelligence Research Institute](https://intelligence.org).


November 2020 Newsletter

MIRI researcher Scott Garrabrant has completed his [Cartesian Frames](https://www.lesswrong.com/s/2A7rrZ4ySx6R8mfoT) sequence. Scott also covers the first two posts' contents [in video form](https://www.youtube.com/watch?v=H1tJdaCvcck).

#### Other MIRI updates

* Contrary to my [previous announcement](https://intelligence.org/2020/10/23/october-2020-newsletter/), MIRI won’t be running a formal fundraiser this year, though we’ll still be participating in Giving Tuesday and other matching opportunities. To donate and get information on tax-advantaged donations, employer matching, etc., see [intelligence.org/donate](http://intelligence.org/donate). We’ll also be doing an end-of-the-year update and retrospective in the next few weeks.
* Facebook's Giving Tuesday matching event takes place this Tuesday (Dec. 1) at 5:00:00am PT.  Facebook will 100%-match the first $2M donated, something that will plausibly take only 2–3 seconds. To get 100%-matched, then, it's even more important than [last year](https://intelligence.org/2019/11/28/giving-tuesday-2019/) to start clicking at 4:59:59AM PT. Facebook will then 10%-match the next $50M of donations that are made. Details on optimizing your donation(s) to [MIRI's Facebook Fundraiser](https://www.facebook.com/donate/405215240508085/) can be found at [EA Giving Tuesday](https://eagivingtuesday.org/), a [Rethink Charity](https://rtcharity.org/) project.
* [Video discussion](https://www.youtube.com/watch?v=JVVj9Dui9es): Stuart Armstrong, Scott Garrabrant, and the [AI Safety Reading Group](https://aisafety.com/reading-group/) discuss Stuart's [If I Were A Well-Intentioned AI…](http://lesswrong.com/posts/gzWb5kWwzhdaqmyTt/if-i-were-a-well-intentioned-ai-i-image-classifier).
* MIRI research associate Vanessa Kosoy raises questions about [AI information hazards](https://www.lesswrong.com/posts/3D3DsX5rMbk3jEZ5h).
* Buck Shlegeris argues that [we're likely at the “hinge of history”](https://forum.effectivealtruism.org/posts/j8afBEAa7Xb2R9AZN/thoughts-on-whether-we-re-living-at-the-most-influential) (assuming we aren't living in a simulation).
* To make it easier to find and cite old versions of MIRI papers (especially ones that aren't on arXiv), we've collected links to obsolete versions on [intelligence.org/revisions](https://intelligence.org/revisions/).

#### News and links

* CFAR's Anna Salamon asks: [Where do (did?) stable, cooperative institutions come from?](https://www.lesswrong.com/posts/R4FX6wDmppvZ2JqpB/where-do-did-stable-cooperative-institutions-come-from)
* The Center for Human-Compatible AI is accepting applications for [research internships](https://humancompatible.ai/jobs#internship) through Dec. 13.
* [AI Safety (virtual) Camp](https://aisafety.camp/) is accepting applications through Dec. 15.
* The 4th edition of *Artificial Intelligence: A Modern Approach* is out, with [expanded discussion of the alignment problem](https://www.lesswrong.com/posts/BNJx2CqfXyiusoJcK/artificial-intelligence-a-modern-approach-4th-edition-on-the).
* DeepMind's Rohin Shah [reviews](https://www.lesswrong.com/posts/gYfgWSxCpFdk2cZfE/the-alignment-problem-machine-learning-and-human-values) Brian Christian's new book [*The Alignment Problem: Machine Learning and Human Values*](https://www.amazon.com/Alignment-Problem-Machine-Learning-Values/dp/153669519X).
* Daniel Filan and Rohin Shah discuss [security mindset and takeoff speeds](https://www.lesswrong.com/posts/Lfk2FXBwrpoM6Jm7p).
* *Fortune* [profiles](https://fortune.com/2020/11/13/jaan-tallinn-ai-safety-bitcoin-cryptocurrency-elon-musk/) existential risk philanthropist Jaan Tallinn.

The post [November 2020 Newsletter](https://intelligence.org/2020/11/30/november-2020-newsletter/) appeared first on [Machine Intelligence Research Institute](https://intelligence.org).


October 2020 Newsletter

Starting today, Scott Garrabrant has begun posting [**Cartesian Frames**](https://www.lesswrong.com/posts/BSpdshJWGAW6TuNzZ/introduction-to-cartesian-frames), a sequence introducing a new conceptual framework Scott has found valuable for thinking about agency.
In Scott's words: Cartesian Frames are “applying *reasoning* like Pearl's to *objects* like game theory's, with a *motivation* like Hutter's”.

Scott will be giving an online talk introducing Cartesian frames this Sunday at 12pm PT on Zoom ([link](https://us02web.zoom.us/j/81328320546)). He'll also be hosting office hours on Gather.Town the next four Sundays; [see here for details](https://www.lesswrong.com/posts/N4uDrgFoZKJXhnHLw/sunday-october-25-12-00pm-pt-scott-garrabrant-on-cartesian).

#### Other MIRI updates

* Abram Demski discusses the problem of [comparing utilities](https://www.lesswrong.com/posts/cYsGrWEzjb324Zpjx), highlighting some non-obvious implications.
* In March 2020, the US Congress passed the [CARES Act](https://www.forbes.com/sites/berniekent/2020/04/03/giving-more-than-60-of-income-to-charity-cares-act-says-deduct-it/), which changes the tax advantages of donations to qualifying NPOs like MIRI in 2020. Changes include:
	+ 1. A new “above the line” tax deduction: up to $300 per taxpayer ($600 for a married couple) in annual charitable contributions for people who take the standard deduction. Donations to donor-advised funds (DAFs) do not qualify for this new deduction.
	+ 2. New charitable deduction limits: Taxpayers who itemize their deductions can deduct much greater amounts of their contributions. Individuals can elect to deduct donations up to 100% of their 2020 AGI — up from 60% previously. This higher limit also does not apply to donations to DAFs.
 As usual, consult with your tax advisor for more information.
* Our fundraiser this year will start on November 29 (two days before Giving Tuesday) and finish on January 2. We're hoping that having our fundraiser straddle 2020 and 2021 will give people more flexibility given the unusual tax law changes above.
* I'm happy to announce that MIRI has received a donation of $246,435 from an anonymous returning donor. Our thanks to the donor, and to [Effective Giving UK](https://www.effectivegiving.org/) for facilitating this donation!

#### News and links

* Richard Ngo tries to provide a relatively careful and thorough version of the standard argument for worrying about AGI risk: [AGI Safety from First Principles](https://www.alignmentforum.org/s/mzgtmmTKKn5MuCzFJ). See also Rohin Shah's [summary](https://mailchi.mp/051273eb96eb/an-122arguing-for-agi-driven-existential-risk-from-first-principles).
* The AI Alignment Podcast [interviews Andrew Critch](https://futureoflife.org/2020/09/15/andrew-critch-on-ai-research-considerations-for-human-existential-safety/?cn-reloaded=1) about his recent overview paper, “[AI Research Considerations for Human Existential Safety](https://arxiv.org/pdf/2006.04948.pdf).”

The post [October 2020 Newsletter](https://intelligence.org/2020/10/23/october-2020-newsletter/) appeared first on [Machine Intelligence Research Institute](https://intelligence.org).


September 2020 Newsletter

Abram Demski and Scott Garrabrant have made [a major update](https://www.lesswrong.com/posts/9vYg8MyLL4cMMaPQJ/updates-and-additions-to-embedded-agency) to "[Embedded Agency](https://www.lesswrong.com/s/Rm6oQRJJmhGCcLvxh/p/i3BTagvt3HbPMx6PN)", with new discussions of ε-exploration, Newcomblike problems, reflective oracles, logical uncertainty, Goodhart's law, and predicting rare catastrophes, among other topics.

Abram has also written an overview of what good reasoning looks in the absence of Bayesian updating: [Radical Probabilism](https://www.lesswrong.com/s/HmANELvkhAZ9eDxFS/p/xJyY5QkQvNJpZLJRo). One recurring theme:

> [I]n general (i.e., *without any special prior which does guarantee convergence for restricted observation models*), a Bayesian relies on a realizability (aka grain-of-truth) assumption for convergence, as it does for some other nice properties. Radical probabilism demands these properties without such an assumption.
>
>
> [… C]onvergence points at a notion of "objectivity" for the radical probabilist. Although the individual updates a radical probabilist makes can go all over the place, the beliefs must eventually settle down to something. The goal of reasoning is to settle down to that answer as quickly as possible.
>
>

Meanwhile, [Infra-Bayesianism](https://www.lesswrong.com/posts/zB4f7QqKhBHa5b37a/introduction-to-the-infra-bayesianism-sequence) is a new formal framework for thinking about optimal reasoning without requiring an reasoner's true environment to be in its hypothesis space. Abram comments: "Alex Appel and Vanessa Kosoy have been working hard at 'Infra-Bayesianism', a new approach to RL which aims to make it easier (ie, possible) to prove safety-relevant theorems (and, also, a new approach to Bayesianism more generally).

#### Other MIRI updates

* Abram Demski writes a parable on the differences between logical inductors and Bayesians: [The Bayesian Tyrant](https://www.lesswrong.com/posts/4tke3ibK9zfnvh9sE).
* Building on the [selection vs. control](https://www.lesswrong.com/posts/ZDZmopKquzHYPRNxq/selection-vs-control) distinction, Abram contrasts ["mesa-search" and "mesa-control"](https://www.lesswrong.com/posts/WmBukJkEFM72Xr397).

#### News and links

* From OpenAI's Stiennon et al.: [Learning to Summarize with Human Feedback](https://openai.com/blog/learning-to-summarize-with-human-feedback/). MIRI researcher Eliezer Yudkowsky [comments](https://twitter.com/ESYudkowsky/status/1301954347933208578):

> A very rare bit of research that is directly, straight-up relevant to real alignment problems! They trained a reward function on human preferences *and then* measured how hard you could optimize against the trained function before the results got actually worse.
>
>
> [… Y]ou can ask for results as good as the best 99th percentile of rated stuff in the training data (a la Jessica Taylor's [quantilization](https://intelligence.org/files/QuantilizersSaferAlternative.pdf) idea).  Ask for things the trained reward function rates as "better" than that, and [it starts to find "loopholes" as seen from outside the system](https://arbital.com/p/goodharts_curse/); places where the trained reward function poorly matches your real preferences, instead of places where your real preferences would rate high reward.
>
>
* Chi Nguyen writes up [an introduction to Paul Christiano's iterated amplification research agenda](https://www.lesswrong.com/posts/PT8vSxsusqWuN7JXp#fnref-7XLtvdgT8DkovpShi-2) that seeks to be the first such resource that is "both easy to understand and [gives] a complete picture". The post includes inline comments by Christiano.
* Forecasters share [visualizations of their AI timelines](https://www.lesswrong.com/posts/hQysqfSEzciRazx8k) on LessWrong.

The post [September 2020 Newsletter](https://intelligence.org/2020/09/10/september-2020-newsletter/) appeared first on [Machine Intelligence Research Institute](https://intelligence.org).


August 2020 Newsletter

#### MIRI updates

* Three questions from MIRI's Abram Demski: [What does it mean to apply decision theory?](https://www.lesswrong.com/posts/wgdfBtLmByaKYovYe), [How “honest” is GPT-3?](https://www.lesswrong.com/posts/c3RsLTcxrvH4rXpBL), and [How should AI debate be judged?](https://www.lesswrong.com/posts/m7oGxvouzzeQKiGJH)
* A transcript from MIRI researcher Scott Garrabrant: [What Would I Do? Self-Prediction in Simple Algorithms](https://www.lesswrong.com/posts/PiXS9kE4qX68KveCt/what-would-i-do-self-prediction-in-simple-algorithms).
* MIRI researcher Buck Shlegeris reviews the debate on [what the history of nuclear weapons implies about humanity's ability to coordinate](https://www.lesswrong.com/posts/y3jDSoTTdBD9Nj3Gx/how-good-is-humanity-at-coordination).
* From MIRI's Evan Hubinger: [Learning the Prior and Generalization](https://www.lesswrong.com/posts/YhQr36yGkhe6x8Fyn) and [Alignment Proposals and Complexity Classes](https://www.lesswrong.com/posts/N64THGX7XNCqRtvPG).
* Rafael Harth's [Inner Alignment: Explain Like I'm 12 Edition](https://www.lesswrong.com/posts/AHhCrJ2KpTjsCSwbt/inner-alignment-explain-like-i-m-12-edition) summarizes the concepts and takeaways from “[Risks from Learned Optimization](https://intelligence.org/learned-optimization/)”.
* Issa Rice [reviews discussion to date](https://www.lesswrong.com/posts/BGxTpdBGbwCWrGiCL) on MIRI's research focus, “To what extent is it possible to have a precise theory of rationality?”, and the relationship between [deconfusion](https://intelligence.org/2018/11/22/2018-update-our-new-research-directions/) research and safety outcomes. (Plus [a short reply](https://www.lesswrong.com/posts/BGxTpdBGbwCWrGiCL?commentId=xTpGttMrLwfXEyTeW).)
* “Pitfalls of Learning a Reward Function Online” ([IJCAI paper](https://www.ijcai.org/Proceedings/2020/221), [LW summary](https://www.lesswrong.com/posts/LpjjWDBXr88gzcYK2/learning-and-manipulating-learning)): FHI researcher and MIRI research associate Stuart Armstrong, with DeepMind's Jan Leike, Laurent Orseau, and Shane Legg, explore ways to discourage agents from manipulating their reward signal to be easier to optimize.

#### News and links

* From Paul Christiano: [Learning the Prior](https://www.alignmentforum.org/posts/SL9mKhgdmDKXmxwE4/learning-the-prior) and [Better Priors as a Safety Problem](https://www.alignmentforum.org/posts/roA83jDvq7F2epnHK/better-priors-as-a-safety-problem).
* From Victoria Krakovna: [Tradeoff Between Desirable Properties for Baseline Choices in Impact Measures](https://www.alignmentforum.org/posts/nLhfRpDutEdgr6PKe/tradeoff-between-desirable-properties-for-baseline-choices).
* Ben Pace [summarizes Christiano's “What Failure Looks Like” post](https://www.lesswrong.com/posts/6jkGf5WEKMpMFXZp2) and the resultant discussion.
* Kaj Sotala collects recent examples of [experiences from people working with GPT-3](https://www.lesswrong.com/posts/6Hee7w2paEzHsD6mn/collection-of-gpt-3-results).

The post [August 2020 Newsletter](https://intelligence.org/2020/08/13/august-2020-newsletter/) appeared first on [Machine Intelligence Research Institute](https://intelligence.org).


July 2020 Newsletter

After completing a study fellowship at MIRI that he began in late 2019, Blake Jones is joining the MIRI research team full-time! Blake joins MIRI after a long career working on low-level software systems such as the Solaris operating system and the Oracle database.
#### Other MIRI updates

* MIRI researcher Evan Hubinger goes on the FLI podcast ([transcript/discussion](https://www.alignmentforum.org/posts/qZGoHkRgANQpGHWnu/evan-hubinger-on-inner-alignment-outer-alignment-and), [audio](https://futureoflife.org/2020/07/01/evan-hubinger-on-inner-alignment-outer-alignment-and-proposals-for-building-safe-advanced-ai/)) to discuss “inner alignment, outer alignment, and proposals for building safe advanced AI”.
* A revised version of Vanessa Kosoy’s “[Optimal Polynomial-Time Estimators: A Bayesian Notion of Approximation Algorithm](https://arxiv.org/abs/1608.04112),” co-authored with Alex Appel, has been accepted to the *Journal of Applied Logics*.
* From MIRI researcher Abram Demski: [Dutch-Booking CDT: Revised Argument](https://www.lesswrong.com/posts/X7k23zk9aBjjpgLd3) argues that “causal” theories (ones using counterfactuals to evaluate expected value) must behave the same as theories using conditional probabilities. [Relating HCH and Logical Induction](https://www.lesswrong.com/posts/R3HAvMGFNJGXstckQ) discusses amplification in the context of reflective oracles. And [Radical Probabilism](https://www.lesswrong.com/posts/ZM63n353vh2ag7z4p/radical-probabilism-transcript) reviews the surprising gap between Dutch-book arguments and Bayes' rule.

#### News and links

* Alex Flint summarizes the Center for Human-Compatible AI's [assistance games research program](https://www.alignmentforum.org/posts/qPoaA5ZSedivA4xJa/my-take-on-chai-s-research-agenda-in-under-1500-words).
* CHAI’s Andrew Critch and MILA’s David Krueger release “[AI Research Considerations for Human Existential Safety (ARCHES)](https://arxiv.org/pdf/2006.04948.pdf)”, a review of 29 AI (existential) safety research directions, each with an illustrative analogy, examples of current work and potential synergies between research directions, and discussion of ways the research approach might lower (or raise) existential risk.
* OpenAI’s Danny Hernandez and Tom Brown [present evidence](https://openai.com/blog/ai-and-efficiency/) that “for AI tasks with high levels of recent investment, algorithmic progress has yielded more gains than classical hardware efficiency”.
* DeepMind's Victoria Krakovna shares her [takeaways from the COVID-19 pandemic for slow-takeoff scenarios](https://www.lesswrong.com/posts/wTKjRFeSjKLDSWyww/possible-takeaways-from-the-coronavirus-pandemic-for-slow-ai).
* AI Impacts’ Daniel Kokotajlo discusses [possible changes the world might undergo before reaching AGI](https://www.lesswrong.com/posts/zjhZpZi76kEBRnjiw/relevant-pre-agi-possibilities).
* 80,000 Hours describes [careers they view as promising but haven’t written up as priority career paths](https://forum.effectivealtruism.org/posts/6x2MjPXhpPpnatJFQ/some-promising-career-ideas-beyond-80-000-hours-priority), including information security ([previously discussed here](https://forum.effectivealtruism.org/posts/ZJiCfwTy5dC4CoxqA/information-security-careers-for-gcr-reduction)).

The post [July 2020 Newsletter](https://intelligence.org/2020/07/08/july-2020-newsletter/) appeared first on [Machine Intelligence Research Institute](https://intelligence.org).


June 2020 Newsletter

MIRI researcher Evan Hubinger reviews “[11 different proposals for building safe advanced AI under the current machine learning paradigm](https://www.alignmentforum.org/posts/fRsjBseRuvRhMPPE5/an-overview-of-11-proposals-for-building-safe-advanced-ai)”, comparing them on outer alignment, inner alignment[,](https://arxiv.org/abs/1906.01820) training competitiveness, and performance competitiveness.
#### Other updates

* We keep being amazed by new shows of support — following our [last](https://intelligence.org/2020/04/27/miris-largest-grant-to-date/) [two](https://intelligence.org/2020/05/29/may-2020-newsletter/) announcements, MIRI has received a donation from another anonymous donor totaling ~$265,000 in euros, facilitated by [Effective Giving UK](https://www.effectivegiving.org/) and the [Effective Altruism Foundation](https://ea-foundation.org/). Massive thanks to the donor for their generosity, and to both organizations for their stellar support for MIRI and other longtermist organizations!
* Hacker News [discusses](https://news.ycombinator.com/item?id=23401328) Eliezer Yudkowsky's [There's No Fire Alarm for AGI](https://intelligence.org/2017/10/13/fire-alarm/).
* MIRI researcher Buck Shlegeris talks about [deference and inside-view models](https://forum.effectivealtruism.org/posts/53JxkvQ7RKAJ4nHc4/some-thoughts-on-deference-and-inside-view-models) on the EA Forum.
* OpenAI [unveils GPT-3](https://arxiv.org/pdf/2005.14165.pdf), a massive 175-*billion* parameter language model that can figure out how to solve a variety of problems without task-specific training or fine-tuning. Gwern Branwen's pithy [summary](https://twitter.com/gwern/status/1267215588214136833):

> GPT-3 is terrifying because it's a tiny model compared to what's possible, trained in the dumbest way possible on a single impoverished modality on tiny data, yet the first version already manifests crazy runtime meta-learning—and the scaling curves *still* are not bending!
>
>

Further discussion [by Branwen](https://www.gwern.net/newsletter/2020/05#gpt-3) and [by Rohin Shah](https://www.lesswrong.com/posts/D3hP47pZwXNPRByj8/an-102-meta-learning-by-gpt-3-and-a-list-of-full-proposals).
* Stuart Russell gives this year's Turing Lecture online, discussing “[provably beneficial AI](https://www.youtube.com/watch?v=_H87qqT8pdY)”.

The post [June 2020 Newsletter](https://intelligence.org/2020/06/08/june-2020-newsletter/) appeared first on [Machine Intelligence Research Institute](https://intelligence.org).


May 2020 Newsletter

MIRI has received an anonymous donation of ~$275,000 in euros, facilitated by [Effective Giving UK](https://www.effectivegiving.org/). Additionally, the Survival and Flourishing Fund, working with funders Jaan Tallinn and Jed McCaleb, has announced [$340,000 in grants to MIRI](http://survivalandflourishing.fund/sff-2020-h1). SFF is a new fund that is [taking over much of BERI's grantmaking work](http://existence.org/tallinn-grants-future/).

To everyone involved in both decisions to support our research, thank you!

#### Other updates

* [An Orthodox Case Against Utility Functions](https://www.lesswrong.com/posts/A8iGaZ3uHNNGgJeaD): MIRI researcher Abram Demski makes the case against utility functions that rely on a microphysical “view from nowhere”.
* Stuart Armstrong’s “[If I were a well-intentioned AI…](https://www.lesswrong.com/s/knbhjv252HshMSwpt/p/gzWb5kWwzhdaqmyTt)” sequence looks at alignment problems from the perspective of a well-intentioned but ignorant agent.
* AI Impacts' Asya Bergal summarizes [takeaways from “safety-by-default” researchers](https://aiimpacts.org/takeaways-from-safety-by-default-interviews/).
* Agarwal and Norouzi report [improvements in offline RL](https://ai.googleblog.com/2020/04/an-optimistic-perspective-on-offline.html).
* Daniel Kokotajlo’s [Three Kinds of Competitiveness](https://www.lesswrong.com/posts/sD6KuprcS3PFym2eM/three-kinds-of-competitiveness) distinguishes performance-competitive, cost-competitive, and date-competitive AI systems.
* The *Stanford Encyclopedia of Philosophy*'s new [Ethics of AI and Robotics](https://plato.stanford.edu/entries/ethics-ai/) article includes a discussion of existential risk from AGI.

The post [May 2020 Newsletter](https://intelligence.org/2020/05/29/may-2020-newsletter/) appeared first on [Machine Intelligence Research Institute](https://intelligence.org).


April 2020 Newsletter

[MIRI has been awarded its largest grant to date](https://intelligence.org/2020/04/27/miris-largest-grant-to-date/) — $7,703,750 split over two years from Open Philanthropy, in partnership with Ben Delo, co-founder of the cryptocurrency trading platform BitMEX!
(**Update:** Open Philanthropy decided not to move forward with its [partnership with Ben Delo](https://openphilanthropy.org/research/co-funding-partnership-with-ben-delo/). This doesn’t affect the size of the grant we received, but means that other funders provided the funds for Open Phil’s grant.)

We have also been awarded generous grants by the Berkeley Existential Risk Initiative ($300,000) and the Long-Term Future Fund ($100,000). Our thanks to everybody involved!

#### Other updates

* Buck Shlegeris of MIRI and Rohin Shah of CHAI discuss Rohin's 2018–2019 overview of technical AI alignment research [on the AI Alignment Podcast](https://www.lesswrong.com/posts/6skeZgctugzBBEBw3/ai-alignment-podcast-an-overview-of-technical-ai-alignment).
* From MIRI's Abram Demski: [Thinking About Filtered Evidence Is (Very!) Hard](https://www.lesswrong.com/posts/fhJkQo34cYw6KqpH3/thinking-about-filtered-evidence-is-very-hard) and [Bayesian Evolving-to-Extinction](https://www.lesswrong.com/posts/u9Azdu6Z7zFAhd4rK/bayesian-evolving-to-extinction). And from Evan Hubinger: [Synthesizing Amplification and Debate](https://www.lesswrong.com/posts/dJSD5RK6Qoidb3QY5/synthesizing-amplification-and-debate).
* From OpenAI's Beth Barnes, Paul Christiano, Long Ouyang, and Geoffrey Irving: [Progress on AI Safety via Debate](https://www.lesswrong.com/posts/Br4xDbYu4Frwrb64a/writeup-progress-on-ai-safety-via-debate-1).
* [Zoom In: An Introduction to Circuits](https://www.lesswrong.com/posts/MG4ZjWQDrdpgeu8wG/zoom-in-an-introduction-to-circuits): OpenAI's Olah, Cammarata, Schubert, Goh, Petrov, and Carter argue, “Features are the fundamental unit of neural networks. They correspond to directions [in the space of neuron activations]. […] Features are connected by weights, forming circuits. […] Analogous features and circuits form across models and tasks.”
* [DeepMind's Agent57](https://www.lesswrong.com/posts/ygb6ryKcScJxhmwQo/atari-early) appears to meet one of the AI benchmarks in [AI Impacts' 2016 survey](https://arxiv.org/abs/1705.08807), “outperform professional game testers on all Atari games using no game specific knowledge”, earlier than NeurIPS/ICML authors predicted.
* From DeepMind Safety Research: [Specification gaming: the flip side of AI ingenuity](https://medium.com/@deepmindsafetyresearch/specification-gaming-the-flip-side-of-ai-ingenuity-c85bdb0deeb4).

The post [April 2020 Newsletter](https://intelligence.org/2020/05/01/april-2020-newsletter/) appeared first on [Machine Intelligence Research Institute](https://intelligence.org).


MIRI’s largest grant to date!

A big announcement today: MIRI has been awarded a two-year **[$7,703,750 grant](https://www.openphilanthropy.org/focus/global-catastrophic-risks/potential-risks-advanced-artificial-intelligence/machine-intelligence-research-institute-general-support-2020)** by Open Philanthropy — our largest grant to date. In combination with the ~$1.06M Open Philanthropy is also disbursing to MIRI this year (the second half of their [2019 grant](https://intelligence.org/2019/04/01/new-grants-open-phil-beri/)), this amounts to $4.38M per year over two years, or roughly 60% of our predicted 2020–2021 budgets.

While ~$6.24M of Open Philanthropy’s new grant comes from [their main funders](https://www.openphilanthropy.org/about/who-we-are), $1.46M was made possible by [Open Philanthropy’s new partnership with Ben Delo](https://www.openphilanthropy.org/blog/co-funding-partnership-ben-delo), co-founder of the cryptocurrency trading platform BitMEX. Ben Delo has teamed up with Open Philanthropy to support their long-termist grantmaking, which includes (quoting Open Philanthropy):

> reducing potential risks from advanced artificial intelligence, furthering biosecurity and pandemic preparedness, and other initiatives to combat global catastrophic risks, as well as much of the work we fund on effective altruism.
>
>

We’re additionally happy to announce a **$300,000 grant** from the [Berkeley Existential Risk Initiative](http://existence.org/). I’ll note that at the time of our [2019 fundraiser](https://intelligence.org/2019/12/02/miris-2019-fundraiser/), we expected to receive a grant from BERI in early 2020, and incorporated this into our reserves estimates. However, we predicted the grant size would be $600k; now that we know the final grant amount, that estimate should be $300k lower.

Finally, MIRI has been awarded a **[$100,000 grant](https://app.effectivealtruism.org/funds/far-future/payouts/3waQ7rp3Bfy4Lwr5sZP9TP)** by the Effective Altruism Funds [Long-Term Future Fund](https://app.effectivealtruism.org/funds/far-future), managed by the Centre for Effective Altruism. The fund plans to release a write-up describing the reasoning for their new round of grants in a couple of weeks.

Our thanks to Open Phil, Ben Delo and [Longview Philanthropy](http://www.longview.org/) (Ben Delo’s philanthropic advisor, formerly known as Effective Giving UK), BERI, and the Long-Term Future Fund for this amazing support! Going into 2020–2021, these new grants put us in an unexpectedly good position to grow and support our research team. To learn more about our growth plans, see our [2019 fundraiser post](https://intelligence.org/2019/12/02/miris-2019-fundraiser/) and our [2018 strategy update](https://intelligence.org/2018/11/22/2018-update-our-new-research-directions/).

---

**Update:** Open Philanthropy decided not to move forward with its [partnership with Ben Delo](https://openphilanthropy.org/research/co-funding-partnership-with-ben-delo/). This doesn’t affect the size of the grant we received, but means that other funders provided the funds for Open Phil’s grant.

The post [MIRI’s largest grant to date!](https://intelligence.org/2020/04/27/miris-largest-grant-to-date/) appeared first on [Machine Intelligence Research Institute](https://intelligence.org).


March 2020 Newsletter

As the COVID-19 pandemic progresses, the LessWrong team has put together [a database of resources](https://www.lesswrong.com/coronavirus-link-database) for learning about the disease and staying updated, and 80,000 Hours has a new write-up on [ways to help in the fight against COVID-19](https://80000hours.org/articles/covid-19-what-should-you-do/). In my non-MIRI time, I've been keeping my own quick and informal notes on various sources' COVID-19 recommendations [in this Google Doc](https://docs.google.com/document/d/10MFFoUMYHqGB3cLCxuqhsM2kahtOQVybPhLG-YNZdaI/edit). Stay safe out there!

#### Updates

* [My personal cruxes for working on AI safety](https://forum.effectivealtruism.org/posts/Ayu5im98u8FeMWoBZ/my-personal-cruxes-for-working-on-ai-safety): a talk transcript from MIRI researcher Buck Shlegeris.
* Daniel Kokotajlo of AI Impacts discusses [Cortés, Pizarro, and Afonso as Precedents for Takeover](https://www.lesswrong.com/posts/ivpKSjM4D6FbqF4pZ/cortes-pizarro-and-afonso-as-precedents-for-takeover).
* O'Keefe, Cihon, Garfinkel, Flynn, Leung, and Dafoe's “[The Windfall Clause](https://www.fhi.ox.ac.uk/windfallclause/)” proposes “an ex ante commitment by AI firms to donate a significant amount of any eventual extremely large profits” that result from “fundamental, economically transformative breakthroughs” like AGI.
* Microsoft announces the 17-billion-parameter language model [Turing-NLG](https://www.microsoft.com/en-us/research/blog/turing-nlg-a-17-billion-parameter-language-model-by-microsoft/).
* Oren Etzioni thinks AGI is [too far off](https://intelligence.org/2017/10/13/fire-alarm/) to deserve much thought, and cites Andrew Ng's “overpopulation on Mars” metaphor approvingly — but he's also moving the debate in a very positive direction [by listing specific observations that would make him change his mind](https://www.technologyreview.com/s/615264/artificial-intelligence-destroy-civilization-canaries-robot-overlords-take-over-world-ai/).

The post [March 2020 Newsletter](https://intelligence.org/2020/04/01/march-2020-newsletter/) appeared first on [Machine Intelligence Research Institute](https://intelligence.org).


February 2020 Newsletter

#### Updates

* Colm [reviews our 2019 fundraiser](https://intelligence.org/2020/02/13/our-2019-fundraiser-review/): taking into account matching, we received a total of $601,120 from 250+ donors. Our thanks again for all the support!
* Evan Hubinger's [Exploring Safe Exploration](https://www.lesswrong.com/posts/NBffcjqm2P4dNbjrE) clarifies points he raised in [Safe Exploration and Corrigibility](https://www.lesswrong.com/posts/87Y7w73phjBxnPyPD/safe-exploration-and-corrigibility). The issues raised here are somewhat subtler than may be immediately apparent, since we tend to discuss things in ways that collapse the distinctions Evan is making.
* Logician Arthur Milchior [reviews the AIRCS workshop and MIRI's application process](https://www.lesswrong.com/posts/EYhWxoaCK2YSpTei9/aircs-workshop-how-i-failed-to-be-recruited-at-miri) based on his first-hand experience with both. See also [follow-up discussion](https://www.lesswrong.com/posts/EYhWxoaCK2YSpTei9/aircs-workshop-how-i-failed-to-be-recruited-at-miri#comments) with MIRI and CFAR staff.
* Rohin Shah posts an in-depth [2018–19 review of the field of AI alignment](https://www.lesswrong.com/posts/dKxX76SCfCvceJXHv).
* From Shevlane and Dafoe's new “[The Offense-Defense Balance of Scientific Knowledge: Does Publishing AI Research Reduce Misuse?](https://arxiv.org/abs/2001.00463)”:

> [T]he existing conversation around AI has heavily borrowed concepts and conclusions from one particular field: vulnerability disclosure in computer security. We caution against AI researchers treating these lessons as immediately applicable. There are important differences between vulnerabilities in software and the types of vulnerabilities exploited by AI. […]
>
>
>  Patches to software are often easy to create, and can often be made in a matter of weeks. These patches fully resolve the vulnerability. The patch can be easily propagated: for downloaded software, the software is often automatically updated over the internet; for websites, the fix can take effect immediately.
>
>
>  [… F]or certain technologies, there is no low-cost, straightforward, effective defence. [… C]onsider biological research that provides insight into the manufacture of pathogens, such as a novel virus. A subset of viruses are very difficult to vaccinate for (there is still no vaccination for HIV) or otherwise prepare against. This lowers the defensive benefit of publication, by blocking a main causal pathway by which publication leads to greater protection. This contrasts with the case where an effective treatment can be developed within a reasonable time period[.]
>
>
* Yann LeCun and Eliezer Yudkowsky [discuss the concept “AGI”](https://twitter.com/ESYudkowsky/status/1204114830174482434).
* CFAR's Anna Salamon contrasts [“reality-revealing” and “reality-masking” puzzles](https://www.lesswrong.com/posts/byewoxJiAfwE6zpep/reality-revealing-and-reality-masking-puzzles).
* Scott Alexander [reviews Stuart Russell's *Human Compatible*](https://slatestarcodex.com/2020/01/30/book-review-human-compatible/).

#### Links from the research team

MIRI researchers anonymously summarize and comment on recent posts and papers:

* Re [ACDT: a hack-y acausal decision theory](https://www.lesswrong.com/posts/9m2fzjNSJmd3yxxKG) — “Stuart Armstrong calls this decision theory a hack. I think it might be more elegant than he's letting on (i.e., a different formulation could look less hack-y), and is getting at something.”
* Re [Predictors exist: CDT going bonkers… forever](https://www.lesswrong.com/posts/Kr76XzME7TFkN937z) — “I don't think Stuart Armstrong's example really adds much over some variants of Death in Damascus, but there's some good discussion of CDT vs. EDT stuff in the comments.”
* Re [Is the term mesa optimizer too narrow?](https://www.lesswrong.com/posts/nFDXq7HTv9Xugcqaw/is-the-term-mesa-optimizer-too-narrow) — “Matthew Barnett poses the important question, '[I]f even humans are not mesa optimizers, why should we expect mesa optimizers to be the primary real world examples of [malign generalization]?'”
* Re [Malign generalization without internal search](https://www.lesswrong.com/posts/ynt9TD6PrYw6iT49m) — “I think Matthew Barnett's question here is an important one. I lean toward the 'yes, this is a problem' camp—I don't think we can entirely eliminate malign generalization by eliminating internal search. But it is possible that this falls into other categories of misalignment (which we don't want to term 'inner alignment').”
* Re [(A -> B) -> A in Causal DAGs](https://www.lesswrong.com/posts/G25RBnBk5BNpv3KyF) and [Formulating Reductive Agency in Causal Models](https://www.lesswrong.com/posts/qrWFvMnRm4SkKnpRZ) — “I've wanted something like this for a while. Bayesian influence diagrams model agents non-reductively, by boldly asserting that some nodes are agentic. Can we make models which represent agents, without declaring a basic 'agent' type like that? John Wentworth offers an approach, representing agents via 'strange loops' across a use-mention boundary; and discusses how this might break down even further, with fully reductive agency. I'm not yet convinced that Wentworth has gotten it right, but it's exciting to see an attempt.”

The post [February 2020 Newsletter](https://intelligence.org/2020/02/23/february-2020-newsletter/) appeared first on [Machine Intelligence Research Institute](https://intelligence.org).


Our 2019 Fundraiser Review

Our [2019 fundraiser](https://intelligence.org/2019/12/02/miris-2019-fundraiser/) ended on December 31 with the month-long campaign raising $601,120[1](https://intelligence.org/feed/?paged=10#fn1) from over[2](https://intelligence.org/feed/?paged=10#fn2) 259 donors. While we didn’t reach our $1M fundraising goal, the amount raised will be a very substantial help to us going into 2020. We’re grateful to all who supported us during the fundraiser.

20% of our fundraiser total was raised in a single minute on December 3 during Facebook’s [Giving Tuesday](https://intelligence.org/2019/11/28/giving-tuesday-2019/) matching event, deserving special mention. Facebook’s matching pool of $7M was exhausted within 14 seconds of their event’s 5am PST start but in spite of the shortness of the event, a cohort of lightning-fast early risers secured a significant amount of Facebook matching for MIRI this year; of the $77,325 donated to MIRI on Giving Tuesday, $45,915 was donated early enough to be matched by Facebook. Thank you to everybody who set their clocks early to support us[3](https://intelligence.org/feed/?paged=10#fn3) and a shout out to the [EA Giving Tuesday](https://www.eagivingtuesday.org/)/[Rethink Charity](https://www.rtcharity.org/) collaboration which helped direct $563k in matching funds to EA nonprofits on Giving Tuesday.

[![](https://intelligence.org/wp-content/uploads/2019/11/GT19Chart-PostMatch.png)](https://intelligence.org/wp-content/uploads/2019/11/GT19Chart-PostMatch2.png)

Beyond Giving Tuesday, we’re very grateful to [the Effective Altruism Foundation](https://ea-foundation.org/) for providing a channel for residents of a number of EU countries including Germany, Switzerland, and the Netherlands to donate to MIRI in [a tax-advantaged way](https://intelligence.org/donate/tax-advantaged-donations/) and also to donors who leveraged their companies’ matching generosity to maximum effect during the fundraiser.

Our fundraiser fell well short of our $1M target this year, and also short of our in-fundraiser support in 2018 ($947k) and 2017 ($2.5M). It’s plausible that some of the following (non-mutually-exclusive) factors may have contributed to this, though we don’t know the relative strength of these factors:

* The value of cryptocurrency, especially ETH, was significantly lower during the fundraiser than [in 2017](https://intelligence.org/2018/01/10/fundraising-success/). Some donors have explicitly told us they’re waiting for more advantageous ETH prices before supporting us again.
* MIRI’s current [nondisclosed-by-default](https://intelligence.org/2018/11/22/2018-update-our-new-research-directions/#section3) research approach makes it difficult for some donors to financially support us at the moment, either because they disagree with the policy itself or because (absent more published research from MIRI) they feel like they personally lack the data they need to evaluate us against other giving opportunities. Several donors have voiced one or both of these sentiments to me, and they were cited prominently in this [2019 alignment research review](https://www.lesswrong.com/posts/SmDziGM9hBjW9DKmf/2019-ai-alignment-literature-review-and-charity-comparison).
* The changes to US tax law in 2018 relating to individual deductions have caused some supporters to adjust the scale and timing of their donations, a trend noticed across US charitable giving in general. It’s plausible that having future MIRI fundraisers straddle the new year (e.g. starting in 2020 and ending in 2021) might provide supporters with more flexibility in their giving; if you would personally find this helpful (or unhelpful), I’m interested to hear about it at colm@intelligence.org.
* This fundraiser saw MIRI involved in less counterfactual matching opportunities than in previous years — one compared to three in 2018 — which may have reduced the attraction for some of our more leverage-sensitive supporters this time around.
* Since MIRI’s budget and size have grown a great deal over the past few years, some donors may think that we’re hitting diminishing returns on marginal donations, at a rate that makes them want to look for other giving opportunities.
* The scale of the funds received during MIRI fundraisers tends to be strongly affected by 1-4 large donors each year, a fair number of whom are one-time or sporadic donors. Since this has certainly given us higher-than-expected results on a number of past occasions, it’s perhaps not surprising that such a randomness-heavy phenomenon would sometimes yield lower-than-expected support by chance. Specifically, we received no donations over $100,000 during this fundraiser, and the two donations over $50,000 were especially welcome.
* Some MIRI supporters who had been previously following an earning-to-give strategy have moved to direct work as [80,000 Hours’ developing thoughts on the subject](https://80000hours.org/2015/07/80000-hours-thinks-that-only-a-small-proportion-of-people-should-earn-to-give-long-term/) continue to influence the EA community.
* In past years, when answering supporters’ questions about the discount rate on their potential donations to MIRI, we’ve leaned towards a “now > later” approach. This plausibly resulted in a front-loading of some donations in 2017 and 2018.

As always, I’m interested in hearing individual supporters’ thoughts about how they personally are thinking about their giving strategy; you’re welcome to shoot me an email at [colm@intelligence.org](mailto:colm@intelligence.org)

Overall, we’re extremely grateful for all the support we received during this fundraiser. Although we didn’t hit our target, the support we received allows us to continue pursuing the majority of our [growth plans](https://intelligence.org/2019/12/02/miris-2019-fundraiser/), with cash reserves of 1.2–1.4 years at the start of 2020. To everyone who contributed, thank you.

---

1. The exact total is still subject to change as we continue to process a small number of donations.[![↩](https://s.w.org/images/core/emoji/14.0.0/72x72/21a9.png)](https://intelligence.org/feed/?paged=10#fnref1)
2. There were significantly more anonymous donors than in previous years — plausibly a result of new data protection legislation like the European Union’s [GDPR](https://en.wikipedia.org/wiki/General_Data_Protection_Regulation) and California’s [CCPA](https://en.wikipedia.org/wiki/California_Consumer_Privacy_Act) — which we aggregate as a single donor.[![↩](https://s.w.org/images/core/emoji/14.0.0/72x72/21a9.png)](https://intelligence.org/feed/?paged=10#fnref2)
3. Including Richard Schwall, Luke Stebbing, Simon Sáfár, Laura Soares, John Davis, Cliff Hyra, Noah Topper, Gwen Murray, and Daniel Kokotajlo. Thanks, all![![↩](https://s.w.org/images/core/emoji/14.0.0/72x72/21a9.png)](https://intelligence.org/feed/?paged=10#fnref3)

The post [Our 2019 Fundraiser Review](https://intelligence.org/2020/02/13/our-2019-fundraiser-review/) appeared first on [Machine Intelligence Research Institute](https://intelligence.org).


January 2020 Newsletter

#### Updates

* Our [2019 fundraiser](https://intelligence.org/2019/12/02/miris-2019-fundraiser/) ended Dec. 31. We'll have more to say in a few weeks in our fundraiser retrospective, but for now, a big thank you to the ~240 donors who together donated more than $526,000, including $67,484 in the first 20 seconds of [Giving Tuesday](https://intelligence.org/2019/11/28/giving-tuesday-2019/) (not counting matching dollars, which have yet to be announced).
* Jan. 15 is the final day of [CFAR's annual fundraiser](https://rationality.org/fundraiser). CFAR also [recently ran an AMA](https://www.lesswrong.com/posts/96N8BT9tJvybLbn5z/we-run-the-center-for-applied-rationality-ama) and [has posted their workshop participant handbook online](https://www.lesswrong.com/posts/Z9cbwuevS9cqaR96h/cfar-participant-handbook-now-available-to-all).
* [Understanding “Deep Double Descent”](https://www.lesswrong.com/posts/FRv7ryoqtvSuqBxuT): MIRI researcher Evan Hubinger describes a fascinating phenomenon in ML, and an interesting case study in ML research aimed at deepening our understanding, and not just advancing capabilities. In a follow-up post, Evan also considers [possible implications for alignment research](https://www.lesswrong.com/posts/nGqzNC6uNueum2w8T).
* [Safe Exploration and Corrigibility](https://www.lesswrong.com/posts/87Y7w73phjBxnPyPD): Evan notes an important (and alignment-relevant) way that notions of exploration in deep RL have shifted.
* “[Learning Human Objectives by Evaluating Hypothetical Behavior](https://deepmind.com/blog/article/learning-human-objectives-by-evaluating-hypothetical-behaviours)”: UC Berkeley and DeepMind researchers “present a method for training reinforcement learning agents from human feedback in the presence of unknown unsafe states”.

#### Links from the research team

This continues my experiment from last month: having MIRI researchers anonymously pick out AI Alignment Forum posts to highlight and comment on.

* Re [(When) is Truth-telling Favored in AI debate?](https://www.lesswrong.com/posts/RQoSCs9SePDMLJvfz) — “A paper by Vojtěch Kovařík and Ryan Carey; it's good to see some progress on the debate model!”
* Re [Recent Progress in the Theory of Neural Networks](https://www.lesswrong.com/posts/KrQvZM8uFjSTJ7hq3) — “Noah MacAulay provides another interesting example of research attempting to explain what's going on with NNs.”
* Re [When Goodharting is optimal](https://www.lesswrong.com/posts/megKzKKsoecdYqwb7) — “I like Stuart Armstrong's post for the systematic examination of why we might be afraid of Goodharting. The example at the beginning is an interesting one, because it seems (to me at least) like the robot really should go back and forth (staying a long time at each side to minimize lost utility). But Stuart is right that this answer is, at least, quite difficult to justify.”
* Re [Seeking Power is Instrumentally Convergent in MDPs](https://www.lesswrong.com/posts/6DuJxY8X45Sco4bS2) and [Clarifying Power-Seeking and Instrumental Convergence](https://www.lesswrong.com/posts/cwpKagyTvqSyAJB7q) — “It's nice to finally have a formal model of this, thanks to Alex Turner and Logan Smith. Instrumental convergence has been an informal part of the discussion for a long time.”
* Re [Critiquing “What failure looks like”](https://www.lesswrong.com/posts/Q8Z8yoG4tBaowBHwk) — “I thought that Grue Slinky's post was a good critical analysis of Paul Christiano's ‘[going out with a whimper](https://www.lesswrong.com/posts/HBxe6wdjxK239zajf/what-failure-looks-like)’ scenario, highlighting some of the problems it seems to have as a concrete AI risk scenario. In particular, I found the analogy given to the simplex algorithm persuasive in terms of showcasing how, despite the fact that many of our current most powerful tools already have massive differentials in how well they work on different problems, those values which are not served well by those tools don't seem to have lost out massively as a result. I still feel like there may be a real risk along the lines of ‘going out with a whimper’, but I think this post presents a real challenge to that scenario as it has been described so far.”
* Re [Counterfactual Induction](https://www.lesswrong.com/posts/EAqHkKtbefvyRs4nw) — “A proposal for logical counterfactuals by Alex Appel. This could use some more careful thought and critique; it's not yet clear exactly how much or little it accomplishes.”
* Re [A dilemma for prosaic AI alignment](https://www.lesswrong.com/posts/jYdAxH8BarPT4fqnb/a-dilemma-for-prosaic-ai-alignment) — “Daniel Kokotajlo outlines key challenges for prosaic alignment: ‘[…] Now I think the problem is substantially harder than that: To be competitive prosaic AI safety schemes must deliberately create misaligned mesa-optimizers and then (hopefully) figure out how to align them so that they can be used in the scheme.’”

The post [January 2020 Newsletter](https://intelligence.org/2020/01/15/january-2020-newsletter/) appeared first on [Machine Intelligence Research Institute](https://intelligence.org).


December 2019 Newsletter

From now through the end of December, [**MIRI's 2019 Fundraiser**](https://intelligence.org/2019/12/02/miris-2019-fundraiser/) is live! See our fundraiser post for updates on our past year and future plans.
One of our biggest updates, I'm happy to announce, is that **we've hired five new research staff**, with a sixth to join us in February. For details, see [Workshops and Scaling Up](https://intelligence.org/2019/12/02/miris-2019-fundraiser/#1) in the fundraiser post.

Also: Facebook's Giving Tuesday matching opportunity is **tomorrow** at 5:00am PT! See [Colm's post](https://intelligence.org/2019/11/28/giving-tuesday-2019/) for details on how to get your donation matched.

#### Other updates

* Our most recent hire, “[Risks from Learned Optimization](https://arxiv.org/abs/1906.01820)” co-author Evan Hubinger, describes [what he'll be doing at MIRI](https://www.alignmentforum.org/posts/ptmmK9PWgYTuWToaZ/what-i-ll-be-doing-at-miri). See also Nate Soares' comment on [how MIRI does nondisclosure-by-default](https://www.alignmentforum.org/posts/ptmmK9PWgYTuWToaZ/what-i-ll-be-doing-at-miri#rfaMzMaKAAsdv38Me).
* Buck Shlegeris discusses [EA residencies as an outreach opportunity](https://forum.effectivealtruism.org/posts/yrSiWNypE6AMNApDi/ea-residencies-as-outreach-activity).
* OpenAI releases [Safety Gym](https://openai.com/blog/safety-gym/), a set of tools and environments for incorporating safety constraints into RL tasks.
* CHAI is [seeking interns](https://boards.greenhouse.io/centerforhumancompatibleartificialintelligence/jobs/4358062002); application deadline Dec. 15.

#### Thoughts from the research team

This month, I'm trying something new: quoting MIRI researchers' summaries and thoughts on recent AI safety write-ups.

I've left out names so that these can be read as a snapshot of people's impressions, rather than a definitive “Ah, researcher X believes Y!” Just keep in mind that these will be a small slice of thoughts from staff I've recently spoken to, not anything remotely like a consensus take.

* Re [Will transparency help catch deception?](https://www.lesswrong.com/posts/J9D6Bi3eFDDhCaovi/will-transparency-help-catch-deception-perhaps-not) — “A good discussion of an important topic. Matthew Barnett suggests that any weaknesses in a transparency tool may turn it into a detrimental middle-man, and directly training supervisors to catch deception may be preferable.”
* Re [Chris Olah’s views on AGI safety](https://www.alignmentforum.org/posts/X2i9dQQK3gETCyqh2/chris-olah-s-views-on-agi-safety) — “I very much agree with Evan Hubinger's idea that collecting different perspectives — different ‘hats’ — is a useful thing to do. Chris Olah's take on transparency is good to see. The concept of microscope AI seems like a useful one, and Olah's vision of how the ML field could be usefully shifted is quite interesting.”
* Re [Defining AI Wireheading](https://www.lesswrong.com/posts/vXzM5L6njDZSf4Ftk) — “Stuart Armstrong takes a shot at making a principled distinction between wireheading and the rest of Goodhart.”
* Re [How common is it to have a 3+ year lead?](https://www.lesswrong.com/posts/yXikQ87FFw3oPPaYh) — “This seems like a pretty interesting question for AI progress models. The expected lead time and questions of expected takeoff speed greatly influence the extent to which winner-take-all dynamics are plausible.”
* Re [Thoughts on Implementing Corrigible Robust Alignment](https://www.lesswrong.com/posts/8W5gNgEKnyAscg8BF) — “Steve Byrnes provides a decent overview of some issues around getting ‘pointer’ type values.”

The post [December 2019 Newsletter](https://intelligence.org/2019/12/05/december-2019-newsletter/) appeared first on [Machine Intelligence Research Institute](https://intelligence.org).


MIRI’s 2019 Fundraiser

**MIRI’s 2019 fundraiser** is concluded.

Over the past two years, huge donor support has helped us double the size of our AI alignment research team. Hitting our $1M fundraising goal this month will put us in a great position to continue our growth in 2020 and beyond, recruiting as many brilliant minds as possible to take on what appear to us to be the central technical obstacles to alignment.

Our fundraiser progress, updated in real time (including donations and matches made during the Facebook [Giving Tuesday](https://intelligence.org/giving-tuesday-2019/) event):

---

---

MIRI is a CS/math research group with a goal of understanding how to reliably “aim” future general-purpose AI systems at known goals. For an introduction to this research area, see [Ensuring Smarter-Than-Human Intelligence Has A Positive Outcome](https://intelligence.org/2017/04/12/ensuring/) and [Risks from Learned Optimization in Advanced Machine Learning Systems](https://arxiv.org/abs/1906.01820). For background on how we approach the problem, see [2018 Update: Our New Research Directions](https://intelligence.org/2018/11/22/2018-update-our-new-research-directions/) and [Embedded Agency](https://www.alignmentforum.org/s/Rm6oQRJJmhGCcLvxh/p/i3BTagvt3HbPMx6PN).

At the end of 2017, we [announced](https://intelligence.org/2017/12/01/miris-2017-fundraiser/) plans to substantially grow our research team, with a goal of hiring “around ten new research staff over the next two years.” Two years later, I’m happy to report that we’re up eight research staff, and we have a ninth starting in February of next year, which will bring our total research team size to 20.[1](https://intelligence.org/2019/12/02/miris-2019-fundraiser/#footnote_0_19472 "This number includes a new staff member who is currently doing a 6-month trial with us.")

We remain excited about our current research directions, and continue to feel that we could make progress on them more quickly by adding additional researchers and engineers to the team. As such, our main organizational priorities remain the same: push forward on our research directions, and grow the research team to accelerate our progress.

While we’re quite uncertain about how large we’ll ultimately want to grow, we plan to continue growing the research team at a similar rate over the next two years, and so expect to add around ten more research staff by the end of 2021.

Our projected budget for 2020 is $6.4M–$7.4M, with a point estimate of $6.8M,[2](https://intelligence.org/2019/12/02/miris-2019-fundraiser/#footnote_1_19472 "These estimates were generated using a model similar to the one I used last year. For more details see our 2018 fundraiser post.") up from around $6M this year.[3](https://intelligence.org/2019/12/02/miris-2019-fundraiser/#footnote_2_19472 "This falls outside the $4.4M–$5.5M range I estimated in our 2018 fundraiser post, but is in line with the higher end of revised estimates we made internally in Q1 2019.") In the mainline-growth scenario, we expect our budget to look something like this:

Looking further ahead, since staff salaries account for the vast majority of our expenses, I expect our spending to increase proportionately year-over-year while research team growth continues to be a priority.

Given our $6.8M budget for 2020, and the cash we currently have on hand, raising $1M in this fundraiser will put us in a great position for 2020. Hitting $1M positions us with cash reserves of 1.25–1.5 years going into 2020, which is exactly where we want to be to support ongoing hiring efforts and to provide the confidence we need to make and stand behind our salary and other financial commitments.

For more details on what we’ve been up to this year, and our plans for 2020, read on!

### 1. Workshops and scaling up

If you lived in a world that [didn’t know calculus](https://www.lesswrong.com/posts/Gg9a4y8reWKtLe3Tn/the-rocket-alignment-problem), but you knew *something was missing*, what general practices would have maximized your probability of coming up with it?

What if you didn’t start off knowing something was missing? Could you and some friends have [gotten together and done research](https://www.lesswrong.com/posts/PqMT9zGrNsGJNfiFR/alignment-research-field-guide) in a way that put you in a good position to notice it, to ask the right questions?

MIRI thinks that humanity is currently missing some of the core concepts and methods that AGI developers will need in order to align their systems down the road. We think we’ve found research paths that may help solve that problem, and good ways to rapidly improve our understanding via experiments; and we’re eager to add more researchers and engineers’ eyes and brains to the effort.

A significant portion of MIRI’s current work is in Haskell, and benefits from experience with functional programming and dependent type systems. More generally, if you’re a programmer who loves hunting for the most appropriate abstractions to fit some use case, developing clean concepts, making and then deploying elegant combinators, or audaciously trying to answer the deepest questions in computer science—then we think you should [apply to work here](https://machineintelligence.typeform.com/to/CDVFE2), get to know us at a [workshop](https://intelligence.org/ai-risk-for-computer-scientists/), or [reach out with questions](mailto:buck@intelligence.org).

As noted above, our research team is growing fast. The latest additions to the MIRI team include:

[![New MIRI Staff](https://intelligence.org/wp-content/uploads/2019/11/evan-jeremy-seraphina-rafe.jpg)](https://intelligence.org/team)**Evan Hubinger**, a co-author on “[Risks from Learned Optimization in Advanced Machine Learning Systems](https://intelligence.org/learned-optimization/)”. Evan previously designed the functional programming language [Coconut](http://coconut-lang.org/), was an intern at OpenAI, and has done software engineering work at Google, Yelp, and Ripple.

**Jeremy Schlatter**, a software engineer who previously worked at Google and OpenAI. Some of the public projects Jeremy has contributed to include OpenAI’s [Dota 2 bot](https://cdn.openai.com/dota-2.pdf) and a debugger for the Go programming language.

**Seraphina Nix**, joining MIRI in February 2020. Seraphina graduates this month from Oberlin College with a major in mathematics and minors in computer science and physics. She has previously done research on ultra-lightweight dark matter candidates, deep reinforcement learning, and teaching neural networks to do high school mathematics.

**Rafe Kennedy**, who joins MIRI after working as an independent existential risk researcher at the Effective Altruism Hotel. Rafe previously worked at the data science startup NStack, and he holds an MPhysPhil from the University of Oxford in Physics & Philosophy.

[![AIRCS](https://intelligence.org/wp-content/uploads/2019/11/crop-aircs.png)](https://intelligence.org/ai-risk-for-computer-scientists/)MIRI’s hires and job trials are typically drawn from our 4.5-day, all-expense-paid **[AI Risk for Computer Scientists](https://intelligence.org/ai-risk-for-computer-scientists/)** (AIRCS) workshop series.

Our workshop program is the best way we know of to bring promising talented individuals into what we think are useful trajectories towards being highly-contributing AI researchers and engineers. Having established an experience that participants love and that we believe to be highly valuable, we plan to continue experimenting with new versions of the workshop, and expect to run ten workshops over the course of 2020, up from eight this year.

These programs have led to a good number of new hires at MIRI as well as other AI safety organizations, and we find them valuable for everything from introducing talented outsiders to AI safety, to leveling up people who have been thinking about these issues for years.

If you’re interested in attending, [apply here](https://docs.google.com/forms/d/e/1FAIpQLSeSJ4G-hMIMIGZ-Ze-I__WfEaarUMC9Q2zRuPpkeG9r6OR4ww/viewform). If you have any questions, we highly encourage you to shoot [Buck Shlegeris](mailto:buck@intelligence.org) an email.

Our [MIRI Summer Fellows Program](https://www.rationality.org/workshops/apply-msfp) plays a similar role for us, but is more targeted at mathematicians. We’re considering running MSFP in a shorter format in 2020. For any questions about MSFP, email [Colm Ó Riain](mailto:colm@intelligence.org).

### 2. Research and write-ups

Our [2018 strategy update](https://intelligence.org/2018/11/22/2018-update-our-new-research-directions/) continues to be a great overview of where MIRI stands today, describing how we think about our research, laying out our [case for working here](https://intelligence.org/2018/11/22/2018-update-our-new-research-directions/#section4), and explaining why most of our work [currently isn’t public-facing](https://intelligence.org/2018/11/22/2018-update-our-new-research-directions/#section3).

Given the latter point, I’ll focus in this section on spotlighting what we’ve written up this past year, providing snapshots of some of the work individuals at MIRI are currently doing (without any intended implication that this is representative of the whole), and conveying some of our current broad impressions about how our research progress is going.

Some of our major write-ups and publications this year were:

* “[Risks from Learned Optimization in Advanced Machine Learning Systems](https://intelligence.org/learned-optimization/),” by Evan Hubinger, Chris van Merwijk, Vladimir Mikulik, Joar Skalse, and Scott Garrabrant. The process of generating this paper significantly clarified our own thinking, and informed Scott and Abram’s discussion of [subsystem alignment](https://intelligence.org/2018/11/06/embedded-subsystems/) in “[Embedded Agency](http://intelligence.org/embedded-agency).”Scott views “Risks from Learned Optimization” as being of comparable importance to “Embedded Agency” as exposition of key alignment difficulties, and we’ve been extremely happy about the new conversations and research that the field at large has produced to date in dialogue with the ideas in “Risks from Learned Optimization”.
* [Thoughts on Human Models](https://intelligence.org/2019/02/22/thoughts-on-human-models/), by Scott Garrabrant and DeepMind-based MIRI Research Associate Ramana Kumar, argues that the AI alignment research community should begin prioritizing “approaches that work well in the absence of human models.”The role of human models in alignment plans strikes us as one of the most important issues for MIRI and other research groups to wrestle with, and we’re generally interested in seeing what new approaches groups outside MIRI might come up with for leveraging AI for the common good in the absence of human models.
* “[Cheating Death in Damascus](https://intelligence.org/2017/03/18/new-paper-cheating-death-in-damascus/),” by Nate Soares and Ben Levinstein. We presented this decision theory paper at the Formal Epistemology Workshop in 2017, but a lightly edited version has now been accepted to *The Journal of Philosophy*, previously [voted](https://leiterreports.typepad.com/blog/2013/07/top-philosophy-journals-without-regard-to-area.html) the second highest-quality academic journal in philosophy.
* The [Alignment Research Field Guide](https://www.lesswrong.com/posts/PqMT9zGrNsGJNfiFR/alignment-research-field-guide), a very accessible and broadly applicable resource both for individual researchers and for groups getting off the ground.

Our other recent public writing includes an Effective Altruism Forum [AMA with Buck Shlegeris](https://forum.effectivealtruism.org/posts/tDk57GhrdK54TWzPY/i-m-buck-shlegeris-i-do-research-and-outreach-at-miri-ama), Abram Demski’s [The Parable of Predict-O-Matic](https://www.lesswrong.com/posts/SwcyMEgLyd4C3Dern), and the many interesting outputs of the [AI Alignment Writing Day](https://www.lesswrong.com/s/YuTinYEzsyHmPoocw) we hosted toward the end of this year’s MIRI Summer Fellows Program.

Turning to our research team, last year we [announced](https://intelligence.org/2018/11/28/miris-newest-recruit-edward-kmett/) that prolific Haskell programmer Edward Kmett joined the MIRI team, freeing him up to do the thing he’s passionate about—improving the state of highly reliable (and simultaneously highly efficient) programming languages. MIRI Executive Director Nate Soares views this goal as very ambitious, though would feel better about the world if there existed programming languages that were both efficient and amenable to strong formal guarantees about their properties.

This year Edward moved to Berkeley to work more closely with the rest of the MIRI team. We’ve found it very helpful to have him around to provide ideas and contributions to our more engineering-oriented projects, helping give some amount of practical grounding to our work. Edward has also continued to be a huge help with recruiting through his connections in the functional programming and type theory world.

Meanwhile, our newest addition, Evan Hubinger, plans to continue working on solving [inner alignment](https://www.lesswrong.com/posts/pL56xPoniLvtMDQ4J/the-inner-alignment-problem) for [amplification](https://www.alignmentforum.org/s/EmDuGeRw749sD3GKd). Evan has outlined his [research plans](https://www.alignmentforum.org/posts/ptmmK9PWgYTuWToaZ/what-i-ll-be-doing-at-miri) on the AI Alignment Forum, noting that [relaxed adversarial training](https://www.alignmentforum.org/posts/9Dy5YRaoCxH9zuJqa/relaxed-adversarial-training-for-inner-alignment) is a fairly up-to-date statement of his research agenda. Scott and other researchers at MIRI consider Evan’s work quite exciting, both in the context of amplification and in the context of other alignment approaches it might prove useful for.

Abram Demski is another MIRI researcher who has written up a large number of his research thoughts over the last year. Abram reports ([fuller thoughts here](https://intelligence.org/files/Nov2019ResearchThoughts.pdf)) that he has moved away from a traditional decision-theoretic approach this year, and is now spending more time on learning-theoretic approaches, similar to MIRI Research Associate Vanessa Kosoy. Quoting Abram:

> Around December 2018, I had a big update against the “classical decision-theory” mindset (in which learning and decision-making are viewed as separate problems), and towards taking a learning-theoretic approach. [… I have] made some attempts to communicate my update against UDT and toward learning-theoretic approaches, including [this write-up](https://www.alignmentforum.org/posts/9sYzoRnmqmxZm4Whf/conceptual-problems-with-udt-and-policy-selection). I talked to Daniel Kokotajlo about it, and he wrote [The Commitment Races Problem](https://www.alignmentforum.org/posts/brXr7PJ2W4Na2EW2q/the-commitment-races-problem), which I think captures a good chunk of it.
>
>

For her part, Vanessa’s recent work includes the paper “[Delegative Reinforcement Learning: Learning to Avoid Traps with a Little Help](https://intelligence.org/2019/04/24/delegative-reinforcement-learning/),” which she presented at the ICLR 2019 SafeML workshop.

I’ll note again that the above are all snapshots of particular research directions various researchers at MIRI are pursuing, and don’t necessarily represent other researchers’ views or focus. As Buck recently [noted](https://forum.effectivealtruism.org/posts/tDk57GhrdK54TWzPY/i-m-buck-shlegeris-i-do-research-and-outreach-at-miri-ama#FdWqNnEh5kGb37P58), MIRI has a pretty flat management structure. We pride ourselves on minimizing bureaucracy, and on respecting the ability of our research staff to form their own inside-view models of the alignment problem and of what’s needed next to make progress. Nate recently expressed similar thoughts about [how we do nondisclosure-by-default](https://www.lesswrong.com/posts/ptmmK9PWgYTuWToaZ/what-i-ll-be-doing-at-miri#rfaMzMaKAAsdv38Me).

As a consequence, MIRI’s more math-oriented research especially tends to be dictated by individual models and research taste, without the expectation that everyone will share the same view of the problem.

Regarding his overall (very high-level) sense of how MIRI’s new research directions are progressing, Nate Soares reports:

> Progress in 2019 has been slower than expected, but I have a sense of steady progress. In particular, my experience is one of steadily feeling less confused each week than the week before—of me and other researchers having difficulties that were preventing us from doing a thing we wanted to do, staring at them for hours, and then realizing that we’d been thinking wrongly about this or that, and coming away feeling markedly more like we know what’s going on.
>
>
> An example of the kind of thing that causes us to feel like we’re making progress is that we’ll notice, “Aha, the right tool for thinking about all three of these apparently-dissimilar problems was order theory,” or something along those lines; and disparate pieces of frameworks will all turn out to be the same, and the relevant frameworks will become simpler, and we’ll be a little better able to think about a problem that I care about. This description is extremely abstract, but represents the flavor of what I mean by “steady progress” here, in the same vein as [my writing last year about “deconfusion.”](https://intelligence.org/2018/11/22/2018-update-our-new-research-directions/#section2)
>
>
> Our hope is that enough of this kind of progress gives us a platform from which we can generate particular exciting results on core AI alignment obstacles, and I expect to see such results reasonably soon. To date, however, I have been disappointed by the amount of time that’s instead been spent on deconfusing myself and shoring up my frameworks; I previously expected to have more exciting results sooner.
>
>
> In research of the kind we’re working on, it’s not uncommon for there to be years between sizeable results, though we should also expect to sometimes see cascades of surprisingly rapid progress, if we are indeed pushing in the right directions. My inside view of our ongoing work currently predicts that we’re on a productive track and should expect to see results we are more excited about before too long.
>
>

Our research progress, then, is slower than we had hoped, but the rate and quality of progress continues to be such that we consider this work very worthwhile, and we remain optimistic about our ability to convert further research staff hours into faster progress. At the same time, we are also (of course) looking for where our research bottlenecks are and how we can make our work more efficient, and we’re continuing to look for tweaks we can make that might boost our output further.

If things go well over the next few years—which seems likely but far from guaranteed—we’ll continue to find new ways of making progress on research threads we care a lot about, and continue finding ways to hire people to help make that happen.

Research staff expansion is our biggest source of expense growth, and by encouraging us to move faster on exciting hiring opportunities, donor support plays a key role in how we execute on our research agenda. Though the huge support we’ve received to date has put us in a solid position even at our new size, further donor support is a big help for us in continuing to grow. If you want to play a part in that, thank you.

---

[Donate Now](https://intelligence.org/donate/)
----------------------------------------------

---

1. This number includes a new staff member who is currently doing a 6-month trial with us.
2. These estimates were generated using a model similar to the one I used last year. For more details see our [2018 fundraiser post](https://intelligence.org/2018/11/26/miris-2018-fundraiser/).
3. This falls outside the $4.4M–$5.5M range I estimated in our [2018 fundraiser post](https://intelligence.org/2018/11/26/miris-2018-fundraiser/), but is in line with the higher end of revised estimates we made internally in Q1 2019.

The post [MIRI’s 2019 Fundraiser](https://intelligence.org/2019/12/02/miris-2019-fundraiser/) appeared first on [Machine Intelligence Research Institute](https://intelligence.org).


Giving Tuesday 2019

**Update January 25, 2020**: $77,325 was donated to MIRI through Facebook on Giving Tuesday. $45,915 of this was donated within 13.5 seconds of the Facebook matching event starting at 5:00AM PT and was matched by Facebook. Thank you to everybody who set their clocks early to support us so generously! Shout out too to the [EA Giving Tuesday](https://www.eagivingtuesday.org) and [Rethink Charity](https://www.rtcharity.org) team for their amazing efforts on behalf of the EA Community.

[![](https://intelligence.org/wp-content/uploads/2019/11/GT19Chart-PostMatch2-300x233.png)](https://intelligence.org/wp-content/uploads/2019/11/GT19Chart-PostMatch2.png)

---

**Update December 2, 2019**: This page has been updated to reflect (a) observed changes in Facebook’s flow for donations of $500 or larger (b) additional information on securing matching for donations of $2,500 or larger during Facebook’s matching event and (c) a pointer to Paypal’s newly announced, though significantly smaller, matching event(s). Please check in here for more updates before the Facebook Matching event begins at 5am PT on December 3.

---

[MIRI’s annual fundraiser](https://intelligence.org/2019/12/02/miris-2019-fundraiser/) begins this Monday, December 2, 2019 and **Giving Tuesday** takes place the next day; starting at 5:00:00am PT (8:00:00am ET) on December 3, [Facebook](https://www.facebook.com/help/332488213787105) will match donations made on fundraiser pages on their platform up to a total of $7,000,000. This post focuses on this Facebook matching event. (You can find information on Paypal’s significantly smaller matching events in the footnotes.[1](https://intelligence.org/feed/?paged=11#paypal))

1. Donations during Facebook’s Giving Tuesday event will be matched dollar for dollar on a first-come, first-served basis until the $7,000,000 in matching funds are used up. Based on trends in previous years, this will probably occur within 10 seconds.
2. Any US-based 501(c)(3) nonprofit eligible to receive donations on Facebook, e.g. MIRI, can be matched.
3. Facebook will match up to a total of $100,000 per nonprofit organization.
4. Each donor can have up to $20,000 in eligible donations matched on Giving Tuesday. There is a default limit of $2,499 per donation. Donors who wish to donate more than $2,499 have multiple strategies to choose from (below) to increase the chances of their donations being matched.

 In [2018](https://intelligence.org/2019/02/07/our-2018-fundraiser-review/), Facebook’s matching pool of $7M was exhausted within 16 seconds of the event starting and in that time, 66% of our lightning-fast donors got their donations to MIRI matched, securing a total of $40,072 in matching funds. This year, we’re aiming for the per-organization $100,000 maximum and since it’s highly plausible that this year’s matching event will end **within 4-10 seconds**, here are some tips to improve the chances of your donation to [MIRI’s Fundraiser Page on Facebook](https://www.facebook.com/donate/2649342025109229/) being matched.

### Pre-Event Preparation (before Giving Tuesday)

* Confirm your FB account is operational.
* Add your preferred credit card(s) as payment method(s) [in your FB settings page](https://secure.facebook.com/settings?tab=payments&section=settings). Credit cards are plausibly mildly preferable to Paypal as a payment option in terms of donation speed.
* Test your payment method(s) ahead of time by donating small amount(s) to [MIRI’s Fundraiser page](https://www.facebook.com/donate/2649342025109229/).
* If your credit card limit is lower than the amount you’re considering donating, it may be possible to (a) overpay the balance ahead of time and/or (b) call your credit card asking them to (even temporarily) increase your limit.
* If you plan to donate more than $2,499, [see below](https://intelligence.org/feed/?paged=11#largedonations) for instructions.
* Sync whatever clock you’ll be using with [time.is](https://time.is/).
* Consider pledging your donation to MIRI at [EA Giving Tuesday](https://www.eagivingtuesday.org/pledges).[2](https://intelligence.org/feed/?paged=11#eagt)

### Donating on Giving Tuesday

On Tuesday, December 3, **BEFORE** 5:00:00am PT — it’s advisable to be alert and ready 10-20 minutes before the event — prepare your donation, so you can make your donation with a single click when the event begins at 5:00:00am PT.

1. Open an accurate clock at [time.is](https://time.is/).
2. In a different browser window alongside, open [MIRI’s Fundraiser Page on Facebook](https://www.facebook.com/donate/2649342025109229/) in your browser.
3. Click on the Donate button. ![](https://intelligence.org/wp-content/uploads/2018/11/DonateButton.png)
4. In the “Donate” popup that surfaces:
* Enter your donation amount — between $5 and $2,499. [See below](https://intelligence.org/feed/?paged=11#largedonations) for larger donations.
* Choose whichever card you’re using for your donation.
* Optionally enter a note and/or adjust the donation visibility.

6. **At 05:00:00 PST, click on the Green Donate button.** If your donation amount is $500 or larger, you may be presented with an additional “Confirm Your Donation” dialog. If so, click on *its* Donate button as quickly as possible.

### Larger Donations

By default, Facebook places a limit of $2,499 per donation (in the US[3](https://intelligence.org/feed/?paged=11#outsideus)), and will match up to $20,000 per donor. If you’re in a position to donate $2,500 or more to MIRI, you can:

1. Use multiple browser windows/tabs for each individual donation: open up [MIRI’s Fundraiser Page on Facebook](https://www.facebook.com/donate/2649342025109229/) in as many tabs as needed in your browser and follow the instructions above in each window/tab so you have multiple Donate buttons ready to click, one in each tab. Then at 5:00:00 PT, channel your lightning and click as fast as you can — one of our star supporters last year made 8 donations within 21 seconds, 5 of which were matched.

**and/or**

- Before the event — ideally not the morning of — follow [EA Giving Tuesday’s instructions](https://docs.google.com/document/d/1GPb9cDFtKmTW8NdyLwwVrQkEMg_CzzfV7zarz4r2v60/edit) on how to increase your per-donation limit on Facebook above $2,499. Our friends at [EA Giving Tuesday](https://www.eagivingtuesday.org/) estimate that “you are likely to be able to successfully donate up to $9,999 per donation” after following these instructions. Their analysis also suggests that going higher than $10,000 for an individual donation plausibly significantly increases the probability of being declined and therefore advise not going beyond $9,999 per donation. It is possible that Facebook may put a cap of $2,499 on individual donations closer to the event.

Using a combination of the above, a generous supporter could, for example, make 2 donations of $9,999 each — in separate browser windows — within seconds of the event starting.

###

---

1. Paypal has 3 separate matching events on Giving Tuesday — all of which add 10% to eligible donations — for [the USA](https://www.paypal.com/fundraiser/112574644767835624?CampaignName=VanityURL-GiveBack&fbclid=IwAR1EnDNyAMM-ZTV8ukAyO7Am8_H8Sw7utsWOTSeFzAP1_SvJMVNEy00P-mo), [Canada](https://www.paypal.ca/giveback), and [the UK](https://www.paypal.co.uk/giveback).
2. Thanks to Ari, William, Rethink Charity and all at EA Giving Tuesday for their work to help EA organizations maximize their share of Facebook’s matching funds.
3. For up-to-date information on Facebook’s donation limits outside the US, check out [EA Giving Tuesday’s doc](https://docs.google.com/document/d/1VkyavA383WKo29wKqokMDr917rAOKwLiWTAnWVKA7PY/edit).

The post [Giving Tuesday 2019](https://intelligence.org/2019/11/28/giving-tuesday-2019/) appeared first on [Machine Intelligence Research Institute](https://intelligence.org).