---

What is the function of a fire alarm?

One might think that the function of a fire alarm is to provide you with important evidence about a fire existing, allowing you to change your policy accordingly and exit the building.

In the classic experiment by Latane and Darley in 1968, eight groups of three students each were asked to fill out a questionnaire in a room that shortly after began filling up with smoke. Five out of the eight groups didn’t react or report the smoke, even as it became dense enough to make them start coughing. Subsequent manipulations showed that a lone student will respond 75% of the time; while a student accompanied by two actors told to feign apathy will respond only 10% of the time. This and other experiments seemed to pin down that what’s happening is pluralistic ignorance. We don’t want to look panicky by being afraid of what isn’t an emergency, so we try to look calm while glancing out of the corners of our eyes to see how others are reacting, but of course they are also trying to look calm.

(I’ve read a number of replications and variations on this research, and the effect size is blatant. I would not expect this to be one of the results that dies to the replication crisis, and I haven’t yet heard about the replication crisis touching it. But we have to put a maybe-not marker on everything now.)

A fire alarm creates common knowledge, in the you-know-I-know sense, that there is a fire; after which it is socially safe to react. When the fire alarm goes off, you know that everyone else knows there is a fire, you know you won’t lose face if you proceed to exit the building.

The fire alarm doesn’t tell us with certainty that a fire is there. In fact, I can’t recall one time in my life when, exiting a building on a fire alarm, there was an actual fire. Really, a fire alarm is *weaker* evidence of fire than smoke coming from under a door.

But the fire alarm tells us that it’s socially okay to react to the fire. It promises us with certainty that we won’t be embarrassed if we now proceed to exit in an orderly fashion.

It seems to me that this is one of the cases where people have mistaken beliefs about what they believe, like when somebody loudly endorsing their city’s team to win the big game will back down as soon as asked to bet. They haven’t consciously distinguished the rewarding exhilaration of shouting that the team will win, from the feeling of anticipating the team will win.

When people look at the smoke coming from under the door, I think they think their uncertain wobbling feeling comes from not assigning the fire a high-enough probability of really being there, and that they’re reluctant to act for fear of wasting effort and time. If so, I think they’re interpreting their own feelings mistakenly. If that was so, they’d get the same wobbly feeling on hearing the fire alarm, or even more so, because fire alarms correlate to fire less than does smoke coming from under a door. The uncertain wobbling feeling comes from the worry that others believe differently, not the worry that the fire isn’t there. The reluctance to act is the reluctance to be seen looking foolish, not the reluctance to waste effort. That’s why the student alone in the room does something about the fire 75% of the time, and why people have no trouble reacting to the much weaker evidence presented by fire alarms.

---

It’s now and then proposed that we ought to start reacting later to the issues of Artificial General Intelligence ([background here](http://econlog.econlib.org/archives/2016/03/so_far_unfriend.html)), because, it is said, we are so far away from it that it just isn’t possible to do productive work on it today.

(For direct argument about there being things doable today, see: Soares and Fallenstein ([2014/2017](https://intelligence.org/files/TechnicalAgenda.pdf)); Amodei, Olah, Steinhardt, Christiano, Schulman, and Mané ([2016](https://arxiv.org/abs/1606.06565)); or Taylor, Yudkowsky, LaVictoire, and Critch ([2016](https://intelligence.org/2017/02/28/using-machine-learning/)).)

(If none of those papers existed or if you were an AI researcher who’d read them but thought they were all garbage, and you wished you could work on alignment but knew of nothing you could do, the wise next step would be to sit down and spend two hours by the clock sincerely trying to think of possible approaches. Preferably without self-sabotage that makes sure you don’t come up with anything plausible; as might happen if, hypothetically speaking, you would actually find it much more comfortable to believe there was nothing you ought to be working on today, because e.g. then you could work on other things that interested you more.)

(But never mind.)

So if AGI seems far-ish away, and you think the conclusion licensed by this is that you can’t do any productive work on AGI alignment yet, then the implicit alternative strategy on offer is: Wait for some unspecified future event that tells us AGI is coming near; and *then* we’ll all know that it’s okay to start working on AGI alignment.

This seems to me to be wrong on a number of grounds. Here are some of them.

**One:** As Stuart Russell observed, if you get radio signals from space and spot a spaceship there with your telescopes and you know the aliens are landing in thirty years, you still start thinking about that today.

You’re not like, “Meh, that’s thirty years off, whatever.” You certainly don’t casually say “Well, there’s nothing we can do until they’re closer.” Not without spending two hours, or at least [five minutes](http://www.readthesequences.com/MotivatedStoppingAndMotivatedContinuation) by the clock, brainstorming about whether there is anything you ought to be starting now.

If you said the aliens were coming in thirty years and you were therefore going to do nothing today… well, if these were [more effective time](https://www.facebook.com/yudkowsky/posts/10155616782514228)s, somebody would ask for a schedule of what you thought ought to be done, starting when, how long before the aliens arrive. If you didn’t have that schedule ready, they’d know that you weren’t operating according to a worked table of timed responses, but just procrastinating and doing nothing; and they’d correctly infer that you probably hadn’t searched very hard for things that could be done today.

In Bryan Caplan’s terms, anyone who seems quite casual about the fact that “nothing can be done now to prepare” about the aliens is [missing a mood](http://econlog.econlib.org/archives/2016/01/the_invisible_t.html); they should be much more alarmed at not being able to think of any way to prepare. And maybe ask if somebody else has come up with any ideas? But never mind.

**Two:** History shows that for the general public, and even for scientists not in a key inner circle, and even for scientists *in* that key circle, it is very often the case that key technological developments still seem decades away, five years before they show up.

In 1901, two years before helping build the first heavier-than-air flyer, Wilbur Wright told his brother that powered flight was [fifty years away](https://books.google.com/books?id=ldxfLyNIk9wC&pg=PA91&dq="i+said+to+my+brother+orville"&hl=en&sa=X&ved=0ahUKEwioiseChcnWAhWL-VQKHab6AqMQ6AEIJjAA#v=onepage&q=%22i%20said%20to%20my%20brother%20orville%22&f=false).

In 1939, three years before he personally oversaw the first critical chain reaction in a pile of uranium bricks, Enrico Fermi voiced [90% confidence](https://books.google.com/books?id=aSgFMMNQ6G4C&pg=PA813&lpg=PA813&dq=weart+fermi&source=bl&ots=Jy1pBOUL10&sig=c9wK_yLHbXZS_GFIv0K3bgpmE58&hl=en&sa=X&ved=0ahUKEwjNofKsisnWAhXGlFQKHbOSB1QQ6AEIKTAA#v=onepage&q=%22ten%20per%20cent%22&f=false) that it was [impossible](http://lesswrong.com/lw/h8m/being_halfrational_about_pascals_wager_is_even/) to use uranium to sustain a fission chain reaction. I believe Fermi also said a year after that, aka two years before the denouement, that *if* net power from fission was even possible (as he then granted some greater plausibility) then it would be fifty years off; but for this I neglected to keep the citation.

And of course if you’re not the Wright Brothers or Enrico Fermi, you will be even more surprised. Most of the world learned that atomic weapons were now a thing when they woke up to the headlines about Hiroshima. There were esteemed intellectuals saying [four years *after* the Wright Flyer](https://www.xaprb.com/blog/flight-is-impossible/) that heavier-than-air flight was impossible, because knowledge propagated more slowly back then.

Were there events that, in [hindsight](https://www.readthesequences.com/Hindsight-Devalues-Science), today, we can see as signs that heavier-than-air flight or nuclear energy were nearing? Sure, but if you go back and read the actual newspapers from that time and see what people actually said about it then, you’ll see that they did not know that these were signs, or that they were very uncertain that these might be signs. Some playing the part of Excited Futurists proclaimed that big changes were imminent, I expect, and others playing the part of Sober Scientists tried to pour cold water on all that childish enthusiasm; I expect that part was more or less exactly the same decades earlier. If somewhere in that din was a superforecaster who said “decades” when it was decades and “5 years” when it was five, good luck noticing them amid all the noise. More likely, the superforecasters were the ones who said “Could be tomorrow, could be decades” both when the big development was a day away and when it was decades away.

One of the major modes by which hindsight bias makes us feel that the past was more predictable than anyone was actually able to predict at the time, is that in hindsight we know what we ought to notice, and we fixate on only one thought as to what each piece of evidence indicates. If you look at what people actually say at the time, historically, they’ve usually got no clue what’s about to happen three months before it happens, because they don’t know which signs are which.

I mean, you *could* say the words “AGI is 50 years away” and have those words happen to be true. People were also saying that powered flight was decades away when it was in fact decades away, and those people happened to be right. The problem is that everything looks the same to you either way, if you are actually living history instead of reading about it afterwards.

It’s not that whenever somebody says “fifty years” the thing always happens in two years. It’s that this confident prediction of things being far away corresponds to an epistemic state about the technology that feels the same way internally until you are very very close to the big development. It’s the epistemic state of “Well, I don’t see how to do the thing” and sometimes you say that fifty years off from the big development, and sometimes you say it two years away, and sometimes you say it while the Wright Flyer is flying somewhere out of your sight.

**Three:** Progress is driven by peak knowledge, not average knowledge.

If Fermi and the Wrights couldn’t see it coming three years out, imagine how hard it must be for anyone else to see it.

If you’re not at the global peak of knowledge of how to do the thing, and looped in on all the progress being made at what will turn out to be the leading project, you aren’t going to be able to see of your own knowledge *at all* that the big development is imminent. Unless you are very good at perspective-taking in a way that wasn’t necessary in a hunter-gatherer tribe, and very good at realizing that other people may know techniques and ideas of which you have no inkling even that you do not know them. If you don’t consciously compensate for the lessons of history in this regard; then you will promptly say the decades-off thing. Fermi wasn’t still thinking that net nuclear energy was impossible or decades away by the time he got to 3 months before he built the first pile, because at that point Fermi was looped in on everything and saw how to do it. But anyone not looped in probably still felt like it was fifty years away while the actual pile was fizzing away in a squash court at the University of Chicago.

People don’t seem to automatically compensate for the fact that the timing of the big development is a function of the peak knowledge in the field, a threshold touched by the people who know the most and have the best ideas; while they themselves have average knowledge; and therefore what they themselves know is not strong evidence about when the big development happens. I think they aren’t thinking about that at all, and they just eyeball it using their own sense of difficulty. If they are thinking anything more deliberate and reflective than that, and incorporating real work into correcting for the factors that might bias their lenses, they haven’t bothered writing down their reasoning anywhere I can read it.

To know that AGI is decades away, we would need enough understanding of AGI to know what pieces of the puzzle are missing, and how hard these pieces are to obtain; and that kind of insight is unlikely to be available until the puzzle is complete. Which is also to say that to anyone outside the leading edge, the puzzle will look more incomplete than it looks on the edge. That project may publish their theories in advance of proving them, although I hope not. But there are unproven theories now too.

And again, that’s not to say that people saying “fifty years” is a certain sign that something is happening in a squash court; they were saying “fifty years” sixty years ago too. It’s saying that anyone who thinks technological *timelines* are actually forecastable, in advance, by people who are not looped in to the leading project’s progress reports and who don’t share all the best ideas about exactly how to do the thing and how much effort is required for that, is learning the wrong lesson from history. In particular, from reading history books that neatly lay out lines of progress and their visible signs that we all know *now* were important and evidential. It’s sometimes possible to say useful conditional things about the consequences of the big development whenever it happens, but it’s rarely possible to make confident predictions about the *timing* of those developments, beyond a one- or two-year horizon. And if you are one of the rare people who can call the timing, if people like that even exist, nobody else knows to pay attention to you and not to the Excited Futurists or Sober Skeptics.

**Four:** The future uses different tools, and can therefore easily do things that are very hard now, or do with difficulty things that are impossible now.

Why do we know that AGI is decades away? In popular articles penned by heads of AI research labs and the like, there are typically three prominent reasons given:

(A) The author does not know how to build AGI using present technology. The author does not know where to start.

(B) The author thinks it is really very hard to do the impressive things that modern AI technology does, they have to slave long hours over a hot GPU farm tweaking hyperparameters to get it done. They think that the public does not appreciate how hard it is to get anything done right now, and is panicking prematurely because the public thinks anyone can just fire up Tensorflow and build a robotic car.

(C) The author spends a lot of time interacting with AI systems and therefore is able to personally appreciate all the ways in which they are still stupid and lack common sense.

We’ve now considered some aspects of argument A. Let’s consider argument B for a moment.

Suppose I say: “It is now possible for one comp-sci grad to do in a week anything that N+ years ago the research community could do with neural networks *at all*.” How large is N?

I got some answers to this on Twitter from people whose credentials I don’t know, but the most common answer was five, which sounds about right to me based on my own acquaintance with machine learning. (Though obviously not as a literal universal, because reality is never that neat.) If you could do something in 2012 period, you can probably do it fairly straightforwardly with modern GPUs, Tensorflow, Xavier initialization, batch normalization, ReLUs, and Adam or RMSprop or just stochastic gradient descent with momentum. The modern techniques are just that much better. To be sure, there are things we can’t do now with just those simple methods, things that require tons more work, but those things were not possible at all in 2012.

In machine learning, when you can do something at all, you are probably at most a few years away from being able to do it easily using the future’s much superior tools. From this standpoint, argument B, “You don’t understand how hard it is to do what we do,” is something of a non-sequitur when it comes to timing.

Statement B sounds to me like the same sentiment voiced by Rutherford [in 1933](https://www.edge.org/conversation/the-myth-of-ai#26015) when he called net energy from atomic fission “moonshine”. If you were a nuclear physicist in 1933 then you had to split all your atoms by hand, by bombarding them with other particles, and it was a laborious business. If somebody talked about getting net energy from atoms, maybe it made you feel that you were unappreciated, that people thought your job was easy.

But of course this will always be the lived experience for AI engineers on serious frontier projects. You don’t get paid big bucks to do what a grad student can do in a week (unless you’re working for a bureaucracy with no clue about AI; but that’s not Google or FB). Your personal experience will *always* be that what you are paid to spend months doing is difficult. A change in this personal experience is therefore not something you can use as a fire alarm.

Those playing the part of wiser sober skeptical scientists would obviously agree in the abstract that our tools will improve; but in the popular articles they pen, they just talk about the painstaking difficulty of this year’s tools. I think that when they’re in that mode they are not even trying to forecast what the tools will be like in 5 years; they haven’t written down any such arguments as part of the articles I’ve read. I think that when they tell you that AGI is decades off, they are literally giving an estimate of [how long it feels to them](https://www.readthesequences.com/UnboundedScalesHugeJuryAwardsAndFuturism) like it would take to build AGI using their current tools and knowledge. Which is why they emphasize how hard it is to stir the heap of linear algebra until it spits out good answers; I think they are not imagining, at all, into how this experience may change over considerably less than fifty years. If they’ve explicitly considered the bias of estimating future tech timelines based on their present subjective sense of difficulty, and tried to compensate for that bias, they haven’t written that reasoning down anywhere I’ve read it. Nor have I ever heard of that forecasting method giving good results historically.

**Five:** Okay, let’s be blunt here. I don’t think most of the discourse about AGI being far away (*or* that it’s near) is being generated by models of future progress in machine learning. I don’t think we’re looking at wrong models; I think we’re looking at no models.

I was once at a conference where there was a panel full of famous AI luminaries, and most of the luminaries were nodding and agreeing with each other that of course AGI was very far off, except for two famous AI luminaries who stayed quiet and let others take the microphone.

I got up in Q&A and said, “Okay, you’ve all told us that progress won’t be all that fast. But let’s be more concrete and specific. I’d like to know what’s the *least* impressive accomplishment that you are very confident *cannot* be done in the next two years.”

There was a silence.

Eventually, two people on the panel ventured replies, spoken in a rather more tentative tone than they’d been using to pronounce that AGI was decades out. They named “A robot puts away the dishes from a dishwasher without breaking them”, and [Winograd schemas](http://www.cs.nyu.edu/faculty/davise/papers/WinogradSchemas/WS.html). Specifically, “I feel quite confident that the Winograd schemas—where we recently had a result that was in the 50, 60% range—in the next two years, we will not get 80, 90% on that regardless of the techniques people use.”

A few months after that panel, there was unexpectedly a big breakthrough on Winograd schemas. The breakthrough didn’t crack 80%, so three cheers for wide credibility intervals with error margin, but I expect the predictor might be feeling slightly more nervous now with one year left to go. (I don’t think it was the breakthrough I remember reading about, but Rob turned up [this paper](https://l.facebook.com/l.php?u=https%3A%2F%2Fwww.ijcai.org%2Fproceedings%2F2017%2F0326.pdf&h=ATMiIliuWNyZbf0ezht51f12W7gL1Gw1AgwfGsF2MUCMMNa_sw9vB1iS6etZYPeiaJuKbxZR92VAbn7uJAZwUHkXm59JK0pBI4cB2ve9rKRpz0vKGXozkvegWE7gbiUWuoP8BwLo0_0mhpnIdbfiO9X9Dpw) as an example of one that could have been submitted at most 44 days after the above conference and gets up to 70%.)

But that’s not the point. The point is the silence that fell after my question, and that eventually I only got two replies, spoken in tentative tones. When I asked for concrete feats that were impossible in the next two years, I think that that’s when the luminaries on that panel switched to trying to build a mental model of future progress in machine learning, asking themselves what they could or couldn’t predict, what they knew or didn’t know. And to their credit, most of them did know their profession well enough to realize that forecasting future boundaries around a rapidly moving field is actually *really hard*, that nobody knows what will appear on arXiv next month, and that they needed to put wide credibility intervals with very generous upper bounds on how much progress might take place twenty-four months’ worth of arXiv papers later.

(Also, Demis Hassabis was present, so they all knew that if they named something insufficiently impossible, Demis would have DeepMind go and do it.)

The question I asked was in a completely different genre from the panel discussion, requiring a mental context switch: the assembled luminaries actually had to try to consult their rough, scarce-formed intuitive models of progress in machine learning and figure out what future experiences, if any, their model of the field definitely prohibited within a two-year time horizon. Instead of, well, emitting socially desirable verbal behavior meant to kill that darned hype about AGI and get some predictable applause from the audience.

I’ll be blunt: I don’t think the confident long-termism has been thought out at all. If your model has the extraordinary power to say what will be impossible in ten years after another one hundred and twenty months of arXiv papers, then you ought to be able to say much weaker things that are impossible in two years, and you should have those predictions queued up and ready to go rather than falling into nervous silence after being asked.

In reality, the two-year problem is hard and the ten-year problem is laughably hard. The future is hard to predict in general, our predictive grasp on a rapidly changing and advancing field of science and engineering is very weak indeed, and it doesn’t permit narrow credible intervals on what can’t be done.

Grace et al. ([2017](https://arxiv.org/abs/1705.08807)) surveyed the predictions of 352 presenters at ICML and NIPS 2015. Respondents’ aggregate forecast was that the proposition “all occupations are fully automatable” (in the sense that “for any occupation, machines could be built to carry out the task better and more cheaply than human workers”) will not reach 50% probability until 121 years hence. Except that a randomized subset of respondents were instead asked the slightly different question of “when unaided machines can accomplish every task better and more cheaply than human workers”, and in this case held that this was 50% likely to occur [within 44 years](http://www.bayesianinvestor.com/blog/index.php/2017/06/01/do-ai-experts-exist/).

That’s what happens when you ask people to produce an estimate they can’t estimate, and there’s a social sense of what the desirable verbal behavior is supposed to be.

---

When I observe that there’s no fire alarm for AGI, I’m not saying that there’s no possible equivalent of smoke appearing from under a door.

What I’m saying rather is that the smoke under the door is always going to be arguable; it is not going to be a clear and undeniable and absolute sign of fire; and so there is never going to be a fire alarm producing common knowledge that action is now due and socially acceptable.

There’s an old trope saying that as soon as something is actually done, it ceases to be called AI. People who work in AI and are in a broad sense pro-accelerationist and techno-enthusiast, what you might call the Kurzweilian camp (of which I am not a member), will sometimes rail against this as unfairness in judgment, as moving goalposts.

This overlooks a real and important phenomenon of adverse selection against AI accomplishments: If you can do something impressive-sounding with AI in 1974, then that is because that thing turned out to be doable in some cheap cheaty way, not because 1974 was so amazingly great at AI. We are uncertain about how much cognitive effort it takes to perform tasks, and how easy it is to cheat at them, and the first “impressive” tasks to be accomplished will be those where we were most wrong about how much effort was required. There was a time when some people thought that a computer winning the world chess championship would require progress in the direction of AGI, and that this would count as a sign that AGI was getting closer. When Deep Blue beat Kasparov in 1997, in a Bayesian sense we did learn something about progress in AI, but we also learned something about chess being easy. Considering the techniques used to construct Deep Blue, most of what we learned was “It is surprisingly possible to play chess without easy-to-generalize techniques” and not much “A surprising amount of progress has been made toward AGI.”

Was AlphaGo smoke under the door, a sign of AGI in 10 years or less? People had previously given Go as an example of What You See Before The End.

Looking over the paper describing AlphaGo’s architecture, it seemed to me that we *were* mostly learning that available AI techniques were likely to go further towards generality than expected, rather than about Go being surprisingly easy to achieve with fairly narrow and ad-hoc approaches. Not that the method scales to AGI, obviously; but AlphaGo did look like a product of *relatively* general insights and techniques being turned on the special case of Go, in a way that Deep Blue wasn’t. I also updated significantly on “The general learning capabilities of the human cortical algorithm are less impressive, less difficult to capture with a ton of gradient descent and a zillion GPUs, than I thought,” because if there were anywhere we expected an impressive hard-to-match highly-natural-selected but-still-general cortical algorithm to come into play, it would be in humans playing Go.

Maybe if we’d seen a thousand Earths undergoing similar events, we’d gather the statistics and find that a computer winning the planetary Go championship is a reliable ten-year-harbinger of AGI. But I don’t actually know that. Neither do you. Certainly, anyone can publicly argue that we just learned Go was easier to achieve with strictly narrow techniques than expected, as was true many times in the past. There’s no possible sign short of actual AGI, no case of smoke from under the door, for which we know that this is definitely serious fire and now AGI is 10, 5, or 2 years away. Let alone a sign where we know everyone else will believe it.

And in any case, multiple leading scientists in machine learning have already published articles telling us their criterion for a fire alarm. They will believe Artificial General Intelligence is imminent:

(A) When they personally see how to construct AGI using their current tools. This is what they are always saying is not currently true in order to castigate the folly of those who think AGI might be near.

(B) When their personal jobs do not give them a sense of everything being difficult. This, they are at pains to say, is a key piece of knowledge not possessed by the ignorant layfolk who think AGI might be near, who only believe that because they have never stayed up until 2AM trying to get a generative adversarial network to stabilize.

(C) When they are very impressed by how smart their AI is relative to a human being in respects that still feel magical to them; as opposed to the parts they do know how to engineer, which no longer seem magical to them; aka the AI seeming pretty smart in interaction and conversation; aka the AI actually being an AGI already.

So there isn’t going to be a fire alarm. Period.

There is never going to be a time before the end when you can look around nervously, and see that it is now clearly common knowledge that you can talk about AGI being imminent, and take action and exit the building in an orderly fashion, without fear of looking stupid or frightened.

---

So far as I can presently estimate, now that we’ve had AlphaGo and a couple of other maybe/maybe-not shots across the bow, and seen a huge explosion of effort invested into machine learning and an enormous flood of papers, we are probably going to occupy our present epistemic state until very near the end.

By saying we’re probably going to be in roughly this epistemic state until almost the end, I *don’t* mean to say we know that AGI is imminent, or that there won’t be important new breakthroughs in AI in the intervening time. I mean that it’s hard to guess how many further insights are needed for AGI, or how long it will take to reach those insights. After the next breakthrough, we still won’t know how many more breakthroughs are needed, leaving us in pretty much the same epistemic state as before. Whatever discoveries and milestones come next, it will probably continue to be hard to guess how many further insights are needed, and timelines will continue to be similarly murky. Maybe researcher enthusiasm and funding will rise further, and we’ll be able to say that timelines are shortening; or maybe we’ll hit another AI winter, and we’ll know that’s a sign indicating that things will take longer than they would otherwise; but we still won’t know *how long.*

At some point we might see a sudden flood of arXiv papers in which really interesting and fundamental and scary cognitive challenges seem to be getting done at an increasing pace. Whereupon, as this flood accelerates, even some who imagine themselves sober and skeptical will be unnerved to the point that they venture that perhaps AGI is only 15 years away now, maybe, possibly. The signs might become so blatant, very soon before the end, that people start thinking it is socially acceptable to say that maybe AGI is 10 years off. Though the signs would have to be pretty darned blatant, if they’re to overcome the social barrier posed by luminaries who are estimating arrival times to AGI using their personal knowledge and personal difficulties, as well as all the historical bad feelings about AI winters caused by hype.

But even if it becomes socially acceptable to say that AGI is 15 years out, in those last couple of years or months, I would still expect there to be disagreement. There will still be others protesting that, as much as associative memory and human-equivalent cerebellar coordination (or whatever) are now solved problems, they still don’t know how to construct AGI. They will note that there are no AIs writing computer science papers, or holding a truly sensible conversation with a human, and castigate the senseless alarmism of those who talk as if we already knew how to do that. They will explain that foolish laypeople don’t realize how much pain and tweaking it takes to get the current systems to work. (Although those modern methods can easily do almost anything that was possible in 2017, and any grad student knows how to roll a stable GAN on the first try using the tf.unsupervised module in Tensorflow 5.3.1.)

When all the pieces are ready and in place, lacking only the last piece to be assembled by the very peak of knowledge and creativity across the whole world, it will still seem to the average ML person that AGI is an enormous challenge looming in the distance, because they still won’t personally know how to construct an AGI system. Prestigious heads of major AI research groups will still be writing [articles](https://www.technologyreview.com/s/608986/forget-killer-robotsbias-is-the-real-ai-danger/) decrying the folly of fretting about the total destruction of all Earthly life and all future value it could have achieved, and saying that we should not let this distract us from *real, respectable concerns* like loan-approval systems accidentally absorbing human biases.

Of course, the future is very hard to predict in detail. It’s so hard that not only do I confess my own inability, I make the far stronger positive statement that nobody else can do it either. The “flood of groundbreaking arXiv papers” scenario is one way things could maybe possibly go, but it’s an implausibly specific scenario that I made up for the sake of concreteness. It’s certainly not based on my extensive experience watching other Earthlike civilizations develop AGI. I do put a significant chunk of probability mass on “There’s not much sign visible outside a Manhattan Project until Hiroshima,” because that scenario is simple. Anything more complex is just one more story full of [burdensome details](https://www.readthesequences.com/Burdensome-Details) that aren’t likely to all be true.

But no matter how the details play out, I do predict in a very general sense that there will be no fire alarm that is not an actual running AGI—no unmistakable sign before then that everyone knows and agrees on, that lets people act without feeling nervous about whether they’re worrying too early. That’s just not how the history of technology has usually played out in much simpler cases like flight and nuclear engineering, let alone a case like this one where all the signs and models are disputed. We already know enough about the uncertainty and low quality of discussion surrounding this topic to be able to say with confidence that there will be no unarguable socially accepted sign of AGI arriving 10 years, 5 years, or 2 years beforehand. If there’s any general social panic it will be by coincidence, based on terrible reasoning, uncorrelated with real timelines except by total coincidence, set off by a Hollywood movie, and focused on relatively trivial dangers.

It’s no coincidence that nobody has given any actual account of such a fire alarm, and argued convincingly about how much time it means we have left, and what projects we should only then start. If anyone does write that proposal, the next person to write one will say something completely different. And probably neither of them will succeed at convincing me that they know anything prophetic about timelines, or that they’ve identified any sensible angle of attack that is (a) worth pursuing at all and (b) not worth starting to work on right now.

---

It seems to me that the decision to delay all action until a nebulous totally unspecified future alarm goes off, implies an order of recklessness great enough that the law of continued failure comes into play.

The law of continued failure is the rule that says that if your country is incompetent enough to use a plaintext 9-numeric-digit password on all of your bank accounts and credit applications, your country is not competent enough to correct course after the next disaster in which a hundred million passwords are revealed. A civilization competent enough to correct course in response to that prod, to react to it the way you’d want them to react, is competent enough not to make the mistake in the first place. When a system fails massively and obviously, rather than subtly and at the very edges of competence, the next prod is not going to cause the system to suddenly snap into doing things intelligently.

The law of continued failure is especially important to keep in mind when you are dealing with big powerful systems or high-status people that you might feel nervous about derogating, because you may be tempted to say, “Well, it’s flawed now, but as soon as a future prod comes along, everything will snap into place and everything will be all right.” The systems about which this fond hope is actually warranted look like they are mostly doing all the important things right already, and only failing in one or two steps of cognition. The fond hope is almost never warranted when a person or organization or government or social subsystem is currently falling massively short.

The folly required to ignore the prospect of aliens landing in thirty years is already great enough that the other flawed elements of the debate should come as no surprise.

And with all of that going wrong simultaneously today, we should predict that the same system and incentives won’t produce correct outputs after receiving an uncertain sign that maybe the aliens are landing in five years instead. The law of continued failure suggests that if existing authorities failed in enough different ways at once to think that it makes sense to try to derail a conversation about existential risk by saying the real problem is the security on self-driving cars, the default expectation is that they will still be saying silly things later.

People who make large numbers of simultaneous mistakes don’t generally have all of the incorrect thoughts subconsciously labeled as “incorrect” in their heads. Even when motivated, they can’t suddenly flip to skillfully executing all-correct reasoning steps instead. Yes, we have various experiments showing that monetary incentives can reduce overconfidence and political bias, but (a) that’s reduction rather than elimination, (b) it’s with extremely clear short-term direct incentives, not the nebulous and politicizable incentive of “a lot being at stake”, and (c) that doesn’t mean a switch is flipping all the way to “carry out complicated correct reasoning”. If someone’s brain contains a switch that can flip to enable complicated correct reasoning at all, it’s got enough internal precision and skill to think mostly-correct thoughts now instead of later—at least to the degree that some conservatism and double-checking gets built into examining the conclusions that people know will get them killed if they’re wrong about them.

There is no sign and portent, [no threshold crossed](http://lesswrong.com/lw/hp5/after_critical_event_w_happens_they_still_wont/), that suddenly causes people to wake up and start doing things systematically correctly. People who can react that competently to any sign at all, let alone a less-than-perfectly-certain not-totally-agreed item of evidence that is *likely* a wakeup call, have probably already done the timebinding thing. They’ve already imagined the future sign coming, and gone ahead and thought sensible thoughts earlier, like Stuart Russell saying, “If you know the aliens are landing in thirty years, it’s still a big deal now.”

---

Back in the funding-starved early days of what is now MIRI, I learned that people who donated last year were likely to donate this year, and people who last year were planning to donate “next year” would quite often this year be planning to donate “next year”. Of course there were genuine transitions from zero to one; everything that happens needs to happen for a first time. There were college students who said “later” and gave nothing for a long time in a genuinely strategically wise way, and went on to get nice jobs and start donating. But I also learned well that, like many cheap and easy solaces, saying the word “later” is addictive; and that this luxury is available to the rich as well as the poor.

I don’t expect it to be any different with AGI alignment work. People who are trying to get what grasp they can on the alignment problem will, in the next year, be doing a little (or a lot) better with whatever they grasped in the previous year (plus, yes, any general-field advances that have taken place in the meantime). People who want to defer that until after there’s a better understanding of AI and AGI will, after the next year’s worth of advancements in AI and AGI, want to defer work until a better future understanding of AI and AGI.

Some people really *want* alignment to *get done* and are therefore *now* trying to wrack their brains about how to get something like a reinforcement learner to [reliably identify a utility function over particular elements in a model of the causal environment instead of a sensory reward term](https://arbital.com/p/pointing_finger/) or [defeat the seeming tautologicalness of updated (non-)deference](https://arbital.com/p/updated_deference/). Others would rather be working on other things, and will therefore declare that there is no work that can possibly be done today, *not* spending two hours quietly thinking about it first before making that declaration. And this will not change tomorrow, unless perhaps tomorrow is when we wake up to some interesting newspaper headlines, and probably not even then. The luxury of saying “later” is not available only to the truly poor-in-available-options.

After a while, I started telling effective altruists in college: “If you’re planning to earn-to-give later, then for now, give around $5 every three months. And never give exactly the same amount twice in a row, or give to the same organization twice in a row, so that you practice the mental habit of re-evaluating causes and re-evaluating your donation amounts on a regular basis. *Don’t* learn the mental habit of just always saying ‘later’.”

Similarly, if somebody was *actually* going to work on AGI alignment “later”, I’d tell them to, every six months, spend a couple of hours coming up with the best current scheme they can devise for aligning AGI and doing useful work on that scheme. Assuming, if they must, that AGI were somehow done with technology resembling current technology. And publishing their best-current-scheme-that-isn’t-good-enough, at least in the sense of posting it to Facebook; so that they will have a sense of embarrassment about naming a scheme that does not look like somebody actually spent two hours trying to think of the best bad approach.

There are things we’ll better understand about AI in the future, and things we’ll learn that might give us more confidence that particular research approaches will be relevant to AGI. There may be more future sociological developments akin to Nick Bostrom publishing *Superintelligence*, Elon Musk tweeting about it and thereby heaving a rock through the Overton Window, or more respectable luminaries like Stuart Russell openly coming on board. The future will hold more AlphaGo-like events to publicly and privately highlight new ground-level advances in ML technique; and it may somehow be that this does *not* leave us in the same epistemic state as having already seen AlphaGo and GANs and the like. It could happen! I can’t see exactly how, but the future does have the capacity to pull surprises in that regard.

But before waiting on that surprise, you should ask whether your uncertainty about AGI timelines is really uncertainty at all. If it feels to you that guessing AGI might have a 50% probability in N years is not enough knowledge to act upon, if that feels scarily uncertain and you want to wait for more evidence before making any decisions… then ask yourself how you’d feel if you believed the probability was 50% in N years, and everyone else on Earth also believed it was 50% in N years, and everyone believed it was right and proper to carry out policy P when AGI has a 50% probability of arriving in N years. If that visualization feels very different, then any nervous “uncertainty” you feel about doing P is not really about whether AGI takes much longer than N years to arrive.

And you are almost surely going to be stuck with that feeling of “uncertainty” no matter how close AGI gets; because no matter how close AGI gets, whatever signs appear will almost surely not produce common, shared, agreed-on public knowledge that AGI has a 50% chance of arriving in N years, nor any agreement that it is therefore right and proper to react by doing P.

And if all that did become common knowledge, then P is unlikely to still be a neglected intervention, or AI alignment a neglected issue; so you will have waited until sadly late to help.

But far more likely is that the common knowledge just isn’t going to be there, and so it will always feel nervously “uncertain” to consider acting.

You can either act despite that, or not act. Not act until it’s too late to help much, in the best case; not act at all until after it’s essentially over, in the average case.

I don’t think it’s wise to wait on an unspecified epistemic miracle to change how we feel. In all probability, you’re going to be in this mental state for a while—including any nervous-feeling “uncertainty”. If you handle this mental state by saying “later”, that general policy is not likely to have good results for Earth.

---

Further resources:

* MIRI’s [research guide](https://intelligence.org/research-guide/) and [research forum](https://agentfoundations.org)
* FLI’s [collection of introductory resources](https://futureoflife.org/2016/02/29/introductory-resources-on-ai-safety-research/)
* CHAI’s alignment bibliography at <http://humancompatible.ai/bibliography>
* 80,000 Hours’ AI job postings on <https://80000hours.org/job-board/>
* The Open Philanthropy Project’s [AI fellowship](http://www.openphilanthropy.org/focus/global-catastrophic-risks/potential-risks-advanced-artificial-intelligence/open-philanthropy-project-ai-fellows-program) and general call for [research proposals](http://www.openphilanthropy.org/blog/concrete-problems-ai-safety)
* My brain-dumps on [AI alignment](https://arbital.com/explore/2v)
* If you’re arriving here for the first time, my long-standing work on [rationality](https://www.lesswrong.com/sequences), and CFAR’s [workshops](http://rationality.org/workshops/upcoming)
* And some general tips from [Ray Arnold](http://effective-altruism.com/ea/17s/what_should_the_average_ea_do_about_ai_alignment/) for effective altruists considering AI alignment as a cause area.

---

The post [There’s No Fire Alarm for Artificial General Intelligence](https://intelligence.org/2017/10/13/fire-alarm/) appeared first on [Machine Intelligence Research Institute](https://intelligence.org).


September 2017 Newsletter

**Research updates**

* “[Incorrigibility in the CIRL Framework](https://intelligence.org/2017/08/31/incorrigibility-in-cirl/)”: a new paper by MIRI assistant researcher Ryan Carey responds to Hadfield-Menell et al.’s “The Off-Switch Game”.
* New at IAFF: [The Three Levels of Goodhart’s Curse](https://agentfoundations.org/item?id=1621); [Conditioning on Conditionals](https://agentfoundations.org/item?id=1624); [Stable Pointers to Value: An Agent Embedded in Its Own Utility Function](https://agentfoundations.org/item?id=1622); [Density Zero Exploration](https://agentfoundations.org/item?id=1627); [Autopoietic Systems and the Difficulty of AGI Alignment](https://agentfoundations.org/item?id=1628)
* Ryan Carey is leaving MIRI to collaborate with the Future of Humanity Institute’s Owain Evans on AI safety work.

**General updates**

* As part of his engineering internship at MIRI, Max Harms assisted in the construction and extension of [RL-Teacher](https://github.com/nottombrown/rl-teacher), an open-source tool for training AI systems with human feedback based on the “[Deep RL from Human Preferences](https://arxiv.org/abs/1706.03741)” OpenAI / DeepMind research collaboration. See [OpenAI’s announcement](https://blog.openai.com/gathering_human_feedback/).
* MIRI COO Malo Bourgon participated in panel discussions on getting things done ([video](https://www.youtube.com/watch?v=trTslOidmq8)) and working in AI ([video](https://www.youtube.com/watch?v=gmL_7SayalM)) at the Effective Altruism Global conference in San Francisco. AI Impacts researcher Katja Grace also spoke on AI safety ([video](https://www.youtube.com/watch?v=r91Co5CeOCY)). Other EAG talks on AI included Daniel Dewey’s ([video](https://www.youtube.com/watch?v=Nfrh4K3d_Z0)) and Owen Cotton-Barratt’s ([video](https://www.youtube.com/watch?v=gATWIWiIy_8)), and a larger panel discussion ([video](https://www.youtube.com/watch?v=JA4vW4oQavk)).
* Announcing two winners of the Intelligence in Literature prize: Laurence Raphael Brothers’ “[Houseproud](https://www.laurencebrothers.com/stories/houseproud/)” and Shane Halbach’s “[Human in the Loop](https://shanehalbach.com/human-in-the-loop/)”.
* [RAISE](https://wiki.lesswrong.com/wiki/Road_to_AI_Safety_Excellence), a project to develop online AI alignment course material, is seeking volunteers.

**News and links**

* The Open Philanthropy Project is accepting applicants to an [AI Fellows Program](http://www.openphilanthropy.org/focus/global-catastrophic-risks/potential-risks-advanced-artificial-intelligence/open-philanthropy-project-ai-fellows-program) “to fully support a small group of the most promising PhD students in artificial intelligence and machine learning”. See also Open Phil’s partial list of [key research topics](http://www.openphilanthropy.org/focus/global-catastrophic-risks/potential-risks-advanced-artificial-intelligence/open-philanthropy-project-ai-fellows-program#examples) in AI alignment.
* [Call for papers](http://www.aies-conference.com/): AAAI and ACM are running a new Conference on AI, Ethics, and Society, with submissions due by the end of October.
* DeepMind’s Viktoriya Krakovna argues for a [portfolio approach to AI safety research](https://vkrakovna.wordpress.com/2017/08/16/portfolio-approach-to-ai-safety-research/).
* “[Teaching AI Systems to Behave Themselves](https://www.nytimes.com/2017/08/13/technology/artificial-intelligence-safety-training.html?referer=)”: a solid article from the *New York Times* on the growing field of AI safety research. The *Times* also has an [opening](https://nytimes.wd5.myworkdayjobs.com/en-US/News/job/New-York-NY/Artificial-Intelligence-FutureTech-Investigative-Reporter_REQ-001480-1) for an investigative reporter in AI.
* UC Berkeley’s Center for Long-term Cybersecurity is [hiring](https://cltc.berkeley.edu/2017/09/15/cltc-is-hiring-check-out-our-job-listings/) for several roles, including researcher, assistant to the director, and program manager.
* [*Life 3.0*](https://smile.amazon.com/Life-3-0-Being-Artificial-Intelligence/dp/1101946598): Max Tegmark releases a new book on the future of AI ([podcast discussion](https://futureoflife.org/2017/08/29/podcast-life-3-0-human-age-artificial-intelligence/)).

The post [September 2017 Newsletter](https://intelligence.org/2017/09/24/september-2017-newsletter/) appeared first on [Machine Intelligence Research Institute](https://intelligence.org).


[![Incorrigibility in the CIRL Framework](https://intelligence.org/files/IncorrigibilityCIRL.png)](https://arxiv.org/abs/1709.06275)

MIRI assistant research fellow Ryan Carey has a new paper out discussing situations where good performance in [Cooperative Inverse Reinforcement Learning](https://arxiv.org/abs/1606.03137) (CIRL) tasks fails to imply that software agents will assist or cooperate with programmers.

The paper, titled “**[Incorrigibility in the CIRL Framework](https://arxiv.org/abs/1709.06275)**,” lays out four scenarios in which CIRL violates the four conditions for *corrigibility* defined in [Soares et al. (2015)](https://intelligence.org/feed/?paged=22). Abstract:

> A value learning system has incentives to follow shutdown instructions, assuming the shutdown instruction provides information (in the technical sense) about which actions lead to valuable outcomes. However, this assumption is not robust to model mis-specification (e.g., in the case of programmer errors). We demonstrate this by presenting some Supervised POMDP scenarios in which errors in the parameterized reward function remove the incentive to follow shutdown commands. These difficulties parallel those discussed by Soares et al. (2015) in their paper on corrigibility.
>
>
> We argue that it is important to consider systems that follow shutdown commands under some weaker set of assumptions (e.g., that one small verified module is correctly implemented; as opposed to an entire prior probability distribution and/or parameterized reward function). We discuss some difficulties with simple ways to attempt to attain these sorts of guarantees in a value learning framework.
>
>

The paper is a response to a paper by Hadfield-Menell, Dragan, Abbeel, and Russell, “[The Off-Switch Game](https://arxiv.org/abs/1611.08219).” Hadfield-Menell et al. show that an AI system will be more responsive to human inputs when it is uncertain about its reward function and thinks that its human operator has more information about this reward function. Carey shows that the CIRL framework can be used to formalize the problem of corrigibility, and that the known assurances for CIRL systems, given in “The Off-Switch Game”, rely on strong assumptions about having an error-free CIRL system. With less idealized assumptions, a value learning agent may have beliefs that cause it to evade redirection from the human.

> [T]he purpose of a shutdown button is to shut the AI system down *in the event that all other assurances failed*, e.g., in the event that the AI system is ignoring (for one reason or another) the instructions of the operators. If the designers of [the AI system] **R** have programmed the system so perfectly that the prior and [reward function] *R* are completely free of bugs, then the theorems of Hadfield-Menell et al. (2017) do apply. In practice, this means that in order to be corrigible, it would be necessary to have an AI system that was uncertain about all things that could possibly matter. The problem is that performing Bayesian reasoning over all possible worlds and all possible value functions is quite intractable. Realistically, humans will likely have to use a large number of heuristics and approximations in order to implement the system’s belief system and updating rules. […]
>
>
> Soares et al. (2015) seem to want a shutdown button that works as a mechanism of last resort, to shut an AI system down in cases where it has observed and refused a programmer suggestion (and the programmers believe that the system is malfunctioning). Clearly, *some* part of the system must be working correctly in order for us to expect the shutdown button to work at all. However, it seems undesirable for the working of the button to depend on there being zero critical errors in the specification of the system’s prior, the specification of the reward function, the way it categorizes different types of actions, and so on. Instead, it is desirable to develop a shutdown module that is small and simple, with code that could ideally be rigorously verified, and which ideally works to shut the system down even in the event of large programmer errors in the specification of the rest of the system.
>
>
> In order to do this in a value learning framework, we require a value learning system that (i) is capable of having its actions overridden by a small verified module that watches for shutdown commands; (ii) has no incentive to remove, damage, or ignore the shutdown module; and (iii) has some small incentive to keep its shutdown module around; even under a broad range of cases where *R*, the prior, the set of available actions, etc. are misspecified.
>
>

Even if the utility function is learned, there is still a need for additional lines of defense against unintended failures. The hope is that this can be achieved by modularizing the AI system. For that purpose, we would need a model of an agent that will behave corrigibly in a way that is robust to misspecification of other system components.

#### Sign up to get updates on new MIRI technical results

*Get notified every time a new technical paper is published.*

*
*

×

The post [New paper: “Incorrigibility in the CIRL Framework”](https://intelligence.org/2017/08/31/incorrigibility-in-cirl/) appeared first on [Machine Intelligence Research Institute](https://intelligence.org).


August 2017 Newsletter

|  |
| --- |
|

**Research updates**
* “[A Formal Approach to the Problem of Logical Non-Omniscience](https://arxiv.org/abs/1707.08747v1)”: We presented our work on [logical induction](https://intelligence.org/2016/09/12/new-paper-logical-induction/) at the [16th Conference on Theoretical Aspects of Rationality and Knowledge](http://tark17.csc.liv.ac.uk/).
* New at IAFF: [Smoking Lesion Steelman](https://agentfoundations.org/item?id=1525); “[Like This World, But…](https://agentfoundations.org/item?id=1527)”; [Jessica Taylor’s Current Thoughts on Paul Christiano’s Research Agenda](https://agentfoundations.org/item?id=1534); [Open Problems Regarding Counterfactuals: An Introduction For Beginners](https://agentfoundations.org/item?id=1591)
* “[A Game-Theoretic Analysis of The Off-Switch Game](http://www.tomeveritt.se/papers/AGI-17-off-switch-game.pdf)”: researchers from Australian National University and Linköping University release a new paper on corrigibility, spun off from a MIRIx workshop.

**General updates**
* Daniel Dewey of the Open Philanthropy Project [writes up his current thoughts](http://effective-altruism.com/ea/1ca/my_current_thoughts_on_miris_highly_reliable/) on MIRI’s highly reliable agent design work, with discussion from Nate Soares and others in the comments section.
* Sarah Marquart of the Future of Life Institute [discusses](https://futureoflife.org/2017/07/18/aligning-superintelligence-with-human-interests/) MIRI’s work on logical inductors, corrigibility, and other topics.
* We attended the [Workshop on Decision Theory & the Future of Artificial Intelligence](http://www.decision-ai.org/) and the [5th International Workshop on Strategic Reasoning](http://sr2017.csc.liv.ac.uk/).

**News and links**
* Open Phil awards [a four-year $2.4 million grant](http://www.openphilanthropy.org/focus/global-catastrophic-risks/potential-risks-advanced-artificial-intelligence/montreal-institute-learning-algorithms-ai-safety-research) to Yoshua Bengio’s group at the Montreal Institute for Learning Algorithms “to support technical research on potential risks from advanced artificial intelligence”.
* A new [IARPA-commissioned report](http://www.belfercenter.org/sites/default/files/files/publication/AI%20NatSec%20-%20final.pdf) discusses the potential for AI to accelerate technological innovation and lead to “a self-reinforcing technological and economic edge”. The report suggests that AI “has the potential to be a worst-case scenario” in combining high destructive potential, military/civil dual use, and difficulty of monitoring with potentially low production difficulty.
* Elon Musk and Mark Zuckerberg [criticize each other’s statements](https://www.washingtonpost.com/news/innovations/wp/2017/07/25/billionaire-burn-musk-says-zuckerbergs-understanding-of-ai-threat-is-limited/?utm_term=.d5ba3c701fae) on AI risk.
* China [makes plans](http://www.zdnet.com/article/china-aims-to-become-global-ai-leader-by-2030/) for major investments in AI ([full text](https://na-production.s3.amazonaws.com/documents/translation-fulltext-8.1.17.pdf), [translation note](http://lesswrong.com/lw/pbn/chinas_plan_to_lead_in_ai_purpose_prospects_and/dw6b)).
* Microsoft opens [a new AI lab](https://www.bloomberg.com/news/articles/2017-07-12/microsoft-creates-new-ai-lab-to-take-on-google-s-deepmind) with a goal of building “more general artificial intelligence”.
* New from the Future of Humanity Institute: “[Trial without Error: Towards Safe Reinforcement Learning via Human Intervention](https://owainevans.github.io/blog/hirl_blog.html).”
* FHI is seeking [two research fellows](https://www.fhi.ox.ac.uk/vacancies/) to study AI macrostrategy.
* Daniel Selsam and others release certigrad ([arXiv](https://arxiv.org/abs/1706.08605), [github](https://github.com/dselsam/certigrad)), a system for creating formally verified machine learning systems; see discussion on Hacker News ([1](https://news.ycombinator.com/item?id=14739491), [2](https://news.ycombinator.com/item?id=14658832)).
* Applications are open for the Center for Applied Rationality’s [AI Summer Fellows Program](http://rationality.org/workshops/apply-aisfp2017), which runs September 8–25.

 |

The post [August 2017 Newsletter](https://intelligence.org/2017/08/16/august-2017-newsletter/) appeared first on [Machine Intelligence Research Institute](https://intelligence.org).


July 2017 Newsletter

|  |
| --- |
|
A number of major [mid-year MIRI updates](https://intelligence.org/2017/07/04/updates-to-the-research-team-and-a-major-donation/): we received our largest donation to date, $1.01 million from an Ethereum investor! Our research priorities have also shifted somewhat, reflecting the addition of four new full-time researchers (Marcello Herreshoff, Sam Eisenstat, Tsvi Benson-Tilsen, and Abram Demski) and the departure of Patrick LaVictoire and Jessica Taylor.
**Research updates**
* New at IAFF: [Futarchy Fix](https://agentfoundations.org/item?id=1493), [Cooperative Oracles: Stratified Pareto Optima and Almost Stratified Pareto Optima](https://agentfoundations.org/item?id=1508)
* New at AI Impacts: [Some Survey Results!](http://aiimpacts.org/some-survey-results/), [AI Hopes and Fears in Numbers](http://aiimpacts.org/ai-hopes-and-fears-in-numbers/)

**General updates**
* We attended the [Effective Altruism Global Boston](https://www.eaglobal.org/events/ea-global-2017-boston/) event. Speakers included Allan Dafoe on “The AI Revolution and International Politics” ([video](https://www.youtube.com/watch?v=Zef-mIKjHAk)) and Jason Matheny on “Effective Altruism in Government” ([video](https://www.youtube.com/watch?v=g05om2NJwco)).
* MIRI COO Malo Bourgon moderated an [IEEE workshop](https://www.eventbrite.com/e/symposium-on-ethics-of-autonomous-systems-seas-north-america-tickets-28733403383?utm_source=eb_email&utm_medium=email&utm_campaign=event_reminder&utm_term=eventname) revising a section from [*Ethically Aligned Design*](http://standards.ieee.org/develop/indconn/ec/ead_v1.pdf).

**News and links**
* New from DeepMind researchers: “[Interpreting Deep Neural Networks Using Cognitive Psychology](https://deepmind.com/blog/cognitive-psychology/)”
* New from OpenAI researchers: “[Corrigibility](https://ai-alignment.com/corrigibility-3039e668638)”
* A collaboration between DeepMind and OpenAI: “[Learning from Human Preferences](https://blog.openai.com/deep-reinforcement-learning-from-human-preferences/)”
* Recent progress in deep learning: “[Self-Normalizing Neural Networks](https://arxiv.org/abs/1706.02515)”
* From Ian Goodfellow and Nicolas Papernot: “[The Challenge of Verification and Testing of Machine Learning](http://www.cleverhans.io/security/privacy/ml/2017/06/14/verification.html)”
* From 80,000 Hours: a [guide to working in AI policy and strategy](https://80000hours.org/articles/ai-policy-guide/) and a related [interview with Miles Brundage](https://80000hours.org/2017/06/the-world-desperately-needs-ai-strategists-heres-how-to-become-one/) of the Future of Humanity Institute.

 |

The post [July 2017 Newsletter](https://intelligence.org/2017/07/25/july-2017-newsletter/) appeared first on [Machine Intelligence Research Institute](https://intelligence.org).


We have several major announcements to make, covering new developments in the two months since our [2017 strategy update](https://intelligence.org/2017/04/30/2017-updates-and-strategy/):

1. On May 30th, we received a surprise **$1.01 million donation** from an [Ethereum](https://en.wikipedia.org/wiki/Ethereum) cryptocurrency investor. This is the single largest contribution we have received to date by a large margin, and will have a substantial effect on our plans over the coming year.

2. **Two new full-time researchers** are joining MIRI: Tsvi Benson-Tilsen and Abram Demski. This comes in the wake of Sam Eisenstat and Marcello Herreshoff’s addition to the team [in May](https://intelligence.org/2017/03/31/two-new-researchers-join-miri/). We’ve also begun working with engineers on a trial basis for our new slate of [software engineer job openings](https://intelligence.org/2017/04/30/software-engineer-internship-staff-openings/).

3. **Two of our researchers have recently left**: Patrick LaVictoire and Jessica Taylor, researchers previously heading work on our “[Alignment for Advanced Machine Learning Systems](https://intelligence.org/2016/07/27/alignment-machine-learning/)” research agenda.

For more details, see below.

---

#### 1. Fundraising

The major donation we received at the end of May, totaling $1,006,549, comes from a long-time supporter who had donated roughly $50k to our research programs over many years. This supporter has asked to continue to remain anonymous.

The first half of this year has been the most successful in MIRI’s fundraising history, with other notable contributions including Ethereum donations from investor Eric Rogstad totalling ~$22k, and a ~$67k donation from [Octane AI](https://octaneai.com/) co-founder Leif K-Brooks as part of a Facebook Reactions challenge. In total we’ve raised about $1.45M in the first half of 2017.

We’re thrilled and extremely grateful for this show of support. This fundraising success has increased our runway to around 18–20 months, giving us more leeway to trial potential hires and focus on our research and outreach priorities this year.

Concretely, we have already made several plan adjustments as a consequence, including:

* moving forward with more confidence on full-time researcher hires,
* trialing more software engineers, and
* deciding to run only one fundraiser this year, in the winter.[1](https://intelligence.org/2017/07/04/updates-to-the-research-team-and-a-major-donation/#footnote_0_16309 "More generally, this will allow us to move forward confidently with the different research programs we consider high-priority, without needing to divert as many resources from other projects to support our top priorities. This should also allow us to make faster progress on the targeted outreach writing we mentioned in our 2017 update, since we won’t have to spend staff time on writing and outreach for a summer fundraiser.")

This likely is a one-time outlier donation, similar to the $631k in cryptocurrency donations we received from Ripple developer Jed McCaleb in 2013–2014.[2](https://intelligence.org/2017/07/04/updates-to-the-research-team-and-a-major-donation/#footnote_1_16309 "Of course, we’d be happy if these large donations looked less like outliers in the long run. If readers are looking for something to do with digital currency they might be holding onto after the recent surges, know that we gratefully accept donations of many digital currencies! In total, MIRI has raised around $1.85M in cryptocurrency donations since mid-2013.") Looking forward at our funding goals over the next two years:

* While we still have some uncertainty about our 2018 budget, our current point estimate is roughly **$2.5M**.
* This year, between support from the Open Philanthropy Project, the Future of Life Institute, and other sources, we expect to receive at least an additional $600k without spending significant time on fundraising.
* Our tentative (ambitious) goal for the rest of the year is to raise **an additional $950k**, or $3M in total. This would be sufficient for our 2018 budget even if we expand our engineering team more quickly than expected, and would give us a bit of a buffer to account for uncertainty in our future fundraising (in particular, uncertainty about whether the Open Philanthropy Project will continue support after 2017).

On a five-year timescale, our broad funding goals are:[3](https://intelligence.org/2017/07/04/updates-to-the-research-team-and-a-major-donation/#footnote_2_16309 "These plans are subject to substantial change. In particular, an important source of variance in our plans is how our new non-public-facing research progresses, where we’re likely to take on more ambitious growth goals if our new work looks like it’s going well.")

* On the low end, once we finish growing our team over the course of a few years, our default expectation is that our operational costs will be roughly $4M per year, mostly supporting researcher and engineer salaries. Our goal is therefore to reach that level in a sustainable, stable way.
* On the high end, it’s possible to imagine scenarios involving an order-of-magnitude increase in our funding, in which case we would develop a qualitatively different set of funding goals reflecting the fact that we would most likely substantially restructure MIRI.
* For funding levels in between—roughly $4M–$10M per year—it is likely that we would not expand our current operations further. Instead, we might fund work outside of our current research after considering how well-positioned we appear to be to identify and fund various projects, including MIRI-external projects. While we consider it reasonably likely that we are in a good position for this, we would instead recommend that donors direct additional donations elsewhere if we ended up concluding that our donors (or other organizations) are in a better position than we are to respond to surprise funding opportunities in the AI alignment space.[4](https://intelligence.org/2017/07/04/updates-to-the-research-team-and-a-major-donation/#footnote_3_16309 "We would also likely increase our reserves in this scenario, allowing us to better adapt to unexpected circumstances, and there is a smaller probability that we would use these funds to grow moderately more than currently planned without a significant change in strategy.")

A new major once-off donation at the $1M level like this one covers nearly half of our current annual budget, which makes a substantial difference to our one- and two-year plans. Our five-year plans are largely based on assumptions about multiple-year [funding flows](https://intelligence.org/2016/11/11/post-fundraiser-update/), so how aggressively we decide to plan our growth in response to this new donation depends largely on whether we can sustainably raise funds at the level of the above goal in future years (e.g., it depends on whether and how other donors change their level of support in response).

To reduce the uncertainty going into our expansion decisions, we’re encouraging more of our regular donors to sign up for [monthly donations](https://intelligence.org/donate/) or other recurring giving schedules—under 10% of our income currently comes from such donations, which limits our planning capabilities.[5](https://intelligence.org/2017/07/04/updates-to-the-research-team-and-a-major-donation/#footnote_4_16309 "Less frequent (e.g., quarterly) donations are also quite helpful from our perspective, if we know about them in advance and so can plan around them. In the case of donors who plan to give at least once per year, predictability is much more important from our perspective than frequency.") We also encourage supporters to [reach out to us](mailto:contact@intelligence.org) about their future donation plans, so that we can answer questions and make more concrete and ambitious plans.

---

#### 2. New hires

Meanwhile, two new full-time researchers are joining our team after having previously worked with us as associates while based at other institutions.

![Abram Demski](https://intelligence.org/wp-content/uploads/2017/06/abram-d.png)**Abram Demski**, who is joining MIRI as a research fellow this month, is completing a PhD in Computer Science at the University of Southern California. His research to date has focused on cognitive architectures and artificial general intelligence. He is interested in filling in the gaps that exist in formal theories of rationality, especially those concerned with what humans are doing when reasoning about mathematics.

Abram made key contributions to the [MIRIxLosAngeles](https://intelligence.org/mirix/) work that produced [precursor results to logical induction](https://intelligence.org/2016/04/21/two-new-papers-uniform/). His other past work with MIRI includes “[Generalizing Foundations of Decision Theory](https://agentfoundations.org/item?id=1302)” and “[Computable Probability Distributions Which Converge on Believing True Π1 Sentences Will Disbelieve True Π2 Sentences](https://intelligence.org/files/Pi1Pi2Problem.pdf).”

![Tsvi Benson-Tilsen](https://intelligence.org/wp-content/uploads/2017/06/tsvi-bt.png)**Tsvi Benson-Tilsen** has joined MIRI as an assistant research fellow. Tsvi holds a BSc in Mathematics with honors from the University of Chicago, and is on leave from the UC Berkeley Group in Logic and the Methodology of Science PhD program.

Prior to joining MIRI’s research staff, Tsvi was a co-author on “[Logical Induction](https://intelligence.org/2016/09/12/new-paper-logical-induction/)” and “[Formalizing Convergent Instrumental Goals](https://intelligence.org/2015/11/26/new-paper-formalizing-convergent-instrumental-goals/),” and also authored “[Updateless Decision Theory With Known Search Order](https://intelligence.org/files/UDTSearchOrder.pdf)” and “[Existence of Distributions That Are Expectation-Reflective and Know It](https://agentfoundations.org/item?id=548).” Tsvi’s research interests include logical uncertainty, logical counterfactuals, and reflectively stable decision-making.

We’ve also accepted our first six [software engineers](https://intelligence.org/2017/04/30/software-engineer-internship-staff-openings/) for 3-month visits. We are continuing to review applicants, and in light of the generous support we recently received and the strong pool of applicants so far, we are likely to trial more candidates than we’d planned previously.

In other news, going forward Scott Garrabrant will be acting as the research lead for MIRI’s [agent foundations](https://intelligence.org/technical-agenda/) research, handling more of the day-to-day work of coordinating and directing research team efforts.

---

#### 3. The AAMLS agenda

Our AAMLS research was previously the focus of Jessica Taylor, Patrick LaVictoire, and Andrew Critch, all of whom joined MIRI in mid-2015. With Patrick and Jessica departing (on good terms) and Andrew on a two-year leave to work with the Center for Human-Compatible AI, we will be putting relatively little work into the AAMLS agenda over the coming year.

We continue to see the problems described in the AAMLS agenda as highly important, and expect to reallocate more attention to these problems in the future. Additionally, we see the AAMLS agenda as a good template for identifying safety desiderata and promising alignment problems. However, we did not see enough progress on AAMLS problems over the last year to conclude that we should currently prioritize this line of research over our other work (e.g., our agent foundations research on problems such as logical uncertainty and counterfactual reasoning). As a partial consequence, MIRI’s current research staff do not plan to make AAMLS research a high priority in the near future.

Jessica, the project lead, describes some of her takeaways from working on AAMLS:

> […] Why was little progress made?
>
>
> [1.] **Difficulty**
>
>
> I think the main reason is that the problems were very difficult. In particular, they were mostly selected on the basis of “this seems important and seems plausibly solveable”, rather than any *strong* intuition that it’s possible to make progress.
>
>
> In comparison, problems in the agent foundations agenda have seen more progress:
>
>
> * Logical uncertainty (Definability of truth, reflective oracles, logical inductors)
> * Decision theory (Modal UDT, reflective oracles, logical inductors)
> * Vingean reflection (Model polymorphism, logical inductors)
>
>
> One thing to note about these problems is that they were formulated on the basis of a strong intuition that they ought to be solveable. Before logical induction, it was possible to have the intuition that some sort of asymptotic approach could solve many logical uncertainty problems in the limit. It was also possible to strongly think that some sort of self-trust is possible.
>
>
> With problems in the AAMLS agenda, the plausibility argument was something like:
>
>
> * Here’s an existing, flawed approach to the problem (e.g. using a reinforcement signal for environmental goals, or modifications of this approach)
> * Here’s a vague intuition about why it’s possible to do better (e.g. humans do a different thing)
>
>
> which, empirically, turned out not to make for tractable research problems.
>
>
> [2.] **Going for the throat**
>
>
> In an important sense, the AAMLS agenda is “going for the throat” in a way that other agendas (e.g. the agent foundations agenda) are to a lesser extent: it is attempting to solve the whole alignment problem (including goal specification) given access to resources such as powerful reinforcement learning. Thus, the difficulties of the whole alignment problem (e.g. specification of environmental goals) are more exposed in the problems.
>
>
> [3.] **Theory vs. empiricism**
>
>
> Personally, I strongly lean towards preferring theoretical rather than empirical approaches. I don’t know how much I endorse this bias overall for the set of people working on AI safety as a whole, but it is definitely a personal bias of mine.
>
>
> Problems in the AAMLS agenda turned out not to be very amenable to purely-theoretical investigation. This is probably due to the fact that there is not a clear mathematical aesthetic for determining what counts as a solution (e.g. for the environmental goals problem, it’s not actually clear that there’s a recognizable mathematical statement for what the problem is).
>
>
> With the agent foundations agenda, there’s a clearer aesthetic for recognizing good solutions. Most of the problems in the AAMLS agenda have a less-clear aesthetic. […]
>
>

For more details, see Jessica’s retrospective [on the Intelligent Agent Foundations Forum](https://agentfoundations.org/item?id=1470).

More work would need to go into AAMLS before we reached confident conclusions about the tractability of these problems. However, the lack of initial progress provides some evidence that new tools or perspectives may be needed before significant progress is possible. Over the coming year, we will therefore continue to spend some time thinking about AAMLS, but will not make it a major focus.

We continue to actively collaborate with Andrew on MIRI research, and expect to work with Patrick and Jessica more in the future as well. Jessica and Andrew in particular intend to continue to focus on AI safety research, including work on AI strategy and coordination.

We’re grateful for everything Jessica and Patrick have done to advance our research program and our organizational mission over the past two years, and I’ll personally miss having both of them around.

In general, I’m feeling really good about MIRI’s position right now. From our increased financial security and ability to more ambitiously pursue our plans, to the new composition and focus of the research team, the new engineers who are spending time with us, and the growth of the research that they’ll support, things are moving forward quickly and with purpose. Thanks to everyone who has contributed, is contributing, and will contribute in the future to help us do the work here at MIRI.

---

1. More generally, this will allow us to move forward confidently with the different research programs we consider high-priority, without needing to divert as many resources from other projects to support our top priorities. This should also allow us to make faster progress on the targeted outreach writing we mentioned in our 2017 update, since we won’t have to spend staff time on writing and outreach for a summer fundraiser.
2. Of course, we’d be happy if these large donations looked less like outliers in the long run. If readers are looking for something to do with digital currency they might be holding onto after the recent surges, know that [we gratefully accept donations of many digital currencies](https://intelligence.org/donate/)! In total, MIRI has raised around $1.85M in cryptocurrency donations since mid-2013.
3. These plans are subject to substantial change. In particular, an important source of variance in our plans is how our new non-public-facing research progresses, where we’re likely to take on more ambitious growth goals if our new work looks like it’s going well.
4. We would also likely increase our reserves in this scenario, allowing us to better adapt to unexpected circumstances, and there is a smaller probability that we would use these funds to grow moderately more than currently planned without a significant change in strategy.
5. Less frequent (e.g., quarterly) donations are also quite helpful from our perspective, if we know about them in advance and so can plan around them. In the case of donors who plan to give at least once per year, predictability is much more important from our perspective than frequency.

The post [Updates to the research team, and a major donation](https://intelligence.org/2017/07/04/updates-to-the-research-team-and-a-major-donation/) appeared first on [Machine Intelligence Research Institute](https://intelligence.org).


June 2017 Newsletter

|  |
| --- |
|
**Research updates**
* A new [AI Impacts](http://aiimpacts.org/some-survey-results/) paper: “[When Will AI Exceed Human Performance?](https://arxiv.org/abs/1705.08807)” News coverage at [*Digital Trends*](https://www.digitaltrends.com/cool-tech/oxford-yale-ai-survey/) and [*MIT Technology Review*](https://www.technologyreview.com/s/607970/experts-predict-when-artificial-intelligence-will-exceed-human-performance/).
* New at IAFF: [Cooperative Oracles](https://agentfoundations.org/item?id=1468); [Jessica Taylor on the AAMLS Agenda](https://agentfoundations.org/item?id=1470); [An Approach to Logically Updateless Decisions](https://agentfoundations.org/item?id=1472)
* Our 2014 technical agenda, “[Agent Foundations for Aligning Machine Intelligence with Human Interests](https://intelligence.org/files/TechnicalAgenda.pdf),” is now available as a book chapter in the anthology [*The Technological Singularity: Managing the Journey*](http://www.creative-science.org/activities/singularity/).

**General updates**
* [readthesequences.com](https://www.readthesequences.com/): supporters have put together a web version of Eliezer Yudkowsky’s [*Rationality: From AI to Zombies*](https://intelligence.org/rationality-ai-zombies/).
* The Oxford Prioritisation Project publishes [a model of MIRI’s work](http://effective-altruism.com/ea/1ae/a_model_of_the_machine_intelligence_research/) as an existential risk intervention.

**News and links**
* From *MIT Technology Review*: “[Why Google’s CEO Is Excited About Automating Artificial Intelligence](https://www.technologyreview.com/s/607894/why-googles-ceo-is-excited-about-automating-artificial-intelligence/).”
* A new alignment paper from researchers at Australian National University and DeepMind: “[Reinforcement Learning with a Corrupted Reward Channel](https://arxiv.org/abs/1705.08417).”
* New from OpenAI: [Baselines](https://blog.openai.com/openai-baselines-dqn/), a tool for reproducing reinforcement learning algorithms.
* The [Future of Humanity Institute](https://www.fhi.ox.ac.uk/join-partnership-ai/) and [Centre for the Future of Intelligence](http://lcfi.ac.uk/news-events/news/2017/may/16/lcfi-joins-partnership-ai/) join the Partnership on AI alongside [twenty other groups](https://www.partnershiponai.org/partners/).
* New AI safety [job postings](https://80000hours.org/job-board/) include research roles at the [Future of Humanity Institute](https://www.fhi.ox.ac.uk/fhi-seeks-research-assistant-book-existential-risk/) and the [Center for Human-Compatible AI](http://humancompatible.ai/jobs/#postdoc), as well as a [UCLA PULSE fellowship](https://recruit.apo.ucla.edu/apply/JPF02981) for studying AI’s potential large-scale consequences and appropriate preparations and responses.

 |

The post [June 2017 Newsletter](https://intelligence.org/2017/06/16/june-2017-newsletter/) appeared first on [Machine Intelligence Research Institute](https://intelligence.org).


May 2017 Newsletter

|  |
| --- |
|
**Research updates**
* New at IAFF: [The Ubiquitous Converse Lawvere Problem](https://agentfoundations.org/item?id=1372); [Two Major Obstacles for Logical Inductor Decision Theory](https://agentfoundations.org/item?id=1399); [Generalizing Foundations of Decision Theory II](https://agentfoundations.org/item?id=1341).
* New at AI Impacts: [Guide to Pages on AI Timeline Predictions](http://aiimpacts.org/guide-to-pages-on-ai-timeline-predictions/)
* “[Decisions Are For Making Bad Outcomes Inconsistent](https://intelligence.org/2017/04/07/decisions-are-for-making-bad-outcomes-inconsistent/)”: Nate Soares dialogues on some of the deeper issues raised by our “[Cheating Death in Damascus](https://intelligence.org/2017/03/18/new-paper-cheating-death-in-damascus/)” paper.
* We ran a machine learning [workshop](https://intelligence.org/workshops/#april-2017) in early April.
* “[Ensuring Smarter-Than-Human Intelligence Has a Positive Outcome](https://intelligence.org/2017/04/12/ensuring/)”: Nate’s talk at Google ([video](https://www.youtube.com/watch?v=dY3zDvoLoao)) provides probably the best general introduction to MIRI’s work on AI alignment.

**General updates**
* Our [strategy update](https://intelligence.org/2017/04/30/2017-updates-and-strategy/) discusses changes to our AI forecasts and research priorities, new outreach goals, a MIRI/DeepMind collaboration, and other news.
* [MIRI is hiring software engineers!](https://intelligence.org/2017/04/30/software-engineer-internship-staff-openings/) If you’re a programmer who’s passionate about MIRI’s mission and wants to directly support our research efforts, [apply here](https://machineintelligence.typeform.com/to/j8LRNq) to trial with us.
* MIRI Assistant Research Fellow Ryan Carey has taken on an additional [affiliation](http://cser.org/new-research-affiliates/) with the Centre for the Study of Existential Risk, and is also helping edit an issue of [*Informatica*](http://www.informatica.si/index.php/informatica/pages/view/csi3) on superintelligence.

**News and links**
* DeepMind researcher Viktoriya Krakovna lists [security highlights from ICLR](https://futureoflife.org/2017/05/01/machine-learning-security-iclr-2017/).
* DeepMind is [seeking applicants](https://deepmind.com/careers/655890/v) for a policy research position “to carry out research on the social and economic impacts of AI”.
* The Center for Human-Compatible AI [is hiring an assistant director](http://humancompatible.ai/jobs/#assistant-director). Interested parties may also wish to apply for the [event coordinator](http://existence.org/jobs/event-coordinator) position at the new Berkeley Existential Risk Initiative, which will help support work at CHAI and elsewhere.
* 80,000 Hours lists other potentially high-impact [openings](https://80000hours.org/job-board/), including ones at Stanford’s AI Index project, the [White House OSTP](https://www.whitehouse.gov/ostp/internship), [IARPA](https://www.iarpa.gov/index.php/careers/become-a-program-manager), and [IVADO](http://www.lecre.umontreal.ca/poste-dagente-de-recherche-en-ethique-de-lintelligence-artificielle-au-cre/).
* New papers: “[One-Shot Imitation Learning](https://arxiv.org/abs/1703.07326)” and “[Stochastic Gradient Descent as Approximate Bayesian Inference](https://arxiv.org/abs/1704.04289).”
* The Open Philanthropy Project summarizes its findings on [early field growth](http://www.openphilanthropy.org/blog/new-report-early-field-growth).
* The Centre for Effective Altruism is collecting donations for the [Effective Altruism Funds](http://effective-altruism.com/ea/17v/ea_funds_beta_launch/) in a range of cause areas.

 |

The post [May 2017 Newsletter](https://intelligence.org/2017/05/10/may-2017-newsletter/) appeared first on [Machine Intelligence Research Institute](https://intelligence.org).