Anders Sandberg on Space Colonization

![](https://intelligence.org/wp-content/uploads/2014/03/Anders-Sandberg.jpg)Anders Sandberg works [at the Future of Humanity Institute](http://www.fhi.ox.ac.uk/about/staff/), a part of the Oxford Martin School and the Oxford University philosophy faculty. Anders’ research at the FHI centres on societal and ethical issues surrounding human enhancement, estimating the capabilities and underlying science of future technologies, and issues of global catastrophic risk. In particular he has worked on cognitive enhancement, whole brain emulation and risk model uncertainty. He is senior researcher for the FHI-Amlin Research Collaboration on Systemic Risk of Modelling, a unique industry collaboration investigating how insurance modelling contributes to or can mitigate systemic risks.

Anders has a background in computer science and neuroscience. He obtained his Ph.D in computational neuroscience from Stockholm University, Sweden, for work on neural network modelling of human memory. He is co-founder and writer for the think tank Eudoxa, and a regular participant in international public debates about emerging technology.

**Luke Muehlhauser**: In your paper with Stuart Armstrong, “[Eternity in Six Hours](http://commonsenseatheism.com/wp-content/uploads/2013/05/Armstrong-Sandberg-Eternity-in-six-hours-intergalactic-spreading-of-intelligent-life-and-sharpening-the-Fermi-paradox.pdf),” you run through a variety of calculations based on known physics, and show that “Given certain technological assumptions, such as improved automation, the task of constructing Dyson spheres, designing replicating probes, and launching them at distant galaxies, become quite feasible. We extensively analyze the dynamics of such a project, including issues of deceleration and collision with particles in space.”

You frame the issue in terms of the Fermi paradox, but I’d like to ask about your paper from the perspective of “How hard would it be for an AGI-empowered, Earth-based civilization to colonize the stars?”

In section 6.3. you comment on the robustness of the result:

> In the estimation of the authors, the assumptions on intergalactic dust and on the energy efficiency of the rockets represent the most vulnerable part of the whole design; small changes to these assumptions result in huge increases in energy and material required (though not to a scale unfeasible on cosmic timelines). If large particle dust density were an order of magnitude larger, reaching outside the local group would become problematic without shielding methods.
>
>

What about the density of *intra*galactic dust? Given your technological assumptions, do you think it would be fairly straightforward to colonize most of the Milky Way from Earth?

---

**Anders Sandberg**: I believe it would be straightforward given the technological assumptions we make, but to demonstrate it requires much more exploratory engineering footwork.

Intergalactic dust appears to be relatively thin, which is unsurprising: most of the contents of intergalactic space are primordial. There are contributions from galaxies, but they are relatively minor. Another way of seeing that the dust is thin is the absence of obscuration of remote galaxies: the total area along the line of sight is small.

This is unfortunately not true inside the galactic plane. At least in some directions dust is optically thick. The probability of a photon (or a fast spacecraft) hitting a dust particle along its trajectory is high. The size distribution of course matters: if it was all in very small particles the problem is different from than rarer pebbles. (If the total mass density is ρ kg/m3, and the average radius is r meters, then the number density N scales as ρ/r3 grains per cubic meter, while the surface area scales as r2N = ρ/r – finer dust can obscure more, down to the diffraction limit.) The actual size distribution for larger grains that would be a hazard is unfortunately not experimentally known – there is a fair deal known about smaller grains, but the ones we care about here are hard to observe.

The approach in our paper consists of just sending enough probes to have a good chance of one reaching the target without any collision. This has an effective distance cutoff rather dependent on the density of too large grains (d ≈ 1/σN, where σ is the probe cross section and N is the number density). One might argue that sending a *lot* of probes could work beyond that, but since the required number grows as exp(distance/d) this soon swamps any local resources. So clearly some shielding is necessary; this reduces N. (Getting a very small σ is also good up until the point where the length of the javelin-shaped probe becomes long enough that it starts to suffer lateral hits)

Now, showing that this shielding is achievable is where things get much more involved. This requires analysing the interaction between relativistic grains and a target. We have done an informal analysis of this (unpublished), and think it is possible to get the shielding up more than enough for interstellar travel if one can build atomically precise structures. This seems to be roughly within the capabilities assumed in the rest of the paper.

Another interesting issue is that in our galaxy most dust exists within the Thin Disk. This is a disk around 3000 lightyear thick (with a scale height of about 400 lightyears), which contain most of the gas and dust (plus star mass). Outside this is the thick disk, which is far clearer. Even interstellar hydrogen gas is a rather nasty proton beam from the perspective of a relativistic traveller, so it might actually be useful to go above the plane of the galaxy. This is a spiral galaxy problem: elliptical galaxies tend to have little dust or gas, so they are much easier to travel through.

---

**Luke**: What if we just talk about colonizing the local group or the local supercluster? How sensitive are those outcomes to the density of intergalactic space dust and the energy efficiency of rockets?

---

**Anders**: There is a nice trade-off between dangerous dust density and safe travel distance: twice the density of largish dust and your distance is halved. So in our paper we assumed intergalactic dust densities larger than a micron to be 10-35 gm/cm3, giving us a reach of 1,194 megaparsecs for using just a single probe. This is more than enough for the local supercluster (which is just 33 megaparsecs across). So even if dust was 36 times denser than we thought this would be achievable.

Note that in galaxies densities can be up to a few million times higher than outside. So there we should expect the distance of travel to be merely a thousand parsecs (if, as seems likely, there is also more grains in the interstellar medium per unit of density distances get even shorter – at least when flying through the dirtiest parts).

Going slower reduces the impact energy, and hence makes the dangerous dust density go down. This effect is however mainly felt for low velocity (<0.1 c) probes: as soon as you start going an appreciable clip of lightspeed you become sensitive to even rather small grains. However, for local colonization if you are not competing with anybody or trying to outrun cosmological expansion, a leisurely pace might be totally acceptable. The energy demands of course decrease a lot too: for 99% c travel you need to supply energy corresponding to 7 times the rest mass of the probe, and even 0.5 c requires 15% of the rest mass energy. 0.1 c is down to just 0.5%.

Some depends on how good rockets can be made. The rocket equation is my least favourite physics equation: it states that to get a velocity change you need to use an exponential amount of reaction mass (and it gets even worse for relativistic rockets). This is why our paper assumed external launchers to send off the probes: using rockets for launch too would have increased the energy and mass demands enormously. The total energy requirement difference between relatively inefficient fission rockets and efficient antimatter rockets is however less than an order of magnitude, since the efficiency boost is eaten up by a longer range and hence the need to send more probes.

Since the paper was written we have tinkered with laser-pushed rocket ideas: if one section of the probe continues ahead, sending back a very intense laser beam to slow the payload, this can be made more effective. There are some very interesting optical possibilities here if designs can be made atomically precise that would make slowing quite energy efficient. But in the end, I don’t think these considerations will provide many orders of magnitude difference.

---

**Luke**: As explained by your paper, your plan assumes the probes would be launched by “fixed launch systems” such as coilguns, quenchguns, laser propulsion, and particle beam propulsion, which are vastly more efficient than rockets. (Rockets are for deceleration near the end of a probe’s trip.) Could you give a brief description of these fixed launch systems? Which ones are you most confident will be technologically feasible?

---

**Anders**: Most people have heard about railguns, devices that uses the Lorenz force law to electromagnetically accelerate projectiles. These are based on an armature sliding along two current-bearing rails, pushed forward by the magnetic field through the loop of rails and armature. While able to generate tremendous acceleration they also tend to be \*messy\*, since the armature experiences friction with the rails, generating plasma discharges and erosion. Nevertheless, such systems exist today and can generate a few km/s velocity launches. While one can imagine scaling things up and use optimal materials, for relativistic launches railguns are likely entirely out.

In our paper we assumed some form of coil-guns: the projectile is never touching the launcher, and is accelerated by electromagnetic coils along the launch path. This has many practical benefits, like only needing currents for one or two coils at a time. The main limitations are switching speeds (which we assume can be handled by advanced technology, especially since the motion is very predictable), magnetic saturation of the projectile (which limits the acceleration per coil, but can be handled by using many more coils) and resistance in the coils. Using superconductors for the coils seems entirely reasonable for a large scale project with big resources. That also allows “quenchguns”, where energy is first stored in currents in superconducting coils that are then quenched, producing a very sharp field gradient that could propel the payload. According to the literature and experts we talked with these can in principle be made very efficient, converting most of the electrical energy into kinetic energy. They are based on well understood physics and can be seen as a modest extrapolation of current engineering to large scales.

We have not investigated particle beam propulsion, but it clearly has some potential. In a way a particle beam is a coilgun-accelerated stream of very small projectiles, and part of their momentum can be gathered by a projectile either by absorbing them (leading to heating and damage issues), or by using a mini-accelerator to “grip” them. There is not as much research on this as for electromagnetic propulsion or laser propulsion, but the physics seems sound. Whether it can be made efficient enough likely depends on whether the high efficiency of the particle accelerator can balance the inefficiencies of capturing their momentum at the receiving end. This is worth investigating further.

Laser propulsion has been experimentally demonstrated on small scales, and again is fairly well understood from a exploratory engineering standpoint. The main limitation on acceleration is heating of the launched object: ideally it would just reflect the beam back, but imperfect reflection means that the back part will heat up. Atomically precise manufacturing appears to allow surprisingly beam-resistant materials, even when based on current materials. Another problem for relativistic projectiles is that the beam will become redshifted as it speeds up, and that a normal beam will spread out as it gets further away from the source, reducing efficiency. It turns out these problems can to some extent be fixed by some more radical optics: it looks like it is physically possible to create free-floating “optical fibres” (actually a long line of very thin lenses) that allow the beam intensity to remain high even over significant distances, adapting as the effective wavelength changes. These ideas were developed more fully after the manuscript was finished, so we do not make use of them in it. But they suggest (if they work out when fully written up and simulated) that laser-launching (and laser-powered slowing) might be the most effective way for an advanced civilization to deliver relativistic payloads.

---

**Luke**: Now let’s talk about deceleration, where rockets may be needed. But before we get to rockets, let’s talk about more exotic solutions for deceleration: magnetic sails and Bussard ramjets. How might those work?

---

**Anders**: The Bussard ramjet is a classic idea for powering an interstellar starship: instead of lugging around a lot of reaction mass (the reason the rocket equation is so painful), why not gather interstellar gas and burn it in a fusion reactor? Unfortunately this requires a pretty high speed in order to get an appreciable mass inflow, and constructing a ramscoop that sucks up enough mass is daunting. It would probably have to be a vast electromagnetic funnel, requiring significant power to generate (and then some more power to compress the input for fusion). Most calculations in the literature end up with the depressing conclusion that it would not be feasible unless the scoop and fusion reaction were very effective, and the exhaust speed very high.

However, one might of course turn the thing around and instead use the funnel to slow the ship against the interstellar medium. Dana Andrews and Robert Zubrin suggested using a magnetic sail, where the field from a superconducting loop deflects charged particles and hence experiences drag. This might be quite effective for slowing relativistic probes; they get a result that implies that a 99% c probe could be slowed to typical interplanetary speeds in about two centuries.

For slowing in intergalactic space where gas is thin ramjets or magnetic sails are of course not very effective until the probe has come very close to the destination (however, if it takes centuries to brake, the braking distance is still merely a few hundred light-years). The lack of reaction mass is very enticing. In our paper we did not explore these possibilities since we wanted a rather conservative estimate, but it looks likely that magnetic sails are worth a serious look in further modelling.

---

**Luke**: Why do you hate the relativistic rocket equation? How did you come up with your estimates for the energy efficiency of future rocket designs that could be used for deceleration?

---

**Anders**: The rocket equation states that the amount of reaction mass you need to expel grows exponentially with the desired ending velocity – this is why standard rockets are mostly fuel and the payload small. You can get better performance by expelling the reaction mass faster, up to the limit of light-speed (which would require a rocket that expels photons).

The relativistic rocket equation is even worse: as you speed up a lot of the energy turns into mass of the rocket rather than velocity. No amount of energy will get you beyond lightspeed, and as you approach it the acceleration decreases (as seen by an outside observer; somebody onboard the rocket will not notice anything). So the overall energy efficiency becomes worse and worse; unless you really want to get to your destination fast it might not be worth it.

The energy estimates were taken from the literature; see our citations.

This is one area where engineering realities might make designs less efficient, and it does have a big effect: a factor of 50% lower efficiency will square the initial mass to payload ratio. Ouch. Still, given the enormous amount of energy, mass and time available the end result is not too different: very large number of probes can be sent by the resources of a single solar system over astronomically insignificant timescales. It is practically irrelevant if it takes 15 years instead of 6 hours to launch the probes. Inefficiencies however start to bite more sharply the larger the replicator is: in the 500 ton case a 45% less efficient fusion reactor means a complete launch of 1.52 billion probes takes a billion years. So the lesson is clearly that boosting efficiency and/or producing smaller payloads have a big effect. If someone were to argue against our analysis they should likely focus here, looking for fundamental limits of efficiency or credible payload sizes. However, as discussed above, there might be alternatives like magnetic sails that vastly improve efficiency too.

---

**Luke**: Which few “next analyses” would be most desirable, from your perspective, for helping to answer the question of how likely it is that an AGI-empowered Earth-based civilization could colonize the galaxy, the supercluster, or the observable universe?

---

**Anders**: Much of the work can be done using exploratory engineering: using known physics, can we make designs that perform well enough to fit into the overall mission profile? They do not have to be optimal to prove that it could be done by an advanced civilisation (indeed, when thinking about the Dyson shell material requirements we considered a “steampunk” approach consisting of iron mirrors focusing sunlight on boilers with water or gas-filled turbines in order to check that even if Mercury or similar materials sources lacked elements for big photovoltaics large-scale energy production was still possible). The more detail, the more convincing, but effort should mainly be directed at the big bottlenecks of efficiency or where our understanding of engineering is a bit weak. (More stars = higher priority.)

★ Obviously better designs for self-replicating installations with materials closure in generic space environments are nice: the classic NASA lunar factory study is ancient by now. This also helps understand constraints on how quickly industrialisation of a system could be done, whether there are large first-mover advantages in the solar system that might pose arms race temptations, and some firmer numbers of seed sizes and replication times.

★ Understanding the AI needs for space mining and engineering would be useful. Full AGI is presumably by assumption able to run the infrastructure we require, but even lower order intelligence might be adequate: many animals construct elaborate constructions adaptively or navigate in complex environments without much general intelligence. This also helps estimate the importance of AI and AGI for human space industry.

★★ Relativistic mass drivers look feasible but cumbersome. We think investigating the limits and possibilities of laser-powered launching might be more productive.

★★★ We need a better understanding of the large particle distribution in the interstellar and intergalactic medium. Much of probe design hinges on whether to attempt a shielded probe or just send a large number of expendable probes.

★★ Slowing down using magnetic sails or lasers appear promising, but requires a good power source that can last for a very long time for intergalactic trips, presumably induced fission, fusion or antimatter. How do long-term storable energy sources scale? What are their minimum mass?

★ Automated navigation and colonization requires automated astronomy. We assumed the probes knew where they were going, but it would be interesting to analyse automated colonisation planning. Large orbiting infrastructures can likely act as very powerful telescopes.

★ From a Fermi question perspective it would be useful to investigate more the properties of spread in the younger universe: is the shorter distances outweighed by higher densities?

★★ From a strategic perspective it would be useful to investigate whether this type of replicating probes can be successfully prevented from invading an occupied galaxy or not. Much of long-term strategy may depend on whether invaders or defenders have the advantage, including whether very large singleton systems are possible or whether significant amount of resources have to be spent on defence.

---

**Luke**: Thanks, Anders!

The post [Anders Sandberg on Space Colonization](https://intelligence.org/2014/03/02/anders-sandberg/) appeared first on [Machine Intelligence Research Institute](https://intelligence.org).


What is the world’s current distribution of computation, and what will it be in the future?

This question is relevant to several issues in AGI safety strategy. To name just three examples:

* If a large government or corporation wanted to quickly and massively upgrade its computing capacity so as to make a push for AGI or WBE, how quickly could it do so?
* If a government thought that AGI or WBE posed a national security threat or global risk, how much computation could it restrict, how quickly?
* How much extra computing is “immediately” available to a successful botnet or government, simply by running existing computers near 100% capacity rather than at current capacity?

To investigate these questions, MIRI recently contracted Vipul Naik to gather data on the world’s current distribution of computation, including current trends. This blog post summarizes our initial findings by briefly responding to a few questions. Naik’s complete research notes are available **[here](http://intelligence.org/wp-content/uploads/2014/02/Naik-Distribution-of-Computation.pdf)** (22 pages). This work is meant to provide a “quick and dirty” launchpad for future, more thorough research into the topic.

**Q**: *How much of the world’s computation is in high-performance computing clusters vs. normal clusters vs. desktop computers vs. other sources?*

**A**: Computation is split between application-specific integrated circuits (ASICs) and general purpose computing: According to Hilbert & Lopez ([2011a](http://www.uvm.edu/~pdodds/files/papers/others/2011/hilbert2011a.pdf), [2012a](http://commonsenseatheism.com/wp-content/uploads/2014/02/Hilbert-Lopez-How-to-Measure-the-World’s-Technological-Capacity-part-1.pdf), [2012b](http://commonsenseatheism.com/wp-content/uploads/2014/02/Hilbert-Lopez-How-to-Measure-the-World’s-Technological-Capacity-part-2.pdf)), the fraction of computation done by general-purpose computing declined from 40% in 1986 to 3% in 2007. The trend line suggests further decline.

Within general-purpose computing, the split for the year 2007 is as follows:

* For installed capacity: 66% PCs (incl. laptops), 25% videogame consoles, 6% mobile phones/PDAs, 3% servers and mainframes, 0.03% supercomputers, 0.3% pocket calculators.
* For effective gross capacity: 52% PCs, 20% videogame consoles, 13% mobile phones/PDAs, 11% servers and mainframes, 4% supercomputers, 0% pocket calculators.

For more detailed data, see Section 2.2 of Naik’s research notesand Section E of [Hilbert & Lopez (2011b)](http://commonsenseatheism.com/wp-content/uploads/2014/02/Hilbert-Lopez-Supporting-online-material-for-The-Worlds-Technological-Capacity.pdf).

**Q**: *What is it being used for, by whom, where?*

**A**: See the answer above, plus Section 3 of Naik’s research notes.

**Q**: *How much capacity is added per year?*

**A**:Growth rates and doubling periods are as follows, using data 1986-2007:

* General-purpose computing capacity: growth rate 58% per annum, doubling period 18 months  (see Section 2.2 of Naik’s research notes).
* Communication: growth rate 28% per annum, doubling period 34 months (see Section 2.3 of Naik’s research notes).
* Storage: growth rate 23% per annum, doubling period 40 months (see Section 2.4 of Naik’s research notes).
* Application-specific computing: growth rate 83% per annum, doubling time 14 months (see Section 2.2 of Naik’s research notes).

Breakdown of data by time periods is available in [Hilbert & Lopez (2011a)](http://www.uvm.edu/~pdodds/files/papers/others/2011/hilbert2011a.pdf), and the most important quotes are included in the relevant sections of Naik’s research notes.

**Q**: *How quickly could capacity be scaled up (in the short or medium term) if demand for computing increased?*

The semiconductor industry is quite responsive to changes in demand, and catches up with book-to-bill ratios as large as 1.4 within 6 months (see Section 3.2 of Naik’s research notes). In addition, the fact that Litecoin, an allegedly ASIC-resistant substitute for Bitcoin, already has ASICs about to be shipped (within two years of launch) also suggests relatively rapid turnaround given large enough economic incentives. In the case of high-frequency trading (HFT), huge investments in nanosecond computing and shaving off milliseconds from Chicago-New York and New York-London cables also suggests quick responsiveness to large incentives.

**Q**: *How much computation would we expect to be available from custom hardware (FPGA’s/ASIC’s and their future analogs)?*

**A**:An increasing fraction (note decline in general-purpose computing’s share from 40% in 1986 to 3% in 2007). However, stuff like ASICs and FPGAs can’t really be repurposed for other use, so existing ones don’t help that much with new tasks.

**Q**: *What is the state of standard conventions for reporting data on such trends?*

**A**:The work of Hilbert, Lopez, and others may eventually lead to uniform conventions for reporting and communicating the data, which would allow for a more informed discussion of these. However, Martin Hilbert in particular is skeptical of standardization in the near future, although he believes it possible in principle, see [Hilbert (2012)](http://ijoc.org/index.php/ijoc/article/download/1318/746) for more. On the other hand, [Dienes (2012)](http://ijoc.org/index.php/ijoc/article/download/1357/737) argues for standardization.

The post [The world’s distribution of computation (initial findings)](https://intelligence.org/2014/02/28/the-worlds-distribution-of-computation-initial-findings/) appeared first on [Machine Intelligence Research Institute](https://intelligence.org).


![Nik Weaver portrait](https://intelligence.org/wp-content/uploads/2014/02/Weaver_w150.jpg)[Nik Weaver](http://www.math.wustl.edu/~nweaver/) is a professor of mathematics at Washington University in St. Louis. He did his graduate work at Harvard and Berkeley and received his Ph.D. in 1994. His main interests are functional analysis, quantization, and the foundations of mathematics. He is best known for his work on independence results in C\*-algebra and his role in the recent solution of the Kadison-Singer problem. His most recent book is *[Forcing for Mathematicians](http://www.amazon.com/Forcing-Mathematicians-Nik-Weaver/dp/9814566950/).*

**Luke Muehlhauser**: In [Weaver (2013)](http://arxiv-web.arxiv.org/abs/1312.3626) you discuss some paradoxes of rational agency. Can you explain roughly what these “paradoxes” are, for someone who might not be all that familiar with provability logic?

---

**Nik Weaver**: Sure. First of all, these are “paradoxes” in the sense of being highly counterintuitive — they’re not outright contradictions.

They all relate to the basic Löbian difficulty that if you reason within a fixed axiomatic system S, and you know that some statement A is provable within S, you’re generally not able to deduce that A is true. This may be an inference that you and I would be willing to make, but if you try to build it into a formal system then the system becomes inconsistent. So, for a rational agent who reasons within a specified axiomatic system, knowing that a proof exists is not as good as actually having a proof.

This leads to some very frustrating consequences. Let’s say I want to build a spaceship, but first I need to be sure that it’s not going to blow up. I have an idea about how to prove this, but it’s extremely tedious, so I write a program to work out the details of the proof and verify that it’s correct. The good news is that when I run the program, it informs me that it was able to fill in the details successfully. The bad news is that I now know that there is a proof that the ship won’t blow up, but I still don’t know that the ship won’t blow up! I’m going to have to check the proof myself, line by line. It’s a complete waste of time, because I *know* that the program functions correctly (we can assume I’ve proven this), so I *know* that the line by line verification is going to check out, but I still have to do it.

You may say that I have been programmed badly. Whoever wrote my source code ought to have allowed me to accept “there is a proof that the ship won’t blow up” as sufficient justification for building the ship. This can be a general rule: for any statement A, let “there exists a proof of A” license all the actions that A licenses. We’re not contradicting Löb’s theorem — we still can’t deduce A from knowing there is a proof of A — but we’re finessing it by stipulating that knowing there’s a proof of A is good enough. But there are still problems. Imagine that I can prove that if the Riemann hypothesis is true, then the ship won’t blow up, and if it’s false then there exists a proof that the ship won’t blow up. Then I’m in a situation where I know that *either* A is true *or* there is a proof that A is true, but I don’t know which one. So even with the more liberal licensing condition, I still can’t build my lovely spaceship.

---

**Luke**: Section 2 of the paper develops an approach to solving this kind of problem via the notion of *warranted assertibility*. How does that work?

---

**Nik**: “Warranted assertibility” is a philosophical term that, in mathematical settings, refers to statements which can be asserted with perfect rational justification. My idea was that this concept could help with these Löbian type problems because it is nicer in some ways than formal provability. For instance, in any consistent axiomatic system there are statements A(n) with the property that for each value of n we can prove A(n), yet there is no single proof of the statement “for all n, A(n)”. However, it is always the case that if each A(n) is warranted then “for all n, A(n)” is also warranted.

So I introduce a predicate Box(A) to express “A is warranted”, and given a suitable axiomatization (which is a little subtle) I show that we can always infer Box(A) from knowing that there is a proof of A. We still can’t infer A — Löb’s theorem is not violated — but, along the lines I suggested earlier, we can program our agents to accept Box(A) as licensing the same actions that A licenses. Thus, it suffices to prove that the statement “the ship won’t blow up” is warranted. If you go this route then all the paradoxes seem to disappear. If an agent is able to reason about warranted assertibility, then it gets to perform all the actions it intuitively ought to be allowed to perform.

---

**Luke**: Later in the paper, you write:

> [[Yudkowsky & Herreshoff (2013)](https://intelligence.org/files/TilingAgents.pdf)] involves a rational agent who is seeking… general permission to delegate the performance of actions to a second agent… The same issues appear in the case of an intelligent machine that is considering modifying its own source code (in order to make itself more intelligent, say). Before doing this it would want to be sure that its post-modiﬁcation state will reason correctly, i.e., any theorem it proves after the modiﬁcation should actually be true. This runs into the familiar Löbian difficulty that the agent is not even able to affirm the soundness of its pre-modiﬁcation reasoning.
>
>
> … In Section 4 of [Yudkowsky & Herreshoff (2013)] two constructions are presented of an infinite sequence of independently acting agents, each of whom can give a provable justification for activating the next one, yet none of whose deductive power falls below an initially prescribed level. The constructions are clever but they have a nonstandard flavor. Probably this is unavoidable, unless the problem description were to be altered in some fundamental way. In the remainder of this section we present another solution which uses nonstandard assertibility reasoning.
>
>

Can you describe the solution you go on to present — again, in terms that someone lacking a background in provability logic might be able to understand, at least in an intuitive way?

---

**Nik**: I will try! First, in regard to my comment that only nonstandard solutions seem possible, I should mention that there is now [a theorem due to Herreshoff, Fallenstein, and Armstrong](http://intelligence.org/wp-content/uploads/2013/12/procrastinati on-paradox.pdf) that makes this idea precise. The point is that it’s not clear I *should* be allowed to build a duplicate copy of myself and delegate my goals to it, for fear that it would then do the same thing, and on ad infinitum (the “procrastination problem”).

Now, going back to my spaceship example, let B be the statement that the ship won’t blow up. A basic feature of assertibility is that we have a general law that A implies Box(A), but we don’t have the converse. So Box(B) is “weaker” than B, and my proposal was that we should allow our agents to accept Box(B) to license the same action (building the spaceship) that B does. This helps with Löbian issues because the statement that B is provable doesn’t imply B, but it does imply Box(B).

But if Box(B) is good enough, why not accept the even weaker statement Box(Box(B))? Or Box(Box(Box(B)))? Let me write Box^k(B) for the general expression with k boxes. The idea about delegating tasks to successors is that if I am willing to accept Box^9(B), say, then I should be willing to build a successor who can’t accept Box^9(B) but can accept Box^8(B). I can reason that if my successor proves Box^8(B), then Box^8(B) is provable, and therefore Box^9(B) is true, which is good enough for me. Then my successor should be happy with a subsequent model that is required to prove Box^7(B), and so on. So we can create finite chains in this way, without doing anything nonstandard.

The way to get infinite chains is to introduce a nonstandard “infinite” natural number kappa, and start with an initial agent who is required to verify Box^kappa(B). Then it can build a successor who is required to verify Box^{kappa – 1}(B), and so on. All the agents in the resulting sequence have the same formal strength because a simple transposition (replacing kappa with kappa + 1) turns each agent into its predecessor, without affecting what they’re able to prove.

---

**Luke**: Now let me ask about research of this type in general. What’s your model for how this kind of research might be useful? It’s pretty distant from current methods for constructing actual software agents.

---

**Nik**: Yes, it’s very theoretical, no question. One model is pure mathematics, which is actually my background. In pure math, most of the time we’re working on questions that don’t have any immediate practical applications. I think experience has shown that this kind of work occasionally becomes very important, while most of it does not, but it’s hard to predict which directions are going to pan out.

Or, you know, maybe we’re all living inside a simulation like Nick Bostrom says. In which case, the fact that we’re trying to figure out how to help machines make themselves smarter may be a clue as to the nature of that simulation.

---

**Luke**: What do you think are some “promising avenues” for making additional progress on reasoning our way through these “paradoxes” of rational agency?

---

**Nik**: I don’t know. If you’re talking about the Löbian obstacle, I feel that something essential is missing from our current problem description. Among the directions that are available now my preference is for Benja Fallenstein’s parametric polymorphism. But I also think we should be looking at ways of modifying the problem description.

---

**Luke**: What is the origin of your interest in paradoxes of rational agency?

---

**Nik**: I’ve been thinking about issues surrounding truth and circularity for around five years. (I just checked: my [earliest paper](http://arxiv.org/pdf/0905.1681.pdf) on the subject was posted on the Mathematics arXiv in May 2009.) This was purely theoretical work, coming from an interest in the foundations of mathematics and logic.

My involvement in paradoxes of rational agency was sparked by [Eliezer Yudkowsky and Marcello Herreshoff’s paper on the Lobian obstacle](https://intelligence.org/files/TilingAgents.pdf). In fact I think I got the term “paradoxes of rational agency” from that paper. When I read it I had the immediate reaction that my ideas about assertibility should be relevant, so that’s how I got interested.

---

**Luke**: Thanks, Nik!

The post [Nik Weaver on Paradoxes of Rational Agency](https://intelligence.org/2014/02/24/nik-weaver-on-paradoxes-of-rational-agency/) appeared first on [Machine Intelligence Research Institute](https://intelligence.org).


MIRI’s May 2014 Workshop

![december-2013-workshop-header-721px](https://intelligence.org/wp-content/uploads/2014/02/december-2013-workshop-header-721px.jpg)
From May 3–11, MIRI will host its **7th Workshop of Logic, Probability, and Reflection**. This workshop will focus on decision theory and tiling agents.

The participants — all veterans of [past workshops](https://intelligence.org/workshops/) — are:

* [Mihály Barasz](http://www.quora.com/Mihaly-Barasz) (Google)
* [Paul Christiano](http://rationalaltruist.com/) (UC Berkeley)
* [Benja Fallenstein](http://lesswrong.com/user/Benja/submitted/) (Bristol U)
* [Marcello Herreshoff](http://www.linkedin.com/pub/marcello-herreshoff/0/8b4/51a) (Google)
* [Patrick LaVictoire](http://www.math.wisc.edu/~patlavic/) (U Wisconsin)
* [Nate Soares](http://so8r.es/) (Google)
* [Nisan Stiennon](http://lesswrong.com/user/Nisan/submitted/) (Stanford)
* [Qiaochu Yuan](http://math.berkeley.edu/~qchu/) (UC Berkeley)
* [Eliezer Yudkowsky](http://yudkowsky.net/) (MIRI)

If you have a strong mathematics background and might like to attend a future workshop, [apply today](https://intelligence.org/get-involved/)! Even if there are no upcoming workshops that fit your schedule, please **still apply**, so that we can notify you of other workshops (long before they are announced publicly).

The post [MIRI’s May 2014 Workshop](https://intelligence.org/2014/02/22/miris-may-2014-workshop/) appeared first on [Machine Intelligence Research Institute](https://intelligence.org).


Recently, Eliezer and I had an email conversation with Holden Karnofsky to discuss future-oriented philanthropy, including MIRI. The participants were:

* [Eliezer Yudkowsky](http://yudkowsky.net/) (research fellow at MIRI)
* [Luke Muehlhauser](http://lukeprog.com/) (executive director at MIRI)
* Holden Karnofsky (co-CEO at [GiveWell](http://www.givewell.org/))

We then edited the email conversation into a streamlined conversation, available [**here**](http://intelligence.org/wp-content/uploads/2014/02/Conversation-with-Holden-Karnofsky-about-Future-Oriented-Philanthropy.pdf).

See also four previous conversations between MIRI and Holden Karnofsky: on [existential risk](http://intelligence.org/2014/01/27/existential-risk-strategy-conversation-with-holden-karnofsky/), on [MIRI strategy](http://intelligence.org/2014/01/13/miri-strategy-conversation-with-steinhardt-karnofsky-and-amodei/), on [transparent research analyses](http://intelligence.org/2013/08/25/holden-karnofsky-interview/), and on [flow-through effects](http://intelligence.org/2013/09/14/effective-altruism-and-flow-through-effects/).

The post [Conversation with Holden Karnofsky about Future-Oriented Philanthropy](https://intelligence.org/2014/02/21/conversation-with-holden-karnofsky-about-future-oriented-philanthropy/) appeared first on [Machine Intelligence Research Institute](https://intelligence.org).


John Baez on Research Tactics

![John Baez portrait](https://intelligence.org/wp-content/uploads/2014/02/Baez_w150.jpg)[John Baez](http://math.ucr.edu/home/baez/) is a professor of mathematics at U.C. Riverside. Until recently he worked on higher category theory and quantum gravity. His internet column [This Week’s Finds](http://math.ucr.edu/home/baez/TWF.html) dates back to to 1993 and is sometimes called the world’s first blog. In 2010, concerned about climate change and the future of the planet, he switched to working on more practical topics and started the [Azimuth Project](http://www.azimuthproject.org/azimuth/show/HomePage), an international collaboration to create a focal point for scientists and engineers interested in saving the planet. His research now focuses on the math of [networks](http://math.ucr.edu/home/baez/networks/) and [information theory](http://math.ucr.edu/home/baez/information/), which should help us understand the complex systems that dominate biology and ecology.

**Luke Muehlhauser**: In a previous interview, I [asked Scott Aaronson](http://intelligence.org/2013/12/13/aaronson/) which “object-level research tactics” he finds helpful when trying to make progress in theoretical research, and I provided some examples. Do you have any comments on the research tactics that Scott and I listed? Which recommended tactics of your own would you add to the list?

---

**John Baez**: What do you mean by “object-level” research tactics? I’ve got dozens of tactics. Some of them are ways to solve problems. But equally important, or maybe more so, are tactics for coming up with problems to solve: problems that are interesting but still easy enough to solve. By “object-level”, do you mean the former?

---

**Luke**: Both! Conceiving of — and crisply posing — good research problems can often be even more important than solving previously-identified research problems.

---

**John**: Okay. Here are some of my tactics.

(1) Learn a lot. Try to understand how the whole universe works, from the philosophical, logical, mathematical and physical aspects to chemistry, biology, and the sciences based on those, to the historical sciences such as cosmology, paleontology, archaeology and history, to the social sciences such as psychology, sociology, anthropology, politics and economics, to the aspects that are captured best in literature, art and music.

It’s a never-ending quest, and obviously it pays to specialize and become more of an expert on a few things – but the more angles you can take on any subject, the more likely you are to stumble on good questions or good answers to existing questions. Also, when you get stuck on a problem, or get tired, it can be really re-energizing to learn new things.

(2) Keep synthesizing what you learn into terser, clearer formulations. The goal of learning is not to memorize vast amounts of data. You need to do serious data compression, and filter out the noise. Very often people will explain things to you in crappy ways, presenting special cases and not mentioning the general rules, stating general rules incorrectly, and so on.

This process goes on forever. When you first learn algebraic topology, for example, they teach you. homology theory. At the beginner’s level, this is presented as a rather complicated recipe for taking a topological space and getting a list of groups out of it. By looking at examples you get insight into what these groups do: the nth one counts the n-dimensional holes, in some sense. You learn how to use them to solve problems, and how to efficiently compute them.

But later—much later, in my case—you learn that algebraic topology of this sort not really about topological spaces, but something more abstract, called “homotopy types”. This is a discovery that happened rather slowly. It crystallized around the 1968, when a guy named Quillen wrote a book on “homotopical algebra”. It’s always fascinating when this happens: when people in some subject learn that its proper object of study is not what they had thought!

But even this was just the beginning: a lot has happened in math since the 1960s. Shortly thereafter, Grothendieck came along and gave us a new dream of what homotopy types might actually be. Very roughly, he realized that they should show up naturally if we think of “equality” as a process—the process of proving two thing are the same—rather than a static relationship.

I’m being pretty vague here, but I want to emphasize that this was a very fundamental discovery with widespread consequences, not a narrow technical thing.

For a long time people have struggled to make Grothendieck’s dream precise. I was involved in that myself for a while. But in the last 5 years or so, a guy named Voevodsky made a lot of progress by showing us how to redo the foundations of mathematics so that instead of treating equality as a mere relationship, it’s a kind of process. This new approach gives an alternative to set theory, where we use homotopy types right from the start as the basic objects of mathematics, instead of sets. It will take about a century for the effects of this discovery to percolate through all of math.

So, you see, by taking something important but rather technical, like algebraic topology, and refusing to be content with treating it as a bunch of recipes to be memorized, you can dig down into deep truths. But it takes great persistence. Even if you don’t discover these truths yourself, but merely learn them, you have to keep simplifying and unifying.

(3) Look for problems, not within disciplines, but in the gaps between existing disciplines. The division of knowledge into disciplines is somewhat arbitrary, and people put most of their energy into questions that lie squarely within disciplines, so it shouldn’t be surprising that many interesting things are lurking in the gaps, waiting to be discovered.

At this point, tactics (1) and (2) really come in handy. If you study lots of subjects and keep trying to distill their insights into terse, powerful formulations, you’re going to start noticing points of contact between these subjects. Sometimes these will be analogies that deserve to be made precise. Sometimes people in one subject know a trick that people in some other subject could profit from. Sometimes people in one subject have invented the hammer, and people in another have invented the nail—and neither know what these things are good for!

(4) Talk to lots of people. This is a great way to broaden your vistas and find connections between seemingly different subjects.

Talk to the smartest people who will condescend to talk to you. Don’t be afraid to ask them questions. But don’t bore them. Smart people tend to be easily bored. Try to let them talk about what’s interesting to them, instead of showing off and forcing them to listen to your brilliant ideas. But make sure to bring them some “gifts” so they’ll want to talk to you again. “Gifts” include clear explanations of things they don’t understand, and surprising facts—little nuggets of knowledge.

One of my strategies for this was to write [This Week’s Finds](http://math.ucr.edu/home/baez/TWF.html), explaining lots of advanced math and physics. You could say that column is a big pile of gifts. I started out as a nobody, but after ten years or so, lots of smart people had found out about me. So now it’s pretty easy for me to blunder into any subject, write a blog post about it, and get experts to correct me or tell me more. I also get invited to give talks, where I meet lots of smart people.

---

**Luke**: You’ve explained some tactics for how to come up with problems to solve. Once you generate a good list, how do you choose among them?

---

**John**: Here are two bits of advice on that.

(1) Actually *write down lists of problems*.

When I was just getting started, I had a small stock of problems to think about – so small that I could remember most of them. Many were problems I’d heard from other people, but most of those were too hard. I would also generate my own problems, but they were often either too hard, too vague, or too trivial.

In more recent years I’ve been able to build up a huge supply of problems to think about. This means I need to actually list them. Often I generate these lists using the ‘data compression’ tactic I mentioned in part (2) of my last answer. When I learn stuff, I ask:

* Is this apparently new concept or fact a special case of some concept or fact I already know?
* Given two similar-sounding concepts or facts, can I find a third having both of these as special cases?
* Can I use the analogy between X and Y to do something new in subject Y that’s analogous to something people have already done in subject X?
* Given a rough ‘rule of thumb’, can I state it more precisely so that it holds always, or at least more often?

as well as plenty of more specific questions.

So, instead of being ‘idea-poor’, with very few problems to work on, I’m now ‘idea-rich’, and the challenge is keeping track of all the problems and finding the best ones.

I always carry around a notebook. I write down questions that seem interesting, especially when I’m bored. The mere act of writing them down either makes them less vague or reveals them to be hopelessly fuzzy. Sometimes I can solve a problem just by taking the time to state it precisely. And the act of writing down questions naturally triggers more questions.

Besides questions, I like ‘analogy charts’, consisting of two or more columns with analogous items lined up side by side. You can see one near the bottom of my 2nd article on [quantropy](http://johncarlosbaez.wordpress.com/2012/02/10/quantropy-part-2/). Quantropy is an idea born of the analogy between thermodynamics and quantum mechanics. This is a big famous analogy, which I’d known for decades, but writing down an analogy chart made me realize there was a hole in the analogy. In thermodynamics we have entropy, so what’s the analogous thing in quantum mechanics? It turns out there’s an answer: quantropy.

I later wrote [a paper with Blake Pollard](http://arxiv.org/abs/1311.0813) on quantropy, but I gave a link to the blog article because that’s another aspect of how I keep track of questions. I don’t just write lists for myself—I write blog articles about things that I want to understand better.

(2) Only work on problems when you think they’re important and you see how to solve them.

This tactic isn’t for everyone, but it works for me. When I was just getting started I would try to solve problems that I had no idea how to solve. People who are good at puzzles may succeed this way, but I generally did not.

It turns out that for me, a better approach is to make long lists of questions, and keep thinking about them on and off for many years. I slowly make progress until—poof!—I think I see something new and important. Only then do a take a problem off the back burner and start intensely working on it.

The physicist John Wheeler put it this way: you should never do a calculation until you already know the answer. That’s a bit of an exaggeration, because it’s also good to fool around and see where things go. But there’s a lot more truth to it than you might think.

Feynman had a different but related rule of thumb: he only worked on a problem when he felt he had an “inside track” on it—some insight or trick up his sleeve that nobody else had.

---

**Luke**: And once you’ve chosen a problem to solve, what are some of your preferred tactics for actually solving it?

---

**John**: By what I’ve said before, it’s clear that I get serious about a problem only after I have a darn good idea of how to solve it. At the very least, I *believe* I know what to do. So, I just do it.

But usually it doesn’t work quite that easily.

If you only officially tackle problems after absolutely every wrinkle has been ironed out by your previous musings, you’re being too cautious: you’ll miss working on a lot of interesting things. Many young researchers seem to fall prey to the opposite error, and waste time being completely stuck. The right balance lies in the middle. You break a problem down into sub-problems, and break those down into sub-subproblems… and you decide you’re ready to go when all these sub-subproblems seem likely to be doable, even before you’ve worked through the details.

How can you tell if they’re doable? This depends a lot on having previous experience with similar problems. If you’re a newbie, things that seem hard to you can be really easy to experts, while things that seem easy can turn out to be famously difficult.

Even with experience, some of sub-subproblems that seem likely to be routine will turn out to be harder than expected. That’s where the actual work comes in. And here it’s good to have lots of tricks. For example:

(1) If you can’t solve a problem, there should be a similar problem that’s a bit easier. Try solving that. And if you can’t solve that one… use the same principle again! Keep repeating until you get down to something you can solve. Then climb your way back up, one step at a time.

Don’t be embarrassed to simplify a problem to the point where you can actually do it.

(2) There are lots of ways to make a problem easier. Sometimes you should consider a special case. In math there are special cases of special cases of special cases… so there’s a lot of room for exploration here. If you see how enough special cases work, you’ll get ideas that may help you for your original problem.

(3) On the other hand, sometimes a problem becomes simpler when you generalize, leaving out potentially irrelevant details. Often people get stuck in clutter. But if it turns out the generalization doesn’t work, it may help you see which details were actually relevant.

(4) Sometimes instead of down or up the ladder of generality it pays to move across, by considering an analogous problem in a related field.

(5) Finally, a general hint: keep a written record of your efforts to solve a problem, including explanations of what didn’t work, and why. Look back at what you wrote from time to time. It’s amazing how often I come close to doing something right, forget about it, and come back later—sometimes years later—and see things from a slightly different angle, which makes everything fall into place. Failure can be just millimeters from success.

---

**Luke**: Thanks, John!

The post [John Baez on Research Tactics](https://intelligence.org/2014/02/21/john-baez-on-research-tactics/) appeared first on [Machine Intelligence Research Institute](https://intelligence.org).


2013 in Review: Friendly AI Research

This is the 4th part of my personal and qualitative [self-review of MIRI in 2013](http://intelligence.org/2013/12/20/2013-in-review-operations/), in which I review MIRI’s 2013 Friendly AI (FAI) research activities.[1](https://intelligence.org/2014/02/18/2013-in-friendly-ai-research/#footnote_0_10726 "What counts as “Friendly AI research” is, naturally, a matter of debate. For most of this post I’ll assume “Friendly AI research” means “what Yudkowsky thinks of as Friendly AI research,” with the exception of intelligence explosion microeconomics, for reasons given in this post.")

### Friendly AI research in 2013

1. In early 2013, we [decided](http://intelligence.org/2013/04/13/miris-strategy-for-2013/) to shift our priorities from research plus public outreach to a more exclusive focus on technical FAI research. This resulted in roughly as much *public-facing* FAI research in 2013 as in all past years combined.
2. Also, our workshops succeeded in identifying candidates for hire. We expect to hire two 2013 workshop participants in the first half of 2014.
3. During 2013, I learned many things about how to create an FAI research institute and FAI research field. In particular…
4. MIRI needs to attract more experienced workshop participants.
5. Much FAI research can be done by a broad community, and need not be labeled as FAI research. But, more FAI progress is made when the researchers themselves conceive of the research as FAI research.
6. Communication style matters a lot.

### The shift to Friendly AI research

From MIRI’s founding in 2000 until our [strategic shift in early 2013](http://intelligence.org/2013/04/13/miris-strategy-for-2013/),[2](https://intelligence.org/2014/02/18/2013-in-friendly-ai-research/#footnote_1_10726 "Until early 2013, the organization currently named “Machine Intelligence Research Institute” was known as the “Singularity Institute for Artificial Intelligence.”") we did some research and *much* public outreach (e.g. the [Singularity Summit](http://intelligence.org/singularitysummit/) and *[The Sequences](http://wiki.lesswrong.com/wiki/Sequences)*).[3](https://intelligence.org/2014/02/18/2013-in-friendly-ai-research/#footnote_2_10726 "From 2000-2004, “MIRI” was just Eliezer Yudkowsky, doing early FAI research. The organization began to grow in 2004, and by 2006 most efforts were outreach-related rather than research-related. This remained true until early 2013.") In early 2013, we decided that enough outreach and movement-building had been done that we could productively shift to a *primary* focus on research, and Friendly AI research specifically.

The task before us was, essentially, to **create a new FAI research institute** (out of what had previously been primarily an outreach organization), and to **create a new field of FAI research**. We still have much to learn about how to achieve these goals (see below).

Our initial steps were to (1) hold a series of [research workshops](http://intelligence.org/workshops/), and to (2) describe open problems in Friendly AI theory to potential research collaborators. Our workshops and open problem descriptions were aimed at **three goals in particular**. We wanted them to:

1. help us identify researchers MIRI should hire to work full-time on Friendly AI theory,
2. expose additional researchers to the Friendly AI research agenda, and
3. spur concrete progress on open problems in Friendly AI.

First, I’ll describe our 2013 Friendly AI research activities. After that, I’ll review “how good” I think these results are, and what lessons I’ve learned.

#### The workshops

The workshops strategy had been suggested by the success of our one-week [November 2012 workshop](http://intelligence.org/workshops/#november-2012), which had been an experiment involving only four researchers, and had produced the core result of “[Definability of Truth in Probabilistic Logic](http://intelligence.org/wp-content/uploads/2013/03/Christiano-et-al-Naturalistic-reflection-early-draft.pdf).”

Our [first workshop of 2013](http://intelligence.org/workshops/#april-2013), held in April, was an attempt to tackle as many open problems as we could, with as many people as we could gather, to quickly learn which problems were most tractable and which researchers were most likely to contribute in the future. It involved 12 participants and lasted 3 weeks, though (due to scheduling constraints) only 5 researchers participated for the entire duration of the workshop. We learned a great deal about the workshop’s participants, and three problems in particular showed the most progress: Christiano’s “[Definability of Truth](http://intelligence.org/wp-content/uploads/2013/03/Christiano-et-al-Naturalistic-reflection-early-draft.pdf)” framework, LaVictoire’s “[Robust Cooperation](https://intelligence.org/files/RobustCooperation.pdf)” framework, and Fallenstein’s “[parametric polymorphism](http://lesswrong.com/lw/e4e/an_angle_of_attack_on_/)” approach to the [Löbian obstacle for self-modifying systems](http://lesswrong.com/lw/hmt/tiling_agents_for_selfmodifying_ai_opfai_2/). The success of this workshop encouraged us to hold more such workshops, albeit at a smaller scale and with tighter research foci.

Our [next workshop](http://intelligence.org/workshops/#july-2013), in July 2013, had 8 participants and lasted one week. It focused on issues related to logical omniscience and the Löbian obstacle / self-reflective agents, and produced less progress-per-day than the April workshop. Its chief result was described in a [blog post](http://lo-tho.blogspot.com/2013/08/progress-in-logical-priors.html) by participant Abram Demski.

Our [September workshop](http://intelligence.org/workshops/#september-2013) focused instead on decision theory. It had 11 participants and lasted one week. Participants brainstormed “well-posed problems” in the area, built on LaVictoire’s robust cooperation framework, made some progress on formalizing [updateless decision theory](http://wiki.lesswrong.com/wiki/Updateless_decision_theory), and formulated additional toy problems such as the [Ultimate Newcomb’s Problem](http://lesswrong.com/lw/ila/the_ultimate_newcombs_problem/).

Our [November workshop](http://intelligence.org/workshops/#november-2013) was our first workshop held outside of Berkeley. [FHI](http://www.fhi.ox.ac.uk/) graciously hosted us at Oxford University. As with the July workshop, this workshop focused on logical omniscience and self-reflective agents. There were 11 participants, and it lasted one week. November’s theoretical progress flowed into the progress made at our [December workshop](http://intelligence.org/workshops/#december-2013) (same topics, 13 participants, one week), which was captured in [7 new technical reports](http://intelligence.org/2013/12/31/7-new-technical-reports-and-a-new-paper/).

Next, some **basic statistics**:

* We held 5 research workshops in 2013, with all but one of them being one week long.
* These workshops were attended by 35 unique researchers, plus 7 first-day-only visitors (e.g. [Hannes Leitgeb](http://www.philosophie.uni-muenchen.de/lehreinheiten/logik_sprachphil/personen/hannes_leitgeb/index.html) and [Nik Weaver](http://www.math.wustl.edu/~nweaver/)).[4](https://intelligence.org/2014/02/18/2013-in-friendly-ai-research/#footnote_3_10726 "Some statistics about 2013’s 35 workshop participants: 15 have a PhD, three are women, and 3 hold a university faculty position of assistant professor or higher rank. In short, our workshop participants have thus far largely been graduate students, post-docs, and independent researchers. Among the 15 participants who have a PhD, 9 have a PhD in mathematics, 4 have a PhD in computer science, one has a PhD in cognitive science, and one has a joint PhD in philosophy and computer science.")
* For first-time attendees, the median reply to the question “How happy are you that you came to the workshop, 0-10?” was 8.5.
* From the time it went live in March 2013 to the end of 2013, about a dozen people contacted us about our [Recommended Courses for MIRI Math Researchers](http://intelligence.org/courses/) page. However, we have reason to believe it has influenced the study patterns of a much larger number of people. Some MIRI supporters have told us they routinely point smart young acquaintances to that page. Moreover, the page received more unique pageviews in 2013 than (e.g.) our [Donate](https://intelligence.org/donate/) or [About](http://intelligence.org/about/) pages, despite not being linked from every page of the site like the Donate and About pages are. The Recommended Courses page [made it possible](http://lesswrong.com/lw/jg3/the_mechanics_of_my_recent_productivity/) for at least one person (Nate Soares) to quickly upgrade his math skills and attend a workshop in 2013, which he couldn’t have done before studying several of the textbooks on the Courses page.
* From the time it went live in June 2013 to the end of 2013, we received 227 non-junk applications[5](https://intelligence.org/2014/02/18/2013-in-friendly-ai-research/#footnote_4_10726 "By “junk applications” I mean to include both spam applications and applications from people who are clearly incapable of math research, e.g. “Hello, I would love to come to America to learn algebra.”") to attend future MIRI workshops, 47 of which are still being processed. So far, 60 applicants are ones we’ve deemed “promising,” 23 of whom attended a workshop in 2013. Of those 23, about half were researchers with whom we had little to no prior contact.

#### Describing open problems in Friendly AI

In 2013, MIRI described open problems in Friendly AI (OPFAIs) to researchers via three standard methods: articles, talks, and tutorials at workshops.

On **OPFAI articles**: Yudkowsky’s article on “OPFAI #1” discussed [intelligence explosion microeconomics](http://lesswrong.com/lw/hbd/new_report_intelligence_explosion_microeconomics/) (*aka* AI takeoff dynamics), which I consider to be an open problem in “strategy research” rather than in Friendly AI theory, so I discussed it in a [previous post](http://intelligence.org/2014/02/08/2013-in-review-strategic-and-expository-research/). From my perspective, the first written OPFAI description of 2013 was on [logical decision theory](http://intelligence.org/2013/04/19/altairs-timeless-decision-theory-paper-published/). Alex Altair (then a MIRI researcher) described the problem in an April 2013 paper called “[A Comparison of Decision Algorithms on Newcomblike Problems](https://intelligence.org/files/Comparison.pdf).” This open problem had been described before, in [Less](http://lesswrong.com/lw/135/timeless_decision_theory_problems_i_cant_solve/) [Wrong](http://lesswrong.com/lw/15z/ingredients_of_timeless_decision_theory/) [posts](http://lesswrong.com/lw/164/timeless_decision_theory_and_metacircular/) and in a [117-page technical report](https://intelligence.org/files/TDT.pdf), but Altair’s presentation of the issue was more succinct and formal than previous presentations had been.

The second written OPFAI description of 2013 was on the [tiling agents](http://lesswrong.com/lw/hmt/tiling_agents_for_selfmodifying_ai_opfai_2/) problem, and specifically the Löbian obstacle to tiling agents. Yudkowsky brought a draft of this paper to the April workshop, and heavily modified the draft as a result of the progress at that workshop, finally publishing the draft in June 2013. The third written OPFAI description of 2013, by Patrick LaVictoire and co-authors, was on the [robust cooperation](http://lesswrong.com/lw/hmw/robust_cooperation_in_the_prisoners_dilemma/) problem. The fourth written OPFAI description of 2013 was on [naturalized induction](http://lesswrong.com/lw/jd9/building_phenomenological_bridges/).

Because the tiling agents paper took ~2 months of FAI researcher time to produce, we decided to experiment with a [process](http://lesswrong.com/lw/jd9/building_phenomenological_bridges/a89w) that would minimize the amount of FAI-researcher-time required to produce new OPFAI descriptions. First, Yudkowsky brain-dumped the OPFAI to [a Facebook group](https://www.facebook.com/groups/233397376818827/). Then, Robby Bensinger worked with several others to produce Less Wrong posts that described the OPFAI more clearly. The first post produced via this process was published in December 2013: [Building Phenomenological Bridges](http://lesswrong.com/lw/jd9/building_phenomenological_bridges/). The rest of the posts explaining this OPFAI will be published in Q1 2014. Because we want to maximize the amount of FAI researcher hours that goes into FAI research rather than exposition, we hope to hire additional expository writing talent in 2014 (see our [Careers](http://intelligence.org/careers/) page).

On **OPFAI talks**: MIRI scheduled [two OPFAI talks](http://intelligence.org/2013/10/01/upcoming-talks-at-harvard-and-mit/) in 2013. Yudkowsky’s Oct. 15th talk, “Recursion in rational agents: Foundations for self-modifying AI,” described both the robust cooperation and tiling agents problems to an audience at MIT. Two days earlier, (MIRI research associate) Paul Christiano gave a talk about probabilistic metamathematics at Harvard, following up on the earlier results from the “[Definability of Truth](http://intelligence.org/wp-content/uploads/2013/03/Christiano-et-al-Naturalistic-reflection-early-draft.pdf)” paper.[6](https://intelligence.org/2014/02/18/2013-in-friendly-ai-research/#footnote_5_10726 "Probabilistic metamathematics is an OPFAI in itself, and also one possible path toward a solution to the tiling agents problem.") Unfortunately, Yudkowsky’s talk was not recorded, but Christiano’s [was](http://intelligence.org/2013/10/23/probabilistic-metamathematics-and-the-definability-of-truth/).

On **OPFAI tutorials at workshops**: Each MIRI workshop in 2013 opened with a day or two of tutorials on the open problems being addressed by that workshop. These tutorials exposed ~35 researchers (participants and first-day visitors) to OPFAIs they weren’t previously very familiar with. (The others — e.g. Yudkowsky, Christiano, and Fallenstein — were already pretty familiar with the OPFAIs described in the tutorials.)

#### How good are these results?

For comparison’s sake, **MIRI’s 2000-2012 FAI research** efforts consisted in:

* Yudkowsky’s early research into the general “shape” of the Friendly AI challenge, resulting in publications such as “[Creating Friendly AI](https://intelligence.org/files/CFAI.pdf)” (2001), “[Coherent Extrapolated Volition](https://intelligence.org/files/CEV.pdf)” (2004), and “[Artificial Intelligence as a Positive and Negative Factor in Global Risk](https://intelligence.org/files/AIPosNegFactor.pdf)” (2008). These publications did not yet describe any OPFAIs as well-defined as the open problems described in [Altair (2013)](https://intelligence.org/files/Comparison.pdf), [Yudkowsky & Herreshoff (2013)](https://intelligence.org/files/TilingAgents.pdf), or [LaVictoire et al (2013)](http://lesswrong.com/lw/hmw/robust_cooperation_in_the_prisoners_dilemma/).[7](https://intelligence.org/2014/02/18/2013-in-friendly-ai-research/#footnote_6_10726 "The open problems in these publications, too, need additional formalization. Such is the current state of research.")
* Yudkowsky’s early decision theory research, which resulted in TDT circa 2005, though this work wasn’t written up in much detail until 2009 ([1](http://lesswrong.com/lw/135/timeless_decision_theory_problems_i_cant_solve/), [2](http://lesswrong.com/lw/15z/ingredients_of_timeless_decision_theory/), [3](http://lesswrong.com/lw/164/timeless_decision_theory_and_metacircular/)) and [2010](https://intelligence.org/files/TDT.pdf).
* Yudkowsky’s early work on Friendly consequentialist AI, in 2003-2009, some of it with Marcello Herreshoff, and one summer (2006) with Peter de Blanc and Nick Hay as well. This work resulted in early versions of many of the OPFAIs described by MIRI in 2013, currently being written up, or currently in Yudkowsky’s queue to write up. It also resulted in the “infinite waterfall” method later described in [Yudkowsky & Herreshoff (2013)](https://intelligence.org/files/TilingAgents.pdf).
* Yudkowsky worked again with Herreshoff in the summer of 2009, in part on the Löbian obstacle.
* MIRI held a decision theory workshop in March 2010, attended by Eliezer Yudkowsky, [Wei Dai](http://www.weidai.com/), [Stuart Armstrong](http://lesswrong.com/user/Stuart_Armstrong/submitted/), [Gary Drescher](http://en.wikipedia.org/wiki/Gary_Drescher), [Anna Salamon](http://annasalamon.com/), and about a dozen others who were present for some but not all discussions.[8](https://intelligence.org/2014/02/18/2013-in-friendly-ai-research/#footnote_7_10726 "For example, Steve Rayhawk and Henrik Jonsson.") This workshop spawned a decision theory mailing list that has, from 2010 through the present day, produced much of the recent progress on [TDT](http://wiki.lesswrong.com/wiki/Timeless_decision_theory)/[UDT](http://wiki.lesswrong.com/wiki/Updateless_decision_theory)-style decision theories, though mostly via non-MIRI researchers like Wei Dai, Vladimir Slepnev, Stuart Armstrong, and Vladimir Nesov.
* (Former MIRI researcher) Peter de Blanc’s work on “convergence of expected utility for universal AI” and ontological crises, resulting in [de Blanc (2009)](https://intelligence.org/files/Convergence-EU.pdf) and [de Blanc (2011)](https://intelligence.org/files/OntologicalCrises.pdf).
* (MIRI research associate) Daniel Dewey’s work on value learning, resulting in [Dewey (2011)](https://intelligence.org/files/LearningValue.pdf).

Thus, MIRI’s *public-facing* Friendly AI research from 2000-2012 consisted in a few non-technical works like “Creating Friendly AI” and “Coherent Extrapolated Volition,” some philosophical writings on TDT, and three somewhat technical papers by Peter de Blanc and Daniel Dewey. Compare this to MIRI’s 2013 public-facing FAI research: [Muehlhauser & Williamson (2013)](https://intelligence.org/files/IdealAdvisorTheories.pdf),[9](https://intelligence.org/2014/02/18/2013-in-friendly-ai-research/#footnote_8_10726 "This short paper lies deep in the “philosophy” end of the philosophy -> math -> engineering spectrum.") [Altair (2013)](https://intelligence.org/files/Comparison.pdf), [Christiano et al. (2013)](http://intelligence.org/wp-content/uploads/2013/03/Christiano-et-al-Naturalistic-reflection-early-draft.pdf), [Yudkowsky & Herreshoff (2013)](https://intelligence.org/files/TilingAgents.pdf), [LaVictoire et al (2013)](http://lesswrong.com/lw/hmw/robust_cooperation_in_the_prisoners_dilemma/), and [these 7 technical reports](http://intelligence.org/2013/12/31/7-new-technical-reports-and-a-new-paper/).[10](https://intelligence.org/2014/02/18/2013-in-friendly-ai-research/#footnote_9_10726 "For both the 2000-2012 and 2013 calendar periods, when I write of “MIRI’s public-facing FAI work” I’m not including work that was “enabled” but not really “produced” by MIRI or its workshops, for example most work on UDT/ADT (which were nevertheless largely developed on MIRI’s LessWrong.com website and its decision theory mailing list).")

Subjectively, it feels to me like **MIRI produced about as much public-facing Friendly AI research progress in 2013 as in all past years combined** (2000-2012), and possibly more. This is good but not particularly surprising, since 2013 was also the first year in which MIRI *tried* to focus on producing public-facing FAI research progress. (But to be clear: if we remove the “public-facing” qualifier, then it’s clear that Yudkowsky alone produced far more FAI research progress in 2000-2012 than MIRI and its workshops produced in 2013 alone.)

So, did our workshops and open problem descriptions **achieve our stated goals?** Let’s check:

1. Yes, they helped us identify candidates for hire. **We expect to hire two 2013 workshop participants in the first half of 2014**. (One of these hires is pending a visa application approval.)
2. Yes, they exposed many new researchers to the Friendly AI research program. But, this exposure didn’t lead to as much independent Friendly AI work as I had hoped, and I have some theories as to why this was (see below).
3. Yes, they spurred concrete research progress on Friendly AI (see above).

While this represents a promising start toward growing an FAI research institute and a new field of FAI research, there are many dimensions on which our output needs to improve for MIRI to have the impact we hope for (see below).

### What have I learned about how to create an FAI research institute, and a new field of FAI research?

Some of my “lessons learned” from 2013’s FAI research activities were things I genuinely didn’t know at the start of the year. Most of them are things I suspected already, and I think they were confirmed by our experiences in 2013. Here are a few of them, in no particular order.

#### 1. Keep operations work away from researchers.

In other words, “Don’t be afraid of a high operations-staff-to-researchers ratio.” Operations talent (including executive talent) is easier to find than FAI research talent, so it’s important to hire sufficient operations talent to make sure the FAI researchers we *do* find can spend approximately *all* their time on FAI research, and almost *none* of their time on tasks that can mostly be handled by operations staff (writing grant proposals, organizing events, fundraising, paper bibliographies, etc.). MIRI should hire enough operations talent to do this even if it makes our operations-staff-to-researcher ratio looks high for a research institute.[11](https://intelligence.org/2014/02/18/2013-in-friendly-ai-research/#footnote_10_10726 "At the end of 2013, we had five full-time staff members: Luke Muehlhauser (executive director), Louie Helm (deputy director), Eliezer Yudkowsky (research fellow), Malo Bourgon (program manager), and Alex Vermeer (program management analyst), totaling 4 operations staff and one researcher. This 4:1 ratio will shrink as we are able to hire more FAI researchers, but I think it would have been a mistake to try to get by with fewer operations staff in 2013.")

Universities often struggle with this (from a research productivity perspective), loading up some of the best research talent in the world with teaching duties, grant writing duties, and university service.[12](https://intelligence.org/2014/02/18/2013-in-friendly-ai-research/#footnote_11_10726 "Link et al. (2008); Marsh & Hattie (2002); NSOPF (2004).") As an independent research institute, MIRI can set its own policies and minimize these problems.

#### 2. We need to attract more experienced workshop participants.

Our workshops attracted some very bright participants, but they were almost exclusively younger than 30, with relatively few publications to their name. More experienced researchers would probably have advantages in (1) knowing related results and formal tools, (2) knowing productive research tactics, and (3) writing up results for peer-review, among other advantages.

#### 3. Much FAI research can be done by a broad community, and need not be labeled as FAI research.

Presently, the Yudkowskian paradigm for “Friendly AI research” describes a very large research program that breaks down into dozens of sub-problems (OPFAIs), e.g. the [tiling agents](http://lesswrong.com/lw/hmt/tiling_agents_for_selfmodifying_ai_opfai_2/) toy problem. Locating and formulating open problems plausibly relevant for Friendly AI is a challenge in itself, one that especially benefits from specializing in Friendly AI for several years.

Many of the OPFAIs themselves, however, can be framed as “ordinary” open problems in AI safety engineering, philosophy, mathematical logic, theoretical computer science, economics, and other fields. These open problems can often be stated without any mention of Friendly AI, and sometimes without any mention of AI in general.

For every OPFAI Yudkowsky has described,[13](https://intelligence.org/2014/02/18/2013-in-friendly-ai-research/#footnote_12_10726 "Sometimes with much help from Robby Bensinger and/or others.") I’ve been able to locate earlier related work.[14](https://intelligence.org/2014/02/18/2013-in-friendly-ai-research/#footnote_13_10726 "I’ll list some examples of earlier related work. (1) Superrationality: getting agents to rationally cooperate with agents like themselves. Before “Robust Cooperation” there was: Rapoport (1966); McAfee (1984); Hofstadter (1985); Binmore (1987); Howard (1988); Tennenholtz (2004); Fortnow (2009); Kalai et al. (2010); Peters & Szentes (2012). (For Rapoport 1966, see especially pages 141-144 and 209-210. (2) Coherent extrapolated volition: figuring out what we would wish if we knew more, thought better, were more the people we wished we were, etc. Before Yudkowsky (2004) there was: Rawls (1971); Harsanyi (1982); Railton (1986); Rosati (1995). (For an overview of this background, see Muehlhauser & Williamson 2013.) (3) Parliamentary methods for values aggregation: using voting mechanisms to resolve challenges in normative uncertainty and values aggregation. Before Bostrom (2009) there was a vast literature on this topic in social choice theory. For recent overviews, see List (2013); Brandt et al. (2012); Rossi et al. (2011); Gaertner (2009). (4) Reasoning under fragility: figuring out how to get an agent not to operate with full autonomy before it has been made fully trustworthy. Before Yudkowsky began to discuss the issue, there was much work on “adjustable autonomy”: Schreckenghost et al. (2010); Mouaddib et al. (2010); Zieba et al. (2010); Pynadath & Tambe (2002); Tambe et al. (2002). (5) Logical decision theory: finding a decision algorithm which can represent the agent’s deterministic decision process. Before Yudkowsky (2010) there was: Spohn (2003); Spohn (2005). (6) Stable self-improvement: getting a self-modifying agent to avoid rewriting its own code unless it has very high confidence that these rewrites will maintain desirable agent properties. Before Yudkowsky & Herreshoff (2013) there was: Schmidhuber (2003); Schmidhuber (2009); Steunebrink & Schmidhuber (2012). (7) Naturalized induction: getting an induction algorithm to treat itself, its data inputs, and its hypothesis outputs as reducible to its physical posits. Before “Building Phenomenological Bridges” there was: Orseau & Ring (2011); Orseau & Ring (2012).") Although this earlier work has not produced what we would regard as good solutions to open problems in FAI, it does suggest that FAI can be framed in ways palatable to academia. FAI need not be an “alien” research program, operating strictly outside mainstream academia, and conducted only by those explicitly motivated by FAI. Instead, FAI researchers should be able to frame their work in the context of mainstream research paradigms if they choose to do so. Moreover, much FAI research can be done even by those who aren’t explicitly motivated by FAI, so long as they find (e.g.) the Löbian obstacle interesting *as mathematics* — or as computer science, or as philosophy, etc.

#### 4. But, more FAI progress is made when the researchers themselves conceive of the research as FAI research.

Still, researchers *do* seem more likely to produce useful work on Friendly AI if they are thinking about the problems from the perspective of Friendly AI, rather than merely thinking about them as interesting open problems in philosophy, computer science, economics, etc. As I said in [my conversation with Jacob Steinhardt](http://intelligence.org/2014/02/11/conversation-with-jacob-steinhardt-about-miri-strategy/):

> People work on different pieces of the problem depending on whether they’re trying to solve the problem for Friendly AI or just for a math journal. If they aren’t thinking about it from the FAI perspective, people can work all day on stuff that’s very close to what we care about in concept-space and yet has no discernable value to FAI theory. Thus, the people who have contributed by far the most novel FAI progress are people explicitly thinking about the problems from the perspective of FAI…
>
>

####

#### 5. Communication style matters a lot.

When I talk to the kinds of top-notch researchers MIRI would like to collaborate with on open problems in Friendly AI, perhaps the most common complaint I hear is that our work is not formal enough, or not described clearly enough for them to understand it without more effort on their part than they are willing to expend. For an example of such a conversation that was recorded and transcribed, see again [my conversation with Jacob Steinhardt](http://intelligence.org/2014/02/11/conversation-with-jacob-steinhardt-about-miri-strategy/).

I’ve thought this for a long time, and my experiences in 2013 have only reinforced the point. I’ll be writing more about this in the future.

---

1. What counts as “Friendly AI research” is, naturally, a matter of debate. For most of this post I’ll assume “Friendly AI research” means “what Yudkowsky thinks of as Friendly AI research,” with the exception of intelligence explosion microeconomics, for reasons given in this post.
2. Until [early 2013](http://intelligence.org/2013/01/30/we-are-now-the-machine-intelligence-research-institute-miri/), the organization currently named “Machine Intelligence Research Institute” was known as the “Singularity Institute for Artificial Intelligence.”
3. From 2000-2004, “MIRI” was just Eliezer Yudkowsky, doing early FAI research. The organization began to grow in 2004, and by 2006 most efforts were outreach-related rather than research-related. This remained true until early 2013.
4. Some statistics about 2013’s 35 workshop participants: 15 have a PhD, three are women, and 3 hold a university faculty position of assistant professor or higher rank. In short, our workshop participants have thus far largely been graduate students, post-docs, and independent researchers. Among the 15 participants who have a PhD, 9 have a PhD in mathematics, 4 have a PhD in computer science, one has a PhD in cognitive science, and one has a joint PhD in philosophy and computer science.
5. By “junk applications” I mean to include both spam applications and applications from people who are clearly incapable of math research, e.g. “Hello, I would love to come to America to learn algebra.”
6. Probabilistic metamathematics is an OPFAI in itself, and also one possible path toward a solution to the tiling agents problem.
7. The open problems in these publications, too, need additional formalization. Such is the current state of research.
8. For example, Steve Rayhawk and Henrik Jonsson.
9. This short paper lies deep in the “philosophy” end of the philosophy -> math -> engineering [spectrum](http://intelligence.org/2013/11/04/from-philosophy-to-math-to-engineering/).
10. For both the 2000-2012 and 2013 calendar periods, when I write of “MIRI’s public-facing FAI work” I’m not including work that was “enabled” but not really “produced” by MIRI or its workshops, for example most work on [UDT](http://wiki.lesswrong.com/wiki/Updateless_decision_theory)/[ADT](http://wiki.lesswrong.com/wiki/Ambient_decision_theory) (which were nevertheless largely developed on MIRI’s [LessWrong.com](http://lesswrong.com/) website and its decision theory mailing list).
11. At the end of 2013, we had five full-time staff members: Luke Muehlhauser (executive director), Louie Helm (deputy director), Eliezer Yudkowsky (research fellow), Malo Bourgon (program manager), and Alex Vermeer (program management analyst), totaling 4 operations staff and one researcher. This 4:1 ratio will shrink as we are able to hire more FAI researchers, but I think it would have been a mistake to try to get by with fewer operations staff in 2013.
12. [Link et al. (2008)](http://commonsenseatheism.com/wp-content/uploads/2014/01/Link-et-al-A-time-allocation-study-of-university-faculty.pdf); [Marsh & Hattie (2002)](http://commonsenseatheism.com/wp-content/uploads/2014/01/Marsh-Hattie-The-relation-between-research-productivity-and-teaching-effectiveness.pdf); [NSOPF (2004)](http://nces.ed.gov/pubs2006/2006176.pdf).
13. Sometimes with much help from Robby Bensinger and/or others.
14. I’ll list some examples of earlier related work. (**1**) [*Superrationality*](http://en.wikipedia.org/wiki/Superrationality): getting agents to rationally cooperate with agents like themselves. Before “[Robust Cooperation](http://intelligence.org/2014/02/01/robust-cooperation-a-case-study-in-friendly-ai-research/)” there was: [Rapoport (1966)](http://www.amazon.com/Two-Person-Game-Theory-Essential/dp/047205015X/); [McAfee (1984)](http://www.mcafee.cc/Papers/PDF/EffectiveComputability.pdf); [Hofstadter (1985)](http://en.wikipedia.org/wiki/Superrationality); [Binmore (1987)](http://commonsenseatheism.com/wp-content/uploads/2014/02/Binmore-Modeling-Rational-Players-Part-I.pdf); [Howard (1988)](http://commonsenseatheism.com/wp-content/uploads/2013/04/Howard-Cooperation-in-the-Prisoners-Dilemma.pdf); [Tennenholtz (2004)](http://mechroom.technion.ac.il/~moshet/progeqnote4.pdf); [Fortnow (2009)](http://commonsenseatheism.com/wp-content/uploads/2014/02/Fortnow-et-al-Program-Equilibria-and-Discounted-Computation-Time.pdf); [Kalai et al. (2010)](http://commonsenseatheism.com/wp-content/uploads/2014/02/Kalai-et-al-A-commitment-folk-theorem.pdf); [Peters & Szentes (2012)](http://commonsenseatheism.com/wp-content/uploads/2014/02/Peters-Szentes-Definable-and-Contractible-Contracts.pdf). (For Rapoport 1966, see especially pages 141-144 and 209-210. (**2**) [*Coherent extrapolated volition*](http://wiki.lesswrong.com/wiki/Coherent_Extrapolated_Volition): figuring out what we would wish if we knew more, thought better, were more the people we wished we were, etc. Before [Yudkowsky (2004)](https://intelligence.org/files/CEV.pdf) there was: [Rawls (1971)](http://www.amazon.com/A-Theory-Justice-John-Rawls/dp/0674000781/); [Harsanyi (1982)](http://commonsenseatheism.com/wp-content/uploads/2014/02/Harsanyi-Morality-and-the-theory-of-rational-behaviour.pdf); [Railton (1986)](http://commonsenseatheism.com/wp-content/uploads/2012/12/Railton-Facts-and-Values.pdf); [Rosati (1995)](http://commonsenseatheism.com/wp-content/uploads/2012/12/Rosati-Persons-perspectives-and-full-information-accounts-of-the-good.pdf). (For an overview of this background, see [Muehlhauser & Williamson 2013](https://intelligence.org/files/IdealAdvisorTheories.pdf).) (**3**) [*Parliamentary methods for values* aggregation](http://www.overcomingbias.com/2009/01/moral-uncertainty-towards-a-solution.html): using voting mechanisms to resolve challenges in normative uncertainty and values aggregation. Before [Bostrom (2009)](http://www.overcomingbias.com/2009/01/moral-uncertainty-towards-a-solution.html) there was a vast literature on this topic in social choice theory. For recent overviews, see [List (2013)](http://plato.stanford.edu/entries/social-choice/); [Brandt et al. (2012)](http://commonsenseatheism.com/wp-content/uploads/2014/02/Brandt-et-al-computational-social-choice.pdf); [Rossi et al. (2011)](http://www.amazon.com/Short-Introduction-Preferences-Artificial-Inetlligence/dp/1608455866/); [Gaertner (2009)](http://www.amazon.com/Primer-Social-Choice-Theory-Perspectives-ebook/dp/B006R4S9ME/). (**4**) *Reasoning under fragility*: figuring out how to get an agent not to operate with full autonomy before it has been made fully trustworthy. Before Yudkowsky [began to discuss](https://www.facebook.com/groups/233397376818827/permalink/255982137893684/) the issue, there was much work on “adjustable autonomy”: [Schreckenghost et al. (2010)](http://commonsenseatheism.com/wp-content/uploads/2014/02/Schreckenghost-et-al-Measuring-performance-in-real-time-during-remote-human-robot-operations-with-adjustable-autonomy.pdf); [Mouaddib et al. (2010)](http://commonsenseatheism.com/wp-content/uploads/2014/02/Mouaddib-et-al-A-decision-theoretic-approach-to-cooperative-control-and-adjustable-autonomy.pdf); [Zieba et al. (2010)](http://commonsenseatheism.com/wp-content/uploads/2014/02/Zieba-et-al.-Principles-of-adjustable-autonomy-a-framework-for-resilient-human–machine-cooperation.pdf); [Pynadath & Tambe (2002)](http://commonsenseatheism.com/wp-content/uploads/2014/02/Pynadath-Tambe-Revisiting-Asimovs-first-law-a-response-to-the-call-to-arms.pdf); [Tambe et al. (2002)](http://commonsenseatheism.com/wp-content/uploads/2014/02/Tambe-et-al-Adjustable-autonomy-for-the-real-world.pdf). (**5**) *Logical decision theory*: finding a decision algorithm which can represent the agent’s deterministic decision process. Before [Yudkowsky (2010)](https://intelligence.org/files/TDT.pdf) there was:[Spohn (2003)](http://commonsenseatheism.com/wp-content/uploads/2012/09/Spohn-Dependency-equilibria-and-the-causal-structure-of-decision-and-game-situations.pdf); [Spohn (2005)](http://commonsenseatheism.com/wp-content/uploads/2012/09/Spohn-5-Questions-on-Formal-Philosophy.pdf). (**6**) *Stable self-improvement*: getting a self-modifying agent to avoid rewriting its own code unless it has very high confidence that these rewrites will maintain desirable agent properties. Before [Yudkowsky & Herreshoff (2013)](https://intelligence.org/files/TilingAgents.pdf) there was: [Schmidhuber (2003)](http://arxiv.org/abs/cs.LO/0309048); [Schmidhuber (2009)](http://www.idsia.ch/~juergen/ultimatecognition.pdf); [Steunebrink & Schmidhuber (2012)](http://www.idsia.ch/~steunebrink/Publications/AGIbook2_self-reflection.pdf). (**7**) [*Naturalized induction*](http://wiki.lesswrong.com/wiki/Naturalized_induction): getting an induction algorithm to treat itself, its data inputs, and its hypothesis outputs as reducible to its physical posits. Before “[Building Phenomenological Bridges](http://lesswrong.com/lw/jd9/building_phenomenological_bridges/)” there was: [Orseau & Ring (2011)](http://www.idsia.ch/~ring/Orseau,Ring%3BSelf-modification%20and%20Mortality%20in%20Artificial%20Agents,%20AGI%202011.pdf); [Orseau & Ring (2012)](http://agi-conference.org/2012/wp-content/uploads/2012/12/paper_76.pdf).

The post [2013 in Review: Friendly AI Research](https://intelligence.org/2014/02/18/2013-in-friendly-ai-research/) appeared first on [Machine Intelligence Research Institute](https://intelligence.org).


MIRI’s February 2014 Newsletter

|  |  |
| --- | --- |
|

|  |
| --- |
|
[Machine Intelligence Research Institute](http://intelligence.org)
 |

 |
|

|  |  |
| --- | --- |
|

|  |
| --- |
|
Dear friends,
See below for news on our new ebook, new research, new job openings, and Google’s new AI ethics board.
**Research Updates**
* New analyses: [Robust Cooperation: A Case Study in Friendly AI Research](http://intelligence.org/2014/02/01/robust-cooperation-a-case-study-in-friendly-ai-research/) and [How Big is the Field of Artificial Intelligence?](http://intelligence.org/2014/01/28/how-big-is-ai/)
* New interviews: [Emil Vassev](http://intelligence.org/2014/01/30/emil-vassev-on-formal-verification/) on formal verification, [Mike Frank](http://intelligence.org/2014/01/31/mike-frank-on-reversible-computing/) on reversible computing, [Ronald de Wolf](http://intelligence.org/2014/02/03/ronald-de-wolf-on-quantum-computing/) on quantum computing, [Gerwin Klein](http://intelligence.org/2014/02/11/gerwin-klein-on-formal-methods/) on formal methods, [André Platzer](http://intelligence.org/2014/02/15/andre-platzer-on-verifying-cyber-physical-systems/) on verifying cyber-physical systems.
* Two MIRI talks from the AGI-11 academic conference have finally [been posted](http://intelligence.org/2014/01/31/two-miri-talks-from-agi-11/) (videos + slides + transcripts).

**News Updates**
* MIRI has a new [Careers](http://intelligence.org/careers/) page, currently with 4 job openings.
* New ebook: [*Smarter Than Us: The Rise of Machine Intelligence*](http://intelligence.org/smarter-than-us/).
* [Want to help MIRI by investing in XRP?](http://intelligence.org/2014/01/18/investing-in-xrp/)
* Two new posts in our 2013 in Review series, on [outreach](http://intelligence.org/2014/01/20/2013-in-review-outreach/) and on [strategic and expository research](http://intelligence.org/2014/02/08/2013-in-review-strategic-and-expository-research/).
* Two new conversations about MIRI strategy, with [Holden Karnofsky](http://intelligence.org/2014/01/27/existential-risk-strategy-conversation-with-holden-karnofsky/) and with [Jacob Steinhardt](http://intelligence.org/2014/02/11/conversation-with-jacob-steinhardt-about-miri-strategy/).
* [MIRI’s experience with Google Adwords](http://intelligence.org/2014/02/06/miris-experience-with-google-adwords/).

**Other Updates**
* If you’re in the UK, go see Martin Rees, Jaan Tallinn, and Huw Price at the Cambridge lecture “[Existential Risk: Surviving the 21st Century](http://lesswrong.com/lw/jog/cambridge_england_lecture_existential_risk/)” on Feb. 26th.
* The big AI news last month was Google’s ~$400M acquisition of DeepMind, and Google’s rumored creation of an AI ethics board. The best coverage I saw was at *Forbes*: [Inside Google’s Mysterious Ethics Board](http://www.forbes.com/sites/privacynotice/2014/02/03/inside-googles-mysterious-ethics-board/). One of DeepMind’s co-founders, Shane Legg, wrote [his PhD thesis](http://www.vetta.org/documents/Machine_Super_Intelligence.pdf) on machine superintelligence, [takes AI risks quite seriously](http://www.huffingtonpost.com/2014/01/29/google-ai_n_4683343.html), and previously wrote about strategies for [funding safe AGI](http://www.vetta.org/2009/08/funding-safe-agi/). (Shane is far from the *only* safety-conscious person at DeepMind; he’s just the one who has *written* the most about AGI safety.)
* Blue Marble Space is running an [essay contest](http://www.bmsis.org/essaycontest/) on preparing for the distant future of civilization. $500 first-place prize.
* Nick Bostrom’s next book is *[Superintelligence: Paths, Dangers, Strategies](http://ukcatalogue.oup.com/product/9780199678112.do)*, now available for pre-order from [Amazon US](http://www.amazon.com/gp/product/0199678111/ref=as_li_ss_tl?tag=miri05-20) and [Amazon UK](http://www.amazon.co.uk/gp/product/0199678111/ref=as_li_ss_tl?tag=miri05-20). I’ve read it three times already, and it is excellent. It’s easily the most cohesive, comprehensive coverage of the subject so far.

As always, please don’t hesitate to let us know if you have any questions or comments.

Best,
Luke Muehlhauser
Executive Director
 |

 |

 |

The post [MIRI’s February 2014 Newsletter](https://intelligence.org/2014/02/17/miris-february-2014-newsletter/) appeared first on [Machine Intelligence Research Institute](https://intelligence.org).


New eBook: ‘Smarter Than Us’

[![SmarterThanUsCover-Angled-200px](https://intelligence.org/wp-content/uploads/2014/02/SmarterThanUsCover-Angled-200px.jpg)](https://intelligence.org/smarter-than-us/)We are pleased to release a new ebook, commissioned by MIRI and written by Oxford University’s [Stuart Armstrong](http://www.fhi.ox.ac.uk/about/staff/), and available in EPUB, MOBI, PDF, and from the Amazon and Apple ebook stores.

What happens when machines become smarter than humans? Forget lumbering Terminators. The power of an artificial intelligence (AI) comes from its intelligence, not physical strength and laser guns. Humans steer the future not because we’re the strongest or the fastest but because we’re the smartest. When machines become smarter than humans, we’ll be handing them the steering wheel. What promises—and perils—will these powerful machines present? Stuart Armstrong’s new book navigates these questions with clarity and wit.

Head over to [intelligence.org/smarter-than-us/](http://intelligence.org/smarter-than-us/) to grab a copy.

The post [New eBook: ‘Smarter Than Us’](https://intelligence.org/2014/02/17/new-ebook-smarter-than-us/) appeared first on [Machine Intelligence Research Institute](https://intelligence.org).