![Scott Aaronson portrait](https://intelligence.org/wp-content/uploads/2013/12/Aaronson_w150.jpg)[Scott Aaronson](http://www.csail.mit.edu/user/1324) is an Associate Professor of Electrical Engineering and Computer Science at MIT. Before that, he did a PhD in computer science at UC Berkeley, as well as postdocs at the Institute for Advanced Study, Princeton, and the University of Waterloo. His research focuses on the capabilities and limits of quantum computers, and more generally on the connections between computational complexity and physics. Aaronson is known for [his blog](http://www.scottaaronson.com/blog/) as well as for founding the [Complexity Zoo](http://complexityzoo.uwaterloo.ca) (an online encyclopedia of complexity classes); he’s also written about quantum computing for Scientific American and the New York Times. His first book, *[Quantum Computing Since Democritus](http://www.amazon.com/Quantum-Computing-since-Democritus-Aaronson-ebook/dp/B00B4V6IZK/ref=nosim?tag=lukeprogcom-20)*, was published this year by Cambridge University Press. He’s received the Alan T. Waterman Award, the PECASE Award, and MIT’s Junior Bose Award for Excellence in Teaching.

**Luke Muehlhauser**: Though you’re best known for your work in theoretical computer science, you’ve also produced some pretty interesting philosophical work, e.g. in *[Quantum Computing Since Democritus](http://www.amazon.com/Quantum-Computing-since-Democritus-Aaronson-ebook/dp/B00B4V6IZK/ref=nosim?tag=lukeprogcom-20)*, “[Why Philosophers Should Care About Computational Complexity](http://www.scottaaronson.com/papers/philos.pdf),” and “[The Ghost in the Quantum Turing Machine](http://www.scottaaronson.com/papers/giqtm3.pdf).” You also taught a fall 2011 MIT class on [Philosophy and Theoretical Computer Science](http://stellar.mit.edu/S/course/6/fa11/6.893/).

Why are you so interested in philosophy? And what is the social value of philosophy, from your perspective?

---

**Scott Aaronson**: I’ve always been reflexively drawn to the biggest, most general questions that it seemed possible to ask. You know, like are we living in a computer simulation? if not, could we upload our consciousnesses into one? are there discrete “pixels” of spacetime? why does it seem impossible to change the past? could there be different laws of physics where 2+2 equaled 5? are there objective facts about morality? what does it mean to be rational? is there an explanation for why I’m alive right now, rather than some other time? What *are* explanations, anyway? In fact, what really perplexes me is when I meet a smart, inquisitive person—let’s say a mathematician or scientist—who claims NOT to be obsessed with these huge issues! I suspect many MIRI readers might feel drawn to such questions the same way I am, in which case there’s no need to belabor the point.

From my perspective, then, the best way to frame the question is not: “why be interested in philosophy?” Rather it’s: “why be interested in anything else?”

But I think the latter question has an excellent answer. A crucial thing humans learned, starting around Galileo’s time, is that even if you’re interested in the biggest questions, usually the only way to make progress on them is to pick off smaller subquestions: ideally, subquestions that you can attack using math, empirical observation, or both. For again and again, you find that the subquestions aren’t nearly as small as they originally looked! Much like with zooming in to the Mandelbrot set, each subquestion has its own twists and tendrils that could occupy you for a lifetime, and each one gives you a new perspective on the big questions. And best of all, you can actually *answer* a few of the subquestions, and be the first person to do so: you can permanently move the needle of human knowledge, even if only by a minuscule amount. As I once put it, progress in math and science — think of natural selection, Godel’s and Turing’s theorems, relativity and quantum mechanics — has repeatedly altered the terms of philosophical discussion, as philosophical discussion *itself* has rarely altered them! (Of course, this is completely leaving aside math and science’s “fringe benefit” of enabling our technological civilization, which is not chickenfeed either.)

On this view, philosophy is simply too big and too important to be confined to philosophy departments! Of course, the word “philosophy” used to mean the entire range of fundamental inquiry, from epistemology and metaphysics to physics and biology (which were then called “natural philosophy”), rather than just close textual analysis, or writing papers with names like “A Kripkean Reading of Wittgenstein’s Reading of Frege’s Reading of Kant.” And it seems clear to me that there’s enormous scope today for “philosophy” in the former sense — and in particular, for people who love working on the subquestions, on pushing the frontiers of neuroscience or computer science or physics or whatever else, but who also like to return every once in a while to the “deep” philosophical mysteries that motivated them as children or teenagers. Admittedly, there have been many great scientists who didn’t care at all about philosophy, or who were explicitly anti-philosophy. But there were also scientists like Einstein, Schrodinger, Godel, Turing, or Bell, who not only read lots of philosophy but (I would say) used it as a sort of springboard into science — in their cases, a wildly successful one. My guess would be that science ultimately benefits from both the “pro-philosophical” and the “anti-philosophical” temperaments, and even from the friction between them.

As for the “social value” of philosophy, I suppose there are a few things to say. First, the world needs good philosophers, if for no other reason than to refute bad philosophers! (This is similar to why the world needs lawyers, politicians, and soldiers.) Second, the Enlightenment seems like a pretty big philosophical success story. Philosophers like Locke and Spinoza directly influenced statesmen like Thomas Jefferson, in ways you don’t have to squint to see. Admittedly, philosophers’ positive influence on humankind’s moral progress is probably less today than in the 1700s (to put it mildly). And also, most of the philosophical questions that have obsessed me personally have been pretty thin in their moral implications. But that brings me to the third point: namely, to whatever extent you see social value in *popularizing basic science* — that is, in explaining the latest advances in cosmology, quantum information, or whatever else to laypeople — to that extent I think you also need to see social value in philosophy. For the popularizer doesn’t have the luxury of assuming the importance of the particular subquestion on which progress has been made. Instead, he or she constantly needs to say what the little tendrils currently being explored do (or just as importantly, don’t) imply about the whole fractal — and when you’re zooming out like that, it’s hard to avoid talking about philosophy.

---

**Luke**: You write that “usually the only way to make progress on [the big questions] is to pick off smaller subquestions: ideally, subquestions that you can attack using math, empirical observation, or both.” This is an idea you wrote about at greater length in [one of your papers](http://arxiv.org/pdf/1306.0159v2.pdf) — specifically, in [this passage](http://lesswrong.com/lw/hok/link_scott_aaronson_on_free_will/9546):

> whenever it’s been possible to make definite progress on ancient philosophical problems, such progress has almost always involved a [kind of] “bait-and-switch.” In other words: one replaces an unanswerable philosophical riddle Q by a “merely” scientific or mathematical question Q′, which captures part of what people have wanted to know when they’ve asked Q. Then, with luck, one solves Q′.
>
>
> Of course, even if Q′ is solved, centuries later philosophers might still be debating the exact relation between Q and Q′! And further exploration might lead to other scientific or mathematical questions — Q′′, Q′′′, and so on — which capture aspects of Q that Q′ left untouched. But from my perspective, this process of “breaking oﬀ” answerable parts of unanswerable riddles, then trying to answer those parts, is the closest thing to philosophical progress that there is.
>
>
> …A good replacement question Q′ should satisfy two properties: (a) Q′ should capture some aspect of the original question Q — so that an answer to Q′ would be hard to ignore in any subsequent discussion of Q, [and] (b) Q′ should be precise enough that one can see what it would mean to make progress on Q′: what experiments one would need to do, what theorems one would need to prove, etc.
>
>

What are some of your favorite examples of illuminating Q-primes that were solved within your own field, theoretical computer science?

---

**Scott**: It’s hard to know where to begin with this question! In fact, my 59-page essay [Why Philosophers Should Care About Computational Complexity](http://www.scottaaronson.com/papers/philos.pdf) was largely devoted to cataloging the various “Q-primes” on which I think theoretical computer science has made progress. However, let me mention four of my favorites, referring readers to the essay for details:

(1) One of the biggest, oldest questions in the philosophy of science could be paraphrased as: “why is Occam’s Razor justified? when we find simple descriptions of past events, why do we have any grounds whatsoever to expect those descriptions to predict future events?” This, I think, is the core of Hume’s “problem of induction.” Now, I think theoretical computer science has contributed large insights to this question — including Leslie Valiant’s Probably Approximately Correct (PAC) learning model, for which he recently won the Turing Award; the notion of Vapnik-Chernonenkis (VC) dimension; and the notion of the universal prior from algorithmic information theory. In essence, these ideas all give you various formal models where Occam’s Razor *provably* works — where you can give “simplicity” a precise definition, and then see *exactly* why simple hypotheses are more likely to predict the future than complicated ones. Of course, a skeptic about induction could still ask: OK, but why are the assumptions behind these formal models justified? But to me, this represents progress! The whole discussion can now start from a more sophisticated place than before.

(2) One of the first questions anyone asks on learning quantum mechanics is, “OK, but do all these branches of the wavefunction really exist? or are they just mathematical constructs used to calculate probabilities?” Roughly speaking, Many-Worlders would say they do exist, while Copenhagenists would say they don’t. Of course, part of what makes the question slippery is that it’s not even completely clear what we mean by words like “exist”! Now, I’d say that quantum computing theory has sharpened the question in many ways, and actually answered some of the sharpened versions — but interestingly, sometimes the answer goes one way and sometimes it goes the other! So for example, we have strong evidence that quantum computers can solve certain specific problems in polynomial time that would require exponential time to solve using a classical computer. Some Many-Worlders, most notably David Deutsch, have seized on the apparent exponential speedups for problems like factoring, as the ultimate proof that the various branches of the wavefunction must literally exist: “if they *don’t* exist,” they ask, “then where was this huge number factored? where did the exponential resources to solve the problem come from?” The trouble is, we’ve also learned that a quantum computer could NOT solve arbitrary search problems exponentially faster than a classical computer could solve them — something you’d probably predict a QC could do, if you thought of all the branches of the wavefunction as just parallel processors. If you want a quantum speedup, then your problem needs a particular structure, which (roughly speaking) lets you choreograph a pattern of constructive and destructive interference involving ALL the branches. You can’t just “fan out” and have one branch try each possible solution — twenty years of popular articles notwithstanding, that’s not how it works! We also know today that you can’t encode more than about n classical bits into n quantum bits (qubits), in such a way that you can reliably retrieve any one of the bits afterward. And we have all lots of other results that make quantum-mechanical amplitudes feel more like “just souped-up versions of classical probabilities,” and quantum superposition feel more like just a souped-up kind of potentiality. I love how the mathematician Boris Tsirelson summarized the situation: he said that “a quantum possibility is more real than a classical possibility, but less real than a classical reality.” It’s an ontological category that our pre-mathematical, pre-quantum intuitions just don’t have a good name for.

(3) Many interesting philosophical puzzles boil down to what it means to know something: and in particular, to the difference between knowing something “explicitly” and knowing it only “implicitly.” For example, I mentioned in my essay the example of the largest “known” prime number. According to the Great Internet [Mersenne Prime Search](http://www.mersenne.org/), that number is currently 2^57885161 – 1. The question is, why can’t I reply immediately that I know a bigger prime number: namely, “the first prime larger than 2^57885161 – 1”? I can even give you an algorithm to find my number, which provably halts: namely, starting from 2^57885161, try each number one by one until you hit a prime! Theoretical computer science has given us the tools to sharpen a huge number of questions of this sort, and sometimes answer them. Namely, we can say that to know a thing “explicitly” means, not merely to have ANY algorithm to generate the thing, but to have a provably polynomial-time algorithm. That gives us a very clear sense in which, for example, 2^57885161 – 1 is a “known” prime number while the next prime after it is not. And, in many cases where mathematicians vaguely asked for an “explicit construction” of something, we can sharpen the question to whether or not some associated problem has a polynomial-time algorithm. Then, sometimes, we can find such an algorithm or give evidence against its existence!

(4) One example that I *didn’t* discuss in the essay — but a wonderful one, and one where there’s actually been huge progress in the last few years — concerns the question of how we can ever know for sure that something is “random.” I.e., even if a string of bits passes every statistical test for randomness that we throw it at, how could we ever rule out that there’s some complicated regularity that we simply failed to find? In the 1960s, the theory of Kolmogorov complexity offered one possible answer to that question, but a rather abstract and inapplicable one: roughly speaking, it said we can consider a string “random enough for our universe” if it has no *computable* regularities, if there’s no program to output the string shorter than the string itself. More recently, a much more practical answer has come from the Bell inequality — and in particular, from the realization that the experimental violation of that inequality can be used to produce so-called “Einstein-certified random numbers.” These are numbers that are *provably* random, assuming only (a) that they were produced by two separated devices that produced such-and-such outputs in response to challenges, and (b) there was no faster-than-light communication between the devices. But it’s only within the last few years that computer scientists figured out how to implement this striking idea, in such a way that you get out more randomness than you put in. (Recently, two MIT grad students proved that, starting from a fixed “seed” of, let’s say, 100 random bits, you can produce *unlimited* additional random bits in this Einstein-certified way — see [Infinite Randomness Expansion and Amplification with a Constant Number of Devices](http://arxiv.org/abs/1310.6755)) And the experimental demonstration of these ideas is just getting started now. Anyway, I’m working on an article for American Scientist magazine about these developments, so rather than cannibalize the article, I’ll simply welcome people to read it when it’s done!

---

**Luke**: What do you think about *philosophy the field* — work published by people in philosophy departments, who publish mostly in philosophy journals like *Mind* and *Noûs*, who are writing mostly for other philosophers?

I’ve previously called philosophy a “[diseased discipline](http://lesswrong.com/lw/4zs/philosophy_a_diseased_discipline/),” for many reasons. For one thing, people working in philosophy-the-field tend to know strikingly little about the philosophical progress made in other fields, e.g. computer science or cognitive neuroscience. For another, books on the history of philosophy seem to be about the musings of old dead guys who were wrong about almost everything because they didn’t have 20th century science or math, rather than about actual philosophical progress, which is instead recounted in books like *[The Information](http://www.amazon.com/Information-History-Theory-Flood-ebook/dp/B004DEPHUC/)*.

Do you wish people in other fields would more directly try to use the tools of their discipline to make philosophical progress on The Big Questions? Do you wish philosophy-the-field would be reformed in certain ways? Would you like to see more crosstalk between disciplines about philosophical issues? Do you think that, as Clark Glymour [suggested](http://choiceandinference.com/2011/12/23/in-light-of-some-recent-discussion-over-at-new-apps-i-bring-you-clark-glymours-manifesto/), philosophy departments should be defunded unless they produce work that is directly useful to other fields (as is the case with Glymour’s department)?

---

**Scott**: Well, let’s start with the positives of academic philosophy!

(1) I liked the philosophy of math and science courses that I took in college. Sure, I sometimes got frustrated by the amount of time spent on what felt like Talmudic exegesis, but on the other hand, those courses offered a scope for debating big, centuries-old questions that my math and science courses hardly ever did.

(2) These days, I go maybe once a year to conferences where I meet professional philosophers of science, and I’ve found my interactions with them stimulating and fun. Philosophers often listen to what you say more carefully than other scientists do, and they’re incredibly good at spotting hidden assumptions, imprecise use of language, that sort of thing. Also, philosophers of science tend to double in practice as science historians: they often know much, much more about what, let’s say, Einstein or Bohr or Godel or Turing wrote and believed than physicists and mathematicians themselves know.

(3) While my own reading of the philosophical classics has been woefully incomplete, I don’t feel like the time I spent with (say) Hume or J. S. Mill or William James or Bertrand Russell was wasted at all. You’re right that these “old dead guys” didn’t know all the math and science we know today, but then again, neither did Shakespeare or Dostoyevsky! I mean, sure, the central questions of philosophy have changed over time, and the human condition has changed as well: we no longer get confused over Zeno’s paradoxes or the divine right of kings, and we now have global telecommunications and the Pill. I just don’t think either human nature or human philosophical concerns have changed *quickly* enough for great literature on them written centuries ago to have ceased being great.

Having said all that, from what I’ve seen of academic philosophy, I do pretty much agree with your diagnoses of its “diseases.” By far the most important disease, I’d say, is the obsession with interpreting and reinterpreting the old masters, rather than moving beyond them. Back in college, after we’d spent an hour debating why *this* passage of Frege seemed to contradict *that* one, I’d sometimes want to blurt out: “so maybe he was having a bad day! I mean, he was also a raving misogynist and antisemite; he believed all kinds of things. Look, we’ve read Frege, we’ve learned from Frege, now can’t we just give the old dude a rest and debate the ground truth about the problems he was trying to solve?” Likewise, when I read books about the philosophy of physics or computing, it sometimes feels like I’m stuck in a time warp, as the contributors rehash certain specific debates from the 1930s over and over (say, about the Church-Turing Thesis or the Einstein-Podolsky-Rosen paradox). I want to shout, “enough already! why not help clarify some modern scientific debates—-say, about quantum computing, or string theory, or the black-hole firewall problem, ones where we don’t already know how everything turns out?” To be fair, today there are philosophers of science who are doing exactly that, and who have interesting and insightful things to say. That’s a kind of philosophy that I’d love to see more of, at the expense of the hermeneutic kind.

Now, regarding Clark Glymour’s suggestion that philosophy departments be defunded unless they produce work useful to other fields — from what I understand, something not far from that is already happening! As bad as our funding woes in the sciences might be, I think the philosophers have it a hundred times worse, with like a quadrillion applicants for every tenure-track opening. So it seems to me like the right question is not how much further those poor dudes should be defunded, but rather: what can philosophy departments do to make themselves more vibrant, places that scientists regularly turn to for clarifying insights, and that deans and granting agencies get excited about wanting to expand? As a non-philosopher, I hesitate to offer unsolicited “advice” about such matters, but I guess I already did in the previous paragraph.

One final note: none of the positive or hopeful things that I said about philosophy apply to the postmodern or Continental kinds. As far as I can tell, the latter aren’t really “philosophy” at all, but more like pretentious brands of performance art that fancy themselves politically subversive, even as they cultivate deliberate obscurity and draw mostly on the insights of Hitler and Stalin apologists. I suspect I won’t ruffle too many feathers here at MIRI by saying this.

---

**Luke**: Suppose a mathematically and analytically skilled student wanted to make progress, in roughly the way you describe, on the Big Questions of philosophy. What would you recommend they study? What should they read to be inspired? What skills should they develop? Where should they go to study?

---

**Scott**: The obvious thing to say is that, as a student, you should follow your talents and passions, rather than following the generic advice of some guy on the Internet who doesn’t even know you personally!

Having said that, I would think broadly about which fields can give you enough scope to address the “Big Questions of Philosophy.”  You can philosophize from math, computer science, physics, economics, cognitive science, neuroscience, and probably a bunch of other fields too.  (My colleague Seth Lloyd philosophizes left and right, from his perch in MIT’s Mechanical Engineering department.)  Furthermore, all of these fields have the crucial advantage that they’ll offer you a steady supply of “fresh meat”: that is, new and exciting empirical or theoretical discoveries in which you can participate, and that will give you something to philosophize ABOUT (not to mention, something to do when you’re not philosophizing).  If I were working in a philosophy department, I feel like I’d have to make a conscious and deliberate effort to avoid falling into a “hermeneutic trap,” where I’d spend all my time commenting on what other philosophers had said about the works of yet other philosophers, and where I’d seal myself off from anything that had happened in the world of science since (say) Godel’s Theorem or special relativity.  (Once again, though, if you find that your particular talents and passions are best served in an academic philosophy department, then don’t let some guy on the Internet stop you!)

Regardless of your major, I recommend taking a huge range of courses as an undergrad: math, computer science (both applied and theoretical), physics, humanities, history, writing, and yes, philosophy. Looking back on my own undergrad years, the most useful courses I took were probably my math courses, and that’s *despite* the fact that most of them were poorly taught!  Things like linear algebra, group theory, and probability have so many uses throughout science that learning them is like installing a firmware upgrade to your brain — and even the math you *don’t* use will stretch you in helpful ways.  After math courses, the second most useful courses I took were writing seminars — the kind where a small group of students reads and critiques one another’s writing, and the professor functions mostly as a moderator.  It was in such a seminar that I wrote my essay “[Who Can Name the Bigger Number?](http://www.scottaaronson.com/writings/bignumbers.html)“, which for better or worse, continues to attract more readers than anything else I’ve written in the fifteen years since.  One writing seminar, if it’s good, can easily be worth the whole cost of a college tuition.

If you’re the kind of person for whom this advice is intended, then you probably don’t have to be told to read widely and voraciously, anything you get curious about.  Don’t limit yourself to one genre, don’t limit yourself to stuff you agree with, and *certainly* don’t limit yourself to the assigned reading for your courses.  When I was an adolescent, my favorites were just what a nerd stereotyper might expect: science fiction (especially Isaac Asimov), books about programming and the software industry, and math puzzle books (especially Martin Gardner).  A few years later, I became obsessed with reading biographies of scientists, like Feynman, Ramanujan, Einstein, Schrodinger, Turing, Godel, von Neumann, and countless lesser luminaries.  I was interested in every aspect of their lives — in their working habits, their hobbies, their views on social and philosophical issues, their love lives — but, I confess, I was particularly interested in what they were doing as teenagers, so that I could compare to what I was doing and sort of see how I measured up.  At the same time, my reading interests were broadening to include politics, history, philosophy, psychology, and some contemporary fiction (I especially like Rebecca Goldstein).  It was only in grad school that I felt I’d sufficiently recovered from high-school English to tackle “real literature” like Shakespeare — but when I did, it was worth it.

As for where to study, well, the “tautological” answer is wherever will give you the best opportunities!  There are certain places, like Boston or the Bay Area, that are famous for having high concentrations of intellectual opportunity, but don’t go somewhere just because of what you’ve heard about the *general* atmosphere or prestige: particularly for graduate school, go where the particular people or programs are that resonate for you.  In quantum computing, for example, one of the centers of the world for the last decade has been Waterloo, Canada — a place many people hadn’t even heard of when I did my postdoc there eight years ago (though that’s changing now).  And one of the intellectually richest years of my life came when I attended The Clarkson School, a program that lets high-school students live and take courses at Clarkson University in Potsdam, NY.  (I went there when I was 15, and was looking for something less prison-like than high school.)  If, for what you personally want to do, there are better opportunities in Topeka, Kansas than at Harvard, go to Topeka.

---

**Luke**: Finally, I’d like to ask about which object-level research tactics — more specific than your general “bait and switch” strategy — you suspect are likely to help with philosophical research, or perhaps with theoretical research of any kind.

For example, some of the tactics we’ve found helpful at [MIRI](http://intelligence.org/) include:

* When you’re confused about a fuzzy, slippery concept, try to build a simple formal model and push on it with the new tools then available to you. Even if the model doesn’t capture the complexity of the world, pushing things into the mathematical realm can lead to progress. E.g. the [VNM axioms](http://en.wikipedia.org/wiki/Von_Neumann%E2%80%93Morgenstern_utility_theorem#The_axioms) don’t exactly capture “rationality,” but it sure is easier to think clearly about rationality once you have them. Or: we’re confused about how to do principled reflective reasoning within an agent, so even though advanced AIs are unlikely to literally run into a “[Löbian obstacle](https://intelligence.org/files/TilingAgents.pdf)” to self-reflection, setting up the problem that way (in mathematical logic) can lead to some interesting insights in (e.g.) [probabilistic metamathematics](http://intelligence.org/wp-content/uploads/2013/03/Christiano-et-al-Naturalistic-reflection-early-draft.pdf) for reflective reasoning.
* Look for tools from other fields that appear to directly map onto the phenomena you’re studying. E.g. [model moral judgment as an error process amenable to Bayesian curve fitting](http://commonsenseatheism.com/wp-content/uploads/2013/12/Beckstead-On-the-overwhelming-importance-of-shaping-the-far-future.pdf).
* Try to think of how your concept could be instantiated with infinite computing power. If you can’t do that, your concept might be fundamentally confused.
* If you’re pretty familiar with [modern psychology](http://www.amazon.com/Handbook-Thinking-Reasoning-Library-Psychology/dp/0199313792/), then… When using your intuitions to judge between options, try to think about which cognitive algorithms could be generating those intuitions, and [whether they are](http://lesswrong.com/lw/74f/are_deontological_moral_judgments_rationalizations/) [cognitive algorithms](http://lesswrong.com/lw/no/how_an_algorithm_feels_from_inside/) [whose outputs](http://lesswrong.com/lw/hw/scope_insensitivity/) [you reflectively endorse](https://intelligence.org/files/CognitiveBiases.pdf).
* To make the thing you’re studying clearer, look just next to it, and around it. [Foer (2009)](http://www.amazon.com/Eating-Animals-Jonathan-Safran-Foer-ebook/dp/B002SSBD6W/) explains this nicely in the context of thinking about one’s values and vegetarianism: “A simple trick from the backyard astronomer: if you are having trouble seeing something, look slightly away from it. The most light-sensitive parts of our eyes (those we need to see dim objects) are on the edges of the region we normally use for focusing. Eating animals has an invisible quality. Thinking about dogs, and their relationship to the animals we eat, is one way of looking askance and making something invisible visible.”

Which object-level thinking tactics, at roughly this level of specificity, do you use in your own theoretical (especially *philosophical*) research? Are there tactics you suspect might be helpful, which you haven’t yet used much yourself?

---

**Scott**: As far as I can remember, I’ve never set out to do “philosophical research,” so I can’t offer specific advice about that. What I *have* often done is research in quantum computing and complexity theory that was motivated by some philosophical issue, usually in foundations of quantum mechanics. (I’ve also written a few philosophical essays, but I don’t really count those as “research.”) Anyway, I can certainly offer advice about doing the kind of research I like to do!

(1) Any time you find yourself in a philosophical disagreement with a fellow scientist, don’t be content just to argue philosophically — even if you’re sure you can win the argument! Instead, think hard about whether you can go further, and find a concrete technical question that captures some little piece of what you’re disagreeing about. Then see if you can answer that technical question. Of course, any time you do this, you have to be prepared for the possibility that the answer will go your opponent’s way, rather than yours! But what’s nice is that you get to publish a paper even then. (One of the best ways to tell whether a given enterprise is scientific at all, rather than ideological, is by asking whether the participants will opportunistically “go to bat for the opposing side” whenever they find a novel truth on that side.) I’d estimate that up to half the papers I’ve written had their origin in my reading or overhearing some claim — for example, “Grover’s algorithm obviously can’t work for searching actual physical databases, since the speed of light is finite,” or “the quantum states arising in Shor’s algorithm are obviously completely different from anything anyone has ever seen in the lab,” or “the interactive proof results obviously make oracle separations completely irrelevant” — and getting annoyed, either because I thought the claim was false, or because I simply didn’t think it had been adequately justified. The cases where my annoyance paid off are precisely the ones where, rather than just getting mad, I managed to get technical!

(2) Often, the key to research is figuring out how to redefine failure as success. Some stories: when Alan Turing published his epochal 1936 paper on Turing machines, he did so with great disappointment: he had recently learned that Alonzo Church had independently arrived at similar results using lambda calculus, and he didn’t know whether anyone would still be interested in his alternative, machine-based approach. In the early 1970s, Leonid Levin delayed publishing about NP-completeness for several years: apparently, his “real” goal was to prove graph isomorphism was NP-complete (something we now know is almost certainly false), and in his mind, he had failed. Instead, he merely had a few “trivialities,” like the definitions of P, NP, and NP-completeness, and the proof that satisfiability was NP-complete. And Levin’s experience is far from unique: again and again in mathematical research, you’ll find yourself saying something like: “goddammit, I’ve been trying for six months to prove Y, but I can only prove the different/weaker statement X! And every time I think I can bridge the gap between X and Y, yet another difficulty rears its head!” Any time that happens to you, think hard about whether you can write a compelling paper that begins: “Y has been a longstanding open problem. In this work, we introduce a new idea: to make progress on Y by shifting attention to the more tractable X.” More broadly, experience has shown that scientists are *terrible* judges of which of their ideas will be interesting or important to others. Pick any scientist’s most cited paper, and there’s an excellent chance that the scientist herself, at one point, considered it a “little recreational throwaway project” that was barely worth writing up. After you’ve seen enough examples of that, you learn you should always err on the side of publishing, and let posterity sort out which of your ideas are most important. (Yet another advantage of this approach is that, the more ideas you publish, the less emotionally invested you are in any one of them, so the less crushed you are when a few turn out to be wrong or trivial or already known.)

(3) Sometimes, when you set out to prove some mathematical conjecture, your first instinct is just to throw an arsenal of theory at it. “Hey, what if I try a topological fixed-point theorem? What if I translate the problem into a group-theoretic language? If neither of those works, what if I try both at once?” Sometimes, you rise so quickly this way into a stratosphere of generality that the original problem is barely a speck on the ground. And yes, some problems *can* be beaten into submission using high-powered theory. But in my experience, there are two enormous risks with this approach. First, you’re liable to get lost on a wild goose chase, where you get so immersed in theory and techniques that you lose sight of your original goal. It’s as if your efforts to break into a computer network lead you to certain complicated questions about the filesystem, which in turn lead you to yet more complicated questions about the kernel… and in the meantime someone else breaks in by guessing people’s birthdays for their passwords. Second, you’re also liable to fool yourself this way into thinking you’ve solved the problem when you haven’t. When you let high-powered machinery take the place of hands-on engagement with the problem, a single mistake in applying the machinery can creep in unbelievably easily. These risks are why I’ve learned over time to work in an extremely different way. Rather than looking for “general frameworks,” I look for easy special cases and simple sanity checks, for stuff I can try out using high-school algebra or maybe a five-line computer program, just to get a feel for the problem. Even more important, when I’m getting started, I don’t think about proof techniques at all: I think instead about obstructions. That is, I ask myself, “what would the world have to be like for the conjecture to be *false*? what goes wrong if I try to invent a simple counterexample? *does* anything go wrong? it does? OK then, what obstruction keeps me from proving this conjecture in the simplest, dumbest way imaginable?” I find that, after you’ve felt out the full space of obstructions and counterexamples, and really honestly convinced *yourself* of why the conjecture should be true, finding the proof techniques by which to convince everyone else is often a more-or-less routine exercise.

Finally, you ask about tactics that I suspect might be helpful, but that I haven’t used much myself. One that springs to mind is to really master a tool like Mathematica, MATLAB, Maple, or Magma — that is, to learn it so well that I can code as fast as I think, and just let it take over all the routine / calculational / example-checking parts of my work. As it is, I use pretty much the same antiquated tools that I learned as an adolescent, and I rely on students whenever there’s a need for better tools. A large part of the problem is that, as a “tenured old geezer,” I no longer have the time or patience to learn new tools just for the sake of learning them: I’m always itching just to solve the problem at hand with whatever tools I know. (The same issue has kept me from learning new mathematical tools, like representation theory, even when I can clearly see that they’d benefit me.)

---

**Luke**: Thanks, Scott!

The post [Scott Aaronson on Philosophical Progress](https://intelligence.org/2013/12/13/aaronson/) appeared first on [Machine Intelligence Research Institute](https://intelligence.org).


2013 Winter Matching Challenge

![](https://intelligence.org/wp-content/uploads/2013/12/workshop-horizontal-1.jpg)
Thanks to [Peter Thiel](http://en.wikipedia.org/wiki/Peter_Thiel), every donation made to MIRI between now and January 15th, 2014 will be **matched dollar-for-dollar**!

Also, **gifts from “new large donors” will be matched 3x!** That is, if you’ve given less than $5k to SIAI/MIRI ever, and you now give or pledge $5k or more, Thiel will donate $3 for every dollar you give or pledge.

We don’t know whether we’ll be able to offer the 3:1 matching ever again, so if you’re capable of giving $5k or more, we encourage you to take advantage of the opportunity while you can. Remember that:

* If you prefer to give monthly, no problem! If you pledge 6 months of monthly donations, your full 6-month pledge will be the donation amount to be matched. So if you give monthly, you can get 3:1 matching for only $834/mo (or $417/mo if you get matching from your employer).
* We accept [Bitcoin](http://en.wikipedia.org/wiki/Bitcoin) (BTC) and [Ripple](http://en.wikipedia.org/wiki/Ripple_(monetary_system)) (XRP), both of which have recently jumped in value. If the market value of your Bitcoin or Ripple is $5k or more on the day you make the donation, this will count for matching.
* If your employer matches your donations at 1:1 (check [here](http://doublethedonation.com/miri)), then you can take advantage of Thiel’s 3:1 matching by giving as little as $2,500 (because it’s $5k after corporate matching).

*Please email [malo@intelligence.org](mailto: malo@intelligence.org) if you intend on leveraging corporate matching or would like to pledge 6 months of monthly donations, so that we can properly account for your contributions towards the fundraiser.*

Thiel’s total match is capped at $250,000. The total amount raised will depend on how many people take advantage of 3:1 matching. We don’t anticipate being able to hit the $250k cap without substantial use of 3:1 matching — so if you haven’t given $5k thus far, please consider giving/pledging $5k or more during this drive. (If you’d like to know the total amount of your past donations to MIRI, just ask [malo@intelligence.org](mailto: malo@intelligence.org).)

$0

$62.5K

$125K

$187.5k

$250k

### We have reached our matching total of $250,000!

Now is your chance to **double or quadruple your impact** in funding our [research program](http://intelligence.org/research/).

 [Donate Today](https://intelligence.org/donate/#donation-methods)
------------------------------------------------------------------

![](https://intelligence.org/wp-content/uploads/2013/12/workshop-horizontal-3.jpg)
### Accomplishments Since Our July 2013 Fundraiser Launched:

* Held three **[research workshops](http://intelligence.org/workshops/#past-workshops)**, including our [first European workshop](http://intelligence.org/2013/08/30/miris-november-2013-workshop-in-oxford/).
* [**Talks** at MIT and Harvard](http://intelligence.org/2013/10/01/upcoming-talks-at-harvard-and-mit/), by Eliezer Yudkowsky and Paul Christiano.
* Yudkowsky is blogging more **Open Problems in Friendly AI**… [on Facebook](https://www.facebook.com/groups/233397376818827/)! (They’re also being written up in a more conventional format.)
* New **papers**: (1) [Algorithmic Progress in Six Domains](http://lesswrong.com/r/discussion/lw/i8i/algorithmic_progress_in_six_domains/); (2) [Embryo Selection for Cognitive Enhancement](http://intelligence.org/2013/10/30/new-paper-embryo-selection-for-cognitive-enhancement/); (3) [Racing to the Precipice](http://intelligence.org/2013/11/27/new-paper-racing-to-the-precipice/); (4) [Predicting AGI: What can we say when we know so little?](https://intelligence.org/files/PredictingAGI.pdf)
* New **ebook**: [*The Hanson-Yudkowsky AI-Foom Debate*](http://intelligence.org/ai-foom-debate/).
* New **analyses**: (1) [From Philosophy to Math to Engineering](http://intelligence.org/2013/11/04/from-philosophy-to-math-to-engineering/); (2) [How well will policy-makers handle AGI?](http://intelligence.org/2013/09/12/how-well-will-policy-makers-handle-agi-initial-findings/) (3) [How effectively can we plan for future decades?](http://intelligence.org/2013/09/04/how-effectively-can-we-plan-for-future-decades/) (4) [Transparency in Safety-Critical Systems](http://intelligence.org/2013/08/25/transparency-in-safety-critical-systems/); (5) [Mathematical Proofs Improve But Don’t Guarantee Security, Safety, and Friendliness](http://intelligence.org/2013/10/03/proofs/); (6) [What is AGI?](http://intelligence.org/2013/08/11/what-is-agi/) (7) [AI Risk and the Security Mindset](http://intelligence.org/2013/07/31/ai-risk-and-the-security-mindset/); (8) [Richard Posner on AI Dangers](http://intelligence.org/2013/10/18/richard-posner-on-ai-dangers/); (9) [Russell and Norvig on Friendly AI](http://intelligence.org/2013/10/19/russell-and-norvig-on-friendly-ai/).
* New **expert interviews**: [Greg Morrisett](http://intelligence.org/2013/11/05/greg-morrisett-on-secure-and-reliable-systems-2/) (Harvard), [Robin Hanson](http://intelligence.org/2013/11/01/robin-hanson/) (GMU), [Paul Rosenbloom](http://intelligence.org/2013/09/25/paul-rosenbloom-interview/) (USC), [Stephen Hsu](http://intelligence.org/2013/08/31/stephen-hsu-on-cognitive-genomics/) (MSU), [Markus Schmidt](http://intelligence.org/2013/10/28/markus-schmidt-on-risks-from-novel-biotechnologies/) (Biofaction), [Laurent Orseau](http://intelligence.org/2013/09/06/laurent-orseau-on-agi/) (AgroParisTech), [Holden Karnofsky](http://intelligence.org/2013/08/25/holden-karnofsky-interview/) (GiveWell), [Bas Steunebrink](http://intelligence.org/2013/10/25/bas-steunebrink-on-sleight/) (IDSIA), [Hadi Esmaeilzadeh](http://intelligence.org/2013/10/21/hadi-esmaeilzadeh-on-dark-silicon/) (GIT), [Nick Beckstead](http://intelligence.org/2013/07/17/beckstead-interview/) (Oxford), [Benja Fallenstein](http://intelligence.org/2013/08/04/benja-interview/) (Bristol), [Roman Yampolskiy](http://intelligence.org/2013/07/15/roman-interview/) (U Louisville), [Ben Goertzel](http://intelligence.org/2013/10/18/ben-goertzel/) (Novamente), and [James Miller](http://intelligence.org/2013/07/12/james-miller-interview/) (Smith College).
* With [Leverage Research](http://www.leverageresearch.org/), we held a San Francisco **book launch party** for James Barratt’s [*Our Final Invention*](http://www.amazon.com/Our-Final-Invention-Artificial-Intelligence-ebook/dp/B00CQYAWRY/), which discusses MIRI’s work at length. (If you live in the Bay Area and would like to be notified of local events, please tell malo@intelligence.org!)

 [Donate Today](https://intelligence.org/donate/#donation-methods)
------------------------------------------------------------------

### How Will Marginal Funds Be Used?

* **Hiring Friendly AI researchers**, identified through our workshops, as they become available for full-time work at MIRI.
* Running **more workshops** (next one begins [Dec. 14th](http://intelligence.org/2013/07/24/miris-december-2013-workshop/)), to make concrete Friendly AI research progress, to introduce new researchers to open problems in Friendly AI, and to identify candidates for MIRI to hire.
* Describing more **open problems in Friendly AI**. Our current strategy is for Yudkowsky to explain them as quickly as possible via Facebook discussion, followed by more structured explanations written by others in collaboration with Yudkowsky.
* Improving humanity’s **strategic understanding** of what to do about superintelligence. In the coming months this will include (1) additional interviews and analyses on our blog, (2) a reader’s guide for Nick Bostrom’s forthcoming [*Superintelligence*](http://ukcatalogue.oup.com/product/9780199678112.do) [book](http://ukcatalogue.oup.com/product/9780199678112.do), and (3) an introductory ebook currently titled *Smarter Than Us.*

Other projects are still being surveyed for likely cost and impact.

We appreciate your support for our work! [Donate now](https://intelligence.org/donate/#donation-methods), and seize a better than usual chance to move our work forward. If you have questions about donating, please contact Louie Helm at (510) 717-1477 or louie@intelligence.org.

The post [2013 Winter Matching Challenge](https://intelligence.org/2013/12/02/2013-winter-matching-challenge/) appeared first on [Machine Intelligence Research Institute](https://intelligence.org).


[![Predicting AGI](https://intelligence.org/wp-content/uploads/2013/12/predicting-agi-thumb.gif)](https://intelligence.org/files/PredictingAGI.pdf)MIRI research associate Benja Fallenstein and UC Berkeley student Alex Mennen have released a new working paper titled “[Predicting AGI: What can we say when we know so little?](https://intelligence.org/files/PredictingAGI.pdf)”

From the introduction:

> This analysis does not attempt to predict when AGI will actually be achieved, but instead, to predict when this epistemic state with respect to AGI will change, such that we will have a clear idea of how much further progress is needed before we reach AGI. Metaphorically speaking, instead of predicting when AI takes off, we predict when it will start taxiing to the runway.
>
>

The paper argues for a Pareto distribution for “time to taxi,” and concludes:

> in general, a Pareto distribution suggests that we should put a much greater emphasis on short-term strategies than a less skewed distribution (e.g. a normal distribution) with the same median would.
>
>

The post [New Paper: “Predicting AGI: What can we say when we know so little?”](https://intelligence.org/2013/12/01/new-paper-predicting-agi-what-can-we-say-when-we-know-so-little/) appeared first on [Machine Intelligence Research Institute](https://intelligence.org).


New Paper: “Racing to the Precipice”

[![Racing to the edge](https://intelligence.org/wp-content/uploads/2013/12/racing-to-the-precipice-thumb.gif)](http://www.fhi.ox.ac.uk/wp-content/uploads/Racing-to-the-precipice-a-model-of-artificial-intelligence-development.pdf)During his time as a MIRI research fellow, Carl Shulman contributed to a paper now available as an FHI technical report: [Racing to the Precipice: a Model of Artificial Intelligence Development](http://www.fhi.ox.ac.uk/wp-content/uploads/Racing-to-the-precipice-a-model-of-artificial-intelligence-development.pdf).

Abstract:

> This paper presents a simple model of an AI arms race, where several development teams race to build the first AI. Under the assumption that the first AI will be very powerful and transformative, each team is incentivized to finish first — by skimping on safety precautions if need be. This paper presents the Nash equilibrium of this process, where each team takes the correct amount of safety precautions in the arms race. Having extra development teams and extra enmity between teams can increase the danger of an AI-disaster, especially if risk taking is more important than skill in developing the AI. Surprisingly, information also increases the risks: the more teams know about each others’ capabilities (and about their own), the more the danger increases.
>
>

**Update:** As of July 2015, this paper has been published in the journal *[AI & Society](http://link.springer.com/article/10.1007%2Fs00146-015-0590-y)*.

The post [New Paper: “Racing to the Precipice”](https://intelligence.org/2013/11/27/new-paper-racing-to-the-precipice/) appeared first on [Machine Intelligence Research Institute](https://intelligence.org).


MIRI’s November 2013 Newsletter

|  |  |
| --- | --- |
|

|  |
| --- |
|
[Machine Intelligence Research Institute](http://intelligence.org)
 |

 |
|

|  |  |
| --- | --- |
|

|  |
| --- |
|
Dear friends,

We’re experimenting with a new, ultra-brief newsletter style. To let us know what you think of it, simply reply to this email. Thanks!

**News Updates**
* You can now [support MIRI for free](http://intelligence.org/2013/11/06/amazonsmile/) by shopping at smile.amazon.com instead of amazon.com. Update your bookmarks!
* Louie Helm will be onsite for the Nov. 24th Marin county screening of Doug Wolens’ new documentary [*The Singularity*](http://thesingularityfilm.com/). Details on this and other screenings are [here](http://thesingularityfilm.com/screenings/).

**Research Updates**
* New paper: “[Embryo Selection for Cognitive Enhancement](http://intelligence.org/2013/10/30/new-paper-embryo-selection-for-cognitive-enhancement/).” (Including a description of iterated embryo selection.)
* New analysis: “[From Philosophy to Math to Engineering](http://intelligence.org/2013/11/04/from-philosophy-to-math-to-engineering/).” (How we see research progress.)
* Eliezer Yudkowsky [gave a talk](http://intelligence.org/2013/10/01/upcoming-talks-at-harvard-and-mit/) at MIT, but unfortunately MIT failed to record the video as scheduled. A later delivery of the talk *may* be recorded and released in the next few months.
* Paul Christiano gave a talk at Harvard about probabilistic metamathematics. Video [here](http://intelligence.org/2013/10/23/probabilistic-metamathematics-and-the-definability-of-truth/).
* Two surveys of what others have written about risks from AGI: [Richard Posner](http://intelligence.org/2013/10/18/richard-posner-on-ai-dangers/) (the most-cited legal scholar of the 20th century) and [Stuart Russell & Peter Norvig](http://intelligence.org/2013/10/19/russell-and-norvig-on-friendly-ai/) (authors of the #1 AI textbook).
* Several new expert interviews: [Ben Goertzel](http://intelligence.org/2013/10/18/ben-goertzel/) (Novamente) on AGI as a field, [Hadi Esmaeilzadeh](http://intelligence.org/2013/10/21/hadi-esmaeilzadeh-on-dark-silicon/) (GIT) on dark silicon’s threat to Moore’s Law, [Bas Steunebrink](http://intelligence.org/2013/10/25/bas-steunebrink-on-sleight/) (IDSIA) on self-reflective programming, [Markus Schmidt](http://intelligence.org/2013/10/28/markus-schmidt-on-risks-from-novel-biotechnologies/) (Biofaction) on risks from novel biotechnologies, [Robin Hanson](http://intelligence.org/2013/11/01/robin-hanson/) (GMU) on “serious futurism,” and [Greg Morrisett](http://intelligence.org/2013/11/05/greg-morrisett-on-secure-and-reliable-systems-2/) (Harvard) on secure and reliable systems.

**Other Updates**
* Many of our friends have said Louie Helm’s [Rockstar Research](http://rockstarresearch.com/) is among their favorite new sources of news; check it out!
* Know an exceptionally bright, ambitious person younger than 20? Tell them to [apply for The Thiel Fellowship](http://www.thielfellowship.org/apply-2014/)! $100,000 to skip college and develop one’s skills and ideas, with an incredible network of mentors in the Bay Area.
* CFAR has [upcoming rationality workshops](http://rationality.org/apply/) in February (Melbourne), March (Bay Area), and April (NYC). Tell your friends!

Cheers,

Luke Muehlhauser
Executive Director
 |

 |

 |
|

|  |
| --- |
|
 |

 |

The post [MIRI’s November 2013 Newsletter](https://intelligence.org/2013/11/18/miri-update-november-2013/) appeared first on [Machine Intelligence Research Institute](https://intelligence.org).


Support MIRI by Shopping at AmazonSmile

[![Support MIRI through AmazonSmile.](https://intelligence.org/wp-content/uploads/2013/11/amazon-smile.png)](http://smile.amazon.com/ch/58-2565917)If you shop at the new [AmazonSmile](http://smile.amazon.com/ch/58-2565917), Amazon donates 0.5% of the price of your eligible purchases to a charitable organization of your choosing.

MIRI is an eligible charitable organization, so the next time you consider purchasing something through Amazon, support MIRI by shopping at [AmazonSmile](http://smile.amazon.com/ch/58-2565917)!

If you get to Amazon.com via an affiliate link, remember to change “amazon.com” to “smile.amazon.com” in the address bar before making your purchase. Or, even easier, use the [SmileAlways](http://intelligence.org/2013/11/06/amazonsmile/) Chrome extension.

The post [Support MIRI by Shopping at AmazonSmile](https://intelligence.org/2013/11/06/amazonsmile/) appeared first on [Machine Intelligence Research Institute](https://intelligence.org).


![Greg Morrisett portrait](https://intelligence.org/wp-content/uploads/2013/11/Morrisett_w150.jpg) [Greg Morrisett](http://www.eecs.harvard.edu/~greg/) is the Allen B. Cutting Professor of Computer Science at Harvard University. He received his B.S. in Mathematics and Computer Science from the University of Richmond in 1989, and his Ph.D. from Carnegie Mellon in 1995. In 1996, he took a position at Cornell University, and in the 2003-04 academic year, he took a sabbatical and visited the Microsoft European Research Laboratory. In 2004, he moved to Harvard, where he has served as Associate Dean for Computer Science and Engineering, and where he currently heads the Harvard Center for Research on Computation and Society.

Morrisett has received a number of awards for his research on programming languages, type systems, and software security, including a Presidential Early Career Award for Scientists and Engineers, an IBM Faculty Fellowship, an NSF Career Award, and an Alfred P. Sloan Fellowship.

He served as Chief Editor for the Journal of Functional Programming and as an associate editor for ACM Transactions on Programming Languages and Systems and Information Processing Letters. He currently serves on the editorial board for The Journal of the ACM and as co-editor-in-chief for the Research Highlights column of Communications of the ACM. In addition, Morrisett has served on the DARPA Information Science and Technology Study (ISAT) Group, the NSF Computer and Information Science and Engineering (CISE) Advisory Council, Microsoft Research’s Technical Advisory Board, and Microsoft’s Trusthworthy Computing Academic Advisory Board.

**Luke Muehlhauser**: One of the [interesting projects](http://www.eecs.harvard.edu/%7Egreg/) in which you’re involved is [SAFE](http://www.crash-safe.org/), a DARPA-funded project “focused on a clean slate design for resilient and secure systems.” What is the motivation for this project, and in particular for its “clean slate” approach?

---

**Greg Morrisett**: I think that from DARPA’s perspective, the primary force is that what we’ve been doing so far in terms of securing systems isn’t working.  I suspect that some high-profile problems with successful attacks from both nation states and other organizations has made them desperate to try new and different things.

At the same time, there was a growing recognition that legacy software (and potentially hardware) was a big part of the problem. If, for instance, all of our critical software were coded in type-safe languages, then a lot of low-level problems (e.g., buffer overruns) would be gone.  So I think this was the thinking behind the whole CRASH program and there are something like 20 different teams trying a range of ideas.

Our particular project (SAFE) took a very radical approach in wanting to re-think everything:  the languages, the operating system, and even the hardware, as well as an emphasis on formal methods for trying to gain some assurance.  The team has designed some very interesting hardware that (a) effectively enforces type-safety at the machine level (i.e., object-level integrity, pointers as capabilities, etc.), and (b) provides novel support for doing fine-grained end-to-end information flow tracking.  We’ve also designed a high-level, dynamically-typed language (Breeze) that has pervasive information flow tracking built in.  This means that we can tell which principal actors may have influenced the construction of any value in the system. We’re now building compiler support for Breeze and systems-level software (e.g., scheduler, garbage collector, etc.) to bridge the high-level code and the hardware.

At the same time, we’ve been trying to apply formal models and reasoning to try to prove key properties about models of the system.  For example, we’ve had machine-checked proofs of security for the (core of the) language from the very beginning, and are currently building core models of the hardware and trying to establish similar properties.

---

**Luke**: Could you explain, preferably with examples, what you mean by “type-safety at the machine level” and “fine-grained end-to-end information flow tracking”?

---

**Greg**: So let’s first start with “type-safety”. In this context, I simply mean that we make a distinction between different values in memory and the machine will only allow appropriate operations to be applied to appropriate values. For instance, we make a distinction between pointers to objects (of a given size) and integers, and the machine will only let you load/store relative to a pointer (and within the bounds of the object to which it points). It will trap if you try to load/store relative to an integer. Similarly, we make a distinction between references to code and data, so you’re only allowed to jump-to or call references to code, and not data. These basic integrity properties block a whole range of basic attacks that rely upon violating abstractions (e.g., code injection, buffer overruns, etc.)

With respect to “information flow control” (IFC): we want to enforce policies like “unauthorized principal actors, say a web site with whom I’m communicating, should never learn my private information, say my users’ social security number of password”. The hard part is the “learn no information” as opposed to literally passing the data. If, for instance, I reverse your password and send it out, or take every other letter from your password, we want those to be seen as violations as well as explicitly copying the data. (There are much more subtle ways to communicate information as well.)

Formally, the policy we aim for is that the observations of the “unauthorized principal actors” should be the same if I change the password to an arbitrary string. The technical term for this is “non-interference”.

The way we implement IFC is by labeling every piece of data with an information flow control policy. When you combine two pieces of data (say add two numbers), we have rules for computing a new policy to place on the newly generated value. For example, if you add a “secret” and a “public” number, then the result is conservatively classified as “secret”.

---

**Luke**: In [a blog post](http://intelligence.org/2013/10/03/proofs/), I warned readers that no system is “provably secure” or “provably safe” in the sense of a 100% guarantee of security or safety, because e.g. a security/safety definition might be subtly wrong, or a particular security reduction might have an error in it. These things have happened before. I said that “Rather, these approaches are meant to provide *more confidence than we could otherwise have, all else equal*, that a given system is secure [or safe].”

Do you agree with that characterization? And, on the terminological issue: do you think terms like “provably secure” and “provably safe” are too misleading, or are they just another inevitable example of specialist terminology that is often but unavoidably interpreted incorrectly by non-specialists?

---

**Greg**: Yes, I completely agree with your blog post! All “proofs of correctness” for a system will be subject to modeling the environment, and specifying what we mean by correct. For real systems, modeling will never be sufficient, and formalizing the notion of correctness may be as complicated as reimplementing the system.

I do think that building explicit models of the environment, and writing formal specifications and constructing formal proofs has a lot of value though. As you noted, it really does help to increase confidence. And, it helps identify assumptions. Many times, an attacker will look at your assumptions and try to violate them to gain purchase. So identifying assumptions through modeling is a great exercise.

Going through the process of constructing a formal proof also leads to a lot of good architectural insight. For instance, when we were trying to construct a proof of correctness for Google’s Native Client checker, we realized it would be too hard with the tools we had at hand. So we came up with a simpler algorithm and architecture which was easier to prove correct. It also turned out to be faster! This is detailed in a technical paper: [RockSalt: better, faster, stronger SFI for the x86](http://dl.acm.org/citation.cfm?doid=2254064.2254111).

Finally, all of this is a great push-back on creeping complexity in design. I think a lot of problems arise because software is so soft, and we write a lot of code without thinking deeply about what it should (not) be doing.

Back to your question as to whether we should use the term “provably secure”: probably not. I would like to use the term “certified code” because we are going through a very rigorous certification process. But unfortunately, the term “certified” has come to mean that we’ve followed some bureaucratic set of development processes that don’t actually provide much in the way of security or reliability. So I’m open to suggestions for better terms.

---

**Luke**: [Harper (2000)](http://commonsenseatheism.com/wp-content/uploads/2013/06/Harper-Challenges-for-designing-intelligent-systems-for-safety-critical-applications.pdf) quotes [an old message](http://www.cs.york.ac.uk/hise/safety-critical-archive/2000/0501.html) by Ken Frith from the [Safety Critical Mailing List](http://www.cs.york.ac.uk/hise/sc_list.php):

> The thought of having to apply formal proofs to intelligent systems leaves me cold. How do you provide satisfactory assurance for something that has the ability to change itself during a continuous learning process?
>
>

How do you think about the opportunities and limitations for applying formal methods to increasingly intelligent, autonomous systems?

---

**Greg**: I’ll be honest and say that I haven’t given it much thought. Perhaps the closest thing that I’ve worked on is trying to formalize information-theoretic security for cryptographic schemes and construct formal proofs in that setting. This is an area where I think the crypto/theory community has provided enough structure, we can hope to make progress, but even here, we have all sorts of shaky assumptions at the foundations.

I suspect that for intelligent and autonomous systems work (especially in the cyber-physical realm), the first real challenge is constructing appropriate models that are over-approximations of potential failure modes, and try to find something that is tractable to work with, similar to the way that crypto tries to model adversaries.

---

**Luke**: In general, for systems that need to meet a high standard of security, to what degree do you think we can take existing systems that were mostly designed to be *effective*, and “bolt security onto them,” versus needing to build things from the ground up to be secure? Perhaps some specific examples inform your view?

---

**Greg**: I think re-architecting and re-coding things will almost always lead to a win in terms of security, when compared to bolt-on approaches. A good example is [qmail](http://cr.yp.to/qmail/guarantee.html) which was designed as a secure alternative to sendmail, and in comparison, had a good architecture that made it much easier to get most things right, including and especially, configuration. As far as I know, there’s only been one bug in qmail (an integer overflow) that could potentially lead to a security problem, whereas there are many, many issues with sendmail (I’ve lost count.)

But, in the end, security is at best a secondary consideration for almost all systems. And as much as I wish we could throw away all of the code and take the time to re-architect and re-code lots of things (especially using better language technology), it’s too expensive. For instance, Windows XP was over 50 million lines of code (I don’t know how big Win7 or Win8 are) so re-coding Windows just isn’t a realistic possibility.

What we need instead are tools and techniques that people can buy into now, and help block problems with legacy systems. Ideally, these same tools should give you a “pay-as-you-go” kind of property: a little bit of effort buys you some assurance, and more effort buys you a lot more.

A good example is the kind of static analysis tools that we’re starting to see for legacy C/C++ code, such as Coverity, HP/Fortify, or Microsoft’s Prefast/Prefix/SLAM/etc. These are tools that automatically identify potential errors which in turn could lead to vulnerabilities. Today, if you run say these tools on a random pile of C/C++ code, you’ll find that it gives lots of false positives. But if you work to minimize those warnings, then you’ll find that the tool becomes much more effective. And, for new code, it works quite well as long as you pay attention to the warnings and work to get rid of them.

---

**Luke**: Thanks, Greg!

The post [Greg Morrisett on Secure and Reliable Systems](https://intelligence.org/2013/11/05/greg-morrisett-on-secure-and-reliable-systems-2/) appeared first on [Machine Intelligence Research Institute](https://intelligence.org).


From Philosophy to Math to Engineering

For centuries, philosophers wondered how we could learn what causes what. Some argued it was impossible, or possible only via experiment. Others kept [hacking away](http://lesswrong.com/lw/8ns/hack_away_at_the_edges/) at the problem, [clarifying ideas](http://commonsenseatheism.com/wp-content/uploads/2013/10/Pearl-The-Art-and-Science-of-Cause-and-Effect.pdf) like *counterfactual* and *probability* and *correlation* by making them more precise and coherent.

Then, in the 1990s, a breakthrough: Judea Pearl and others [showed](http://www.amazon.com/Causality-ebook/dp/B00AKE1VYK/) that, in principle, we can sometimes infer causal relations from data even without experiment, via the mathematical machinery of probabilistic graphical models.

Next, engineers used this mathematical insight to write [software](http://cran.r-project.org/web/packages/pcalg/index.html) that can, in seconds, infer causal relations from a data set of observations.

Across the centuries, researchers had toiled away, pushing our understanding of causality from philosophy to math to engineering.

[![From Philosophy to Math to Engineering (small)](https://intelligence.org/wp-content/uploads/2013/10/From-Philosophy-to-Math-to-Engineering-small.jpg)](https://intelligence.org/wp-content/uploads/2013/10/From-Philosophy-to-Math-to-Engineering.jpg)
And so it is with [Friendly AI](http://en.wikipedia.org/wiki/Friendly_artificial_intelligence) research. Current progress on each sub-problem of Friendly AI lies somewhere on a spectrum from philosophy to math to engineering.

We began with some fuzzy philosophical ideas of what we want from a Friendly AI (FAI). We want it to be benevolent and powerful enough to eliminate suffering, protect us from natural catastrophes, help us explore the universe, and otherwise make life *awesome*. We want FAI to allow for moral progress, rather than immediately reshape the galaxy according to whatever our current values happen to be. We want FAI to remain beneficent even as it rewrites its core algorithms to become smarter and smarter. And so on.

Small pieces of this philosophical puzzle have been broken off and [turned into](http://lesswrong.com/lw/hok/link_scott_aaronson_on_free_will/9546?context=1#comments) math, e.g. [Pearlian causal analysis](http://www.amazon.com/Causality-ebook/dp/B00AKE1VYK/) and [Solomonoff induction](http://lesswrong.com/lw/dhg/an_intuitive_explanation_of_solomonoff_induction/). Pearl’s math has since been used to produce causal inference software that can be run on today’s computers, whereas engineers have thus far succeeded in implementing (tractable approximations of) Solomonoff induction only for [very limited applications](http://arxiv.org/pdf/0909.0801v2.pdf).

Toy versions of two pieces of the “stable self-modification” problem were transformed into math problems in [de Blanc (2011)](https://intelligence.org/files/OntologicalCrises.pdf) and [Yudkowsky & Herreshoff (2013)](https://intelligence.org/files/TilingAgents.pdf), though this was done to enable further insight via formal analysis, not to assert that these small pieces of the philosophical problem had been *solved* to the level of math.

Thanks to Patrick LaVictoire and other [MIRI workshop](http://intelligence.org/get-involved/#workshop) participants,[1](https://intelligence.org/2013/11/04/from-philosophy-to-math-to-engineering/#footnote_0_10548 "And before them, Moshe Tennenholtz.") Douglas Hofstadter’s FAI-relevant philosophical idea of “[superrationality](http://en.wikipedia.org/wiki/Superrationality)” seems to have been, for the most part, successfully [transformed](https://intelligence.org/files/RobustCooperation.pdf) into math, and a bit of the engineering work has also [been done](https://github.com/klao/provability/blob/master/modal.hs).

I say “seems” because, while humans are fairly skilled at turning math into feats of practical engineering, we seem to be [much](http://lesswrong.com/lw/4zs/philosophy_a_diseased_discipline/) [*less*](http://commonsenseatheism.com/wp-content/uploads/2012/10/Brennan-Scepticism-about-philosophy.pdf) [skilled](http://consc.net/papers/progress.pdf) at turning philosophy into math, without leaving anything out. For example, some very sophisticated thinkers have [claimed](http://commonsenseatheism.com/wp-content/uploads/2013/10/Rathmanner-Hutter-A-philosophical-treatise-of-universal-induction.pdf) that “Solomonoff induction solves the problem of inductive inference,” or [that](http://www.amazon.com/Introduction-Kolmogorov-Complexity-Applications-Computer/dp/0387339981/) “Solomonoff has successfully invented a perfect theory of induction.” And indeed, it certainly *seems* like a truly universal induction procedure. However, it [turns out](http://lesswrong.com/lw/cw1/open_problems_related_to_solomonoff_induction/) that Solomonoff induction *doesn’t* fully solve the problem of inductive inference, for relatively subtle reasons.[2](https://intelligence.org/2013/11/04/from-philosophy-to-math-to-engineering/#footnote_1_10548 "Yudkowsky plans to write more about how to improve on Solomonoff induction, later.")

Unfortunately, philosophical mistakes like this could be fatal when humanity builds the first self-improving AGI ([Yudkowsky 2008](https://intelligence.org/files/AIPosNegFactor.pdf)).[3](https://intelligence.org/2013/11/04/from-philosophy-to-math-to-engineering/#footnote_2_10548 "This is a specific instance of a problem Peter Ludlow described like this: “the technological curve is pulling away from the philosophy curve very rapidly and is about to leave it completely behind.”") FAI-relevant philosophical work is, as Nick Bostrom says, “philosophy with a deadline.”

---

1. And before them, [Moshe Tennenholtz](http://mechroom.technion.ac.il/~moshet/progeqnote4.pdf).
2. Yudkowsky plans to write more about how to improve on Solomonoff induction, later.
3. This is a specific instance of a problem Peter Ludlow described like [this](http://leiterreports.typepad.com/blog/2013/10/progress-in-philosophy-revisited.html): “the technological curve is pulling away from the philosophy curve very rapidly and is about to leave it completely behind.”

The post [From Philosophy to Math to Engineering](https://intelligence.org/2013/11/04/from-philosophy-to-math-to-engineering/) appeared first on [Machine Intelligence Research Institute](https://intelligence.org).