![Benjamin C. Pierce portrait](http://intelligence.org/wp-content/uploads/2014/05/Pierce_w271.jpg) Benjamin C. Pierce is Henry Salvatori Professor of Computer and Information Science at the University of Pennsylvania and a Fellow of the ACM. His research interests include programming languages, type systems, language-based security, computer-assisted formal verification, differential privacy, and synchronization technologies. He is the author of the widely used graduate textbooks *[Types and Programming Languages](http://smile.amazon.com/Types-Programming-Languages-Benjamin-Pierce/dp/0262162091/ref=nosim?tag=793775876-20)* and *Software Foundations*.

He has served as co-Editor in Chief of the *Journal of Functional Programming*, as Managing Editor for *Logical Methods in Computer Science*, and as editorial board member of *Mathematical Structures in Computer Science*, *Formal Aspects of Computing*, and *ACM Transactions on Programming Languages and Systems*. He is also the lead designer of the popular [Unison](http://en.wikipedia.org/wiki/Unison_(file_synchronizer)) file synchronizer.

**Luke Muehlhauser**: I previously [interviewed](http://intelligence.org/2013/11/05/greg-morrisett-on-secure-and-reliable-systems-2/) Greg Morrisett about the [SAFE](http://www.crash-safe.org/) project, and about computer security in general. You’ve also contributed to the SAFE project, gave an “early retrospective” talk on it ([slides](http://www.cis.upenn.edu/%7Ebcpierce/papers/safe-PiP-140126.pdf)), and I’d like to ask you some more detailed questions about it.

In particular, I’d like to ask about the “[verified information-flow architecture](http://prosecco.gforge.inria.fr/personal/hritcu/publications/verified-ifc-draft.pdf)” developed for SAFE. Can you give us an overview of the kinds of information flow security properties you were able to prove about the system?

---

**Benjamin C. Pierce**: Sure. First, to remind your readers: SAFE is a clean-slate design of a new hardware / software stack whose goal is to build a network host that is highly resilient to cyber-attack. One pillar of the design is pervasive mechanisms for tracking information flow. The SAFE hardware offers fine-grained tagging and efficient propagation and combination of tags on each instruction dispatch. The operating system virtualizes these generic facilities to provide an “information-flow abstract machine,” on which user programs run.

Formal verification has been part of the SAFE design process right from the beginning. We’d originally hoped to be able to verify the actual running code of the OS in the style of [sel4](http://ssrg.nicta.com.au/projects/seL4/), but we found that the codebase was too big and moving too fast for this to be practical with a small verification team. Instead, we’ve developed a methodology that combines full formal verification of *models* of the system’s key features with “property-based random testing” (*à la* [QuickCheck](http://en.wikipedia.org/wiki/QuickCheck)) of richer subsets of the system’s functionality.

Our [most interesting formal proof](http://www.cis.upenn.edu/%7Ebcpierce/papers/verified-ifc.pdf) so far shows that the key security property of the information-flow abstract machine — the fact that a program’s secret inputs cannot influence its public outputs — is correctly preserved by our implementation on (a simpified version of) the SAFE hardware. This is interesting because the behavior of the abstract machine is achieved by a fairly intricate interplay between a hardware “rule cache” and a software layer that fills the cache as needed by consulting a symbolic representation of the current security policy. Since this mechanism lies at the very core of the SAFE architecture’s security guarantees, we wanted to be completely sure it is correct (and it is!).

---

**Luke**: You mention that since you aren’t able to formally verify the entire OS with your available resources, you’re instead formally verifying models of the system’s key features and also using property-based random testing of some of the rest of the system’s functionality. What level of subjective confidence (betting odds) does that approach give you for the security of the system, compared to what you could get if you had the resources to successfully verify the entire OS? (A rough “gut guess” is acceptable!)

---

**Benjamin**: Let’s say that an unverified but well-engineered and road-tested system gets a 1 on the security scale and a fully verified OS gets an 8. And let’s assume that we’re talking about a finished and polished version of SAFE (recognizing that what we have now is an incomplete prototype), together with correctness proofs for models of core mechanisms and random testing of some larger components. My gut would put this system at maybe a 4 on this scale. The proofs we’ve done are limited in scope, but they’ve been extremely useful in deepening our understanding of the core design and guiding us to a clean implementation.

Why is the top of my scale 8 and not 10? Because even a full-blown machine-checked proof of correctness doesn’t mean that your OS is truly invulnerable. Any proof is going to be based on assumptions — e.g., that the hardware behaves correctly, which might not be the case if the attacker has a hair dryer and physical access to the CPU! So, to get higher than 8 requires some additional assurance that the degree of potential damage is limited *even if* your formal assumptions are broken.

This observation suggests a defense-in-depth strategy. Our goal in designing the SAFE software has been to make the “omniprivileged” part of the code (i.e., the part that could cause the machine to do absolutely anything, if the attacker could somehow get it to misbehave) as small as possible — vastly smaller than conventional operating systems and even significantly smaller than microkernels. The code that checks security policy and fills the hardware rule cache is highly privileged, as is the garbage collector in our current design, but we believe that the rest of the OS can be written as a set of “compartments” with more limited privilege, using the tagging hardware to enforce separation between components. (Demonstrating this claim is ongoing work.)

---

**Luke**: I’ve spoken to some people who are skeptical that formal verification adds much of a security boost, since so many security problems come from not understanding the vulnerabilities well enough to capture them all in a system’s formal requirements in the first place, as opposed to coming from a system which fails to match the design specification in ways that could have been caught by formal verification. Yet you seem to rank a verified system as far more secure than a non-verified system. What’s your perspective on this?

---

**Benjamin**: Yes, other things being equal I would put quite a bit of money on a verified system being more secure than an unverified one — not because a the verification process eliminates the possibility of vulnerabilities that were not considered in the specification, but because the process of *writing* the formal specification requires someone to sit down and think carefully about at least some of the (security and other) properties that the system is trying to achieve. A clear understanding of at least some of the issues is a lot better than nothing!

Actually carrying out a proof that the system matches the specification adds significant further value, because specifications themselves are complex artifacts that require debugging just as much as the system being specified does! But property-based random testing — i.e., checking that the system satisfies its specification for a large number of well-distributed random inputs — can be a very effective intermediate step, giving a large part of the bang for relatively small buck.

---

**Luke**: In general, do you think that highly complex systems for which “high assurance” of safety or security (as relevant) are required need to be designed “from the ground up” for safety or security in order to obtain such high assurance, or are there many cases in which a highly complex but high assurance system can be successfully developed by optimizing development for general performance and then “patching” the system for security and safety?

And, do you have any guesses as to what the most common reply to this question might be in your own subfield(s) and other adjacent subfields?

---

**Benjamin**: Of course it depends a bit what you mean by “high assurance”! Quite a few people define the term so that it *only* applies to software developed using methods that consider safety and/or security from day 1. But even if we take a broader definition of HA — “offering very good resistance to malicious penetration and catastrophic failure,” or some such — my feeling is that this is extremely difficult to retrofit.

I personally saw this effect very clearly while building Unison, a cross-platform file synchronization tool that I made with Trevor Jim and Jérome Vouillon a few years ago. I had done several synchronizers before Unison, but I always discovered edge cases where the behavior seemed wrong, so we began the Unison design by trying to write down a very precise (though dramatically simplified) specification of its core behavior. Beyond helping us understand edge cases, this exercise had a *huge* effect on the way we wrote the code. For example, the specification says that, at the end of a run of the synchronizer, every part of the filesystems being synchronized should be either *synchronized* (brought up to date with the other copy) or *unchanged* from when the synchronizer started running (for example, if there was a conflict that the user chose not to resolve). But, since the synchronizer can terminate in all sorts of ways (finishing normally, getting killed by the user, the network connection getting dropped, one machine getting rebooted, …) this requirement means that the filesystems must be in one of these two states *at every moment* during execution. This is actually quite hard to achieve: there are many parts of the code that have to do extra work to buffer things in temporary locations until they can be moved atomically into place, use two-phase commits to make sure internal data structures don’t get corrupted, etc., etc., and the reasons why the extra work is needed are often quite subtle. Having the spec beforehand forced us to do this work as we went. Going back later and rediscovering all the places where we should have paid extra attention would have been well nigh impossible.

As for what others in my research area might say to your question, I decided to poll some friends instead of just assuming. As expected, most of the comments I got back were along the lines of “Making a complex system really secure after the fact is basically impossible.” But a few people did propose systems that might count as examples, such as SELinux and Apache. (Actually, SELinux is an interesting case. On one hand, its goal was only to add mandatory access control to Linux; it explicitly didn’t even attempt to increase the assurance of Linux itself. So for attacks that target kernel vulnerabilities, SELinux is no better than vanilla Linux. On the other hand, I’m told that, in certain situations — in particular, for securing web applications and other network-facing code against the sorts of attacks that come over HTTP requests — SELinux can provide very strong protection.)

---

**Luke**: Thanks, Benjamin!

The post [Benjamin Pierce on clean-slate security architectures](https://intelligence.org/2014/05/11/benjamin-pierce/) appeared first on [Machine Intelligence Research Institute](https://intelligence.org).


![Michael Fisher portrait](http://intelligence.org/wp-content/uploads/2014/05/Fisher_w449.jpg) Michael Fisher is a professor of Computer Science, specialising in logical methods and automated formal verification, and Director of the multi-disciplinary [Centre for Autonomous Systems Technology](http://www.liv.ac.uk/cast) at the [University of Liverpool](https://theconversation.com/institutions/university-of-liverpool) and member of the [Logic and Computation group](http://intranet.csc.liv.ac.uk/research/logics/) in the Department of Computer Science.

Professor Fisher is also a Fellow of both the [BCS](http://www1.bcs.org.uk/) and the [IET](http://www.theiet.org/), and a member of the [UK Computing Research Committee](http://www.ukcrc.org.uk/). He is Chair of the Department’s [Industrial Liaison Committee](http://www.csc.liv.ac.uk/~michael/industrial.html).

**Luke Muehlhauser**: In “[Verifying Autonomous Systems](http://commonsenseatheism.com/wp-content/uploads/2013/08/Fisher-et-al-Verifying-Autonomous-Systems.pdf)” (2013), you and your co-authors summarize some recent approaches to formally verifying autonomous systems. At one point, you write:

> Many autonomous systems, ranging over unmanned aircraft, robotics, satellites, and even purely software applications, have a similar structure, namely layered architectures as summarized in Figure 1. Although purely connectionist/sub-symbolic architectures remain prevalent in some areas, such as robotics, there is a broad realization that separating out the important/difficult choices into an identifiable entity can be very useful for development, debugging, and analysis. While such layered architectures have been investigated for many years they appear increasingly common in autonomous systems.
>
>

The accompanying figure is:

![Figure 1. Typical hybrid autonomous figure architecture—with suitable analysis techniques noted.](http://intelligence.org/wp-content/uploads/2014/05/Fisher-Typical-hybrid-autonomous-system-architecture.png)
What can you say about how common you think something like your “typical hybrid autonomous system architecture” is? E.g. do you think that kind of architecture is in use in ~20 different real-world applications, or ~200 different real-world applications, or perhaps more? Do you think there’s a trend in favor of such architectures? Is there a trend toward such architectures even in robotics, do you think?

---

**Michael Fisher**: It is difficult to get definitive evidence about the internal architectures of autonomous systems, especially commercial ones, so I only have anecdotal information about systems constructed in this way. However, from my discussions with engineers, both in academia and industry (aerospace and automotive), it appears that the use of this sort of architecture is increasing. It might not be termed an “agent architecture”, indeed the “decision maker” might not be termed an “agent” at all, but systems with a separate “executive” or “control module” that explicitly deals with high-level decisions, fit into our model.

I am not sure whether robotic architectures are also moving in this direction, but there seems to be a theme of adding a “monitor” or “governor” that can assess the safety or ethics of proposed actions before the robot takes them[1](https://intelligence.org/2014/05/09/michael-fisher/#footnote_0_11068 "Arkin, R. “Governing Lethal Behavior: Embedding Ethics in a Hybrid Deliberative/Reactive Robot Architecture“. Technical Report GIT-GVU-07-11, Georgia Tech, 2007."),[2](https://intelligence.org/2014/05/09/michael-fisher/#footnote_1_11068 "Woodman, R., Winfield, A., Harper, C. and Fraser, M. “Building Safer Robots: Safety Driven Control“. International Journal of Robotic Research 31(13):1603–1626, 2012.") . This can alternatively be viewed as an additional agent making high-level decisions (what to permit, what to block) and so this approach might also be seen as a variety of hybrid agent architecture.

---

**Luke**: In that same paper, you and your co-authors explain that

> Since autonomous software has to make its own decision, it is often vital to know not only what the software does and when it does it but also *why* the software chooses to do it.
>
>
> …A very useful abstraction for capturing such autonomous behavior within complex, dynamic systems turns out to be the concept of an agent. Since the agent concept came into prominence in the 1980s, there has been vast development within both academia and industry. It has become clear this agent metaphor is very useful for capturing many practical situations involving complex systems comprising flexible, autonomous, and distributed components. In essence, agents must fundamentally be capable of flexible autonomous action.
>
>
> However, it turns out the “agent” concept on its own is still not enough! Systems controlled by neural networks, genetic algorithms, and complex control systems, among others, can all act autonomously and thus be called agents, yet the reasons for their actions are often quite opaque. Because of this, such systems are very difficult to develop, control, and analyze.
>
>
> So, the concept of a *rational* agent has become more popular. Again, there are many variations but we consider this to be an agent that *has explicit reasons for making the choices it does, and should be able to explain these if necessary*.
>
>

I previously explored the relative transparency of different AI methods in [Transparency in Safety-Critical Systems](http://intelligence.org/2013/08/25/transparency-in-safety-critical-systems/), where I asked:

> How does the transparency of a method change with scale? A 200-rules logical AI might be more transparent than a 200-node Bayes net, but what if we’re comparing 100,000 rules vs. 100,000 nodes? At least we can *query* the Bayes net to ask “what it believes about X,” whereas we can’t necessarily do so with the logic-based system.
>
>

What are your thoughts on how well the transparency of rational agent models will scale? Or, do you think the “rational agent” part of future autonomous systems will be kept fairly small (e.g. to enable transparency and verification) even as the subsystems at lower “layers” in the architecture may grow massively in complexity?

---

**Michael**: I would say that the high-level rules within rational agents remain relatively transparent even as their numbers increase. It’s just that the proof/analysis mechanisms do not scale so well.

So, as you suggest, the intention is to keep the rational agent part as small as possible. Formal verification is quite costly, so any verification of very large agents is likely to be infeasible.

By only dealing with the high-level, rational agent decisions, we aim to avoid handling the vast state-space of the whole system. The use of logical (specifically, BDI) rules abstracts us from much low-level detail.

Concerning your question about “100,000 rules vs. 100,000 nodes”, I would hope that the symbolic logic language used helps us reduce this to 1000, or even 100, rules (vs. 100,000 nodes).

---

**Luke**: Does your hierarchical autonomous system approach, with a Belief-Desire-Intention rational agent “on top,” integrate with [hybrid system control](http://intelligence.org/2014/03/26/anil-nerode/) approaches, or would it be an alternative to hybrid system control approaches?

---

**Michael**: Yes, it fits together with hybrid system approaches and, certainly, the continuous and stochastic aspects are important for modelling real-world interactions. You can possibly see our approach as providing further detail about the discrete steps within hybrid systems.

---

**Luke**: In the future, do you think the logical agent specification in the top-level role in hierarchical autonomous systems will eventually need to be replaced by a more flexible, probabilistic/fuzzy reasoning approach, so as to be a better match for the environment? Or could a logical agent specification play that role even as we build systems (several decades from now) that approach human-like, fully general reasoning and planning abilities?

---

**Michael**: The latter. We are working towards being able to specify and verify probabilistic, and even real-time, agent behaviours at this top level. (Currently we can do either agent verification or probabilistic verification, but we are developing combination mechanisms.) The high-level agent will likely become increasingly complex as the sophistication of applications develop and may well incorporate different reasoning and decision components. Our approach at present is that planning components are within the separate, low-level, modules. The agent just calls these and then makes decisions between the plan options produced.

---

**Luke**: Interesting. Could you say a bit more about the combination mechanisms you’re developing?

---

**Michael**: Formally verifying basic logical properties is already difficult. However, once we tackle autonomous systems that might well be used in realistic situations, then we are often led to much more complex logical queries. For example, imagine we wish to verify a property such as

> when asked, the probability of the robot replying within 30 seconds is at least 80%
>
>

Already we have moved beyond basic properties to incorporate both probabilities (>80%) and real-time aspects (within 30s). But, with autonomous systems we would like to verify not just what the system (e.g. a robot) does but what its intentions and beliefs are. So we might even want to verify

> when asked, the probability of the robot believing the request has been made and then intending to reply within 30 seconds is at least 80%
>
>

The problem here is that the logical framework we need to describe such requirements is already quite complex. Combinations of logics are often difficult to deal with and combinations of logics of probability, real-time, intention, belief, etc., have not yet been studied in great detail.

We are aiming to develop and extend some theoretical [work we have been doing](http://dx.doi.org/10.1016/j.tcs.2013.07.012) on model checking of complex logical combinations in order to tackle this problem of more comprehensive autonomous system verification.

---

**Luke**: Thanks, Michael!

---

1. Arkin, R. “[Governing Lethal Behavior: Embedding Ethics in a Hybrid Deliberative/Reactive Robot Architecture](http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.110.1610)“. Technical Report GIT-GVU-07-11, Georgia Tech, 2007.
2. Woodman, R., Winfield, A., Harper, C. and Fraser, M. “[Building Safer Robots: Safety Driven Control](http://dx.doi.org/10.1177/0278364912459665)“. International Journal of Robotic Research 31(13):1603–1626, 2012.

The post [Michael Fisher on verifying autonomous systems](https://intelligence.org/2014/05/09/michael-fisher/) appeared first on [Machine Intelligence Research Institute](https://intelligence.org).


![Harry Buhrman portrait](http://intelligence.org/wp-content/uploads/2014/05/Buhrman_w185.jpg) [Harry Buhrman](http://www.cwi.nl/people/552) is head of the research group ‘[Algorithms and Complexity](http://www.cwi.nl/research-groups/algorithms-and-complexity)’ at the Centrum Wiskunde & Informatica, Amsterdam, which he joined in 1994. Since 2000 he also has a joint appointment as full professor of computer science at the [University of Amsterdam](http://www.uva.nl/en/home). Buhrman’s research focuses on quantum computing, algorithms, complexity theory, and computational biology. In 2003 he obtained a prestigious [Vici award](http://www.nwo.nl/en/research-and-results/programmes/Talent+Scheme/awards/vici+awards) and was coordinator of several national and international projects.

Buhrman is editor of several international journals and is member of various advisory and scientific boards, such as the advisory board of the Institute for Quantum Computing (Waterloo, Canada).

**Luke Muehlhauser**: In theory, [position-based quantum cryptography](http://ercim-news.ercim.eu/en85/special/position-based-quantum-cryptography) could ensure that certain tasks could only be performed at certain locations:

> For example, a country could send a message in such a way that it can only be deciphered at the location of its embassy in another country.
>
>

In [Buhrman et al. (2011)](http://commonsenseatheism.com/wp-content/uploads/2014/02/Buhrman-et-al.-Position-Based-Quantum-Cryptography-Impossibility-and-Constructions.pdf), you and your co-authors showed that

> if collaborating adversaries are allowed to pre-share an arbitrarily large entangled quantum state, then… position-based cryptography… is impossible… in the quantum setting.
>
>
> On the positive side, we show that for adversaries that are restricted to not share any entangled quantum states, secure position-verification is achievable. Jointly, these results suggest the interesting question whether secure position-verification is possible in the bounded case of a bounded amount of entanglement.
>
>

What is the current state of our knowledge on whether secure position-verification is possible given a bounded amount of entanglement?

---

**Harry Buhrman**: Classically such schemes will always be insecure, also if one restricts the running time or other resources of the adversaries. Quantumly the situation is more complicated, although we and others have shown that with an exponential amount of entanglement, also quantum schemes can be broken and are insecure. The quest is thus on for schemes that 1) can be easily implemented by the honest parties 2) require unreasonable amounts of entanglement for adversaries to break the scheme. We have seen some partial progress along these lines. For example a [paper](http://arxiv.org/pdf/1101.1065.pdf) by Beigi and Koenig shows that there are schemes where a linear amount of entanglement is necessary to break them. It turns out that the question whether such schemes exists and if they do, prove for certain schemes that they are secure, is a very hard problem that we are currently working on. It turns out that the approach we have taken implies, if successful, results in classical complexity theory, and thus are very hard. On the other hand, it may still be possible that there simply do not exist schemes that are secure in this sense.

---

**Luke**: Based on your knowledge of the current state of quantum computing and quantum algorithm design, what recommendations to governments or citizens might you make?

E.g. Ronald de Wolf recommended we switch to [post-quantum cryptography](http://en.wikipedia.org/wiki/Post-quantum_cryptography) for sensitive data, especially since “even though there’s no QC yet, the security services… are probably hoovering up RSA-encrypted communications that they store for the time being, waiting for the QC that will allow them to decrypt these messages later.”

---

**Harry**: Yes I agree with Ronald completely. Switch as soon as possible to quantum proof security (although we don’t know what exactly is quantum proof, see next paragraph). Indeed, all of our electronic encrypted communication, such as e-mail, telephone conversations, etc. that are encrypted using RSA-like techniques, can not yet be decrypted, but NSA and other organisations collect these encrypted data and wait till the day comes when they can decipher them. For example when a sufficiently large quantum computer is built such data can be decrypted very easily. So if we want to keep our data secure for a reasonable amount of time we should switch to other encryption techniques and standards. This applies more to governments, banks, big companies, than to normal users, but still I feel that a lot of people don’t realise this. The common incorrect way of approaching this is as follows: This quantum computer, if built at all, is still 15 years ahead of us and when that time comes we switch then to more secure encryption. This reasoning is of course incorrect.

So what encryption should we use instead? Problematic is that this requires much more research. However a few encryption schemes have been proposed that *appear* to be quantum proof. I suggest that we swith to those in combination with good old RSA. So double encryption. This way the encryption security remains at least the same as what it is now, but most likely it is also quantum proof. One caveat of this approach is that encryption and decryption become (much) more inefficient since RSA like techniques are conveniently fast. This is the price one has to pay to be secure well into the future. So one has to make a decision when encrypting. Is it only necessary to be secure in the next year, e.g. when transmitting a credit card number it is mostly fine if it is only secure for a year or two, or do we need security well into the future? In the latter case I advise to switch as soon as possible.

---

**Luke**: Are there additional recommendations you can make to governments or citizens, based on your knowledge of the current state of quantum computing and quantum algorithm design?

---

**Harry**: Cryptographic implications are the ones that pertain immediately to society. What most people don’t realise is that there are many problems in science and technology related fields, like eg chemistry, biology, physics and medicine design, that are computational very demanding and often impossible to carry out even on the most powerful supercomputers. Quantum computers wil help to solve some of these more efficiently notably when they concern simulation of quantum mechanical systems and properties. So research in QC will probably have an impact there as well.

---

**Luke**: From your perspective, what are the most significant factors blocking or slowing greater progress on the theoretical half (not the experimental half) of quantum computing research? In other words, what feasible but difficult changes to the field (or related fields) would most accelerate theoretical quantum computing research?

---

**Harry**: That is a difficult question to answer. If I knew what to change I would do it immediately. One of the problems is that we would like to have more quantum algorithms that demonstrate the power of quantum computation. The community has developed several techniques, like quantum Fourier sampling and quantum random walks. We would like to have more of those techniques and ideas. Maybe more important we also need to isolate computational problems or tasks that are amenable to quantum speedup. That is to say where quantum computers outperform classical computers.

Another area that could improve is that of fault tolerance and error correcting codes. A lot of progress has been made but we still don’t know the exact value of how many hardware errors can be tolerated by algorithmic design. This value is called the fault tolerant  threshold. It  is important to know the exact value when  implementing qubits and running quantum  algorithms and protocols on them. We  will need to perform error correction and fault tolerant computation that fight the errors that popup due to imperfections and noise. But if the errors that occur are too big such correction by computation is not possible. Also it will be important to keep the overhead as small as possible.

A third problem is that of few qubit applications. We are beginning to see the demonstration of few qubit computers in the lab, but there are no good problems that could be used to demonstrate the usefulness of small quantum computers that operate on say 30-100 qubits.

---

**Luke**: In [Buhrman et al. (1997)](http://staff.science.uva.nl/%7Eleen/PDFpapers/726.pdf), you and your co-authors listed 6 open problems in computational complexity theory. Have any of them been solved in the intervening time, perhaps at least with relativized results?

---

**Harry**: In that paper we listed 6 hypotheses that all are probably false. The goal was and still is to show that they are equivalent to P=NP. We also listed 6 open problems of which two were [solved the next year](http://www.sciencedirect.com/science/article/pii/S0022000099916537) by D. Sivakumar (# 2 and 5 from that list). As far as I know not much progress has been made towards these problems or showing that all the hypothesis are equivalent to P=NP. I think this is indication for how difficult the P versus NP question is. There currently is a new approach to tackling this problem, not the six hypotheses but the P vs NP problem itself. This approach is called geometric complexity and involves algebraic geometry and representation theory. Difficult topics in Mathematics.

There actually is a connection between quantum information and these types of classical questions. The language of quantum information allows one to state some of these problems in a different language which allows one to approach the problem from a different angle. In general this has been quite successful. For example [the result of Ronald de Wolf and co-authors](http://homepages.cwi.nl/%7Erdewolf/publ/qc/stoc130-fiorini.pdf) which uses quantum communication at its core, or [the paper I have](https://arxiv.org/abs/0911.4007) with Jop Briet, Troy Lee and Thomas Vidick which settles a problem in Banach space theory by solving a problem in quantum information. There also is [a survey](http://theoryofcomputing.org/articles/gs002/gs002.pdf) about other such results.

Perhaps we should revisit the 6 hypotheses paper with quantum techniques and see if we can make a little progress. Though at the moment I don’t see a clear plan of attack.

---

**Luke**: Thanks, Harry!

The post [Harry Buhrman on quantum algorithms and cryptography](https://intelligence.org/2014/05/07/harry-buhrman/) appeared first on [Machine Intelligence Research Institute](https://intelligence.org).


[![Problems of self-reference](https://intelligence.org/wp-content/uploads/2014/05/Problems-of-self-reference.png)](https://intelligence.org/files/ProblemsofSelfReference.pdf)We’ve released a new working paper by Benja Fallenstein and Nate Soares, “[Problems of self-reference in self-improving space-time embedded intelligence](https://intelligence.org/files/ProblemsofSelfReference.pdf).”

Abstract:

> By considering agents to be a part of their environment, Orseau and Ring’s space-time embedded intelligence is a better fit for the real world than the traditional agent framework. However, a self-modifying AGI that sees future versions of itself as an ordinary part of the environment may run into problems of self-reference. We show that in one particular model based on formal logic, naive approaches either lead to incorrect reasoning that allows an agent to put off an important task forever (the *procrastination paradox*), or fail to allow the agent to justify even obviously safe rewrites (the *Löbian obstacle*). We argue that these problems have relevance beyond our particular formalism, and discuss partial solutions.
>
>

This working paper also cites a brief new technical report by Fallenstein, “[Procrastination in probabilistic logic](http://intelligence.org/wp-content/uploads/2014/04/procrastination-probabilistic-logic.pdf).”

**Update** 05/14/14: This paper has been accepted to [AGI-14](http://agi-conf.org/2014/).

The post [New paper: “Problems of self-reference in self-improving space-time embedded intelligence”](https://intelligence.org/2014/05/06/new-paper-problems-of-self-reference-in-self-improving-space-time-embedded-intelligence/) appeared first on [Machine Intelligence Research Institute](https://intelligence.org).


Liveblogging the SV Gives Fundraiser

[![SVGives logo lrg](https://intelligence.org/wp-content/uploads/2014/05/SVGives-logo-lrg-300x231.jpg)](http://intelligence.org/2014/05/04/calling-all-miri-supporters/)Today MIRI is participating in a massive 24-hour fundraiser called SV Gives. Strategy details [here](http://intelligence.org/2014/05/04/calling-all-miri-supporters/), donate [here](http://svgives.razoo.com/story/Machine-Intelligence-Research-Institute).

This blog post will be updated many times throughout the day.

Total donated to MIRI: **$110,245**.

Total prizes & matching won for MIRI: **$61,330**.

**12:12am Pacific:** That’s a wrap, folks! Our thanks once again to [SVCF](http://www.siliconvalleycf.org/) for putting together a thrilling fundraiser with a neck-and-neck race toward the end. Our thanks to everyone who participated in SV Gives. And of course many thanks to our dedicated, well-organized donors who made this such a surprising success for MIRI. And finally, my thanks to the MIRI staff who worked very long hours to run the fundraiser, especially Malo Bourgon, who worked 38 hours straight, and now must go to sleep. ![🙂](https://s.w.org/images/core/emoji/14.0.0/72x72/1f642.png)

**12:07am Pacific:** For the last hour, MIRI once again won both the $2,000 prize and the $150 prize! Total raised is $110,245 plus $61,330 in prizes and matching funds!

**12:04am Pacific:** Fundraiser complete! $7.9M raised for local nonprofits. Congrats to SVCF for a rather thrilling fundraiser, and to leaderboard winners Sankara Eye Foundation and Burlingame Community for Education!

**11:58pm Pacific:** With two minutes to spare, we passed 2000 total donations today!

**11:06pm Pacific**: Another $2,000! And also another $150 for a random winner, Samuel Brooks!

**10:10pm Pacific**: And another $2,000!

**9:22pm Pacific:** MIRI has won $2,000 again!

**8:07pm Pacific:** MIRI wins $2,000 again!

**7:57pm Pacific**: Wow, MIRI and Sankara are still neck-and-neck for Total Unique Donors! Right now Sankara has a 5 person lead.

**7:09pm Pacific:** MIRI won another $2,000 for unique donors last hour, plus $6,980 in matching! But in other news, Sankara Eye Foundation has now overtaken us in the race for the Grand Prize!

**6:21pm Pacific:** MIRI won $3,030 in matching at 6pm. Thanks, everyone!

**6:05pm Pacific:** MIRI won another $2,000 for unique donors last hour. Great job everyone! Get ready for more matching funds being avilable at 7pm sharp!

**5:54pm Pacific:** Hurray! MIRI was one of 50 organizations to win $1,000 from the 5pm matching funds. The 6pm matching funds are coming up, so try to send your donation in the first ~2 seconds after the hour!

**5:11pm Pacific:** We won the hour again ($2,000), and we also won a 2nd randomly chosen $150 Golden Ticket, courtesy of a donation by Simon Safar!

**4:36pm Pacific:** Amazingly, MIRI is now 12% of all donations so far during SV Gives day.

**4:27pm Pacific:** We’ve reached 250 unique donors for the day; thanks so much! As it happens, there’s a MIRI research workshop going on right now, so [here is a photo](https://intelligence.org/wp-content/uploads/2014/05/May-6th.jpg) of what you’re all funding (several times over)!

**4:06pm Pacific:** Another $2,000 for MIRI. Note that we’re coming up on the 5pm matching opportunity, so you should try to donate in the first 5 seconds after 5pm to try to win the match!

**3:16pm Pacific:** KFOX will be [covering](http://www.kfox.com/pages/9131811.php) SV Gives between 5-7pm today.

**3:06pm Pacific**: Another $2,000! The [strategy](http://intelligence.org/2014/05/04/calling-all-miri-supporters/) is still working!

**2:10pm Pacific:** MIRI won the hourly $2,000 again. Thanks, everyone!

**1:36pm Pacific:** Noon matching results have been released. MIRI got $2910 of the $50,000 in matching, which was apparently all taken in less than 4 seconds!

**1:11pm Pacific**: Another $2,000 prize won. Our deep and sincere thanks to [SVCF](http://www.siliconvalleycf.org/) for organizing [SV Gives](http://svgives.razoo.com/giving_events/svg14/home), and to MIRI’s donors for being so dedicated and well-organized!

**12:06pm Pacific:** And we won the $2,000 again! Many thanks to our astoundingly well-coordinated donors.

**11:07am Pacific**: We’ve won the $2,000 Most Unique Donors Each Hour prize again. Who will break MIRI’s winning streak?

**10:34am Pacific:** We’ve hit the 200 unique donors (for the whole day) mark!

**10:08am Pacific:** And yet another $2,000 prize for Most Unique Donors Each Hour! I bet the very gracious Sobrato Family Foundation wasn’t expecting to be one of MIRI’s largest donors of the year so far.

**9:29am Pacific:** On Twitter, SVCF asked us: “What’s your secret? You’ve just got the 9 a.m. $2,000 Golden Ticket prize for a sixth time! Congrats!” We [replied](https://twitter.com/siliconvalleycf/status/463711885628411904): “Our secret is: 70+ person-hours of planning and donor coordination, 10+ energy drinks, and many enthusiastic supporters!” Seriously, I think our participation in SV Gives should be sponsored by Rockstar energy drinks, given how many of them we are consuming over here!

**9:07am Pacific:** MIRI wins another $2,000. Three cheers for donor coordination!

**8:22am Pacific:** $1,970 more in matching came in at the beginning of this hour! (The first 14 seconds of this hour, to be precise.)

**8:10am Pacific:** Another hour, another $2,000 prize for MIRI! Thanks again, everyone.

**7:48am Pacific:** In the first ~30 seconds after 7am, we captured $2,340 in matching. Thanks, everyone!

**7:07am Pacific:** MIRI wins the $2,000 prize once again! (For the 6am-7am hour.)

**6:16am Pacific:** 150 unique donors today so far.

**6:08am Pacific:** Excellent work, everyone! MIRI won the $2,000 prize for “Most Unique Donors Each Hour” for a third time! (For the 5am-6am hour.)

**5:27am Pacific:** According to SV Gives’ twitter account, NBC will be covering the SV Gives fundraiser in 3 minutes.

**5:18am Pacific:** Our thanks to Mick Porter, whose donation won the $150 prize for “random winner” during the 4am hour! We also won the $2,000 “Most Unique Donors Each Hour” again for the 4am-5am hour!

**4:41am Pacific:** So far we’ve won the $2,000 prize for Most Unique Donors Each Hour once, but not yet twice, so it looks like marginal donors giving as little as $10 each hour can make a difference!

**3:50am Pacific:** If you’re visiting this page due to MIRI’s SV Gives participation, and would like to know what it looks like to do *technical* work on ensuring good outcomes from future self-improving AI programs, check out our just-released new working paper, “[Problems of self-reference in self-improving space-time embedded intelligence](http://intelligence.org/2014/05/06/new-paper-problems-of-self-reference-in-self-improving-space-time-embedded-intelligence/).”

**3:04am Pacific**: Here’s a [live feed](https://plus.google.com/hangouts/_/7ecpjeqskmanhhclhe8ug0nmis) of MIRI HQ. Louie, Malo, myself, and Nate are there now.

**2:23am Pacific:** We won the $2,000 Golden Ticket for the 1am-2am hour! We also won the $2500 “Local Area Code Prize” by way of Patrick LaVictoire making the 408th donation (to any charity) of the day! Our thanks to the Sobrato Family Foundation and the Shackleton Family Foundation, respectively.

The post [Liveblogging the SV Gives Fundraiser](https://intelligence.org/2014/05/06/liveblogging-the-svgives-fundraiser/) appeared first on [Machine Intelligence Research Institute](https://intelligence.org).


**Update:** We are liveblogging the fundraiser [here](http://intelligence.org/2014/05/06/liveblogging-the-svgives-fundraiser/).

Read our strategy below, then [give here](http://svgives.razoo.com/story/Machine-Intelligence-Research-Institute)!
------------------------------------------------------------------------------------------------------------------

[![SVGives logo lrg](https://intelligence.org/wp-content/uploads/2014/05/SVGives-logo-lrg.jpg)](http://svgives.razoo.com/story/Machine-Intelligence-Research-Institute)

As previously [announced](http://intelligence.org/2014/04/25/may-6th-miri-participating-in-massive-24-hour-online-fundraiser/ "Help MIRI in a Massive 24-Hour Fundraiser on May 6th"), MIRI is participating in a massive 24-hour fundraiser on May 6th, called [SV Gives](http://svgives.razoo.com/giving_events/svg14/home). This is a unique opportunity for all MIRI supporters to increase the impact of their donations. To be successful we’ll need to pre-commit to a strategy and see it through. **If you plan to give at least $10 to MIRI sometime this year, during this event would be the best time to do it!**

The plan
--------

We need all hands on deck to help us win the following prize as many times as possible:

> $2,000 prize for the nonprofit that has the most individual donors in an hour, every hour for 24 hours.
>
>

To paraphrase, *every hour*, there is a $2,000 prize for the organization that has the most individual donors during that hour. **That’s a total of $48,000 in prizes, from sources that wouldn’t normally give to MIRI.**

The minimum donation is $10, and an individual donor can give as many times as they want. Therefore we ask our supporters to:

1. **[give](http://svgives.razoo.com/story/Machine-Intelligence-Research-Institute) $10 an hour, during *every hour* of the fundraiser that they are awake (I’ll be up and donating for all 24 hours!)**;
2. for those whose giving budgets won’t cover all those hours, see below for list of which hours you should privilege; and
3. publicize this effort as widely as possible.

### International donors, we especially need your help!

MIRI has a strong community of international supporters, and this gives us a distinct advantage! While North America sleeps, you’ll be awake, ready to target all of the overnight $2,000 hourly prizes.

Hours to target in order of importance
--------------------------------------

To increase our chances of winning these prizes we want to preferentially target the hours that will see the least donation traffic from donors of other participating organizations. Below are the top 12 hours we’d like to target in order of importance. Remember that all times are in Pacific Time. (Click on an hour to see what time it is in your timezone.)

* [1 am hour](http://www.timeanddate.com/worldclock/fixedtime.html?iso=20140506T01&p1=224&ah=1)  (01:00–01:59 PT)
* [2 am hour](http://www.timeanddate.com/worldclock/fixedtime.html?iso=20140506T02&p1=224&ah=1) (02:00–02:59 PT)
* [3 am hour](http://www.timeanddate.com/worldclock/fixedtime.html?iso=20140506T03&p1=224&ah=1) (03:00–03:59 PT)
* [4 am hour](http://www.timeanddate.com/worldclock/fixedtime.html?iso=20140506T04&p1=224&ah=1) (04:00–04:59 PT)
* [5 am hour](http://www.timeanddate.com/worldclock/fixedtime.html?iso=20140506T05&p1=224&ah=1) (05:00–05:59 PT)
* [6 am hour](http://www.timeanddate.com/worldclock/fixedtime.html?iso=20140506T06&p1=224&ah=1) (06:00–06:59 PT)
* [11 pm hour](http://www.timeanddate.com/worldclock/fixedtime.html?iso=20140506T23&p1=224&ah=1) (23:00–23:59 PT)
* [7 am hour](http://www.timeanddate.com/worldclock/fixedtime.html?iso=20140506T07&p1=224&ah=1) (07:00–07:59 PT)
* [10 pm hour](http://www.timeanddate.com/worldclock/fixedtime.html?iso=20140506T22&p1=224&ah=1) (22:00–22:59 PT)
* [8 am hour](http://www.timeanddate.com/worldclock/fixedtime.html?iso=20140506T08&p1=224&ah=1) (08:00–08:59 PT)
* [5 pm hour](http://www.timeanddate.com/worldclock/fixedtime.html?iso=20140506T17&p1=224&ah=1) (17:00–17:59 PT)
* [9 pm hour](http://www.timeanddate.com/worldclock/fixedtime.html?iso=20140506T21&p1=224&ah=1) (21:00–21:59 PT)

For the 5 pm hour there is an additional prize I think we can win:

> $1,000 golden ticket added to the first 50 organizations receiving gifts in the 5 pm hour.
>
>

**So if you are giving in the 5 pm hour try and give right at the beginning of the hour.**

### Bottom line, for every hour you are awake, [give](http://svgives.razoo.com/story/Machine-Intelligence-Research-Institute) $10 an hour.

###  Give preferentially to the hours above, if unable to give during all waking hours.

We also have plans to target the $300,000 in matching funds up for grabs during the event. If you would like to contribute $500 or more to this effort, shoot me an email at [malo@intelligence.org](mailto:malo@intelligence.org).

**For those who want to follow along and contribute to the last minute planning, as well as receive updates and giving reminders during the event, sign up below.**

The post [Calling all MIRI supporters for unique giving opportunity!](https://intelligence.org/2014/05/04/calling-all-miri-supporters/) appeared first on [Machine Intelligence Research Institute](https://intelligence.org).


![Kasper Støy portrait](http://intelligence.org/wp-content/uploads/2014/04/Støy_w1300.jpg) [Kasper Stoy](http://www.itu.dk/~ksty/) is a robotics and embodied artificial intelligence researcher holding an associate professor position at the Software and Systems Section of the [IT University of Copenhagen](http://www.itu.dk/en/). He is interested in the construction and design of complete robot systems, but being a computer scientist he has made most of his personal contributions in distributed control of multi-robot systems and modular robotics. He has published more than sixty papers in international conference proceedings or journals and is the author of the book “*[Self-Reconfigurable Robots: An Introduction](http://smile.amazon.com/Self-Reconfigurable-Robots-Introduction-Intelligent-Autonomous/dp/0262013711/ref=nosim?tag=793775876-20)*” published by MIT Press. He also co-founded Universal Robots, a company focuses on user-friendly robot arms for industry. He is an active player in the international robot research community and reviews for all major journals and conferences in robotics. He has stayed for extended periods at University of Southern California, Harvard University, University of Tarapacá (Chile), and Seam Reap (Cambodia). He holds a M.Sc. degree in computer science and physics from the University of Aarhus, Denmark (1999) and a Ph.D. degree in computer system engineering from the University of Southern Denmark (2003) where he also worked as assistant professor (2003-2006) and associate professor (2006-2013). He is married and has two kids.

**Luke Muehlhauser**: In [Larsen et al. (2013)](http://www.itu.dk/%7Eksty/uploads/publications/larsen2013ir.pdf), you and your co-authors write:

> Using a bottom-up, model-free approach when building robots [is] often seen as a less scientific way, compared to a top-down model-based approach, because the results are not easily generalizable to other systems… In this paper we will show how the use of well-known experimental methods from bio-mechanics are used to measure and locate weaknesses in our bottom-up, model-free implementation of a quadruped walker and come up with a better solution.
>
>

From looking at the paper I could see how your experimental method allowed you to find a better solution for your walker robot, but I couldn’t understand how you addressed the challenge of generalizing the solution to other systems despite the bottom-up, model-free approach. Could you explain that part?

---

**Kasper Stoy**: The problem is that researchers who are doing cutting edge robotics want to explore how materials and their interaction can aid the movement of the robot. For instance, researchers have been working on passive-dynamic walkers for a long time now that exploit the mechanical system to achieve walking without using sensors or actuators. The energy comes from walking downhill and the control is open-looped – the mechanical system itself is self-stabilising. For these systems our current engineering approach is ok, but not great. We can just about model these kinds of walkers so we can get a good guess of the initial parameters to get the system walking, but there is still an extended phase of tinkering before the system actually walks. It is in this tinkering phase that our precise motion capture as used in biomechanics comes into play. Given measured paths of all parts of the robot we can analyse the data and come up with reasonable hypotheses about how to improve the robot. Hence, the tinkering becomes much more systematic even though the underlying physical processes are too complex to be modelled. It may not be apparent, but just modelling the impact of a foot with the ground which takes all types of frictions into account, the deformation of the foot, the spring effect, and so on is largely intractable. Hence, the underlying assumption of our work is that all models of locomotion are fundamentally wrong. They may give a high-level picture of what is going on, but fundamentally they cannot be used to predict what will happen just two steps later. However, if we turn this around and we have gotten our robot to walk and we record the data of how it walks. Although difficult, we can build a model that match this data which is based on the ground truth and where there is no modelling bias on the part of the researcher. We now have a better informed model that can be used to build the next generation of robots. Hence, the model is a generalisation of our specific implementation which you can copy, but you can also replace elements of which you have better implementations. In locomotion research like many other fields the models and physical systems are drifting apart because it is researchers with different skill and interests who work on them. In our work, we are clearly working on physical systems and just provide a hint to how our experimental results can be generalised in a way that is meaningful to modelling oriented researchers. We hope.

---

**Luke**: You co-authored [a book](http://smile.amazon.com/Self-Reconfigurable-Robots-Introduction-Intelligent-Autonomous/dp/0262013711/ref=nosim?tag=793775876-20) on self-reconfigurable robots in 2010. What do you mean by that term, and what are some recent examples of practical self-reconfigurable robots? (Perhaps, more recent than the book?)

---

**Kasper**: Self-reconfigurable robots are an idea inspired by multi-cellular organisms. The idea is that instead of building a robot as a single expensive, monolithic, and fragile piece of hardware you would built a robot from many relative simple robotic cells. We refer to these robotic cells as modules and the research community as modular robotics. Self-reconfigurable robots then take this a step further in that not only is the robot multi-cellular, the modules can also rearrange the way they are connected automatically. Hence as the modules wander around each other the robot as a whole change shape. While this is rare in biology there is a small aquatic animal called hydra which has this ability. In fact, it has been shown that if you cut it in half it transform itself into two smaller replicas of the original hydra. The advantage of this type of robot is extreme robustness and versatility. Features that are particular important in for instance extra-terrestrial exploration. Regarding, application this was from the beginning curiosity driving research. Is it possible to build a robot that is multi-cellular and able to chance its own shape? The answer is yes. The concept has been implemented successfully in about five systems internationally. However, modules are relatively expensive, big (10cm diameter), complex, and quite heavy. Hence, direct applications are still lacking given that the cost-benefit of these robots is not good enough for a commercial market. However, the more modest modular robots have seen some recent success on KickStarter with the Modbots. A modular construction kit for building your own robots.

---

**Luke**: What are each of the “about five systems” you refer to?

---

**Kasper**: The most developed system is the M-TRAN III self-reconfigurable robot developed at AIST in Japan. The runner ups are ATRON which was developed mostly by my former colleagues at University of Southern Denmark. However, both of the above systems are not being developed further. Another runner up is the Roombots developed at EPFL in Switzerland. Recent exciting development are the M-Blocks coming of of MIT and SMORES coming ot of UPenn. There is a large number of robots which come very close, but not quite achieving convincing three dimensional self-reconfiguration.

Also, check out this Wikipedia article for a [more comprehensive list](http://en.wikipedia.org/wiki/Self-reconfiguring_modular_robot).

---

**Luke**: What are some specific breakthroughs or kinds of progress that you expect to see in the next 10-15 years of research into self-reconfiguring modular robots?

---

**Kasper**: The field has very much been about whether it is possible to build robots that can change their own shape or not. Within the last decade, we have established that this is indeed the case. However, currently the robotic cells are too complex, heavy, and expensive for most applications. Hence, one of the goals is to reduce all of these obstacles, but given the current state of the art in mechatronics this is quite difficult. Hence, I see a mainstream area of research focussing on various simplifications of the mechatronics to make it attractive for specific applications in space, robotic prototyping tools and the like. However, it also clear that other implementations have to be considered. For this some have turned to DNA-based implementations or using micro fabrication technology. Another unexplored opportunity is in soft robotics that is gain traction in the robotic community as a whole. The question is then where this would lead in 10-15 years. I think all areas have potential so if for instance the soft modular robotics effort is successful this could lead to robotic multi-cellular systems much closer to biological cells in terms of material properties. Where would this find applications is anyones guess, but maybe in furniture, clothing or in healthcare in the form of adaptive casts.

---

**Luke**: Thanks, Kasper!

The post [Kasper Stoy on self-reconfigurable robots](https://intelligence.org/2014/05/02/kasper-stoy/) appeared first on [Machine Intelligence Research Institute](https://intelligence.org).


MIRI’s May 2014 Newsletter

|  |  |
| --- | --- |
|

|  |
| --- |
|
[Machine Intelligence Research Institute](http://intelligence.org)
 |

 |
|

|  |  |
| --- | --- |
|

|  |
| --- |
|
On May 6th, there is $250,000+ in matching funds and prizes available from sources that normally wouldn’t contribute to MIRI at all. [Details here](http://intelligence.org/2014/04/25/may-6th-miri-participating-in-massive-24-hour-online-fundraiser/).

**Research Updates**
* New paper: “[The errors, insights, and lessons of famous AI predictions](http://intelligence.org/2014/04/30/new-paper-the-errors-insights-and-lessons-of-famous-ai-predictions/).”
* [Botworld update](http://lesswrong.com/lw/k5u/exploring_botworld/) with 4 new games, including Prisoner’s Dilemma and Stag Hunt.
* [12 new expert interviews](http://intelligence.org/category/conversations/), including e.g. [Suzana Herculano-Houzel](http://intelligence.org/2014/04/22/suzana-herculano-houzel/) on cognitive ability and brain size.

**News Updates**
* [Can We Really Upload Johnny Depp’s Brain?](http://www.slate.com/articles/technology/future_tense/2014/04/transcendence_science_can_we_really_upload_johnny_depp_s_brain.html) (in *Slate*)
* [Why MIRI?](http://intelligence.org/2014/04/20/why-miri/)

**Other Updates**
* Call for Papers: [Special Issue. Confronting Future Catastrophic Threats To Humanity](http://www.journals.elsevier.com/futures/call-for-papers/call-for-papers-special-issue-confronting-future-catastrophi)
* [AI Risk, New Executive Summary](http://lesswrong.com/lw/k37/ai_risk_new_executive_summary/) by Stuart Armstrong
* **[Transcending complacency on superintelligent machines](http://www.huffingtonpost.com/stephen-hawking/artificial-intelligence_b_5174265.html)**, by Hawking, Tegmark, Russell, and Wilczek

As always, please don’t hesitate to let us know if you have any questions or comments.
Best,
Luke Muehlhauser
Executive Director
 |

 |

 |
|  |

The post [MIRI’s May 2014 Newsletter](https://intelligence.org/2014/05/01/miris-may-2014-newsletter/) appeared first on [Machine Intelligence Research Institute](https://intelligence.org).


[![AI predictions paper](https://intelligence.org/wp-content/uploads/2014/04/AI-predictions-paper.png)](https://intelligence.org/files/ErrorsInsightsLessons.pdf)During his time as a MIRI researcher, Kaj Sotala contributed to a paper now published in the *Journal of Experimental & Theoretical Artificial Intelligence*: “[The errors, insights and lessons of famous AI predictions – and what they mean for the future](http://www.fhi.ox.ac.uk/wp-content/uploads/FAIC.pdf).”

Abstract:

> Predicting the development of artificial intelligence (AI) is a difficult project – but a vital one, according to some analysts. AI predictions are already abound: but are they reliable? This paper starts by proposing a decomposition schema for classifying them. Then it constructs a variety of theoretical tools for analysing, judging and improving them. These tools are demonstrated by careful analysis of five famous AI predictions: the initial Dartmouth conference, Dreyfus’s criticism of AI, Searle’s Chinese room paper, Kurzweil’s predictions in the *Age of Spiritual Machines*, and Omohundro’s ‘AI drives’ paper. These case studies illustrate several important principles, such as the general overconfidence of experts, the superiority of models over expert judgement and the need for greater uncertainty in all types of predictions. The general reliability of expert judgement in AI timeline predictions is shown to be poor, a result that fits in with previous studies of expert competence.
>
>

The post [New Paper: “The errors, insights, and lessons of famous AI predictions”](https://intelligence.org/2014/04/30/new-paper-the-errors-insights-and-lessons-of-famous-ai-predictions/) appeared first on [Machine Intelligence Research Institute](https://intelligence.org).


![Suresh Jagannathan portrait](http://intelligence.org/wp-content/uploads/2014/04/Jagannathan_w1300.jpg) [Dr. Suresh Jagannathan](http://go.usa.gov/kp4F) joined DARPA in September 2013. His research interests include programming languages, compilers, program verification, and concurrent and distributed systems.

Prior to joining DARPA, Dr. Jagannathan was a professor of computer science at Purdue University. He has also served as visiting faculty at Cambridge University, where he spent a sabbatical year in 2010; and as a senior research scientist at the NEC Research Institute in Princeton, N.J.

Dr. Jagannathan has published more than 125 peer-reviewed conference and journal publications and has co-authored one textbook. He holds three patents. He serves on numerous program and steering committees, and is on the editorial boards of several journals.

Dr. Jagannathan holds Doctor of Philosophy and Master of Science degrees in Electrical Engineering and Computer Science from the Massachusetts Institute of Technology. He earned a Bachelor of Science degree in Computer Science from the State University of New York, Stony Brook.

**Luke Muehlhauser**: From your perspective, what are some of the most interesting or important developments in higher-order verification in the past decade?

---

**Suresh Jagannathan**: I would classify the developments of the past decade into four broad categories:

1. The development of higher-order recursion schemes (or higher-order model checking). This approach is based on defining a language semantics in terms of a (recursive) tree grammar. We can now ask whether the tree generated by the grammar satisfies a particular safety property. Because recursion schemes are very general and expressive structures, they can be viewed as natural extensions of model checkers based on pushdown automata or finite-state systems. In the past decade, there have been substantial advances that have made higher-order model checking a practical endeavor, accompanied by realistic implementations.
2. Liquid Types combine facets of classic type inference techniques found in functional language with predicate abstraction techniques used in model checkers to automatically infer dependent type refinements with sufficient precision to prove useful safety properties about higher-order functional programs. Implementations of this approach found in OCaml, SML, and Haskell have demonstrated that it is indeed possible to verify non-trivial program properties without significant type annotation burden.
3. There have been substantial advances in the development of new languages built around rich dependent type systems that enable the expression and static verification of rich safety and security specifications and properties of higher-order programs. These include the languages that support mechanized proof assistants like Coq, Agda, or Epigram, languages like F\* geared towards secure distributed programming, libraries like Ynot that encapsulate imperative (stateful) features and Hoare-logic pre/post-conditions within a higher-order dependent type framework, and features such as GADTs and type classes found in languages like GHC.
4. Polyvariant control-flow analyses like CFA2 or higher-order contracts are two techniques that facilitate verification and analysis of dynamic higher-order languages. CFA2 adopts pushdown models used in program analyses for first-order languages to a higher-order setting, enabling precise matching of call/return sequences. Higher-order contracts allow runtime verification and blame assignment of higher-order programs and are fully incorporated into Racket, a multi-paradigm dialect of Lisp/Scheme.

---

**Luke**: What are some of the “realistic implementations” of higher-order model checking that you refer to?

---

**Suresh**: The most scalable implementation to date is the tool Preface ([Ramsay et. al, POPL’14](http://hopa.cs.rhul.ac.uk/hopa-2013/files/submissions/hopa2013_submission_4.pdf)) that is capable of handling recursion schemes of several thousand rules (i.e., function definitions). Earlier work such as TRecS ([Kobayshi, POPL’09](http://commonsenseatheism.com/wp-content/uploads/2014/04/Kobayashi-Types-and-higher-order-recursion-schemes-for-verification-of-higher-order-programs.pdf)) have good performance for recursion schemes of up to hundreds of rules.

---

**Luke**: What are your recommended readings on recent interesting work on liquid types and on polyvariant control-flow analyses?

---

**Suresh**:

*Liquid Types*

1. Vazou, Rondon, Jhala, “[Abstract Refinement Types](http://goto.ucsd.edu/~rjhala/papers/abstract_refinement_types.pdf)”, European Symposium on Programming, 2013, p. 209-228
2. Rondon, Kawaguchi, Jhala, “[Low-Level Liquid Types](http://goto.ucsd.edu/~rjhala/papers/low_level_liquid_types.pdf)”, ACM Symposium on Principles of Programming Languages (POPL), 2010, pp. 131-144.
3. Rondon, Kawaguchi, Jhala, “[Liquid Types](http://goto.ucsd.edu/~rjhala/papers/liquid_types.pdf)”, ACM Conference on Programming Language Design and Implementation, 2008, pp. 159-169.
4. Kawaguchi, Rondon, Bakst, Jhala, “[Deterministic Parallelism via Liquid Effects](http://goto.ucsd.edu/~rjhala/papers/deterministic_parallelism_via_liquid_effects.pdf)”, ACM Conference on Programming Language Design and Implementation, 2012, pp. 45-54.
5. Zhu, Jagannathan, “[Compositional and Lightweight Dependent Type Inference for ML](https://www.cs.purdue.edu/homes/zhu103/pubs/vmcai13.pdf)”, Conference on Verification, Model-Checking and Abstract Interpretation, 2013, pp. 295-314.

*Polyvariant Control-Flow Analysis*

1. Midtgaard, “[Control-Flow Analysis of Functional Programs](http://cs.au.dk/~jmi/Midtgaard-CSur-final.pdf)”, ACM Computing Surveys, 2012, 44(3).
2. Earl, Sergey, Might, Van Horn, , “[Introspective Pushdown Analysis of Higher-Order Programs](http://matt.might.net/papers/might2010mcfa.pdf)”, International Conference on Functional Programming, 2012, pp. 177-188.
3. Might, Smaragdakis, Van Horn, “[Resolving and Exploiting the k-CFA Paradox: Illuminating Functional vs. Object-Oriented Program Analysis](http://matt.might.net/papers/might2010mcfa.pdf)”, ACM Conference on Programming Langauge Design and Implementation, 2010, pp. 305-315.
4. Vardoulakis, Shivers, “[CFA2: A Context-Free Approach to Control-Flow Analysis](http://arxiv.org/pdf/1102.3676.pdf)”, Logical Methods in Computer Science, 2011, 7(2).

---

**Luke**: What are some key developments and solutions you hope to see in the next 5-10 years of research into higher-order verification techniques? Which breakthroughs could be most important, if they could be achieved, given our current understanding?

---

**Suresh**: Currently, much of the work on higher-order verification centers around foundations (the definition of expressive models for describing salient properties that arise in HO programs, as is the case with current work in higher-order recursion schemes, dependent type systems, or control-flow analysis), or pragmatics (reducing type annotation burden as in Liquid Types). Going forward, I would expect to see a convergence of these techniques that enable applying new expressive and foundational models at scale.

One important breakthrough would be the integration of these systems into realistic compilers that would go beyond simple safety and property checking, to inform new kinds of verifiable optimizations and program transformations. One can envision using these techniques to use the specifications HO verification techniques support, as a way of realizing tangible code improvements and performance gains in verified compilers and runtime systems. Another important advance would be a deeper understanding of the potential synergistic interplay between higher-order verification techniques and other kinds of specification systems (e.g., session types). The issues here deal with how we would express rich protocols that capture not only safety requirements, but liveness and other kinds of sophisticated modalities.

---

**Luke**: Thanks, Suresh!

The post [Suresh Jagannathan on higher-order program verification](https://intelligence.org/2014/04/30/suresh-jagannathan-on-higher-order-program-verification/) appeared first on [Machine Intelligence Research Institute](https://intelligence.org).


Ruediger Schack on quantum Bayesianism

![Ruediger Schack portrait](http://intelligence.org/wp-content/uploads/2014/04/Schack_w100.jpg) [Ruediger Schack](https://www.ma.rhul.ac.uk/rschack) is a Professor at the [Department of Mathematics](https://www.royalholloway.ac.uk/mathematics/home.aspx) at [Royal Holloway, University of London](http://www.royalholloway.ac.uk). He obtained his PhD in Theoretical Physics at the University of Munich in 1991 and held postdoctoral positions at the Max Planck Institute for Quantum Optics, the University of Southern California, the University of New Mexico, and Queen Mary and Westfield College before joining Royal Holloway in 1995. His research interests are quantum information theory, quantum cryptography and quantum Bayesianism.

**Luke Muehlhauser**: In [Fuchs et al. (2013)](http://arxiv.org/abs/1311.5253), you and your co-authors provide an introduction to [quantum Bayesianism](http://en.wikipedia.org/wiki/Quantum_Bayesianism) aka “QBism,” which you more or less co-invented with [Carlton Caves](http://en.wikipedia.org/wiki/Carlton_M._Caves) and [Christopher Fuchs](http://perimeterinstitute.ca/personal/cfuchs/). But before I ask about QBism, let me ask one of the questions asked of the interviewees in *[Elegance and Enigma: The Quantum Interviews](http://www.amazon.com/Elegance-Enigma-Interviews-Frontiers-Collection/dp/3642208797/)* (including [Fuchs](http://arxiv.org/abs/1207.2141)): “What first stimulated your interest in the foundations of quantum mechanics?”

---

**Ruediger Schack**: I can trace the beginning of my interest in quantum foundations to reading one paper: “[Where do we stand on maximum entropy?](http://bayes.wustl.edu/etj/articles/stand.on.entropy.pdf)” by Ed Jaynes, and one book: *Du Microscopique au Macroscopique* by Roger Balian. Jaynes’s paper introduced me to Bayesian probability theory, and Balian’s book taught me that one can think of quantum states as representing Bayesian probabilities.

---

**Luke**: What is your own summary of the message of QBism? And what do you think its practical import for the world could be?

---

**Ruediger**: In two words the message of QBism is that people matter. According to QBism, quantum mechanics is a theory that any agent can use to organize his experience. More precisely, quantum mechanics permits any agent to quantify, on the basis of his past experiences, his probabilistic expectations for his future experiences. QBism takes measurement outcomes as well as quantum states to be personal to the agent using the theory. Quantum mechanics does therefore not provide an objective, agent-independent description of the world – it rules out a “view from nowhere”. By clarifying thus the role of quantum mechanics and of science in general, QBism avoids all the interpretational difficulties usually associated with quantum foundations. In QBism, there are no objective elements of reality that determine either measurement outcomes or probabilities of measurement outcomes. Rather, every quantum measurement is an action on the world by an agent that results in the creation of something entirely new. QBism holds this to be true not only for laboratory measurements on microscopic systems, but for any action an agent takes on the world to elicit a new experience. It is in this sense that agents – people – have a fundamental creative role in the world.

Any interpretation of quantum mechanics by definition makes the same predictions as quantum mechanics. Nevertheless, I expect QBism to have practical import for the world. By shifting the focus away from interpretations that regard quantum states as real, QBism opens up new possibilities: in the search for a compelling physical principle that would explain the quantum formalism, and in the search for new physics.

---

**Luke**: I heard that Fuchs, at least, thinks that the case for QBism would be more compelling if it turns out that SIC-POVMs (symmetric informationally complete positive operator-valued measures) existed in every finite Hilbert space dimension, which is currently an unsolved question. Is that your understanding as well? If so, what’s the reasoning?

---

**Ruediger**: QBism as an interpretation of quantum mechanics is independent of the existence of SICs and can be formulated without referring to SICs. But QBism is also a program, ultimately with the aim of discovering new physics. A more immediate goal is to find a simple and compelling physical principle underpinning the quantum formalism. Now one of QBism’s central tenets is that a measurement does not reveal a preexisting outcome but results in the creation of something new. In the quantum formalism, this idea finds a simple expression in the fact that the classical probability sum rule does not apply to the – necessarily hypothetical – outcomes of an unperformed experiment. For instance, in the double-slit experiment the probability distribution for the measured particle position on the screen cannot be obtained by adding the weighted probabilities given the particle goes through one or the other slit.

So far this is a purely negative statement. If a SIC exists in every finite Hilbert-space dimension, it turns into a powerful positive statement. If the hypothetical measurement is a SIC measurement, the Born rule takes the form of a very simple modification of the probability sum rule. What is more, from the modified probability sum rule, a large part of the structure of quantum mechanics can be derived. In this picture, instead of the purely negative statement that the probability sum rule cannot be used, we would have a simple physically motivated principle that implies a substantial part of the quantum formalism. In that sense, the existence of SICs in all dimensions would strengthen the case for QBism.

---

**Luke**: Roughly how many people are actively advocating or contributing to QBism? Do you think it’s particularly difficult to draw funding and cognitive talent toward this work because of its theoretical nature, or for other reasons?

---

**Ruediger**: As a matter of fact, the mathematical aspects of QBism (such as the structure of the SICs or the quantum de Finetti theorems) have attracted significant funding over the years. At present, however, only a small number of people are actively contributing to QBism. When QBism holds that science is as much about the scientist as it is about the world external to the scientist, it challenges one of the most deeply held prejudices that most physicists subscribe to. This prejudice is exemplified by the following quote from Landau and Lifshitz: “By measurement, in quantum mechanics, we understand any process of interaction between classical and quantum objects occurring apart from and independently of any observer.” Another commonly held prejudice is that a probability-1 assignment implies the existence of an objective mechanism that brings about the event. Physicists find it very hard to accept the QBist principle that probability-1 judgments are still judgments, like any other probability assignments. Let me finish with a prediction: In twenty-five years when a new generation of scientists have been exposed to QBist ideas, QBism will be taken for granted and quantum foundations will have disappeared as a problem.

---

**Luke**: Thanks, Ruediger!

The post [Ruediger Schack on quantum Bayesianism](https://intelligence.org/2014/04/29/ruediger-schack/) appeared first on [Machine Intelligence Research Institute](https://intelligence.org).


David J. Atkinson on autonomous systems

![David J. Atkinson portrait](http://intelligence.org/wp-content/uploads/2014/04/Atkinson_w384.jpg) David J. Atkinson, Ph.D, is a Senior Research Scientist at the [Florida Institute for Human and Machine Cognition](http://www.ihmc.us/) (IHMC). His current area of research envisions future applications of intelligent, autonomous agents, perhaps embodied as robots, who work alongside humans as partners in teamwork or provide services. Dr. Atkinson’s major focus is on fostering appropriate reliance and interdependency between humans and agents, and the role of social interaction in building a foundation for mutual trust between humans and intelligent, autonomous agents. He is also interested in cognitive robotics, meta-reasoning, self-awareness, and affective computing. Previously, he held several positions at California Institute of Technology, JPL (a NASA Center), where his work spanned basic research in artificial intelligence, autonomous systems and robotics with applications to robotic spacecraft, control center automation, and science data analysis. Recently, Dr. Atkinson delivered an invited plenary lecture on the topic of “Trust Between Humans and Intelligent Autonomous Agents” at the 2013 IEEE/WIC/ACM International Conference on Web Intelligence and Intelligent Agent Technology (WI-IAT 2013). Dr. Atkinson holds a Bachelor’s degree in Psychology from University of Michigan, dual Master of Science and Master of Philosophy degrees in Computer Science (Artificial Intelligence) from Yale University, and the Doctor of Technology degree (d.Tekn) in Computer Systems Engineering from Chalmers University of Technology in Sweden.

**Luke Muehlhauser**: One of your projects at [IHMC](http://www.ihmc.us/) is “[The Role of Benevolence in Trust of Autonomous Systems](http://www.ihmc.us/groups/datkinson/wiki/8b432/The_Role_of_Benevolence_in_Trust_of_Autonomous_Systems.html)“:

> The exponential combinatorial complexity of the near-infinite number of states possible in autonomous systems voids the applicability of traditional verification and validation techniques for complex systems. New and robust methods for assessing the trustworthiness of autonomous systems are urgently required if we are to have justifiable confidence in such applications both pre-deployment and during operations…  The major goal of the proposed research is to operationalize the concept of benevolence as it applies to the trustworthiness of an autonomous system…
>
>

Some common approaches for ensuring desirable behavior from AI systems include testing, formal methods, hybrid control, and simplex architectures. Where does your investigation of “benevolence” in autonomous systems fit into this landscape of models and methods?

---

**David J. Atkinson**: First let me point out that test, formal methods and such other techniques that you point out have little to do with ensuring desirable behavior and more to do with avoiding errors in behavior due to design or implementation flaws. These are not equivalent. Existing techniques improve reliability, but that is only one component of trust and it only concerned with behavior “as designed”. Furthermore, as your quote from my material points out, when it comes to the near-infinite state spaces of autonomy, those “common approaches” are inherently limited and cannot make the same strong claims regarding system behavior that they could with machines where the envelope of behavior could be completely known.

Therefore, I chose to look for answers regarding trust, trustability and trustworthiness in the operations phase of the autonomous system lifecycle because it is here, not in testing and evaluation, where the full complexity of autonomous behavior will manifest itself in response to the uncertainty, dynamics, and real-world complexity that is its major strength. My approach is to focus is on the applicability of human interpersonal trust to operation of autonomous systems. The principle reasons for this are 1) Autonomy is limited — humans make the decisions to rely upon an intelligent agent, subject to a variety of individual and situational factors and constraints, and; 2) Eons of evolution have created a reasonably good mechanism in humans for trust. It is fundamental to every human social transaction and it works. Beyond reliability, studies in multiple disciplines have shown that people want evidence of capability, predictability, openness (aka transparency), and safety before granting a measure of trustworthiness to a machine. Key questions revolve around the nature of that evidence, how it is provided, and what inferences can be reasonably made. People use both cognitive and affective mental processes for evaluating trustworthiness.  My central claim is that if we can reverse engineer these mechanisms and make intelligent autonomous agents (IATs) that are compliant with the human trust process (two big ifs), then we will have created a new way for humans to have trusting relationships with they machines they rely upon. It will be a transition from seeing IATs as tools to treating them as partners.

Benevolence is interesting for a couple of reasons. First, it is a complex attribution built upon a structure of beliefs about another person that include good will, competency, predictability, lack of a hidden agenda, agency and other beliefs, most which are likely to play a role in many kinds of interactions between human and IAT. Second, just looking at that list of beliefs makes it a hard problem, although my colleagues think people will have no problem attributing agency to a machine. Third, there are important applications of IATs, perhaps embodied as robots, where the unique psychology of benevolence is critical to success. For example, disaster rescue. It is long known that disaster victims have a unique psychology borne of stress, fear and the psycho-physiological effects these evoke. Human first responders undergo special training on victim psychology. One of the reasons is that a victim, justifiably afraid for their life, may be very reluctant to trust a rescuer and without trust there is no cooperation and rescue can become very difficult. Benevolence seems to be a part of the trust that is required. Today, we have no idea whatsoever whether a real disaster victim will trust and cooperate with a robot rescuer.

Circling back to your question about “fitting in to the landscape of models and methods”, the ultimate goal of my research is to formulate design requirements, interaction methods, operations concepts, guidelines and more that, if followed, will result in an IAT that can itself engender well-justified human trust.

---

**Luke**: As you say, formal methods and other techniques are of limited help in cases where we don’t know how to formulate comprehensive and desirable design requirements, as is often the case for autonomous systems operating in unknown, dynamic environments. What kinds of design requirements might your approach to trustworthy systems suggest? Would these be formal design requirements, or informal ones?

---

**David**: There will certainly be gaps in what we can do today regarding the specific requirements related to unknown, dynamic environments. One of our objectives is to narrow those gaps so precise questions can be studied. It is one thing to wave hands and moan about uncertainty, and another entirely to do something about it.

Generally speaking, we are working towards formal specification of the traditional types of requirements: Functional, Performance, Design Constraint, and Interface (both Internal and External). By “formal”, I mean “complete according to best practices”. I do not mean “expressed in formal logic or according to a model-based language” — that is a step beyond. Requirements may be linked in numerous ways to each other forming a directed graph (hopefully with no cycles!). Relationships include Source, Required-By, Depends-upon and so forth.

For example, an attribution of benevolence by a human “Trustor” requires a belief (among numerous others) that the candidate “Trustee” (the autonomous system) has “no hidden ill will”. This is a very anthropomorphic concept that some might scoff at, but studies have demonstrated its importance to attribution of benevolence and so it is a factor we must reckon in designing trustworthy and trustable autonomous systems. But what does it even mean?

Just to give you an idea of how we are breaking this down, here are some of the derived requirements. I should emphasize that it is very premature to make any claims about the quality or completeness of what we done with requirements engineering thus far — mostly that work is on the schedule for next year so I’ll only give you the types and titles: We will have as a matter of course a generic Level 1 Functional requirement to “Provide Information to Human User”. Derived from this is an Interface requirement something like “Volunteer Information: The Autonomous System shall initiate communication and provide information that is import to humans” (actually, that is two requirements). This in turn is elaborated by a number of Design Constraints such as “Disposition: The Autonomous System shall disclose any disposition that could result in harm to the interests of a human”. These Design Constraints are in turn linked to numerous other detailed requirements such as this Performance Requirement “Behavior – Protective: The Autonomous System shall recognize when its behavior could result in harm to the interests of a human”. A designer of an autonomous system will recognize a number of very hard problems in this simple example that need to be solved to effectively address the hypothetical need for a benevolent autonomous system.. Our focus in this project is on “what” needs to be done, not “how” to do it.

Without a doubt, this requirements process will generate plenty of questions that will require further study. It is also likely that some requirements may conflict and any particular application of autonomy must do tradeoff studies to prioritize. Nevertheless, our goal is to spell out as much as we can. We will differentiate mandatory requirements from goals or objectives. Where it is possible, we will allocate requirements to particular elements of an autonomous system’s architecture, such as “goal selection mechanism” (where issues relating to prioritization may arise and affect predictability and therefore trust). For every requirement, we will provide a rationale with links to studies and empirical data or other discussion that can be used in analysis. That part is very high priority in my mind. Too many times I have encountered requirements where it is impossible to understand how they were derived. I’d also like to take a risk-driven view of the requirements so that individual requirements, or groups of requirements, can be associated with particular risks. That is another area that will have to be quantified by further application-specific analysis. Finally, a good requirement has to be verifiable. There is considerable work to be done on this topic with respect to autonomous systems.

Requirements engineering is a lot of work of course, and the history of large scale system development is replete with horror stories of poor requirements. That’s why I want to express our trust-related requirements formally following requirements engineering standards  the extent possible. From my previous experience at NASA, I know that the less work a project has to do, the more likely it is to to adopt existing requirements. So we are developing our trust-related requirements consciously with the goal of making them easy to understand and easy to reuse. Finally, given the scope of what is required it is likely that we will only be able to go just so far under the auspices of our current project to provide a vector for future work (hint hint to potential sponsors out there!)

---

**Luke**: From your description, this research project seems somewhat interdisciplinary, and the methodology seems less clear-cut than is the case with many other lines of research aimed at similar goals (e.g. some new project to model-check a particular software design for use in robotics). It’s almost “pre-paradigmatic,” in the Kuhnian sense. Do you agree? If not, are there are other research groups who are using this methodology to explore related problems?

---

**David**: Yes, the project is very interdisciplinary but with the primary ones being social and cognitive psychology, social robotics, and artificial intelligence we well as the rigor contributed by solid systems engineering. The relevant content within each of those disciplines can be quite broad. It is not a small undertaking. My hope is that I can plant some memes in each community to help bring them together on this topic of trust and these memes will foster further work. As far as methodology, we are pursuing both theoretical development and experimentation, and trying to be as rigorous as exploratory work of this nature will permit. We have to understand the previous psychological work on human interpersonal trust, and human factors studies on human-automation trust, to find those results that may have important implications for human trust of an intelligent, autonomous agent. The importance of agency is a good example.

We know from psychological studies that an attribution of “free-will”, or more narrowly, the ability to choose, is an important component of deciding whether someone else is benevolent or not. That is, if a person feels the other is compelled to help then the less likely they are to believe that other person is benevolent. Apart from philosophers, most people don’t think very deeply about this. They make a presumption of free-will and then look to see if there are reasons it is limited, for example, is the person just following orders, or required by their profession or social norms to act in a particular way? With machines, we start from the other side: people assume machines have no free-will because they believe machines are (just) programmed. However, there are [some studies](http://scholar.google.com/scholar?q=related:5FS9KTLPVGkJ:scholar.google.com/&hl=en&as_sdt=1,5) that suggest that when machine behavior is complex (enough), and somewhat unpredictable because there are numerous possible courses of action, then people begin to attribute mental states including the ability to choose. My hunch is this is an evoked response of our innate demand to interpret the actions of others in a framework of intentionality. At some point, enough features are present that those evolution-designed heuristics kick in to aid in understanding. WHY that may be, I will leave to the evolutionary socio-biologists.

Back to methodology now: This is a phenomena for which we can design a study involving humans and machines, with systematic variation of various factors to see what qualities a machine actually requires in order to evoke a human attribution of the ability to choose to help. This year we will be conducting just such a study. I hope to begin running participants this summer. We have been rigorous with experimental design, choice of what data to collect and the statistical methods we will use to analyze the results. While we will fudge a little bit on the robot implementation, using simulation in places and “wizard-of-oz” techniques for language interaction for example, the products of the study ought to be recognized as solid science by researchers in multiple disciplines if we do it right. In general, I think this a very hard goal to achieve because each discipline community has certain preferences and biases about what they like to see in methodology before they are convinced. There are a couple of other groups working on social robotics who use this approach, and a very few number of psychologists who are working on human-centered design of automation. I don’t want to start listing names because I’m sure there are others of whom I’m not yet aware. I do know that I am certainly not the first to confront this challenge. Multidisciplinary research of this type is always, in some sense, pre-paradigmatic because it is a struggle for understanding and legitimacy at the boundaries of separate disciplines, not the core. Artificial intelligence has always been multidisciplinary, a strength as well as a weakness as far as more general acceptance among related disciplines. I don’t worry too much about what other people think. I just do what I believe has to be done.

---

**Luke**: What are some concrete outcomes you hope will be achieved during the next 5 years of this kind of research?

---

**David**: As a wise man once said, “It’s hard to make predictions, especially about the future.”

My hopes. This one is concrete to me, but perhaps not what you had in mind: I hope that the value of our approach will be convincingly demonstrated and other research groups and early-career researchers will join in. There is much to be explored and plenty of opportunity to make discoveries that can have a real impact.

On balance, it seems that in many potential application domains the biggest challenge is not mistrust or absence of trust, but excessive trust. It is abundantly clear that people have no trouble trusting machines in most contexts, and, as this is somewhat tied to generational attitudes, the condition will probably increase. Sometimes this leads to over-reliance and complacency, and then surprising (and potentially dangerous) conditions if things go sour. A funny but real example is the driver who set cruise control on his RV on a long stretch of straight highway, got out of the driver’s seat, and went in back to make a sandwich. You can guess what happened. Effective teamwork and cooperation requires that each team member understands the strengths and limitations of the others so under- and over-reliance do not occur. Intelligent, autonomous teammates need that same ability to participate effectively in this essential team-building process — a process that takes time and experience to build mutual familiarity. We will contribute to that solution.

I believe our work will lead directly to an understanding of When, Why, What, and (some of) How a machine needs to interact with human teammates to inoculate against (and/or correct for) under- and over-reliance. This technology is key for solving what many people claim is a lack of transparency in intelligent systems. It will help “users” to better understand the competency of a machine (the most important quality) in a given context consisting of dynamic situational factors, tasks and goals. This will in turn increase predictability (another important quality) and thereby help mitigate concerns about risks and safety. Ultimately, a deep solution that is broadly applicable will require a higher degree of machine meta-reasoning and self-awareness than we can engineer today, but this is an area of active research where useful results ought to be appearing more and more frequently. (The field of cognitive (developmental) robotics is very exciting.) However, I do expect concrete and useful results for early applications in some semi-structured task domains. A few examples of domains containing “low hanging fruit” for applications are transportation (e.g., autonomy-assisted driving, long haul trucking), healthcare (patient monitoring, therapy assistance, assisted living), and some defense-related applications. My group is actively working towards all of these possibilities. I don’t want to leave you with the impression that creating effective applications will be easy because many hard basic research challenges remain, and we will undoubtedly discover others when we start to transition the technology into real-world applications. Nevertheless, I’m optimistic!

---

**Luke**: Thanks, David!

The post [David J. Atkinson on autonomous systems](https://intelligence.org/2014/04/28/david-j-atkinson/) appeared first on [Machine Intelligence Research Institute](https://intelligence.org).