![Thomas Bolander portrait](http://intelligence.org/wp-content/uploads/2014/04/Bolander_284.jpg) [Thomas Bolander, Ph.D.](http://www.imm.dtu.dk/~tobo/), is associate professor at DTU Compute, Technical University of Denmark.

He is doing research in logic and artificial intelligence with primary focus on the use of logic to model human-like planning, reasoning and problem solving.

Of special interest is the modelling of social phenomena and social intelligence with the aim of creating computer systems that can interact intelligently with humans and other computer systems.

**Luke Muehlhauser**: [Bolander (2003)](http://www2.mat.dtu.dk/publications/phd-thesis/2003/phd-2003-ThomasBolander.pdf) and some of your subsequent work studies paradoxes of [self-reference](http://plato.stanford.edu/entries/self-reference/) in the context of logical/computational agents, as does e.g. [Weaver (2013)](http://arxiv.org/abs/1312.3626). Do you think your work on these paradoxes will have practical import for AI researchers who are designing computational agents, or are you merely using the agent framework to explore the more philosophical aspects of self-reference?

---

**Thomas Bolander**: First of all, let me explain why I even started to study self-reference in the context of logical/computational agents. In [my PhD](http://www2.mat.dtu.dk/publications/phd-thesis/2003/phd-2003-ThomasBolander.pdf) I was working on how to formalise agent introspection, that is, agents representing and reasoning about their own mental states. One of the main motivations for this is that agents need introspection in order to understand their own role in social contexts, e.g. to ensure not to “be in the way of others” and to reason about how you can help other agents to achieve their goals. Another motivation is that introspection is essential in learning: realising the shortcomings of one’s own knowledge and routines is required for a deliberate, goal-directed approach to gaining new knowledge and improve ones problem-solving skills.

Introspection can be into one’s own knowledge, beliefs, intentions, plans, etc. In my PhD I focused on knowledge. Representing a propositional attitude such as knowledge in a logical formalisms can be done in two distinct ways, either by a predicate or by a modal operator. In most logical settings, the predicate solution is more expressive than the modal operator solution, but with high expressivity sometimes also comes trouble like paradoxical sentences leading to inconsistency. When formalising knowledge as a predicate in a first-order predicate logic, assuming a fairly reasonable, though idealised, set of properties (axioms) of knowledge leads to trivialising inconsistency (this is Montague’s theorem, see e.g. my entry on [Self-reference](http://plato.stanford.edu/entries/self-reference/) in the [Stanford Encyclopedia of Philosophy](http://plato.stanford.edu/)). Inconsistency arises here because the logic becomes sufficiently expressive to express paradoxical self-referential sentences concerning knowledge like “this is sentence is not known”. To remedy this problem, there are several possible routes to take: 1) move to a non-classical logic, e.g. a paraconsistent one; 2) restrict the set of axioms of knowledge or their applicability; 3) choose a modal operator approach instead.

In my PhD, I primarily considered option 2. Today my preferred option is 3, using modal logic instead of first-order predicate logic. Option 3 seems to be the currently most preferred option in both the logic and AI communities, with massive amounts of work on epistemic logic, temporal epistemic logic and dynamic epistemic logic based on modal logic. So why do people choose the modal logic approach? Well, one reason is certainly to avoid the problems of self-reference and inconsistency just mentioned. Another is that the modal logic approach often has lower computational complexity, which is essential for practical applications. So in a sense, to answer the question of “whether these paradoxes will have a practical import for AI researchers”, a first answer could be: they already have. They already have in the sense that people shifted attention to other logics where these problems don’t appear.

This answer is of course not entirely satisfactory, though. The way modal logic approaches solve problems of self-reference is by choosing logics in which self-referential sentences can not even be expressed. So the next question becomes: Are we going to miss them (the self-referential sentences)? Do they have an important role to play in reasoning that will eventually be required to build computational agents with strong introspective capabilities? This question is much harder to answer. The modal approach implies that knowledge becomes explicitly stratified. On the modal approach one can have first-order knowledge (knowledge about ontic facts), second-order knowledge (knowledge about knowledge about ontic facts), third-order knowledge (knowledge about second-order knowledge) etc., but there is no fixed-point allowing knowledge statements to refer to themselves. This stratification seems to be cognitively feasible, however. From studies in cognitive science, in particular in the so-called false-belief tasks, it is known that children only start to master second-order knowledge around the age of 4, and only third-order knowledge around the age of 7-8. And even adults seldom reason beyond third-order or at most fourth-order. This seems to talk in favour of some kind of explicit stratification also in the case of humans, and with a fairly fixed limit on the depth of reasoning. This picture fits better with the modal approach, where the stratification is also explicit (modal depth/depth of models). Furthermore, in the modal approach it is more straightforward to mimic limits on the depth of reasoning. Of course, having a fixed limit on one’s depth of reasoning means that there will be certain problems you can’t solve, like [the muddy children puzzle](http://sierra.nmsu.edu/morandi/coursematerials/MuddyChildren.html) with more children than your depth limit. But even within such limitations, the experience from human cognition suggests that it is still possible to do quite advanced social cognition and introspection.

So maybe the self-reference problems that are likely to follow from choosing a non-stratified approach are indeed more relevant in other areas such as the foundation of mathematics (set theory, incompleteness theorems) and the foundations of semantics (theories of truth) than for computational agents. I still find it a bit surprising, though, that I am more happy to accept an explicitly stratified approach to propositional attitudes of agents than as a solution to the self-reference problems in set theory: In set theory, I find the non-stratified approach of [Zermelo-Fraenkel set theory](http://en.wikipedia.org/wiki/Zermelo%E2%80%93Fraenkel_set_theory) much more reasonable than the stratified approach of [type theory](http://en.wikipedia.org/wiki/Type_theory). But even if self-reference is not going to play a big practical role in the development of computational agents, it will probably always lurk in the background and affect people’s choices of formalisms – as well as play an important foundational role in the same way it has in mathematics.

---

**Luke**: What are some examples of work that addresses (roughly) the kind of agent introspection you discuss in your thesis, but uses the modal logic approach to the challenge, or some other stratified approach?

---

**Thomas**: Most of the examples I give in the thesis involve quantifying over knowledge or beliefs, like “I believe that some of my beliefs are false” or “I believe that Alice knows more about X than I do”. To quantify over knowledge or beliefs in the operator approach (modal logic approach), one needs at least first-order modal logic. When I left the predicate approach for the operator approach after my PhD, I essentially went all the way “down” to propositional modal logic, so I am not up to date with the uses of first- or higher-order modal logic for representations of introspective knowledge. I can give a couple of relevant references, but they are not very recent. One is the situation calculus extended with a modal knowledge operator as presented in the book *[The Logic of Knowledge Bases](http://smile.amazon.com/Logic-Knowledge-Bases-Hector-Levesque/dp/0262122324/)* by Levesque & Lakemeyer (MIT Press, 2001). Another one is the book *[Reasoning about Rational Agents](http://smile.amazon.com/Reasoning-Rational-Intelligent-Robotics-Autonomous/dp/0262515563/)* by Wooldridge (MIT Press, 2000). Both of these presents logics that extend first-order predicate logic by, among others, a knowledge operator, and they investigate the use of these logics as the basis for computational agents.

My choice of “downgrading” to propositional modal logic was not because I don’t find the quantifying over beliefs non-interesting or non-important, but simply because I started to become very interested in the dynamics of knowledge, i.e., how the knowledge of an agent changes when an action is executed or an exogenous event occurs. This was essential for my goal of being able to construct planning agents with higher-order reasoning capabilities. I found a logical framework that suited my needs perfectly, dynamic epistemic logic (DEL). Since DEL is mostly studied in the context of propositional modal logic, this is where I currently find myself working. To my knowledge there is in only one paper on first-order dynamic epistemic logic, “[Dynamic term-modal logic](http://www.philos.rug.nl/~barteld/kcpdtml.pdf)” by Barteld Kooi.

---

**Luke**: Are there examples of self-reflective computational agents (of the sort we’ve been discussing) in current industrial or governmental use, or is this work still entirely occurring at a more theoretical stage?

---

**Thomas**: To my knowledge it is still at the theoretical stage. However, its importance is becoming increasingly appreciated also among more applied AI researchers. As mentioned in my previous answer, I am now looking into epistemic planning: planning agents with higher-order reasoning capabilities (agents reasoning about the knowledge of themselves and other agents). The automated planning community is starting to become very interested in this. Bernhard Nebel, who is a very prominent figure in automated planning and robotics, even called epistemic planning the “next big thing”. The reason is that planning formalisms and systems have to become general and expressive enough to deal with robots acting in uncertain environments, and environments containing other agents (multi-agent systems). This calls for epistemic reasoning in such systems.

There is also an increased awareness of the importance of social intelligence in AI systems, if these systems are to efficiently communicate and cooperate with humans. This could be in something like hospital robots (e.g. the [TUG robot](http://www.sciencedirect.com/science/article/pii/S0262407909632474), which is very well-known for its current *lack* of social intelligence [Colin Barras, *New Scientist*, vol. 2738, 2009]), intelligent personal assistants like Siri on iPhone or Google Now on Android, and earthquake rescue robots that have to work in mixed teams with humans.

But integrating the existing formalisms for introspection and social intelligence into e.g. planning systems and then embed all of that into a robot is something which is still in its infancy. The most successful attempt up to now is probably the social robot bartender by Ron Petrick and co-authors [Petrick & Foster, ICAPS, 2013][1](https://intelligence.org/2014/04/13/thomas-bolander/#footnote_0_10963 "Petrick, R. and Foster, M. Planning for Social Interaction in a Robot Bartender Domain. ICAPS 2013. Slides available."). They won a best paper award at the major international planning competition, ICAPS, for this last summer. Their underlying logical formalism is a restricted version of first-order epistemic logic (modal operator approach). The logic is restricted in many important ways in order to quickly get something with reasonable computational performance. The full logics of introspection and higher-order reasoning are still too expressive to be computationally feasible, and currently more work is needed to find the best trade-off between expressivity and computational efficiency that will make higher-order reasoning agents more attractive to the more applied researchers and the industry.

---

**Luke**: In general, what kinds of heuristics do you and your colleagues use to decide how to proceed with theoretical research in a broad problem category like “self-reflective reasoning in computational agents,” which may be 10-30 years from practical application? How do you decide which subproblems to work on, and when to pivot to a different sub-problem or a different approach? And how willing are grantmakers to fund that kind of work, in your experience?

---

**Thomas**: Since I’m using logical formalisms to try to capture “self-reflective reasoning”, any development of a new “theory” has a number of standard and clearly defined steps: 1) You define a suitable logical language and its semantics; 2) You define suitable calculi for reasoning within the logic (proof theory); 3) You give examples to show that your formalisms can capture important aspects of self-reflective reasoning; 3) You investigate the computational complexity and relative expressive strength of your logic; 4) You implement the calculi for reasoning, and optimise them if scalability is important for your proof of concept. Of course there are many choices to be made when you define your logical language, but this is usually driven by concrete examples: I want to be able to express these kinds of sentences or these lines of reasoning. Or, oppositely: I don’t want my language to be able to express this, because it is known to lead to inconsistency or undecidability.

For instance, in my new work, a source of inspiration has been the false-belief tasks used in cognitive psychology to test the [Theory of Mind](http://en.wikipedia.org/wiki/Theory_of_mind) (ToM) of humans. Theory of Mind is the ability to attribute mental states – beliefs, desires, intentions, etc. – to oneself and others. A goal for my logical formalism is that it can robustly deal with any of the existing false-belief tasks used on humans, and some suitably general closure of that set. This naturally drives research activities into finding logics that are suitable for representing that kind of reasoning in a natural way. It might be 20-50 years down the line to see e.g. hospital robots have any kind of general Theory of Mind, but this is also exactly because our understanding of Theory of Mind reasoning and how to represent it formally and in a computer is still rather limited. First we need to understand the concept, then we need to find out how to formalise it, then we need to tame the inherent computational complexity, and then finally we are hopefully able to implement it in a practical application. This is the process that e.g. [description logics](http://en.wikipedia.org/wiki/Description_logic) have been going through. Description logics are logics for representing and reasoning about terminological knowledge. Today reasoning calculi for description logic have been commercialised and are being used e.g. on large medical databases, but it was obviously a long process since the early days of description logic (description logics are in fact also a type of modal logics, more precisely a type of hybrid logics).

There is of course also a risk that what we are doing theoretically about self-reflective reasoning in computational agents will *never* have any practical impact. This is probably why some people start in the other end, and insist that everything they produce should from the outset be implemented in, say, a working robot. However, if you do this, you face another set of problems. Working on practical implementations of robots before the underlying concepts and mechanisms have been fully understood can often lead to solutions that are somewhat ad-hoc and far from being general and robust. There are e.g. researchers who have built robots able to pass certain false-belief tasks, but they can only pass first-order false-belief tasks, and it is not clear how the architecture can be generalised.

In terms of grants, there are always more money in applied research than in theoretical/foundational research. I don’t think getting funding of theoretical work on self-reflective reasoning in computational agents is any easier or harder than other foundational work.

---

**Luke**: You give the example of description logics, which I take it were developed several decades before they saw significant commercial application? What other examples of that kind are you aware of, where there was some early theoretical work inspired by fairly abstract motivations (“In the future, we’ll want systems that do X-ish type things, so let’s build some toy models of X and see where things go…”) that wasn’t commercialized until a decade or more later?

---

**Thomas**: Description logics go back to the beginning of the 80s. Efficient reasoners with good practical performance on large databases described in expressive description logics started appearing in the beginning of the 00s. Reasoners for these expressive description logics have very bad worst-case complexities (e.g. 2EXP), so it has been quite impressive, and surprising, to see how well they scale to large databases.

With respect to other examples, I think that most of the genuinely new theories in AI start out with only being applied to toy examples. Also, almost all of mathematics falls into the category of “theoretical work inspired by fairly abstract motivations”. If you e.g. look at all the mathematics needed to construct a physics engine for a modern 3D computer game, then none of this was originally made with computer games in mind, and most of it was developed without any applications in mind at all. In AI, of course, we tend to always have certain applications or sets of applications in mind, and we aim at shorter time intervals from theoretical developments to commercialisation. The example of the physics engine is about creating mathematical models of real-world physical phenomena and then implement these models in a computer. AI is not that different from this. AI is also about creating mathematical models of some part of our reality, but in this case it is mathematical models of (human) reasoning. There is no doubt that reaching this goal involves, as in the case of physics engines, significant work at all levels from the most theoretical foundations to the most applied implementation-oriented stuff.

I think it is dangerous to dismiss any new ideas in AI that at first only applies to toy examples. That might be a necessary first step, as was e.g. the case in description logics. But it is equally dangerous to forget all about scalability, and think that “this is not my business to worry about”. If you don’t worry at all about scalability and applicability, then you are certainly much more likely to come up with ideas that will never scale or be applied. The researchers I respect the most are those who help make the ends meet: theoreticians with strong knowledge and intuition about applications, and applied researchers with strong theoretical knowledge and interest in applying new theory.

---

**Luke**: Thanks, Thomas!

---

1. Petrick, R. and Foster, M. [Planning for Social Interaction in a Robot Bartender Domain](http://homepages.inf.ed.ac.uk/rpetrick/papers/icaps2013.pdf). [ICAPS 2013](http://icaps13.icaps-conference.org/). [Slides available](http://homepages.inf.ed.ac.uk/rpetrick/papers/icaps2013-slides.pdf).

The post [Thomas Bolander on self-reference and agent introspection](https://intelligence.org/2014/04/13/thomas-bolander/) appeared first on [Machine Intelligence Research Institute](https://intelligence.org).


![Jonathan Millen portrait](http://intelligence.org/wp-content/uploads/2014/04/Jonathan_Millen.jpg) [Jonathan Millen](http://jonmillen.com) started work at the [MITRE Corporation](http://www.mitre.org/) in 1969, after graduation from [Rensselaer Polytechnic Institute](http://www.rpi.edu/) with a Ph.D. in Mathematics. He retired from MITRE in 2012 as a Senior Principal in the Information Security Division. From 1997 to 2004 he enjoyed an interlude as a Senior Computer Scientist in the [SRI International Computer Science Laboratory](http://www.csl.sri.com/). He has given short courses at RPI Hartford, University of Bologna Summer School, ETH Zurich, and Taiwan University of Science and Technology. He organized the [IEEE Computer Security Foundations Symposium](http://www.ieee-security.org/CSFWweb/) (initially a workshop) in 1988, and co-founded (with S. Jajodia) the [Journal of Computer Security](http://www.iospress.nl/journal/journal-of-computer-security/) in 1992. He has held positions as General and Program Chair of the IEEE Security and Privacy Symposium, Chair of the IEEE Computer Society Technical Committee on Security and Privacy, and associate editor of the ACM Transactions on Information and System Security.

The theme of his computer security interests is verification of formal specifications, of security kernels and cryptographic protocols. At MITRE he supported the DoD Trusted Product Evaluation Program, and later worked on the application of [Trusted Platform Modules](http://en.wikipedia.org/wiki/Trusted_Platform_Module). He wrote several papers on information flow as applied to covert channel detection and measurement. His 2001 paper (with V. Shmatikov) on the Constraint Solver for protocol analysis received the SIGSAC Test of Time award in 2011. He received the ACM SIGSAC Outstanding Innovation award in 2009.

**Luke Muehlhauser**: Since you were a relatively early researcher in the field of covert channel communication, I’d like to ask you about the field’s early days, which are usually said to have begun with [Lampson (1973)](http://www.cs.umd.edu/%7Ejkatz/TEACHING/comp_sec_F04/downloads/confinement.pdf). Do you know when the first covert channel attack was uncovered “in the wild”? My impression is that Lampson identified the general problem a couple *decades* before it was noticed being exploited in the wild; is that right?

---

**Jonathan Millen**: We might never know when real covert channel attacks were first noticed, or when they first occurred. When information is stolen by covert channel, the original data is still in place, so the theft can go unnoticed. Even if an attack is discovered, the victims are as reluctant as the perpetrators to acknowledge it. This is certainly the case with classified information, since a known attack is often classified higher than the information it compromises. The only evidence I have of real attacks before 1999 is from Robert Morris (senior), a pioneer in UNIX security, and for a while the Chief Scientist of the National Computer Security Center, which was organizationally within NSA. He stated at a security workshop that there had been real attacks. He wouldn’t say anything more; it was probably difficult enough to get clearance for that much.

---

**Luke**: From your own perspective, what are some of the most interesting developments in covert channel communication since you wrote “[20 Years of Covert Channel Modeling and Analysis](http://commonsenseatheism.com/wp-content/uploads/2014/03/Millen-20-Years-of-Covert-Channel-Modeling-and-Analysis.pdf)” in 1999?

---

**Jonathan**: That’s a tough question, for several reasons. Like many research areas that have been in existence for many years, covert channel research has splintered into several specialties. Some of the subtopics are noninterference models, language-based information control via type theory, steganography, and so on.

While specialization is necessary for deep progress, I wish there were more attention given to the pragmatic “so what” questions about covert channels. For example, information flow security is often defined in terms of noninterference, which can, unfortunately, be defined in several subtly different ways. All of them rely on some form of behavioral equivalence, a strong condition guaranteeing that an unauthorized observer cannot tell when a more privileged user is present. Even one bit, or a fraction of a bit, of information about the protected party’s data or actions is considered too much. Yet, what we really want to know is whether a recurrent channel can be set up, yielding a large or unbounded amount of information. Furthermore, information in the Shannon sense may or may not be useful. One paper that attracted my attention is “[Quantifying information flow with beliefs](https://www.cs.cornell.edu/andru/papers/jbelief.pdf)”, by Clarkson, Myers, and Schneider, 2009, because it dealt with the accuracy of information.

Another pragmatic issue is the trustworthiness of the system functionality. In general, I am distrustful of security approaches that depend on access control software or the design of the language in which it is written. Even the firmware and hardware below the operating system kernel is suspect. In the last few years, my main research interest was in trusted computing, as supported by trusted platform modules, with their capability — available but largely unused in modern platforms — to cryptographically check the integrity of system software by a bootstrap sequence starting with the tamper-resistant TPM. And still, having done that, we know only that the system software is genuine, not that it is correct or free from covert channels. It is encouraging that work in software and hardware verification continues, but commonly used systems are still beyond its reach.

Complexity is the enemy of security. The more you have to depend on, the less likely it is that you can understand it well enough to exclude vulnerabilities. A couple of corollaries: the more complex your security policy is, the less robust it is, because more complex software is needed for it. And the more specialized a system component is to support security, the more attractive it is as a target for an attack. I suppose that the optimistic view of this is that if we use a typically complex modern system, covert channels are the least of our worries!

---

**Luke**: From your many decades of experience in computer security, do you know of cases where someone was worried about a computer security or safety challenge that wasn’t imminent but maybe one or two decades away, and they decided to start doing research to prepare for that challenge anyway — e.g. perhaps because they expected the solution would require a decade or two of “serial” research and/or engineering work, with each piece building on the ones before it, and they wanted to be prepared to meet the challenge near when it arrived? Lampson’s early identification of the “confinement problem” looks to me like it might be one such example, but maybe I’m misreading the history there.

---

**Jonathan**: In computer security, what usually happens is that someone realizes that a vulnerability already exists, but it is not clear how long it will take for a malicious party to take advantage of it. Another factor delaying the onset of relevant research and development is that long-term efforts aimed at removing vulnerabilities are risky and their results may be inadequate or eclipsed by different problems in the future.

If the difficulty is social, such as getting standards updated, the difficulty is real, but the basic engineering knowledge is mostly available early on. For example, the TCP sequence number attack for session hijacking was pointed out by Morris in 1985, taken advantage of by Mitnick in 1995, led to Bellovin’s standards-related recommendations in 1996, and only recently has the availability of encryption-based authentication and IPv6 brought a widespread solution within reach.

If the difficulty is in the engineering, such as the development of security kernels and mandatory access policies to defeat Trojan horses, or the installation and use of TPMs, there is an industry to be moved, one driven by the need for commercial viability of new products. And after all that effort, some vulnerabilities remain.

Computer scientists can help by developing powerful general-purpose tools and techniques with fundamental computer science goals — tools that are not problem-dependent. By this, I mean such things as verification tools and model checkers, as well as encryption algorithms and applications. Such tools can ease the engineering burden when new problems and design ideas emerge. Development of these tools deserve decades of research building on prior work, and those who are capable of conducting this activity successfully should be encouraged and supported.

---

**Luke**: You mention verification tools and model checkers. I was recently speaking to a significant figure in the computer security and safety community (I don’t have permission to give her name) about the common claim that formal methods can catch corner bugs which are unlikely to be caught by testing alone. She mentioned that if the formal methods community has ever subjected formal methods to careful experimentation to demonstrate that verification really does catch bugs that aren’t likely to be found by testing, she wasn’t aware of it. Are you aware of any such demonstrations? If not, how would *you* describe the value in formal methods for safety and security purposes?

---

**Jonathan**: It seems unfair to me to ask for careful experimentation to justify formal methods in comparison to testing. One reason for this that was pointed out when formal methods were being recommended for security kernel verification, is that very few security kernels are actually developed, so any experimental results are not going to be statistically significant.

One experiment that was done in the very beginning of the history of formal specifications was by David Parnas, credited by some with inventing the notion of “formal nonprocedural specification.”[1](https://intelligence.org/2014/04/12/jonathan-millen/#footnote_0_10967 "Parnas (1972).") In fact, many of us referred to such specifications at first as “Parnas specifications”, but he let it be known (through Peter Neumann) that he disapproved of that. Parnas’ experiment was a small classroom experiment designed to investigate, not the efficacy of verification of formal specifications, but rather the plausible idea that specifications determine implementations. Parnas’ classroom experiment, in which several groups were given the same specification to implement, disproved that (or at least cast doubt on that) because different groups implemented their specified task in very different ways, despite the very detailed constraints on input-output behavior in the specification. (I heard the experiment described at a conference presentation by Parnas, but I can’t find the paper or the citation now.)

This result turned out to be useful to me a few years later when DoD sponsors argued that analysis of the formal specifications of security kernels should be classified higher than the programs themselves, even when the analysis proved the absence of access control violations and storage channels, because the specifications allowed readers to deduce features of the implementation that would tell them about timing and other channels. I pointed out that Parnas had given evidence that their fears were not justified.

One indicator of the value of formal methods, model checking in particular, is the fact that it has been used routinely in the design of CPUs, which have rather complicated internal strategies for managing memory and register caching. (One citation I found with a quick Google to confirm this is “[Fifteen years of formal property verification in Intel](http://commonsenseatheism.com/wp-content/uploads/2014/04/Fix-Fifteen-Years-of-Formal-Property-Verification-in-Intel.pdf)”, by Limor Fix, of Intel Research Pittsburgh, published in a 2008 Springer-Verlag book, [*25 Years of Model Checking*](http://www.amazon.com/Years-Model-Checking-Achievements-Perspectives/dp/3540698493/).)

---

**Luke**: You also wrote that “Computer scientists can help by developing powerful general-purpose tools and techniques… [e.g.] verification tools and model checkers, as well as encryption algorithms and applications… Development of these tools deserve decades of research building on prior work, and those who are capable of conducting this activity successfully should be encouraged and supported.”

What are some specific examples of tools for software system safety or security that you wish were receiving more development talent and funding than is currently the case? E.g. probabilistic model-checkers, or verified libraries for program synthesis, or whatever you think is most urgent or underfunded.

---

**Jonathan**: My gut reaction is, all of the above. When it comes to specific tools, each of us has preferences based on what we needed, what we have used, and what is most accessible. When I was at SRI International, I used the PVS theorem prover, and later, SRI’s SAL environment, which hosts several flavors of model checking algorithms. Both of those have benefited from the highest levels of talent; the challenge is that the job is never finished, especially if it is a thriving, well-used system. Funding sources typically emphasize urgent applications or brand new ideas, as they should, but they tend to neglect the steady, unromantic work needed to keep a software suite up to date with respect to the evolution of hardware, operating systems, algorithmic advances, and so on. And tools that support formal methods do not have the same opportunities for income as Windows, Apple’s OS X, and other commercial systems, even with a strong user community.

Incidentally, although I have emphasized general-purpose tools, that does not mean that I am against tools designed for particular applications such as security. After all, I spent a fair amount of time on a protocol security analyzer (called the Constraint Solver). Specialized tools, however, benefit in brevity and readability from being built on top of an expressive language with powerful primitives — for my application, Prolog, specifically SWI-Prolog.

A word of caution, though: it’s one thing to build an analysis tool with a complex system under it, and another to build a supposedly secure system that way. A lesson learned from studying covert channels is that the whole system under the secure interface is a potential source of vulnerabilities. So how can we build secure systems? The main requirement, I think, is simple: don’t let the enemy program your computer (the computer holding your data). But how can we prevent that? That’s the hard part.

---

**Luke**: Thanks, Jonathan!

---

1. [Parnas (1972)](http://commonsenseatheism.com/wp-content/uploads/2014/04/Parnas-A-Technique-for-software-module-specification-with-examples.pdf).

The post [Jonathan Millen on covert channel communication](https://intelligence.org/2014/04/12/jonathan-millen/) appeared first on [Machine Intelligence Research Institute](https://intelligence.org).


Wolf Kohn on hybrid systems control

![Wolf Kohn portrait](http://intelligence.org/wp-content/uploads/2014/04/Kohn_w1300.jpg) [Dr. Wolf Kohn](http://depts.washington.edu/ie/people/kohn) is the Chief Scientist at [Atigeo, LLC](http://atigeo.com/), and a Research Professor in Industrial and Systems Engineering at the University of Washington. He is the founder and co-founder of two successful start-up companies: [Clearsight Systems, Corp.](http://www.clearsightsystems.com/), and Kohn-Nerode, Inc. Both companies explore applications in the areas of advanced optimal control, rule-based optimization, and quantum hybrid control applied to enterprise problems and nano-material shaping control. Prof. Kohn, with Prof. Nerode of Cornell, established theories and algorithms that initiated the field of hybrid systems. Prof. Kohn has a Ph.D. in Electrical Engineering and Computer Science from MIT, at the Laboratory of Information and Decision Systems. Dr. Kohn is the author or coauthor of over 100 referred papes, 6 book chapters and with Nerode and Zabinsky has written a book in Distributed Cooperative inferencing. Dr. Kohn Holds 10 US and international patents.

**Luke Muehlhauser**: You co-founded the field of hybrid systems control with Anil Nerode. Anil gave his impressions of the seminal 1990 Pacifica meeting [here](http://intelligence.org/2014/03/26/anil-nerode/). What were your own impressions of how that meeting developed? Is there anything in particular you’d like to add to Anil’s account?

---

**Wolf Kohn**: The discussion on the first day of the conference centered on the problem of how to incorporate heterogeneous descriptions of complex dynamical systems into a common representation for designing large scale automation. What came almost immediately were observations from Colonel Mettala and others that established as a goal the finding of alternatives to classic approaches based on combining expert systems with conventional control and system identification techniques.

These approaches did not lead to robust designs. More important, they did not lead to a theory for the systematic treatment of the systems DOD was deploying at the time. I was working on control architectures based on constraints defined by rules, so after intense discussions among the participants Nerode and I moved to a corner and came up with a proposal to *amalgamate* models by extending the concepts of automata theory and optimal control to characterize the evolution of complex dynamical systems in a manifold in which the topology was defined by rules of operation, and behavior constraints and trajectories were generated by variational methods. This was the beginning of what we would be defined later on as “hybrid systems.”

---

**Luke**: Which commercial or governmental projects would you name as being among the most significant success stories of the hybrid systems research program, from 1990 to the present day?

---

**Wolf**: There are many applications today that use hybrid systems as the basic technology. These are a few of the ones I am I personally familiar with:

* A demand forecaster and an inventory control and management system being deployed by the Microsoft Dynamics group.
* A battlefield simulator deployed by the Army’s Piccatiny Arsenal.
* A generic people and resource scheduling system deployed by Clearsight Systems.
* A cooperative distributed inference system deployed by Atigeo with applications on medical informatics and smart electric power network management systems.
* A quantum hybrid control system for capturing and storing sunlight being prototyped by Kohn-Nerode LLC.

---

**Luke**: What are the new theoretical developments in hybrid systems of the past 10 years that are most impressive or interesting to you? What kinds of advances do you think we might see in the next 10 years?

---

**Wolf**: For me the most important advances in hybrid systems are in three areas:

1. *Representation*: We have found that the behavior of dynamical systems characterized by multiple heterogeneous models can be effectively characterized by Hamiltonian functions. The interaction with multiple models and the transfer of information from one model to another is defined by the interaction of Hamiltonian forms. This fact allows hybrid systems to be a preferred theory and implementation technology for the development of new control approaches such as mean field agent-based distributed control and gauge theory and most importantly, control and optimization specified and implemented by rules.
2. *Hybrid systems control design*, which allows for the specification of architectural structures and the control requirements as part of the formulation of a control problem. This fuses the physical and empirical data about the dynamic process to be controlled. The control specification and the computational requirements are achieved because control performance specifications and architectural constraints *are* models of the controlled process.
3. *Agent-based distributed control*: Nerode and I, both together and separately have developed a variational theory, based on hybrid systems, for dynamic synchronization of multiple agents participating in the control of a distributed process. The variational theory generalizes a principle in network theory, called Tellegen’s Theorem. The theory provides for active synchronization with no umpire for network interaction between the agents. A version of this theory has been used to implement an agent-based architecture for control, uncertainty management, and learning in several applications. This architecture is known by the acronym MAHCA (Multiple Agent Hybrid Control Architecture).
4. *Metacontrol*: This is a theory for controlling the performance and behavior of *implemented* algorithms. The dynamics in this case is a computational multitask process. The objective is to make it run faster with less memory and with active synchronization between tasks. We are developing a computational hybrid systems theory based on metacontrol. Preliminary implementations of this theory on optimization algorithms has shown very promising results in terms of reduction of compute time, real time synchronization and rule based optimization.

---

**Luke**: I’m particularly interested in the safety challenges presented by the increasingly autonomous AI systems of the future. Self-driving cars are on their way, the U.S. military is working toward [autonomous battlefield robots](http://www.amazon.com/Governing-Lethal-Behavior-Autonomous-Robots-ebook/dp/B008I9YG9G/) of various types, etc. Do you think hybrid systems control, and relatively modest extensions to it, will be sufficient to gain high assurances of the safety for the more-capable autonomous systems we’re likely to have in 10 or 20 years, or do you think other contemporary control and verification approaches have a better shot at addressing that problem, or do you think entirely new approaches will need to be developed?

---

**Wolf**: I will break down my answer in two parts: (1) the autonomy issue and (2) the verification issue.

*Autonomy*: This was one of the central questions brought up by the initial sponsors of hybrid systems (ARPA, NIST, ARO, SAP). Our answer early on came with the following proposition which we implemented in a battlefield dynamics simulator: given a process to be controlled, build a model of desired behavior based on the performance specifications, regulations, and economic operation, and construct the controlled process dynamics. This is a hybrid system; let’s call it S.

Then we build a model, another hybrid system, say C, representing specified safety rules and constraints and hybridize S and C to produce a new Hybrid control system S1. This approach is only successful in semi-closed, quasi-stationary systems.

For the large scale autonomous systems, one needs to allow the hybrid system controller to detect, learn and dispose of un-programmed situations. To do this, a class of hybrid controllers (known as agents in our papers) contain directives that implement Learning from Sensory Data. One example is what we might call “Learning by Failure Predictor and Repair” systems. This particular system operates as follows: a safety agent (or agents) is (are) monitoring the controlled system operation, and infer(s) whether it is operating in a feasible region and what is the likelihood that it is going to leave the feasibility region (a Failure) in the near future (“near” is a concept depending on the controlled system). This likelihood determines the response to the failure: a Repair operation.

Then, another agent application designs a controller that implements the Repair operation. Note this is not necessarily an adaptation of an existing controller. We use the fact that the design procedure of hybrid controllers is itself a hybrid system defining the synthesis approach. We term the resulting design procedure a repair hybrid control. We have used this approach in a prototype microgrid management and control system that is near deployment with considerable success: i.e., high robustness, resiliency, recovery: all this while maintaining good performance.

Another element that agent-based hybrid systems provide is the ability to improve safety via redundancy.

In short, my answer to the first part of the question may be summarized as follows:

Hybrid systems is a first principles platform that allows for the incorporation of safety, learning and repair. What I believe the research in this area should be focused on is how to populate the system with *information* about the application and to allow for heuristics and empirical constraints and rules, and to provide the structural Failure and Repair mechanisms I outlined above. This approach builds on top on existing control techniques but incorporates a new concept of structural adaptation that is essential for the level of autonomy posed in your question.

*Verification*: many researchers have proposed methods for hybrid systems validation and verification prior to deployment. In our approach we are happy to use some of these techniques. Our contribution in this area called for verification of online designs, as discussed above. So, we are developing hybrid systems to model verification principles, with the idea to amalgamate them in near real time to our applications.

---

**Luke**: Do you think our capacity to make systems more autonomous and capable will outpace our capacity to achieve confident safety assurances for those systems?

---

**Wolf**: I believe we have made great progress on increasing autonomy in most of the systems we are designing today. I also believe we have paid far less attention to develop the methods, sensory redundancy principles, and theory of design for safety performance.

Nerode and I are working on developing an advanced magnetic battery using quantum hybrid control methodology. We found that the key ingredients of this battery for it to operate safely are principles of safety that have been used in non-autonomous systems for the last 100 years. We are encoding these principles formally as part of our design algorithms. Perhaps this way may be generalized to obtain acceptable levels of safety.

---

**Luke**: Thanks, Wolf!

The post [Wolf Kohn on hybrid systems control](https://intelligence.org/2014/04/11/wolf-kohn/) appeared first on [Machine Intelligence Research Institute](https://intelligence.org).


MIRI’s April 2014 Newsletter

|  |  |
| --- | --- |
|

|  |
| --- |
|
[Machine Intelligence Research Institute](http://intelligence.org)
 |

 |
|

|  |  |
| --- | --- |
|

|  |
| --- |
|
**Research Updates**
* New technical report: “**[Botworld 1.0](http://intelligence.org/2014/04/10/new-report-botworld/)**.”
* 9 new expert interviews: [Max Tegmark](http://intelligence.org/2014/03/19/max-tegmark/) on the mathematical universe, [Randal Koene](http://intelligence.org/2014/03/20/randal-a-koene-on-whole-brain-emulation/) on whole brain emulation, [Michael Carbin](http://intelligence.org/2014/03/23/michael-carbin/) on integrity properties in approximate computing, [Anil Nerode](http://intelligence.org/2014/03/26/anil-nerode/) on hybrid systems control, [Lyle Ungar](http://intelligence.org/2014/03/26/lyle-ungar/) on forecasting, [Erik DeBenedictis](http://intelligence.org/2014/04/03/erik-debenedictis/) on supercomputing, [Will MacAskill](http://intelligence.org/2014/04/08/will-macaskill/) on normative uncertainty, [Diana Spears](http://intelligence.org/2014/04/09/diana-spears/) on the safety of adaptive agents, and [Paulo Tabuada](http://intelligence.org/2014/04/09/paulo-tabuada/) on program synthesis for cyber-physical systems.

**News Updates**
* We are [actively hiring](http://intelligence.org/careers/) for three positions: math researcher, science writer, and executive assistant. Compensation is competitive and visa are possible.
* We’re also [accepting applications](http://intelligence.org/get-involved/#workshop) on a rolling basis for future MIRI research workshops. Please apply if you’re a curious academic looking for exposure to our material. If you’re accepted, we’ll contact you about potential workshops over time, as we schedule them.
* We published our [2013 in Review: Fundraising](http://intelligence.org/2014/04/02/2013-in-review-fundraising/) post.
* Louie was [interviewed](http://io9.com/why-asimovs-three-laws-of-robotics-cant-protect-us-1553665410) at *io9* about the unsuitability of Asimov’s Three Laws of Robotics for machine ethics.
* Luke was interviewed at *The Information* about MIRI’s research. Unfortunately, the interviewer’s editing introduced several errors: see [here](https://www.facebook.com/lukeprog/posts/10104392165212640?stream_ref=10).

**Other Updates**
* April 11th is your last chance to **[vote for Nick Bostrom](http://www.prospectmagazine.co.uk/worldthinkers/)** and others in *Prospect*‘s World Thinkers 2014 poll.
* Oxford University is hosting an effective altruism conference in July. [Speakers](http://www.gooddoneright.com/#!programme/c1dj9) include Nick Bostrom and Derek Parfit. Register [here](http://www.gooddoneright.com/#!register/c10fk).
* Registration is also open for [Effective Altruism Summit](http://www.effectivealtruismsummit.com/) 2014, in August in the Bay Area.
* Our frequent collaborators at the Future of Humanity Institute are [currently hiring](http://www.fhi.ox.ac.uk/vacancy-population-ethics/) a research fellow in philosophy, for a focus on population ethics.

As always, please don’t hesitate to let us know if you have any questions or comments.
Best,
Luke Muehlhauser
Executive Director
 |

 |

 |
|  |

The post [MIRI’s April 2014 Newsletter](https://intelligence.org/2014/04/10/miris-april-2014-newsletter/) appeared first on [Machine Intelligence Research Institute](https://intelligence.org).


New Report: Botworld

![robot swarm](https://intelligence.org/wp-content/uploads/2014/04/robot-swarm.jpg)
Today MIRI releases a new technical report: “[Botworld 1.0](https://intelligence.org/files/Botworld.pdf)” (pdf) by [recent hires](http://intelligence.org/2014/03/13/hires/) Nate Soares and Benja Fallenstein. The report is a “[literate](http://www.haskell.org/haskellwiki/Literate_programming)” Haskell file, available from [MIRI’s Github page](https://github.com/machine-intelligence).

Soares explains the report on his accompanying [Less Wrong post](http://lesswrong.com/lw/k1o/botworld_a_cellular_automaton_for_studying/), which is also the preferred place for discussion of the report:

> This report introduces *Botworld*, a cellular automaton that provides a toy environment for studying self-modifying agents.
>
>
> The traditional agent framework, used for example in Markov Decision Processes and in Marcus Hutter’s universal agent AIXI, splits the universe into an agent and an environment, which interact only via discrete input and output channels.
>
>
> Such formalisms are perhaps ill-suited for real self-modifying agents, which are embedded within their environments. Indeed, the agent/environment separation is somewhat reminiscent of Cartesian dualism: any agent using this framework to reason about the world does not model itself as part of its environment. For example, such an agent would be unable to understand the concept of the environment interfering with its internal computations, e.g. by inducing errors in the agent’s RAM through heat.
>
>
> Intuitively, this separation does not seem to be a fatal flaw, but merely a tool for simplifying the discussion. We should be able to remove this “Cartesian” assumption from formal models of intelligence. However, the concrete non-Cartesian models that have been proposed (such as Orseau and Ring’s formalism for space-time embedded intelligence, Vladimir Slepnev’s models of updateless decision theory, and Yudkowsky and Herreshoff’s tiling agents) depart significantly from their Cartesian counterparts.
>
>
> Botworld is a toy example of the type of universe that these formalisms are designed to reason about: it provides a concrete world containing agents (“robots”) whose internal computations are a part of the environment, and allows us to study what happens when the Cartesian barrier between an agent and its environment breaks down. Botworld allows us to write decision problems where the Cartesian barrier is relevant, program actual agents, and run the system.
>
>
> As it turns out, many interesting problems arise when agents are embedded in their environment. For example, agents whose source code is readable may be subjected to Newcomb-like problems by entities that simulate the agent and choose their actions accordingly.
>
>
> Furthermore, certain obstacles to self-reference arise when non-Cartesian agents attempt to achieve confidence in their future actions. Some of these issues are raised by Yudkowsky and Herreshoff; Botworld gives us a concrete environment in which we can examine them.
>
>
> One of the primary benefits of Botworld is concreteness: when working with abstract problems of self-reference, it is often very useful to see a concrete decision problem (“game”) in a fully specified world that directly exhibits the obstacle under consideration. Botworld makes it easier to visualize these obstacles.
>
>
> Conversely, Botworld also makes it easier to visualize suggested agent architectures, which in turn makes it easier to visualize potential problems and probe the architecture for edge cases.
>
>
> Finally, Botworld is a tool for communicating. It is our hope that Botworld will help others understand the varying formalisms for self-modifying agents by giving them a concrete way to visualize such architectures being implemented. Furthermore, Botworld gives us a concrete way to illustrate various obstacles, by implementing Botworld games in which the obstacles arise.
>
>
> Botworld has helped us gain a deeper understanding of varying formalisms for self-modifying agents and the obstacles they face. It is our hope that Botworld will help others more concretely understand these issues as well.
>
>

The post [New Report: Botworld](https://intelligence.org/2014/04/10/new-report-botworld/) appeared first on [Machine Intelligence Research Institute](https://intelligence.org).


![Paulo Tabuada portrait](http://intelligence.org/wp-content/uploads/2014/04/Tabuada_w460.jpg) [Paulo Tabuada](http://www.seas.ucla.edu/~tabuada/) was born in Lisbon, Portugal, one year after the Carnation Revolution. He received his “Licenciatura” degree in Aerospace Engineering from Instituto Superior Tecnico, Lisbon, Portugal in 1998 and his Ph.D. degree in Electrical and Computer Engineering in 2002 from the Institute for Systems and Robotics, a private research institute associated with Instituto Superior Tecnico. Between January 2002 and July 2003 he was a postdoctoral researcher at the University of Pennsylvania. After spending three years at the University of Notre Dame, as an Assistant Professor, he joined the Electrical Engineering Department at the University of California, Los Angeles, where he established and directs the [Cyber-Physical Systems Laboratory](http://www.cyphylab.ee.ucla.edu/).

Paulo Tabuada’s contributions to cyber-physical systems have been recognized by multiple awards including the NSF CAREER award in 2005, the Donald P. Eckman award in 2009 and the George S. Axelby award in 2011. In 2009 he co-chaired the [International Conference Hybrid Systems: Computation and Control](http://hscc-conference.org) (HSCC’09) and in 2012 he was program co-chair for the 3rd IFAC Workshop on Distributed Estimation and Control in Networked Systems (NecSys’12). He also served on the editorial board of the IEEE Embedded Systems Letters and the IEEE Transactions on Automatic Control. His latest [book](http://www.springer.com/mathematics/applications/book/978-1-4419-0223-8), on verification and control of hybrid systems, was published by Springer in 2009.

**Luke Muehlhauser**: In “[Abstracting and Refining Robustness for Cyber-Physical Systems](http://arxiv.org/abs/1310.5199),” you and your co-author write that:

> …we present a design methodology for robust cyber-physical systems (CPS) [which]… captures two intuitive aims of a robust design: bounded disturbances have bounded consequences and the effect of sporadic disturbances disappears as time progresses.
>
>

You use an “abstraction and refinement” procedure for this. How does an abstraction and refinement procedure work, in this context?

---

**Paulo Tabuada**: Cyber-physical systems are notoriously difficult to design and verify because of the complex interactions between the cyber and the physical components. Although control theorists have developed powerful techniques for designing and analyzing physical components, say described by differential equations, and computer scientists have developed powerful techniques for designing and analyzing cyber components, say described by finite-state models, these techniques are for the most part incompatible. The latter rely on discrete mathematics while the former rely on continuous mathematics. Our approach is based on replacing all the physical components by cyber abstractions so that all the remaining design and verification tasks can be done in the cyber world. The construction of these abstractions is based on rigorous numerical simulation combined with an analysis of the differential equation models to guarantee that the original physical components and its abstractions are equivalent up to a desired precision. Technically, “equivalent up to a desired precision” means approximately bisimilar and intuitively this means that both models generate the same set of behaviors up to a desired precision.

---

**Luke**: [Last summer](https://excape.cis.upenn.edu/summer-school.html) you gave a four-part tutorial ([p1](https://excape.cis.upenn.edu/documents/Tutorialthree_partone.pdf), [p2](https://excape.cis.upenn.edu/documents/Tutorialthree_parttwo.pdf), [p3](https://excape.cis.upenn.edu/documents/Tutorialthree_partthree.pdf), [p4](https://excape.cis.upenn.edu/documents/Sat_handson_pessoa.pdf)) on program synthesis for cyber-physical systems. For someone who isn’t familiar with program synthesis, can you describe how it’s done in the context of cyber-physical systems, and give an example of such a system that has been implemented?

---

**Paulo**: Program synthesis is already a challenging problem for software only systems. In the context of cyber-physical systems it becomes even more challenging since the objective is to synthesize a program that will make a physical system behave as intended. I am currently working on two projects related to program synthesis. One of these has for objective the synthesization programs that control the behavior of bipedal robots. The very same techniques are being used in the same project to synthesize programs for adaptive cruise control and lane departure control systems in collaboration with Toyota and Ford. To give you an idea of where the challenge lies, let me recall how this project started. My colleagues Jessy Grizzle at UMich and Aaron Ames at Texas A&M came to me and shared their frustration with the programs they developed to control their robots. While they were very satisfied making their robots walk on flat and unobstructed surfaces, they faced large problem on uneven terrain. Their approach was to develop a set of rules, i.e., a reactive program that responds to the stimuli provided by the sensors by determining which actuators should be used and how. Although the rules were developed based on common sense, it soon became clear it was impossible to predict how the execution of these rules would impact the motion of the robot. Moreover, a small change in the rules would lead to completely different and unexpected behavior. Our approach is to construct a finite-state abstraction of the robot dynamics and then to synthesize a reactive program that forces this abstraction to satisfy a desired specification. The synthesis of these reactive programs is done via the solution of a two player game where the synthesized program plays against the robot’s environment and enforces the specification no matter which action the environment takes. In this way, rather than verifying a reactive program, we synthesize one that is guaranteed to be correct by construction.

---

**Luke**: Robots and other AI systems are becoming increasingly autonomous in operation: [self-driving cars](http://en.wikipedia.org/wiki/Autonomous_car), robots that [navigate disaster sites](http://www.darpa.mil/Our_Work/TTO/Programs/DARPA_Robotics_Challenge.aspx), [HFT](http://en.wikipedia.org/wiki/High-frequency_trading) programs that trade stocks quickly enough to “[flash crash](http://en.wikipedia.org/wiki/2010_Flash_Crash)” the market or [nearly bankrupt](http://en.wikipedia.org/wiki/Knight_Capital_Group#2012_stock_trading_disruption) a large equities trader, etc.

How might current AI safety methods (formal verification, program synthesis, simplex architectures, etc.) be scaled up to meet the safety challenges raised for highly autonomous systems operating in unknown, continuous, dynamic environments? Will our capacity to make systems more autonomous and capable outpace our capacity to achieve confident safety assurances for those systems?

---

**Paulo**: Indeed, that is what has been happening so far. Our ability to create large complex systems is far ahead of our understanding of the basic scientific principles ensuring their safe operation. In my opinion, synthesis based approaches have the best chance to scale to the level of existing applications. Formal verification of these systems is extremely hard since the design space is very very large. No formal verification technique can handle the wide variety of systems from such large design space. When using synthesis, however, we can reduce the design space and thus obtain much more structured designs for which formal guarantees can be obtained. What we loose, in turn, is the ability to find all the design solutions. This tradeoff is well understood in other areas of engineering, e.g., brick and mortar or wood framed building construction. I believe that it wont take long until we discover a few synthesis techniques that guarantee safety and correctness of highly autonomous systems, even if these result in somewhat more conservative solutions.

---

**Luke**: You write that “it wont take long until we discover a few synthesis techniques that guarantee safety and correctness of highly autonomous systems, even if these result in somewhat more conservative solutions.” Could you clarify what you have in mind? For example, what might a “conservative solution” look like for a high-assurance program synthesis solution for the software of a self-driving car? (Feel free to give a different example if you prefer.)

---

**Paulo**: By conservative I mean that we will have to restrict the synthesized software so that it falls within a class for which synthesis techniques are available. A simple example is software that can be described by timed-automata. Although it might be convenient to make a decision based on a generic function of several timers, to stay within the timed-automata class we are only allowed to compare timers against constants or compare differences between timers against constants. For a self-driving car, this could mean taking decisions based on delays exceeding thresholds but not being allowed to make decisions based on the average of several delays.

---

**Luke**: Thanks, Paulo!

The post [Paulo Tabuada on program synthesis for cyber-physical systems](https://intelligence.org/2014/04/09/paulo-tabuada/) appeared first on [Machine Intelligence Research Institute](https://intelligence.org).


![Diana Spears portrait](http://intelligence.org/wp-content/uploads/2014/04/Spears_w730.jpg) [Diana Spears](http://www.swarmotics.com/Contact_Information.html) is an Owner and Research Scientist at [Swarmotics, LLC](http://www.swarmotics.com/Swarmotics.html). Previously, she worked at US government laboratories ([Goddard](http://www.nasa.gov/centers/goddard/home/), [NIST](http://www.nist.gov/), [NRL](http://www.nrl.navy.mil/)) and afterwards she was an Associate Professor of Computer Science at the [University of Wyoming](http://www.uwyo.edu/). She received both the MS and PhD (1990) degrees in Computer Science from the [University of Maryland, College Park](http://www.umd.edu/).

Dr. Spears’s research interests include machine learning, adaptive swarm robotic sensing networks/grids, computational fluid dynamics based algorithms for multi-robot chemical/biological plume tracing and plume mapping, and behaviorally-assured adaptive and machine learning systems. Dr. Spears pioneered the field of safe adaptive agents with her award-winning (2001 NRL Alan Berman Research Publication Award) publication entitled, [“Asimovian adaptive agents.”](https://www.jair.org/media/720/live-720-1895-jair.pdf) Most recently she and her husband co-edited the book [“*Physicomimetics: Physics-Based Swarm Intelligence*,”](http://www.springer.com/computer/ai/book/978-3-642-22803-2) published by Springer-Verlag in 2012.

**Luke Muehlhauser**: In [Spears (2006)](http://www.swarmotics.com/uploads/chap.pdf) and other publications, you’ve discussed the challenge of ensuring the safety of adaptive (learning) agents:

> a designer cannot possibly foresee all circumstances that will be encountered by the agent. Therefore, in addition to supplying an agent with a plan, it is essential to also enable the agent to learn and modify its plan to adapt to unforeseen circumstances. The introduction of learning, however, often makes the agent’s behavior significantly harder to predict. The goal of this research is to verify the behavior of adaptive agents. In particular, our objective is to develop efficient methods for determining whether the behavior of learning agents remains within the bounds of prespecified constraints… after learning…
>
>
> …Our results include proofs that… with respect to important classes of properties… if the [safety] property holds for the agent’s plan prior to learning, then it is guaranteed to still hold after learning. If an agent uses these “safe” learning operators, it will be guaranteed to preserve the properties with no reverification required. This is the best one could hope for in an online situation where rapid response time is critical. For other learning operators and property classes our a priori results are negative. However, for these cases we have developed incremental reverification algorithms that can save time over total reverification from scratch.
>
>

What do you mean by “incremental” reverification algorithms, as in the last sentence I quoted?

---

**Diana Spears**: Verification (model checking, in particular) consists of proving that a computer program satisfies a desirable property/constraint and, if it does not, a counterexample is provided. In my work, I assume that this program is a (multi)agent plan for action.  In most real-world applications, plans are typically enormous and therefore verification may be quite time-consuming. Suppose the safety property/constraint that agent A must always obey is that “agent A should always be at least M units away from agent B” (to prevent collisions). Let’s assume that initial verification proved that the entire plan (consisting of all action sequences that agent A could ever possibly take) is guaranteed to obey this property in all circumstances. Furthermore, let’s assume that adaptation is required after the agent has been fielded, where adaptation consists of applying a machine learning operator to modify the plan. For example, suppose a specific part of the plan states that agent A should “move from location 3 to location 4 if there is a clear path between 3 and 4, the ground is fairly level (e.g., nowhere higher than X or lower than Y between locations 3 and 4), and if the schedule permits such movement at this time.” Then an example machine learning operator might change the “4” to “6” based on new information about the task being performed.

Note that this learning operator only modifies one condition in one miniscule portion of the entire plan.Therefore, why re-verify that the entire plan still satisfies the desired property after learning? Why not re-verify only the specific portion of the plan that was modified, as well as any parts of the plan that depend on the modified portion?  That is what “incremental re-verification” does. It localizes post-adaptation verification to only those parts of the plan that are essential to re-verify.  In doing so, it improves the time complexity of re-verification. Time complexity is a very important and practical consideration for online systems – especially those that operate in real-time or in time-critical situations. In my research I ran numerous experiments comparing the CPU time of re-verifying an entire plan versus localized incremental re-verification of the plan after learning. My results showed as much as a 1/2-billion-fold speedup using incremental re-verification! And that’s using a plan that’s tiny compared to what agents would typically use in the real world.

---

**Luke**: With what kinds of agent programs have you explored this issue? What do the agents do, in what environment, and what kinds of safety properties do you prove about them?

---

**Diana**: Because of the strong relevance of the topic “safe adaptation” to aerospace applications, I chose to focus on NASA-related (multi-)agent programs. I depict a scenario in which a spacecraft has landed on another planet, such as Mars, and from which multiple mobile rovers emerge. The plans (programs) for the spacecraft  lander, as well as the planetary rovers, are for collecting, retrieving and transmitting/delivering data and/or samples from the surface of the planet. I prove “safety” and “liveness” properties. An example of “safety” is, “It is always the case that agent R is not delivering at the same time that agent L is transmitting.” Here, L is the lander and R is one of the rovers. This property/constraint prevents problems that may arise from the lander simultaneously receiving new data from R while transmitting older data to Earth. An example of “liveness” is, “If agent R executes the ‘deliver’ action, then eventually agent L will execute the ‘receive’ action.”

---

**Luke**: Have you or anyone else built on this specific line of work since 2006? What are some natural “next steps” for this particular line of research?

---

**Diana**: There were essentially three main offshoots of my research that I’m aware of – from NASA Ames, SRI, and USC. I’ll start with the NASA Ames offshoot. Around the year 2000 I gave a talk about “Asimovian Adaptive Agents” at NASA Ames. My instincts about the strong relevance of this work to NASA, and more generally aerospace, proved to be correct. (Furthermore, it appears to be highly relevant to *any* automated transportation, including automated cars/highways.) The researchers at NASA Ames rapidly and eagerly followed up on my talk with a flurry of related work, including research and publications. These researchers focused on “reference models,” which are used for online, run-time I/O checking. Instead of using temporal logic properties to verify, they used control theoretic properties such as “stability” and “performance.” Perkins and Barto also used Lyapunov stability as the property of interest when building adaptive learning agents[1](https://intelligence.org/2014/04/09/diana-spears/#footnote_0_10954 "Perkins, T. and Barto, A. “Lyapunov design for safe reinforcement learning control.” Proceedings of AAAI’02."). For examples of the NASA Ames research and other related work, see the papers that appeared in the NIPS’04 workshop on “Verification, validation, and testing of learning systems”[2](https://intelligence.org/2014/04/09/diana-spears/#footnote_1_10954 " Margineantu, Schumann, Gupta, Drumheller, and Fresnedo (co-chairs). Workshop on “Verification, validation, and testing of learning systems.” NIPS’04. "). Dietterich gave a follow-up talk at NIPS’06 on this topic[3](https://intelligence.org/2014/04/09/diana-spears/#footnote_2_10954 "Dietterich, T. “Research issues in deployed adaptive systems.” NIPS’06."). The NASA Ames offshoot has continued to be active post-2006, as exemplified by the many recent contributing papers in Schumann’s 2010 book[4](https://intelligence.org/2014/04/09/diana-spears/#footnote_3_10954 "Schumann, J. “Applications of Neural Networks in High Assurance Systems.” Springer-Verlag, 2010."). Furthermore, Vahram Stepanyan and others at NASA Ames have been working on a project called “Integrated Resilient Aircraft Control” (IRAC), whose goal is to validate multidisciplinary integrated aircraft control design tools and techniques that will enable safe flight despite unexpected adverse conditions[5](https://intelligence.org/2014/04/09/diana-spears/#footnote_4_10954 "Stepanyan, V. et al., “Stability and performance metrics for adaptive flight control.” AIAA’09.").

Shortly after my Ames talk, a second offshoot was initiated by John Rushby at SRI International. The SRI follow-on research continued to focus on formal methods with model checking, which is what I had originally worked with. However more recently this work has moved in a more similar direction to that of Ames[6](https://intelligence.org/2014/04/09/diana-spears/#footnote_5_10954 "Rushby, J. “A safety-case approach for certifying adaptive systems.” AIAA’09."). For example, in this paper Rushby introduces the idea of using a “safety case” that leads to an online performance monitor. And even more recently, Ashish Tiwari at SRI has worked on bounded verification of adaptive neural networks in the context of the IRAC project[7](https://intelligence.org/2014/04/09/diana-spears/#footnote_6_10954 "Tiwari, A. “Bounded verification of adaptive flight control systems.” AIAA’10.").

Next, consider a third offshoot. This is the research at the University of Southern California (USC) by Milind Tambe and others. These USC researchers built on my specific line of work, but they decided to address the important issue of mixed-initiative situations (also called “adjustable autonomy”), where humans and artificial agents collaborate to achieve a goal. Their multiagent plans are in the form of Partially Observable Markov Decision Processes (POMDPs) and they check safety constraints in this context. The first paper of theirs that I’m aware of on the topic of Asimovian adaptive (multi)agents appeared in 2006[8](https://intelligence.org/2014/04/09/diana-spears/#footnote_7_10954 "Schurr, N. et al. “Asimovian multiagents: Laws of robotics to teams of humans and agents.” 2006."). In 2007, Nathan Schurr got his Ph.D. on this topic[9](https://intelligence.org/2014/04/09/diana-spears/#footnote_8_10954 "Schurr, N. “Toward human-multiagent teams.” USC Ph.D. dissertation, 2007."). Milind Tambe continues to teach a very popular courses on “Artificial Intelligence and Science Fiction” in which he discusses his research on Asimovian multiagents.

Finally, I’ll mention miscellaneous post-2006 research that continues to build on my earlier line of work. For one, during 2006-2008 I was part of a DARPA Integrated Learning initiative that focused on methods for airspace deconfliction. Two of my graduate students, Antons Rebguns and Derek Green, along with Geoffrey Levine (U of Illinois) and Ugur Kuter (U of Maryland), applied safety constraints to planners[10](https://intelligence.org/2014/04/09/diana-spears/#footnote_9_10954 "Levine, G. et al. “Learning and verifying safety constraints for planners in a knowledge-impoverished system.” Computational Intelligence 28 (3), 2012."). Their work was inspired by my earlier research on Asimovian agents. There are also researchers currently building on the NASA Ames work: an international group[11](https://intelligence.org/2014/04/09/diana-spears/#footnote_10_10954 "Tamura, G. et al. “Towards practical runtime verification and validation of self-adaptive software systems.” Self-Adaptive Systems, LNCS 7475, Springer-Verlag, 2013."), Zhang and other researchers at Michigan State University[12](https://intelligence.org/2014/04/09/diana-spears/#footnote_11_10954 "Zhang et al. “Modular verification of dynamically adaptive systems.” AOSD’09."), and Italian researchers who are building on Zhang’s work[13](https://intelligence.org/2014/04/09/diana-spears/#footnote_12_10954 "Sharifloo, A. and Spoletini, P. “LOVER: Light-weight formal verification of adaptive systems at run time.” Formal Aspects of Component Software. Lecture Notes in Computer Science Volume 7684, pp. 170-187, 2013."). Also, Musliner and Pelican (Honeywell Labs), along with Goldman (SIFT, LLC) started building on my *incremental* re-verification work in particular – back in 2005[14](https://intelligence.org/2014/04/09/diana-spears/#footnote_13_10954 "Musliner , D. et al. “Incremental verification for on-the-fly controller synthesis.” MoChArt’05."), and apparently Musliner is still doing verification and validation (V&V) of adaptive systems.

Now I’ll respond to your second question about natural “next steps” for this particular line of research. I believe that all of the research mentioned above is exciting and shows promise. But I want to particularly emphasize the NASA/SRI direction as potentially fruitful for the future. This is based on my personal experiences with machine learning, formal methods, V&V, and AI in general. I have always found that formal methods and other logic-based approaches are, for computational reasons, difficult to scale up to complex real-world problems. Over the course of my career, I have leaned more towards stochastic methods for machine learning, and run-time checking for V&V. Therefore I hope that the aerospace researchers will continue in the directions they have adopted. However I also believe that they should widen their horizons. There are numerous techniques currently available for runtime monitoring and checking, e.g., see the online software self-checking, self-testing, and self-correcting methods of Ronitt Rubinfeld[15](https://intelligence.org/2014/04/09/diana-spears/#footnote_14_10954 "Rubinfeld, R. Checking."), or the run-time monitoring and checking of Insup Lee and Oleg Sokolsky[16](https://intelligence.org/2014/04/09/diana-spears/#footnote_15_10954 "Sokolsky, O. Selected Recent Publications by Subject")**.** I believe it would be potentially very fruitful to explore how many of the available monitoring and checking techniques are applicable to the behavioral assurance of adaptive systems.

But, most importantly, there is a topic that is critical to the future of building trustworthy adaptive systems and needs to be explored in great depth. That’s the issue of self-recovery/repair. Since around 1998-1999, my colleagues and I have been addressing self-repair in the context of swarm robotics[17](https://intelligence.org/2014/04/09/diana-spears/#footnote_16_10954 "Gordon, D. et. al. “Distributed spatial control, global monitoring and steering of mobile physical agents.” ICIIS’99.")[18](https://intelligence.org/2014/04/09/diana-spears/#footnote_17_10954 "Spears, W. and Spears, D. (Eds.) “Physicomimetics: Physics-Based Swarm Intelligence.” Springer-Verlag, 2012."). Our research focuses primarily on physics-based approaches to controlling swarm robotic formations – because physics naturally obeys the “principle of least action,” i.e., if a formation is disturbed then it will automatically perform the minimal actions required to repair the disturbance. This repair is locally optimal but is not necessarily globally optimal. In other words, we have dropped the requirement of global optimality, focusing on “satisficing” behavior instead.  Organic and natural physical systems are not perfect, but their lack of perfection often makes them more robust. There are systems where we need precise guarantees of behavior (e.g., the dynamic control of an airplane wing, to ensure that the plane does not stall and crash). But for other tasks, perfection and optimality are not even relevant (e.g., the Internet). We have demonstrated the feasibility of our research both in simulation and on real robots on numerous tasks, including uniform coverage, chain formations, surveillance, the movement of formations through environments with obstacles, and chemical source localization[19](https://intelligence.org/2014/04/09/diana-spears/#footnote_18_10954 "Spears, W. and Spears, D. (Eds.) 2012."). Hopefully other researchers will also explore physics-based systems. I believe that an excellent “safe adaptive (multi)agent” architecture would consist of a physics-based controller at the lower level, with a monitor/checker at a higher level to provide strict behavioral guarantees when needed. In particular, a more sophisticated version of our original design in [[17](http://intelligence.org/?p=10954&preview=true#footnote_16_10954)] would be quite promising.

In summary, I sincerely hope that the above-mentioned research will continue in the fruitful directions it has already taken, and I also hope that students and researchers will pursue additional, novel research along these lines. It seems to me that the topic of “safe adaptation” is “low-hanging fruit.” DARPA[20](https://intelligence.org/2014/04/09/diana-spears/#footnote_19_10954 "DARPA-sponsored ISAT meeting on “Trustable Deployed Adaptive Systems” at SRI, 2006.") and other funding agencies have expressed to me their desire to fund research on this topic – because it must be satisfactorily addressed if we are to have deployable, adaptive systems that we can trust.

---

**Luke**: In the lines of work you outlined above, what kinds of AI-like functionality are included in the parts of the code that are actually verified? E.g. does the verified code include classical planning algorithms, modern planning algorithms, logical agent architectures, or perhaps even machine learning algorithms in some cases?

---

**Diana**: The code that gets verified consists of reactive, “anytime” plans, which are plans that get continually executed in response to internal and external environmental conditions. Each agent’s plan is a finite-state automaton (FSA), which consists of states and state-to-state transitions. Each state in the FSA corresponds to a subtask of the overall task (which is represented by the entire FSA). And each transition corresponds to an action taken by the agent. In general, there are multiple transitions exiting each state, corresponding to the choice of action taken by the agent. For example, consider the scenario I described in one of my previous answers in this interview, i.e., that of a planetary lander along with rovers. Two possible states for a planetary lander L might be “TRANSMITTING DATA” and “RECEIVING DATA.” Suppose the lander is in the former state. If it takes the action “PAUSE” then it will stay in its current state, but if it takes the action “TRANSMIT” then after this action has completed the lander will transition to the latter state.  Furthermore, the conditions for transitioning from one state to the next depend not only on what action the agent takes, but also on what’s going on in the environment, including what this agent observes the other agents (e.g., the rovers) doing. For this reason, I call the plans “reactive.”

Every FSA has an initial state, but no final state. The philosophy is that the agents are indefinitely reactive to environmental conditions subsequent to task initiation, and their task is continually ongoing. FSAs are internally represented as finite graphs, with vertices (nodes) corresponding to behavioral states and directed edges corresponding to state-to-state transitions.

Machine learning (ML) is applied to the FSA plans for the purposes of agent initialization and adaptation. Learning is done with evolutionary algorithms (EAs), using traditional generalization and specialization operators. These operators add, delete, move, or modify vertices and edges, as well as actions associated with the edges.  For example, suppose the lander’s transition from its “TRANSMITTING DATA” to its “RECEIVING DATA” state depends not only on its own “TRANSMIT” action, but it also requires that rover R1 successfully received the data transmitted by lander L before the lander can make this state-to-state transition in its FSA. This is a very reasonable requirement. Now suppose that R1’s communication apparatus has catastrophically failed. Then L will need to adapt its FSA by modifying the requirement of checking R1’s receipt of the transmission before it can transition to its “RECEIVING DATA” state. One possibility is that it replaces “R1” with “R2” in its FSA. Many other alternative learning operators are of course also possible, depending on the circumstances.

Machine learning is assumed to occur in two phases: offline then online. During the offline initialization phase, each agent starts with a randomly initialized population of candidate FSA plans, which is then evolved using evolutionary algorithms. The main loop of the EA consists of selecting parent plans from the population, applying ML operators to produce offspring, evaluating the fitness of the offspring, and then returning the offspring to the population if they are sufficiently “fit.” After evolving a good population of candidate plans, the agent then selects the “best” (according to its fitness criteria) plan from its population. Verification is then performed to this plan as well as repair, if required. During the online phase, the agents are fielded and plan execution is interleaved with learning (adaptation to environmental changes, such as agent hardware failures), re-verification, and plan repair as needed.

The main point of my “Asimovian adaptive agents” paper is that by knowing what adaptation was done by the agent, i.e., what machine learning operator was applied to the FSA, we can streamline the re-verification process enormously.

---

**Luke**: AI systems are becoming increasingly autonomous in operation: [self-driving cars](http://en.wikipedia.org/wiki/Autonomous_car), robots that [navigate disaster sites](http://www.darpa.mil/Our_Work/TTO/Programs/DARPA_Robotics_Challenge.aspx), [HFT](http://en.wikipedia.org/wiki/High-frequency_trading) programs that trade stocks quickly enough to “[flash crash](http://en.wikipedia.org/wiki/2010_Flash_Crash)” the market or [nearly bankrupt](http://en.wikipedia.org/wiki/Knight_Capital_Group#2012_stock_trading_disruption) a large equities trader, etc.

How might current AI safety methods (formal verification and reverification, program synthesis, simplex architectures, hybrid systems control, [layered architectures](http://commonsenseatheism.com/wp-content/uploads/2013/08/Fisher-et-al-Verifying-Autonomous-Systems.pdf), etc.) be extended to meet the safety challenges that will be raised by the future’s highly autonomous systems operating in unknown, continuous, dynamic environments? Do you think our capacity to make systems more autonomous and capable will outpace our capacity to achieve confident safety assurances for those systems?

---

**Diana**: My response to your first question is problem- and context-dependent. I know of many AI communities that are built around a single algorithm, and the researchers in those communities try to apply this algorithm to as many problems as possible. I believe that’s a misguided approach to research. Instead, I have always tried to adopt a problem-driven approach to research. The best way to scientifically solve a problem is to study it in great depth, and based on the *a priori* problem/task analysis select the most appropriate solution — including the planner or problem-solver, the properties/constraints to be verified, the adaptation method(s), etc. This will require a large suite of different AI safety/verification methods from which to choose. I cannot foresee the nature of this suite in advance; it’ll have to be constructed based on experience. As we tackle more complex autonomous systems, our repertoire of verification techniques will grow commensurately.

Your second question about whether autonomy will outpace safety is an extremely important one, Luke. Based on the applications you listed in your first paragraph, I see that your concerns extend to security. In fact, your security concerns also apply to “[the internet of things](http://www.economist.com/news/science-and-technology/21594955-when-internet-things-misbehaves-spam-fridge),” which includes electronic, interconnected, remotely-accessible autonomous devices such as washing machines, ovens, and thermostats that will be installed in “smart homes” of the future. Businesses usually lack the motivation to install safety and security measures without some incentive. For example, leading software companies release beta versions of their programs with the expectation that the public will find and report the bugs. This is unacceptable as we transition to increasingly capable and potentially hazardous autonomous systems. I believe that the primary incentive will be government regulations. But we can’t wait until disasters arise before putting these regulations in place! Instead, we need to be proactive.

In 2008-2009 I was a member of a [American Association for the Advancement of Artificial Intelligence (AAAI) Presidential Panel](http://research.microsoft.com/en-us/um/people/horvitz/AAAI_Presidential_Panel_2008-2009.htm) to study these issues. This was a fabulous panel, and it brought awareness to the AI community of researchers. Nevertheless it’s time raise awareness beyond the community of AI researchers. One suggestion I have is to assign a new or existing member of the [United States President’s Council of Advisors on Science and Technology](http://en.wikipedia.org/wiki/United_States_President%27s_Council_of_Advisors_on_Science_and_Technology) the task of studying and assessing the safety and security of autonomous systems.  This council member should consult the following people:

1. AI researchers who have extensive experience in developing autonomous systems
2. Engineers from aerospace, transportation, and other applications where safety is paramount
3. Lawyers and lawmakers who are cognizant of the legal issues that could arise
4. Cyber security experts.

I assume this council member would research the topic, consult others, conduct meetings, and conclude with a report and recommendations. Furthermore, I strongly believe that such a task should be assigned as soon as possible. We are *already* in a state where autonomy is outpacing safety and security, particularly in the commercial sector outside of the transportation industry.

---

**Luke**: Given that “autonomy is outpacing safety and security,” what are some other ideas you have for increasing the odds of reliably good outcomes from future autonomous systems?

By the way, I’ve only ever seen an “interim” [report](http://research.microsoft.com/en-us/um/people/horvitz/note_from_AAAI_panel_chairs.pdf) from that AAAI panel. Is there supposed to be a follow-up report at some point?

---

**Diana**: I haven’t heard about any follow-up or final report for the AAAI panel, unfortunately.

One idea is that we should have extensive safety and security testing prior to product release, based on established industry/government standards. We may not be able to enforce 100% compliance, but the presence of something like a “Safe and Secure Autonomous Product” certification could motivate consumers to favor purchasing tested and certified products over others lacking compliance. This would be like the existing [UL product certification](http://www.ul.com/global/eng/pages/offerings/perspectives/newtoul/ulmarkproductcertification/).

Another idea is to have monitoring, safety shutoff, self-recovery, and self-repair capabilities associated with autonomous devices. Furthermore, for security reasons these mechanisms should be decoupled from the autonomous system’s control, and they should also be detached from communications (e.g., not connected to the Internet) so as to avoid malicious tampering.

I don’t believe it’s possible to ensure complete safety and security at all times with autonomous systems. As you stated above, the best we can do is to increase “the odds of reliably good outcomes.” Nevertheless, I believe that substantial progress can be made if there is financial, technical, legal and programmatic support in this direction.

---

**Luke**: Thanks, Diana!

---

1. Perkins, T. and Barto, A. “[Lyapunov design for safe reinforcement learning control.](http://www.aaai.org/Papers/Symposia/Spring/2002/SS-02-07/SS02-07-005.pdf)” Proceedings of AAAI’02.
2. Margineantu, Schumann, Gupta, Drumheller, and Fresnedo (co-chairs). Workshop on “[Verification, validation, and testing of learning systems.](http://www.dmargineantu.net/nips2004/)” NIPS’04.
3. Dietterich, T. “[Research issues in deployed adaptive systems.](http://www.dmargineantu.net/NIPS06-TDLDS/TDLDS-NIPS06.Dietterich.pdf)” NIPS’06.
4. Schumann, J. “[Applications of Neural Networks in High Assurance Systems.](http://www.springer.com/engineering/computational+intelligence+and+complexity/book/978-3-642-10689-7)” Springer-Verlag, 2010.
5. Stepanyan, V. et al., “[Stability and performance metrics for adaptive flight control.](http://arc.aiaa.org/doi/abs/10.2514/6.2009-5965)” AIAA’09.
6. Rushby, J. “[A safety-case approach for certifying adaptive systems.](http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.159.905)” AIAA’09.
7. Tiwari, A. “[Bounded verification of adaptive flight control systems.](http://arc.aiaa.org/doi/abs/10.2514/6.2010-3362)” AIAA’10.
8. Schurr, N. et al. “[Asimovian multiagents: Laws of robotics to teams of humans and agents.](http://www.aptima.info/publications/2006_Schurr_Varakantham_Bowring_Tambe_Grosz.pdf)” 2006.
9. Schurr, N. “[Toward human-multiagent teams.](http://teamcore.usc.edu/schurr/SchurrThesis.pdf)” USC Ph.D. dissertation, 2007.
10. Levine, G. et al. “[Learning and verifying safety constraints for planners in a knowledge-impoverished system.](http://onlinelibrary.wiley.com/doi/10.1111/j.1467-8640.2012.00416.x/abstract)” Computational Intelligence 28 (3), 2012.
11. Tamura, G. et al. “[Towards practical runtime verification and validation of self-adaptive software systems.](http://link.springer.com/chapter/10.1007%2F978-3-642-35813-5_5#page-1)” Self-Adaptive Systems, LNCS 7475, Springer-Verlag, 2013.
12. Zhang et al. “[Modular verification of dynamically adaptive systems.](http://www.cse.msu.edu/~mckinley/Pubs/files/Zhang.AOSD.2009.pdf)” AOSD’09.
13. Sharifloo, A. and Spoletini, P. “[LOVER: Light-weight formal verification of adaptive systems at run time.](http://link.springer.com/chapter/10.1007/978-3-642-35861-6_11)” Formal Aspects of Component Software. Lecture Notes in Computer Science Volume 7684, pp. 170-187, 2013.
14. Musliner , D. et al. “[Incremental verification for on-the-fly controller synthesis.](http://rpgoldman.goldman-tribe.org/papers/mochart05.pdf)” MoChArt’05.
15. Rubinfeld, R. [Checking](http://www.cs.cornell.edu/faculty/home/ronitt/ronitt.html).
16. Sokolsky, O. [Selected Recent Publications by Subject](http://www.cis.upenn.edu/~sokolsky/papers.html)
17. Gordon, D. et. al. “[Distributed spatial control, global monitoring and steering of mobile physical agents.](http://repository.upenn.edu/cgi/viewcontent.cgi?article=1310&context=cis_papers)” ICIIS’99.
18. Spears, W. and Spears, D. (Eds.) “[Physicomimetics: Physics-Based Swarm Intelligence.](http://www.springer.com/computer/ai/book/978-3-642-22803-2)” Springer-Verlag, 2012.
19. Spears, W. and Spears, D. (Eds.) 2012.
20. DARPA-sponsored ISAT meeting on “Trustable Deployed Adaptive Systems” at SRI, 2006.

The post [Diana Spears on the safety of adaptive agents](https://intelligence.org/2014/04/09/diana-spears/) appeared first on [Machine Intelligence Research Institute](https://intelligence.org).