John Ridgway on safety-critical systems

![John Ridgway portrait](http://intelligence.org/wp-content/uploads/2014/03/John_Ridgway_w680.jpg)John Ridgway studied physics at the University of Newcastle Upon Tyne and Sussex University before embarking upon a career in software engineering. As part of that career he worked for 28 years in the field of Intelligent Transport Systems (ITS), undertaking software quality management and systems safety engineering roles on behalf of his employer, [Serco Transportation Systems](http://www.serco.com). In particular, John provided design assurance for Serco’s development of the Stockholm Ring Road Central Technical System (CTS) for the Swedish National Roads Administration (SNRA), safety analysis and safety case development for Serco’s M42 Active Traffic Management (ATM) Computer Control System for the UK Highways Agency (HA), and safety analysis for the National Traffic Control Centre (NTCC) for the HA.

John is a regular contributor to the Safety Critical Systems Club (SCSC) Newsletter, in which he encourages fellow practitioners to share his interest in the deeper issues associated with the conceptual framework encapsulated by the terms ‘uncertainty’, ‘chance’ and ‘risk’. Although now retired, John recently received the honour of providing the after-banquet speech for the SCSC 2014 Annual Symposium.

**Luke Muehlhauser**: What is the nature of your expertise and interest in safety engineering?

---

**John Ridgway**: I am not an expert and I would not wish to pass myself off as one. I am, instead, a humble practitioner, and a retired one at that. Having been educated as a physicist, I started my career as a software engineer, rising eventually to a senior position within Serco Transportation Systems, UK, in which I was responsible for ensuring the establishment and implementation of processes designed to foster and demonstrate the integrity of computerised systems. The systems concerned (road traffic management systems) were not, initially, considered to be safety-related, and so lack of integrity in the delivered product was held to have little more than a commercial or political significance. However, following a change of safety policy within the procurement departments of the UK Highways Agency, I recognised that a change of culture would be required within my organisation, if it were to continue as an approved supplier.

If there is any legitimacy in my contributing to this forum, it is this: Even before safety had become an issue, I had always felt that the average practitioner’s track record in the management of risk would benefit greatly from taking a closer interest in (what some may deem to be) philosophical issues. Indeed, over the years, I became convinced that many of the factors that have hampered software engineering’s development into a mature engineering discipline (let’s say on a par with civil or mechanical engineering) have at their root, a failure to openly address such issues. I believe the same could also be said with regard to functional safety engineering. The heart of the problem lies in the conceptual framework encapsulated by the terms ‘uncertainty’, ‘chance’ and ‘risk’, all of which appear to be treated by practitioners as intuitive when, in fact, none of them are. This is not an academic concern, since failure to properly apprehend the deeper significance of this conceptual framework can, and does, lead practitioners towards errors of judgement. If I were to add to this the accusation that practitioners habitually fail to appreciate the extent to which their rationality is undermined by cognitive biases, then I feel there is more than enough justification for insisting that they pay more attention to what is going in the world of academia and research organisations, particularly in the fields of cognitive science, decision theory and, indeed, neuroscience. This, at least, became my working precept.

---

**Luke**: What do you think was the origin of your concern for the relevance of philosophy and cognitive science to safety engineering? For example did you study philosophy, or [Kahneman and Tversky](http://www.amazon.com/Judgment-under-Uncertainty-Heuristics-Biases/dp/0521284147/), in university?

---

**John**: As a physics student I was offered little opportunity (and to be honest had little ambition) to pursue an education in philosophy. Of course it is true that quantum mechanics made a mockery of my cherished belief in the existence of unobserved, objective reality, but I was very much in the ‘*I know it makes no sense whatsoever but just shut up and carry on calculating’* school of practice. In fact, it wasn’t until I had become a jobbing software engineer, investigating the use of software metrics to predict development timescales, that philosophical issues started to take on an occupational importance.

The practice within my department had been to use single-point estimations for the duration of tasks, then feed these figures and the task dependencies into a Gantt chart and simply use the chart to read off the predicted project end date. Sometimes the results were very impressive: predicted project end dates often proved to be in the right decade! Then one bright spark said, ‘*You’re living in the dark ages. What you need are three-point estimations for the duration of each task (i.e. estimate a most likely duration together with an upper and lower band). You then use Monte Carlo Simulation to create a project outturn curve; a curve which indicates a statistical spread of possible project durations. Everyone does this nowadays. And it just so happens I have an expensive risk management tool to do all the nasty calculations for you*’. Imagine his look of disgust when I told him that I still thought the old way was better!

Was he talking to a smug idiot? I like to think not. The basis of my objection was this: As a physicist, I was well aware of the benefits of using Monte Carlo Simulation in areas such as solid state and nuclear physics. Here it is used to determine the macroscopic behaviour of a physical system, where probability distribution curves can be used to model stochastic variability of the micro behaviour of the system under study. Now, however, I was being invited to use Monte Carlo methods to draw conclusions based upon the averaging of various levels of posited (and decidedly non-stochastic) ignorance regarding the future. In such circumstances, no one could provide me with a convincing argument for deciding the most appropriate form for the probability distribution curve upon which the simulation would be based. In fact, I was told this choice didn’t matter, though quite clearly it did. If, as seemed likely, a realistic distribution curve would have a fat tail, the results would be hugely influenced by the choice of curve. Furthermore, the extra two estimates (i.e. the upper and lower bands for task duration) were supposed to represent a level of uncertainty, but the uncertainty behind their selection was at least of the same order of magnitude as the uncertainty these bounds were supposed to represent. In other words, one could not be at all certain how uncertain one was! It occurred to me that no real information was being added to the risk model by using these three-point estimates, and so no amount of Monte Carlo Simulation would help matters.

This led me to philosophical musing about the nature of uncertainty and the subtleties of its relationship with risk. These musings had a potentially practical value because I thought that a lot of people were spending a lot of time and money using inappropriate techniques to give themselves false confidence in the reliability of their predictions. Unfortunately, none of my colleagues seemed to share my concern and all I could do to try and persuade them was to wave my hands whilst speaking vaguely about second order uncertainty, model entropy and the important distinction between variability and incertitude. So I decided there was a gap in my education that needed filling.

As it happened, my investigations soon led me to the work of [Professor Norman Fenton](http://www.eecs.qmul.ac.uk/%7Enorman) of Queen Mary University London, and I familiarised myself with concepts such as subjective probability and epistemic versus aleatoric uncertainty. Furthermore, once subjectivity had been placed centre stage, the relevance of cognitive sciences loomed large and although I can’t claim to have studied Tversky and Kahneman, I became familiar with ideas associated with decision theory that owe existence to their work.

Suddenly, my career seemed so much more interesting. And once I moved on to safety engineering, the same issues cropped up again in the form of over-engineered fault tree diagrams replete with probability values determined by ‘expert’ opinion. Now it seemed all the more important that practitioners should think more deeply about the philosophical and psychological basis for their confident proclamations on risk.

---

**Luke**: In many of your articles for the Safety-Critical Systems Club (SCSC) newsletter, you briefly discuss issues in philosophy and cognitive science and their relevance to safety-critical systems (e.g. [1](http://commonsenseatheism.com/wp-content/uploads/2014/02/Ridgway-Whats-Luck-Got-To-Do-With-It.pdf), [2](http://commonsenseatheism.com/wp-content/uploads/2014/02/Ridgway-Risk-%C2%AD-A-Coffee-Break-Philosophers-Viewpoint.pdf), [3](http://commonsenseatheism.com/wp-content/uploads/2014/02/Ridgway-Reflections-on-risk-and-the-importance-of-understanding-the-nature-of-uncertainty.pdf), [4](http://commonsenseatheism.com/wp-content/uploads/2014/02/Ridgway-Having-Confidence-in-Your-Confidence.pdf), [5](http://commonsenseatheism.com/wp-content/uploads/2014/02/Ridgway-Armageddon-and-Other-Failure-Modes.pdf), [6](http://commonsenseatheism.com/wp-content/uploads/2014/02/Ridgway-The-Philosophers-Chances.pdf)). During your time working on safety engineering projects, how interested did your colleagues seem to be in such discussions? Were many of them deeply familiar with the issues already? Does philosophy of probability and risk, and cognitive science of (e.g.) heuristics and biases, seem to be part of the standard training for those working safety engineering — at least, among the people you encountered?

---

**John**: Perhaps my experience was atypical, but the sad fact is that I found it extremely difficult to persuade any of my colleagues to share an interest in such matters, and I found this doubly frustrating. Firstly, I thought it to be a missed opportunity on my colleagues’ part, as I felt certain that application of the ideas would be to their professional advantage. Their diffidence was to a certain extent understandable, however, since there was nothing in the occupational training provided for them that hinted at the importance of philosophy or psychology. However, what really frustrated me was the fact that no one appeared to be at all excited by the prospect of introducing these subjects into the workplace. How could that be? How could my colleagues fail to be anything other than utterly fascinated? In fact, their lack of interest seemed to me to represent nothing less than a wanton refusal to enjoy their job!

The key to the problem lay, of course, with the training provided by my employer, and it didn’t help that the internal department that provided such training glorified under the title of ‘The Best Practice Centre’. Clearly, anything that I might say that differed from the company endorsed view was, by definition, less then best! And I soon found that berating the centre’s risk management course for failing to explore the concept of uncertainty was, if anything, counter-productive. Upon reflection, I think that some of these frustrations led me to seek an alternative forum in which I could express my thinking. Publishing articles for the Safety Critical Systems Club newsletter provided such an outlet.

---

**Luke**: You say that you “felt certain that application of the ideas would be to their professional advantage.” Can you give me some reasons and/or example for why you felt certain of that?

---

**John**: I think that my concerns were the product of working within a profession that appears to see the world rather too much in frequentist terms, in which the assumption of aleatoric uncertainty would be valid. In reality, it is increasingly the case that the risks a systems safety engineer has to analyse are predicated predominantly upon epistemic uncertainty. I cite, in particular, safety risks associated with complex software-based systems, adaptive systems and so-called Systems of Systems (SoS), or indeed any system that interacts with its environment in a novel or inherently unpredictable manner. Whilst it is true that analysing stochastic failure of physical components may play a significant role in predicting system failure, the probabilistic techniques involved in such analysis simply cannot address epistemic concerns, i.e. where the parameters of any posited probability distribution curve may be a matter for pure speculation. (I am aware that Monte Carlo simulation is sometimes used to probabilistically model the parametric uncertainty in probabilistic models, but this strikes me as an act of desperation reminiscent of the invention of epicycles upon epicycles to shore up the Ptolemaic cosmology).

There are a number of suitable approaches available to the safety analyst, which seek to accommodate epistemic uncertainty (Bayesian methods, Possibility Theory and Dempster-Schafer, to name but three). However, whilst the practitioner is not even aware that there is an issue, and continues to assume the objectivity of all probability, there is little hope that these methods will attract the attention they deserve.

Then, of course, we have to consider the pernicious effect that cognitive bias has upon the analyst’s assessment of likelihood. It is in the nature of such biases that the individual is unaware of their impact. Surely, therefore, even the most basic training in this area would be of considerable benefit to the practitioner. On a similar theme, I have become concerned that the average safety analyst is insufficiently mindful of the distinction to be made between risk aversion and ambiguity aversion. This may lead to a failure to adequately understand the rationality that lies behind a particular decision, but it may also explain why my colleagues didn’t appear to appreciate the importance of undertaking uncertainty management alongside risk management.

Finally, when one considers the interconnectivity of risks, and the complications introduced by multi-stakeholders, it becomes very difficult to think about risk management strategies without having to address ethical issues associated with risk transfer, optimisation and sufficing. But perhaps that is another story.

---

**Luke**: Yes, can you say more about the “ethical issues associated with risk transfer, optimization, and sufficing”?

---

**John**: In UK health and safety legislation there is an obligation to reduce existing risk levels ‘So Far As Is Reasonably Practicable’ (SFAIRP). This leads to the idea of residual risks being ‘As Low As Reasonably Practicable’ ([ALARP](https://intelligence.org/feed/As Low As Reasonably Practicable)). The ALARP concept assumes that an upper limit can be defined, above which the risk is considered ‘intolerable’. In addition, there is a lower limit, below which the risk is considered to be ‘broadly acceptable’. A risk may be allowed to lie between these two limits as long as it can be demonstrated that further reduction would require disproportionate cost and effort. Apart from the vagueness of the terminology used here, the main problem with this view is that it says nothing about the possibility that the management of one risk may bear upon the scale of another. Indeed, one can envisage a network of inter-connected risks in which this knock-on effect will propagate, resulting in the net level of risk increasing (keep in mind that the propagation may include both positive and negative feedback loops). For this reason, there exists the Globally At Least Equivalent (GALE) principle, which holds that upon modifying a system, one must assess the overall, resulting risk level posed by the system rather than focussing purely upon the risk that the modification was intended to address. The idea, of course is that the overall level should never increase.

So far this has all been very basic risk management theory and, on the face of it, the ALARP and GALE principles appear to complement each other. But do they always? Well, in the simple case where all risks are owned and managed by a single authority, this may be the case. But what if the various risks under consideration have differing owners and stakeholders? In such circumstances, parties who own risks and seek to reduce them SFAIRP may find themselves in conflict with each other, with the various stakeholders and with any body that may exist to ensure that the global risk level is not increased.

Perhaps we are now in the province of game theory rather than decision theory. If so, it seems reasonable to insist that the game be played in accordance with ethical constraints, but has anyone declared what these might be? Some seem obvious; for example, never transfer risk to another party without their knowledge and consent. Others may not be so straightforward. I think we are all familiar with the old chestnut of the signalman who can prevent a runaway train from ploughing into a group of schoolchildren by changing the points, but only by causing the certain death of the train driver. Does the signalman have the moral authority to commit murder? Would it be murder whether or not he or she switches the points? If we find this difficult to answer, one can easily envisage similar difficulties when deciding the ethical framework associated with the management of risk collectives.

---

**Luke**: Which changes do you most desire for the safety-critical systems industry?

---

**John**: I think that there is a lot to be said for making membership of a suitably constructed professional body a legal imperative for undertaking key systems safety engineering roles. Membership would require demonstration of a specified level of competence, adherence to formulated codes of conduct and the adoption of appropriate ideologies. Given my responses to earlier questions, your readers will probably be unsurprised to hear that I hope that this would provide the opportunity to promote a greater understanding of the conceptual framework lying behind the terms ‘risk’ and ‘uncertainty’. In particular, I would like to see a professional promotion of the philosophical, ethical and cognitive dimensions of system safety engineering.

I appreciate that the various engineering disciplines are already well served by a number of professional societies and that, for example, an Independent Safety Assessor (ISA) in the UK would be expected to be a chartered engineer and a recognised expert in the field (whatever that means). However, the epistemic uncertainties surrounding the development and application of complex computer-based systems introduce issues that perhaps the commonly encountered engineering psyche may not be fully equipped to appreciate. It may be that the safety engineering professional may need to think more like a lawyer. Consequently, the professional body I am looking for could be modelled upon the legal profession, as much as upon the existing engineering professions. I know for some people ‘lawyer’ is a dirty word, but so is ‘subjectivity’ in some engineers’ minds. Being pragmatic, and in order to stay in the game, we may all have to become sophists.

---

**Luke**: Thanks, John!

The post [John Ridgway on safety-critical systems](https://intelligence.org/2014/03/08/john-ridgway-on-safety-critical-systems/) appeared first on [Machine Intelligence Research Institute](https://intelligence.org).


David Cook on the VV&A process

![Emil Vassev portrait](http://intelligence.org/wp-content/uploads/2014/03/David_A_Cook_w678.jpg)Dr. David A. Cook is Associate Professor of Computer Science at [Stephen F. Austin State University](http://www.sfasu.edu/), where he [teaches](https://intelligence.org/feed/<a href=) Software Engineering, Modeling and Simulation, and Enterprise Security. Prior to this, he was Senior Research Scientist and Principal Member of the Technical Staff at [AEgis Technologies](http://www.aegistg.com/), working as a Verification, Validation, and Accreditation agent supporting the Airborne Laser. Dr. Cook has over 40 years’ experience in software development and management. He was an associate professor and department research director at [USAF Academy](http://www.usafa.af.mil/) and former deputy department head of [Software Professional Development Program](http://www.afit.edu/ls/spdp_list.cfm) at [AFIT](http://www.afit.edu/). He has been a consultant for the Software Technology Support Center, Hill AFB, UT for 19 years.

Dr. Cook has a Ph.D. in Computer Science from [Texas A&M University](http://www.tamu.edu), is a Team Chair for [ABET](http://www.abet.org/), Past President for the [Society for Computer Simulation, International](http://www.scs.org/), and Chair of [ACM SIGAda](http://www.sigada.org/).

**Luke Muehlhauser**: In various articles and talks (e.g. [Cook 2006](http://www.ieee-stc.org/proceedings/2006/pdfs/DC1252.pdf)), you’ve discussed the software verification, validation, and accreditation (VV&A) process. Though the general process is used widely, the VV&A term is often used when discussing projects governed by [DoD 5000.61](http://www.dtic.mil/whs/directives/corres/pdf/500061p.pdf). Can you explain to whom DoD 5000.61 applies, and how it is used in practice?

---

**David A Cook**: DOD 5000.81 applies to all Department of Defense activities involving modeling and simulation.  For all practical purposes, it applies to all models and simulations that are used by the DOD.  This implies that it also applies to all models and simulations created by civilian contractors that are used for DOD purposes.

The purpose of the directive, aside from specifying who is the “accreditation authority” (more on this later) is to require Verification and Validation for all models and simulation, and then also to require that each model and simulation by accredited for its intended use.  This is the critical part, as verification and validation has almost universally been a part of software development within the DOD.  Verification asks the question “Are we building the system in a quality manner?”, or “are we building the system right?”.  Verification, in a model (and the resulting execution of the model providing a simulation) goes a bit further – and asks the question “Does the model build and the results of the simulation actually represent the conceptual design and specifications of the system we built?”  The difference is that in a model and simulation, you have to show that your design and specifications of the system you envision are correctly translated into code, and that the data provided to the code also matches specification.

Validation asks the question “are we building a system that meets the users’ actual needs?”, or “are we building the right system?”  Again, the verification of a model and resulting simulation is a bit more complex than non-M&S ”verification”.  In modeling and simulation, verification has to show that the model and the simulation both accurately represent the “real world” from the perspective of the intended use.

These two activities are extremely difficult when you are building models and providing simulation results for notional systems that might not actually exist in the real world.  For example, it would be difficult to provide V&V for a manned Mars mission, because, in the real world, there is not a manned Mars lander yet!   Therefore, for notional systems, V&V might require estimation and guesswork.  However, guesswork and estimation might be the best you can do!

5000.61 further requires that there be an ultimate authority, the “accreditation authority”, that is willing to say “based on the Verification and Validation performed on this model, I certify that it provides answers that are acceptable for its intended use”.  Again, if you are building a notional system, this requires experts to say “These are guesses, but they are the best guesses available, and the system is as close a model to the real world as possible.  We accredit this system to provide simulation results that are acceptable.”   If, for example, an accredited  simulation shows that a new proposed airplane would be able to carry 100,000 pounds of payload – but the result airplane, once built, can only carry 5,000 pounds – the accreditation authority would certainly bear some of the blame for the problem.

In practice, there are process for providing VV&A.  Military Standard  3022 provides a standard template for recording VV&A activities, and many DOD agencies have their own VV&A repository where common models and simulation VV&A artifacts (and associated documentation) are kept.

There are literally hundreds of ways to verify and validate a model (and it’s associated simulation execution).  The V&V “agents” (who have been tasked with performing V&V) provide a recommendation to the Accreditation Authority, listing what are acceptable uses, and (the critical part) the limits of the model and simulation.  For example, a model and simulation might provide an accurate representation of the propagation of a laser beam (in the upper atmosphere) during daylight hours, but not be a valid simulation at night, due to temperature-related atmospheric propagation.  The same model and simulation might be a valid predictor of a laser bouncing off of a “flat surface”, but not bouncing off of uneven terrain.

---

**Luke**: Roughly how many accreditation authorities are there for such projects? Do accreditation authorities tend to specialize in accrediting V&V in certain domains — e.g. some for computer software, some for airplanes, etc.? Are there accreditation authorities that the DoD doesn’t recognize as “legitimate”?

---

**David**: Accreditation authorities are simply the component heads who sign a letter saying “Model and Simulation X is approved for the following purposes”.  The letter then states what the intended uses are, lists any special conditions, and lists the limitations of the model and the simulation.  The accreditation authority is more of a position rather than a person.  It can be a person (usually the head of the organization), or a committee.

Each DOD agency is responsible for models and simulations that it develops or uses – they must either VV&A their own models and simulations, or use models and simulations (from a trusted source) that has performed their own VV&A.  Note however, that each DOD agency probably has their own data – which must also be be accredited.  Every project has it’s own M&S, and probably has domain experts who perform the VV&A.  Rather than go to the top for accreditation, each project probably has been delegated the authority to perform its own VV&A.

There are no non-legimate accreditation authorities per se; accreditation authorities are not authorized based on knowledge, simply on position.  However it is assumed that each M&S area has domain experts who have the specialized knowledge in the application area to perform reliable VV&A.  These domain experts span many areas – application domain experts (who might, for example, be an expert on a laser beam), coding domain experts (who can verify that the code is a good representation of the requirements), data domain experts (who verify that the targeting data represents a valid target), and perhaps many others.  Typically, each project has a VV&A team or “agent” who perform the V&V, and recommend accreditation (usually in a formal letter) that restates the intended uses and limitations,  The recommendation includes all associated artifacts, such as test results, reviews, reports of individual Verification and Validation activities, other models and simulation used to compare against, real-life data (to show validity), and possibly many other items.

If a particular DOD agency is using a model and simulation in its exercises, it is responsible for VV&A of its own M&S.  If, on the other hand, an allied agency is using a model that includes artifacts from an another agency – the outside agency is responsible for working to make sure that the model, simulation and data repressing them is valid.  In essence, each DOD agency is responsible to other other DOD component to ensure that their forces and capabilities are appropriately represented to all outside agencies utilizing models and simulation that involve them.

---

**Luke**: Are there historical cases in which a model or simulation completed the VV&A process, but then failed to perform as it should have given the accreditation, and then the accreditation authority was held to account for their failure? If so, could you give an example? (Preferably but not necessarily in software.)

---

**David**: Because I worked as a consultant on many modeling and simulation projects, I am ethically prevented from discussing actual failures that I know about – mainly because most of the projects I worked on were classified, and I signed non-disclosure agreements.

However, by shifting into hypothetical scenarios, there are several stories that I can use that best illustrate this. One is a story taught in many simulation classes – and I only have secondhand knowledge of it. The other two are ancedotal – but good lessons!

In the first instance, a model was used which predicted “safe runway distance” for an airplane. Feed into the model the weight, altitude, temperature, and humidity, and run the simulation to predict how much runway was needed.

Unfortunately, the day the model was used, it took several hours for the airplane to actually takeoff. It had a bit more fuel that estimated – adding weight. By takeoff time, the temperature had risen, giving “stickier” tires and runway, and decreasing air density (giving less lift) Also – the humidity had changed, also affecting lift characteristics.

The model did not have a large allowance for error (it tried to give a relatively precise answer) – and with all the factors changing, the airplane went from “enough runway” to “marginal” after the simulation had been run. Combined with a relatively inexperienced pilot (who did not advance the throttle fast enough) – and the airplane overshot the end of the runway. Not much damage (other than a bruised ego) – but the simulation – while accurate, was not used properly.

The other two stories are certainly imaginary – but are passed around like legend in our field. In the first story, in the early days of the Airborne Laser, a very simple model was used to predict laser propagation. Code was reused to model the laser – basically, code for a missile, with the speed of the missile increased to the speed of light. The targeting acquisition, target recognition, etc. were all similar, and once fired, the simulation would show if the target was hit. Until the first time they ran it, halfway to the target, the laser beam “ran out of fuel” and fell into the sea.

The second (certainly imaginary) story involves modeling a battle scenario for the Australian Air Force – using helicopters. One of the problems with landing a helicopter was making sure it had a clear landing field – and kangaroos were a problem. So – the developers of the battle simulation, who used object-oriented development, took some code which was basically used to model a ground soldier and modified its behavior to “run at the sound of helicopters”. They then changed its appearance on the simulation to show a small image of a kangaroo. When to model was executed, the simulation showed the kangaroos running away from the helicopter. Until it landed, and then the kangaroos reversed direction, and attacked the helicopter with rifles!

Ok – the last two examples are cute and funny, but show the problems with invalid assumptions and imperfect data.

I leave this question with a quote that I always use in my M&S classes – “All models are inaccurate. Some models are useful anyway.” It is extremely difficult to model the real world totally inside of code. I don’t care how well you model a hot tub in a simulation – it is NOT really a hot tub.

There are always things you do not consider, data that is not perfect, or constraints that you miss. A model is a best-guess approximation of what will happen in the “ dal world” – but it is NOT the “real world”. All models have limitations. In spite of that. the resulting model and the simulation still give useful data. The accreditation authority is simply acknowledging that the model and simulation are useful “for their intended use”, and that “limitations exist”. No reputable modeling and simulation expert (nor any accreditation authority) trusts a single model and its resulting simulation to produce data that is used in life-or-death decisions. Multiple sources of validity are required, multiple independently-developed models and simulations are used, and domain experts are consulted to see if the results “feel right”. And tolerances must always be given. An aircraft might encounter a puddle of water when trying to takeoff. It might hit a bird. Both of these decrease speed, requiring longer takeoff distance. It’s hard to model unforeseen circumstances. If you include a “fudge factor” – how much “fudge factor” is correct? Before an accreditation authority accepts a model or simulation as reliable, many, many steps must be taken to make sure that it produces credible results, and equally as many steps must be taken to make sure that the limitations of the model and simulation are listed and observed before accepting the result of the simulation as valid.

---

**Luke**: How did the VV&A process develop at DoD? When did it develop? Presumably it developed in one or more domains first, and then spread to become a more universal expectation?

---

**David**: Interesting question. Before we can discuss VV&A, we have to take a slight detour through the history of M&S. And I need to tie several threads of though together.

VV&A is, of course, tied to the use of models and simulations. To be honest, the VV&A of models goes back to the civil war (and probably earlier) – when mathematical models were used to predict firing data (given desired range, here was amount of powder and elevation required. Obviously – the models needed a lot of V&V. However, all it too to V&V the model was to load a cannon and fire it. Not a complex process. The accreditation part was implicit – the Secretary of War used to “authorize” the data to be printed. To really need VV&A, however, complex simulations were needed – and it took computing power to achieve complex M&S.

Over the years, modeling became more and more important, as models and simulation were used for problems that could not easily be solved by traditional mathematical methods. To quote from [Wikipedia article](http://en.wikipedia.org/wiki/Computer_simulation),

> Computer simulation developed hand-in-hand with the rapid growth of the computer, following its first large-scale deployment during the Manhattan Project in World War II to model the process of nuclear detonation. It was a simulation of 12 hard spheres using a Monte Carlo algorithm. Computer simulation is often used as an adjunct to, or substitute for, modeling systems for which simple closed form analytic solutions are not possible. There are many types of computer simulations; their common feature is the attempt to generate a sample of representative scenarios for a model in which a complete enumeration of all possible states of the model would be prohibitive or impossible.
>
>

VV&A really did not become a serious issue until models and simulations were computerized, and computers did not become available until the late 1940s. Starting in the late 1940s, both digital and analog computers became available. However, very few (if any) engineers were trained on how to use this newly-developed computing power. There are various stories of how modeling and simulation became a powerful force in the DOD, but the story I have personal knowledge of is the story of John McLeod – an engineer working at the Naval Air Missile Test Center at Point Mugu on the California coast north of Los Angeles. John was an innovator and after working on analog computers and simulations in the early 1950s, , was John McLeod, who took delivery of a new analog computer sometime in 1952. John was not the only engineer in the aerospace community in Southern California facing the same problems, and a few of them decided to get together as an informal user group to exchange ideas and experiences. To make a long story short, John helped found what became the Society for Computer Simulation (SCS). This organization, over the years, has had members who were leaders and innovators in the field of modeling, simulation, and VV&A. [Note that I had the privilege to be the President of SCS from 2011 – 2012, so I am a bit biased]. The SCS has, to this day, the McLeod award to commemorate the advances John McLeod made in the M&S arena. It is only awarded to those that have made significant contributions to the profession.

The SCS published newsletters. M&S conferences were organized. Leaders in the field were able to meet, publish, and share their expertise. All of which help integrate M&S into more and more domains. As a result of leaders in the field being able to share M&S information, and also as a result huge increase in capabilities and availability of computers to run M&S, the need for VV&A also increased. Over the years, modeling and simulation became more and more important in many domains within the DOD. It helped develop fighters (in fact, aircraft of all types). It helped train our astronauts to land on the moon. It modeled the space shuttle. Complex models and simulations helped us model ballistic missile defense, fight war games with minimal expense (and no lives lost!), and design complex weapon systems. In fact, it’s hard to imagine any technologically sophisticated domain that does not use M&S to save money, save time, and ensure safety. But – these increasingly complex models needed verification and validation, and frequently accreditation,

So – the proliferation in the use of M&S lead to an increased need for VV&A. M&S became so complex that VV&A could not be accomplished without “domain experts” – usually referred to as “Subject Matter Experts” (SMEs) to help. Increased complexity of the M&S lead to increased complexity of the VV&A. Various elements within the DOD were performing VV&A on their own, with little official coordination. To leverage the experience of various DOD components and multiple domains, the DOD saw the need for a single point of coordination. As a result, in the 1990s, the DOD formed the Defense Modeling and Simulation Office (DMSO). The DMSO served as a single point of coordination for all M&S (and VV&A) efforts within the DOD. One of the best DMSO contributions was the VV&A Recommended Practices Guide (VV&A RPG) – first published in 1996. The guide has been updated several times over the years, reflecting the uncreased importance of VV& in the DOD. In 2008 DMSO was renamed the Modeling and Simulation Coordination Office. The MSCO web site (and the latest version of the VV&A Recommended Practices Guide) can be found at [msco.mil](http://www.msco.mil/)

For those of you interested in M&S and VV&A, I cannot recommend the MSCO resource enough. It costs nothing (not even email registration), and contains a vast amount of information about M&S and VV&A. The [RPG Key Concepts document](http://www.msco.mil/documents/Key01_Key_Concepts.pdf) alone contains 34 pages of critical “background” information that you should read before going any further in VV&A.

---

**Luke**: In [Cook (2006)](http://www.ieee-stc.org/proceedings/2006/pdfs/DC1252.pdf) you write that one of the reasons V&V is so difficult comes from “having to ‘backtrack’ and fill in blanks long after development.” What does this mean? Can you give an example?

---

**David**: Let’s imagine you are designing a new fighter aircraft. It is still on the drawing board, and only plans exist.

Rather than spend money building an actual prototype first, you develop mathematical models of the jet to help verify the performance characteristics. You might actually build a very small model of the body – maybe 1/10th size for wind tunnel experiments.

You also build computer-based models and execute them to estimate flight characteristics. The wind-tunnel experiences (even though only on a 1/10th size model) will give data that might make you modify or change the computer-based model. This feedback loop consists of “build model – run simulation – examine data – adjust model” and repeat.

Eventually, you build a working prototype of the jet. Almost certainly, the actual flight characteristics will not exactly match the computer-based model. The prototype is “real world” – so you have to readjust the computer-based model. The “real-world” prototype is just a prototype – and probably not used for high-speed fighting and turns – but the basic data gathered from the flying of the prototype leads to changes in the computer-based model, which will now be used to predict more about high-speed maneuvering.

Back when I worked on the Airborne Laser – we had models that predicted the laser performance before the laser was actually built or fired! The models were based on mathematical principles, on data from other lasers, and from simpler, earlier models that were being improved on. Once a working Airborne Laser was built and fired – we had “real world” data. It was no surprise to find out that the actual characteristics of the laser beam were slightly different that those predicted by the models. For one thing, the models were simplistic – it was impossible to take everything into account. The result was that we took the real-world data, and modified the computer models to permit them to better predict future performance.

The bottom line is that the model is *never* finished. Every time you have additional data from the “real world” that is not an exact match to what the model predicts, the model should be examined, and the model adjusted as necessary.

There are two terms I like to use for models when it comes to VV&A – “anchoring” and “benchmarking”. If I can get another independently-developed model to predict the same events as my model, I have a source of validation. I refer this as benchmarking. Subject matter experts, other simulations, similar events that lend credence to your model – all improve the validity, and provide benchmarking. Anchoring, on the other hand, is when I tie my model directly to real-world data.

As long as the model is being used to predict behavior – it needs to continually be tied or anchored to real-world performance, if possible. If no real-world data is available, then similar models, expert opinions, etc. can be used to also increase the validity.

Just a final note. Models can become so engrained in thoughts that they become “real. For example, I remember when the recent Star Trek movie (the 2009 version) came out. A friend of mine said, after viewing the movie, that he had trouble the the bridge of the USS Enterprise. It did not “look real”. I asked what “real” was – and my friend replied “You know, like the REAL USS Enterprise, the NCC 1701 (referring to the original series). Think about it – all are notional and imaginary (sorry, fellow Trekers) – yet he viewed one as “real” and the other as inaccurate. Models – when no real-world artifact exists – have the potential to become “real” in your mind. It’s worth remembering that a model is NOT real, but only an artifact built to resemble or predict what might (or might not) eventually become real one day.

---

**Luke**: Do you have a sense of how common [formal verification](http://en.wikipedia.org/wiki/Formal_verification) is for software used in DoD applications? Is formal verification of one kind or another *required* for certain kinds of software projects? (Presumably, DoD also uses much software that is not amenable to formal methods.)

---

**David**: I have not worked on any project that uses formal V&V methods.

I used to teach the basics of formal methods (using ‘Z’- pronounced Zed) – but it is very time consuming, and not really fit for a lot of project.

Formal notation shows the correctness of the algorithm from a mathematical standpoint. For modeling and simulation, however, they do not necessarily help you with accreditation – because the formal methods check the correctness of the code, and not necessarily the correlation of the cod eight real-world data.

I have heard that certain extremely critical applications (such as reactor code, and code for the Martian Lander) use formal methods to make sure that the code is correct. However, formal methods take a lot of training and education to use correctly, and it also consumes a lot of time in actual use. Formal methods seldom (never?) speed up the process – they are strictly used to validate the code.

From my experience, I have not work on on any project that made any significant use of formal methods – and in fact, I do not have any colleagues that have used formal methods, either.

---

**Luke**: Thanks, David!

The post [David Cook on the VV&A process](https://intelligence.org/2014/03/07/david-cook/) appeared first on [Machine Intelligence Research Institute](https://intelligence.org).


![](http://intelligence.org/wp-content/uploads/2014/03/Constable.jpg)[Robert L. Constable](http://www.cs.cornell.edu/home/rc/) heads the Nuprl research group in automated reasoning and software verification, and joined the Cornell faculty in 1968. He has supervised over forty PhD students in computer science, including the very first graduate of the CS department. He is known for his work connecting programs and mathematical proofs, which has led to new ways of automating the production of reliable software. He has written three books on this topic as well as numerous research articles. Professor Constable is a graduate of Princeton University where he worked with Alonzo Church, one of the pioneers of computer science.

**Luke Muehlhauser**: In some of your work, e.g. [Bickford & Constable (2008)](http://commonsenseatheism.com/wp-content/uploads/2014/02/Bickford-Constable-Formal-Foundations-of-Computer-Security.pdf), you discuss “correct-by-construction” and “secure-by-construction” methods for designing programs. Could you explain what such methods are, and why they are used?

---

**Robert Constable**:

*Short Summary*

The history of programming languages shows that types are valuable because compilers can check programs for type correctness and inform programmers whether or not their code exactly matches its type specification. Types describe requirements on programming tasks, and very rich type systems of the kind used in mathematics and increasingly in computing describe certain technical tasks completely as we’ll see below. *The more expressive the type system, the more specifications help programmers understand their goals and the programs that achieve them.* System designers can specify many requirements and computing tasks precisely with types and organize them into modules (or objects) that become a blueprint for analyzing and building a computing system.

When types are rich enough to completely specify a task, the methodology we just outlined becomes an example of *correct by construction* programming. This approach has been investigated for mathematical problems for decades and for programming problems since the late 60s and early 70s. For example, if we want to implement a correct by construction factorization based on the *fundamental theorem of arithmetic* (FTA), that every natural number greater than 1 can be factored uniquely into a product of primes, we can first express this task as a programming problem using types. The problem is then solved by coding a computable function *factor with the appropriate type.*  For programmers the type is probably just as clear as the FTA theorem. It is given below. It requires taking a natural number n greater than 1 as input and returning a list of pairs of numbers (p,e) where p is a prime and e is its exponent. We write the list as [(p1,e1), (p2,e2), …, (pn,en)] and require that n =p1e1 x … x pnen where pe means prime p to the power e, and the ei are all natural numbers, and x is multiplication (and is overloaded as the Cartesian product of two types). In addition the full FTA theorem requires that the factorization is unique. We discuss uniqueness later. For the factorization part, consider the example factor(24,750) = [(2,1),(3,2),(5,3),(11,1)]. This is correct since 24,750 = 2 x 9 x 125 x 11, where 53 is 125. *In a rich type system the factorization piece of the fundamental theorem of arithmetic can be described completely as a programming task* *and the resulting program,* factor*, can be seen as a constructive proof accomplishing that factorization. A type checker verifies that the program has the correct type. The program* factor *is* correct by construction *because it is known that it has the type of the problem it was coded to solve.*

*Dependent Types*

This paragraph is a bit technical and can be skipped – or used as a guide for coding factorization in a functional language with dependent types. Here is roughly how the above *factorization task* is described in the type systems of the programming languages of the Agda, Coq and Nuprl proof assistants – a bit more on them later. In English the type specification requires *Nat*, the type of the natural numbers, 0,1,2… and says this: given a natural number *n* larger than 1, compute a list of pairs of numbers, (p,e), where p is a prime number, and e is a natural number such that the input value n can be written as a product of all the prime numbers p on the list, each raised to the power e (exponent e) that is paired with it in the *list lst*. Symbolically this can be written as follows were *N++* is the type *(n:Nat where n>1)*. We also define the type *Prime* as *(n:Nat where prime(n))* for a Boolean valued expression *prime(n)* which we code as a function from *Nat to Booleans.* Here is *the FTA factorization type*.

*(n:N++) ->( lst:List[Prime x Nat] where n = prod(lst) ).*

Note that the symbol *x* in *(Prime x Nat*) denotes the type constructor for *ordered pairs* of numbers. Its elements are ordered pairs *(p,e)* of the right types, *p* a *Prime* and *e* a *Nat*. Of course we need to define the product function, *prod(lst),* over a list *lst* of pairs in the right way. This is simple using functional programming in languages such as Agda, Coq, and Nuprl, in which we have the function type used above. We can assign this kind of programming problem to first year undergraduates, and they will solve it quite easily, especially if we give them a fast *prime(n)* procedure.

Another very nice way to write a compact solution is to use *map* and *fold\_left* functions that are usually supplied in the library of list processing functions. That solution uses the well known *map reduce paradigm* taught in functional programming courses for years and widely used in industry. Solving with map and reduce is a good exercise since the reduce function *fold\_right* is mathematical induction on lists defined as a specific very general recursive program.

*Historical Perspective*

The development of mainstream programming languages from Algol 60 to Java and OCaml shows that type systems have become progressively more expressive. For example, now it is standard that functions can be treated as objects, e.g. passed as inputs, produced as outputs, and created as explicit values as we did in the FTA example above. Type checking and type inference algorithms have kept up with this trend, although OCaml can’t yet check the type we created for FTA, Agda, Coq, Nuprl and other programming languages supported by proof assistants can (they all use some form of *automated reasoning* to help). Static type checking of programs defined using very rich types is a demonstrably effective technology for precisely specifying tasks, documenting code as it’s being written, rendering code more readable by others (including yourself when you revisit it several months later), finding errors, and for safely making changes. With higher-order types such as function types, more programming tasks can be precisely defined, and code can be generated that is completely *correct by construction*.

By 1984 researchers created an extreme point in this type checking technology in which the type systems are so rich that they can describe essentially any mathematical programming task in complete “formal” detail. Types also express assumptions on the data and the computing environment on which the specifications depend. These extremely rich type systems can essentially express any problem in mathematics. Indeed, Coq has been used to formally prove the Four Color Theorem as well as other important theorems in mathematics. The type checking by proof assistants often requires human intervention as well as sophisticated reasoning algorithms. The type systems we have mentioned are strong enough to also describe the uniqueness of the FTA factorization. They can do this by converting any factorization into the standard one produced by the function *factor* defined earlier. That function might be called *normalize*, and it does the job of converting any factorization into the standard one in which the primes come in order and are raised to the highest power possible.

Among the best known modern proof assistants are Agda, Coq, HOL, Isabelle, MetaPRL, Minlog, Nuprl, and PVS. Nuprl was the first of these, built in 1984, and the Coq assistant is written in OCaml and supported by INRIA through the French government. Coq is widely taught and used for programming. These proof assistants help programmers check extremely rich specifications using automated reasoning methods. The types are rich enough to guarantee that programs having the type are correct for the task specified. This is a dominant form of correct by construction programming and sometimes includes the technology of extracting a program from a proof that a specification is achievable; this is called using *proofs as programs*.

*Research Goal*

An overarching research goal is to design type systems and programming assistants to express any computational problem and provide mechanisms for type checking, perhaps with human aid and using automated reasoning tools, no matter how expressive the type system becomes.

Researchers in programming languages, formal methods, and logic are working on this goal worldwide. There are many references in the research literature citing progress toward this goal. The recent book by Adam Chlipala *[Certified Programming with Dependent Types](http://www.amazon.com/Certified-Programming-Dependent-Types-Introduction/dp/0262026651/)*, from MIT Press, 2013, provides an up to date view of the subject focused on the widely used Coq proof assistant. The 1991 book by Simon Thompson, *[Type Theory and Functional Programming](http://www.amazon.com/Functional-Programming-International-Computer-Science/dp/0201416670/)*, from Addison-Wesley, is one of the early comprehensive text books on the subject as it was just gaining momentum. Nuprl is the oldest proof assistant of this kind that is still active since 1984. It was described in the 1986 book [*Implementing Mathematics with The Nuprl Proof Development System*](http://www.nuprl.org/book/). The Nuprl literature and an overview of the area circa 2006 can be found in the technical article “[Innovations in computational type theory using Nuprl](http://www.nuprl.org/documents/Allen/05-jal-final.pdf)” from the *Journal of Applied Logic*, 4, 428-469, 2006. Also the author’s 2010 article on the “[The Triumph of Types](http://www.cs.uoregon.edu/research/summerschool/summer11/lectures/Triumph-of-Types-Extended.pdf)” at the 2010 celebration for *Principia Mathematica* in Cambridge, England recounts the 100 year old story of type theory and connects that to current research on type systems for correct by construction programming. It is available at [www.nuprl.org](http://www.nuprl.org).

When a type system can specify security properties, the checked programs are *secure by construction*. In both correct and secure by construction programming, the specifications include assumptions about the computing environment, say about the network topology. If these assumptions fail to hold, say because the network topology changed in an unexpected way, then the type specifications are no longer sufficient to guarantee correctness. So it is very important to document the assumptions on which the type specification depends.

For many problems these assumptions are stable, and in those cases, this correctness methodology is extremely effective. There are very good examples in the literature; at the web site [www.nuprl.org](http://www.nuprl.org) we recently posted a simple example for the purpose of illustrating the method to newcomers. It is a complete formal proof that allowed us to build a correct by construction program to solve the “maximum segment sum” problem that is used to teach formal program development starting from type specifications.

*Recent Work*

The author and his colleagues at Cornell have recently used Nuprl to create a correct by construction version of the Multi-Paxos distributed consensus protocol that is being used in a database system. This required an implemented theory of distributed computing which is steadily evolving. We have made this protocol *attack-tolerant*, which is a form of security. Consensus protocols are critical to Cloud computing, and industry works very hard to build them correctly and make them secure as they are revised and improved. Researchers have created hundreds of correct by construction programs since 1984 and secure by construction programs as well. Worldwide many more are being built.

Modern proof assistants are steadily becoming more powerful and effective at correct and secure by construction programming because there is a research imperative and an economic incentive to use them. There is also a strong incentive to teach them, as the work on *[Software Foundations](http://www.cis.upenn.edu/~bcpierce/sf/)* by Benjamin Pierce and his colleagues at the University of Pennsylvania demonstrates. Proof assistants are an addictive technology partly because they improve themselves the more they are used which steadily increases their appeal and value.

---

**Luke**: What do you think are the prospects for applying correct-by-construction and secure-by-construction approaches to methods commonly labeled “artificial intelligence”?

---

**Robert**: The *proof assistants* such as Agda, Coq, and Nuprl that support correct by construction programming are themselves examples of artificial intelligence in the sense that they use automated reasoning tools developed in AI. The earliest proof assistants such as the Boyer-Moore prover came out of the AI Department at Edinburgh University. Also one of the seminal books in the field, *The Computer Modeling of Mathematical Reasoning,* by Alan Bundy at Edinburgh was a landmark in AI, published in 1983 by Academic Press.

In due course these proof assistants will use other AI tools as well such as those that can translate formal proofs into natural language proofs. There has already been interesting progress on this front. For instance the article “Verbalization of High-Level Formal Proofs” by Holland-Minkley, Barzilay and me in the 16th National Conference on AI in 1999, 277 – 284 translates Nuprl proofs in number theory into natural language.

It is also the case that these systems are extended using each other. For example, right now my colleagues Vincent Rahli and Abhishek Anand are using the Coq prover to check that new rules they want to add to Nuprl are correct with respect to the semantic model of Nuprl’s constructive type theory that they painstakingly formalized in Coq. They have also checked certain rules using Agda. This activity is precisely what you asked about, and the answer is a clear yes.

When it comes to looking at machine learning algorithms on the other hand, the criteria for their success is more empirical. Do they actually give good performance? Machine learning algorithms were used by professors Regina Barzilay (MIT) and Lillian Lee (Cornell) to improve the performance of machine translation from Nuprl mathematics to natural language in their 2002 article on “bootstrapping lexical choice.” For this kind of work, it does not necessarily make sense to use correct by construction programming. On the other hand, one can imagine a scenario where a correct translation of the mathematics might be critical. In that case, we would probably not try to prove properties of the machine learning algorithm, but we would instead try to capture the meaning of the natural language version and compare that to the original mathematics.

The work of “understanding natural language mathematics” would benefit from correct by construction programming. Already there is excellent work being done to use constructive type theory to provide a semantic basis for natural language understanding. The book by Aarne Ranta, *Type-theoretical grammar,* Oxford, 1994 is an excellent example of this kind of work. It also happens to be one of the best introductions to constructive type theory.

Eventually machine learning algorithms are going to have a major impact on the effectiveness of proof assistants. That is because we can improve the ability of proof assistants to work on various subtasks on their own by teaching them how the expert humans handle these tasks. This kind of work will be an excellent example of both aspects of AI working together. Machine learning guides the machine to a proof, but the automated reasoning tools check that it is a correct proof.

---

**Luke**: What developments do you foresee, during the next 10-20 years, in proof assistants and related tools?

---

**Robert**: I’ve been working on proof assistants, programming logics, and constructive type theory for about forty years, and part of the job is to make predictions five to seven years out and try to attract funding to attain a feasible goal that will advance the field. To maintain credibility, you need to be right enough of the time. I’ve been more right than wrong on eight or so of these predictions. So I am confident about predicting five to seven years out, and I have ideas and hopes about ten years out, and then I have beliefs about the long term.

*The Short Term*

Over the next five years we will see more striking examples of correct by construction programming in building significant software and more examples of verifying software systems as in the work on the seL4 kernel, the SAFE software stack, features of Intel hardware, and important protocols for distributed computing, especially for cloud computing. This will be done because it is becoming cost effective to do it. I don’t see a major industrial effort even for the next ten years. This area will be a focus of continued academic and industrial research, and it will have incremental effects on certain key systems. However, it will remain too expensive for wide spread deployment in industry.

We will see formal tools being used to make both offensive and defensive weapons of cyber warfare more effective. One of the key lessons driving this is the fact that the stuxnet weapon, one of the best and most expensive ever created, was apparently defeated because it had a bug that led to its discovery. Another lesson is that the offensive weapons are improving all the time, and the defense must be more agile, not only building secure by construction “fortress like” systems, but thinking about systems as living responsive entities that learn how to adapt, recover, and maintain their core functionality.

In the realm of warfare, cost has rarely been the decisive factor, and formal methods tools are not outrageously costly compared to other required costs of maintaining a superior military. In this realm, unlike in building and maintaining ships, planes, and rockets and defenses against them, the cost lies in training and recruiting people. These people will have valuable skills for the private sector as well, and that sector will remain starved for well educated talent.

So overall, investing in advanced formal methods technology will be seen as cost effective. Proof assistants are the best tools we have in this sector, and we know how to make them significantly better by investing more. These tools include Agda, Coq, Nuprl, Minlog, and MetaPRL on the synthesis side, and ACL2, Isabelle-HOL, HOL-Light, HOL, and PVS on the verification side. All of them are good, each has important unique features. All have made significant contributions. All of them have ambitions for becoming more effective at what they do best. They will attract more funding, and new systems will emerge as well.

*Medium Term*

Most of the current proof assistants are also being used in education. It seems clear to me that the proof assistants that are well integrated with programming languages, such as Coq with OCaml and ACL2 with Lisp will prosper from being used in programming courses. Other proof assistants will move in this direction as well, surely Nuprl will do that. What happens in education will depend on the forces that bring programming languages in and out of fashion in universities. It seems clear to me that the Coq/OCaml system will have a chance of being widely adopted in university education, at least in Europe and the United States. It has already had a major impact in the Ivy League – Cornell, Harvard, Princeton, UPenn, and Yale are all invested as is MIT. I predict that within five years we will see this kind of programming education among the MOOCs, and we will see it move into select colleges and into top high schools as well.

All of the proof assistants have some special area of expertise. If there is sufficient funding, we will see them capitalize on this and advance their systems in various directions. Nuprl has been very effective in building correct by construction distributed protocols that are also attack tolerant. It is likely that this work will continue and result in building communication primitives into the programming language to match the theory of events already well developed formally. Coq has been used to build a verified C compiler, at UPenn and Harvard Coq is being used to verify the Breeze compiler for new hardware designed to support security features, and at MIT they are using Coq to verify more of the software stack with the Bedrock system. There are other applications of Coq too numerous to mention. We will see additional efforts to formally prove important theorems in mathematics, but it’s hard to predict which ones and by which systems. Current effortsto use proof assistants in homotopy theory will continue and will produce important mathematical insights. Proof assistants will also support the effort to build more certified algorithms along the lines of Kurt Mehlhorn’s impressive work.

*Long Term*

People who first encounter a proof assistant operated by a world class expert is likely to be stunned and shocked. This human/machine partnership can do things that seem impossible. They can solve certain kinds of mathematics problems completely precisely in real time, leaving a readable formal proof in Latex as the record of an hour’s worth of work. Whole books are being written this way as witnessed by the *Software Foundations* project at UPenn mentioned previously. I predict that we will see more books of readable formal mathematics and computing theory produced this way. Around them will emerge a technology for facilitating this writing and its inclusion in widely distributed educational materials. It might be possible to use proof assistants in grading homework from MOOCs. Moreover, the use of proof assistants will sooner or later reach the high schools.

I think we will also see new kinds of remote research collaborations mediated by proof assistants that are shared over the internet. We will see joint research teams around the world attacking certain kinds of theoretical problems and building whole software systems that are correct by construction. We already see glimpses of this in the Coq, Agda, and Nuprl community where the proof assistants share very similar and compatible type theories.

There could be a larger force at work for the very long term, a force of nature that has encoded mechanisms into the human gene pool that spread and preserve information, the information that defines our species. We might not be *Homo sapiens* after all, but *Homo informatis*. The wise bits have not always been so manifest, but the information bits advance just fine – at least so far. That is the aspect of our humanness for which we built the proof assistant partners. They are part of an ever expanding information eco system. There is a chance that proof assistants in a further evolved form will be seen by nature as part of us.

---

**Luke**: Thanks, Bob!

The post [Robert Constable on correct-by-construction programming](https://intelligence.org/2014/03/02/bob-constable/) appeared first on [Machine Intelligence Research Institute](https://intelligence.org).


![](https://intelligence.org/wp-content/uploads/2014/03/Armando-Tacchella.png)[Armando Tacchella](https://sites.google.com/site/armandotacchella/) is Associate Professor of Information Systems at the Faculty of Engineering, at the University of Genoa. He obtained his Ph.D in Electrical and Computer Engineering from the University of Genoa in 2001 and his “Laurea” (M.Sc equivalent) in Computer Engineering in 1997. His teaching activities include graduate courses in AI, formal languages, compilers, and machine learning as well as undergraduate courses in design and analysis of algorithms. His research interest are mainly in the field of AI, with a focus on systems and techniques for automated reasoning and machine learning, and applications to modeling, verification and monitoring of cyber-physical systems. His recent publications focus on improving the dependability of complex control architectures using formal methods, from the design stage till the operational stage of the system. He has published more than forty papers in international conferences and journals including AAAI, IJCAI, CAV, IJCAR, JAI, JAIR, IEEE-TCAD. In 2007 he was awarded by the Italian Association of Artificial Intelligence (AI\*IA) the prize “Marco Somalvico” for the best young Italian researcher in AI.

**Luke Muehlhauser**: My summary of [Transparency in Safety-Critical Systems](http://intelligence.org/2013/08/25/transparency-in-safety-critical-systems/) was:

> Black box testing can provide some confidence that a system will behave as intended, but if a system is built such that it is transparent to human inspection, then additional methods of reliability verification are available. Unfortunately, many of AI’s most useful methods are among its least transparent. Logic-based systems are typically more transparent than statistical methods, but statistical methods are more widely used. There are exceptions to this general rule, and some people are working to make statistical methods more transparent.
>
>

The last sentence applies to a [2009 paper](http://commonsenseatheism.com/wp-content/uploads/2014/01/Pulina-Tacchella-An-abstraction-refinement-approach-to-verification-of-artificial-neural-networks.pdf) you co-authored with [Luca Pulina](https://sites.google.com/site/lpulina/), in which you show formal guarantees about the behavior of a trained multi-layer perceptron (MLP). Could you explain roughly how that works, and what kind of guarantees you were able to prove?

---

**Armando Tacchella**: A trained MLP is just a mathematical function and, in principle, its properties could be verified in a logic that deals with real numbers and interpreted linear functions. However, useful MLPs, even the single hidden layer ones that we consider in our paper, use non-linear transcendental functions in their “activation” units (e.g., logistic or hyperbolic tangent). Any logic language that deals natively with such functions is bound to have a non-decidable decision problem — for a discussion see the introduction of [this paper](http://jsat.ewi.tudelft.nl/content/volume1/JSAT1_11_Fraenzle.pdf).

Our approach to establish safety of MLPs is thus based on an abstraction-refinement paradigm, whereby an “abstract” MLP is obtained from the “concrete” one using some over-approximation whose properties are decidable. This is a technique [introduced by P. Cousot and R. Cousot](http://dl.acm.org/citation.cfm?id=512973) some time ago and more recently [exploited in the model checking community](http://dl.acm.org/citation.cfm?id=876643) to attempt the verification of systems whose properties are not decidable.

If the abstract MLP is found  to satisfy the properties, then also the concrete one does. Otherwise, we obtain an “abstract” counterexample, i.e., a set of inputs which violates a property in the abstract MLP.

If the set of inputs is found to violate the property in the concrete MLP, then the concrete MLP is faulty, and must be fixed. On the other hand, if the concrete MLP works fine when supplied the abstract counterexample, such counterexample is “spurious”, meaning that it is an artifact of the abstraction which was too coarse to be useful.

The abstraction is thus “refined”, i.e., made less coarse, perhaps more expensive to check, and the process iterates, until either the MLP is proven safe, or the process exhausts available resources.

Clearly, abstraction-refinement, does not work around the initial undecidability issue, because an infinite chain of refinements due to spurious counterexamples might always exist. However, in practice, we are often able to establish the validity of a property with a reasonable number of refinements.

In our paper, we instantiate the approach above to the specific case of single hidden layer MLPs which are nevertheless [universal approximators](http://www.sciencedirect.com/science/article/pii/0893608089900208) when subject to some reasonable conditions that we ensure. In particular:

1. We abstract MLPs to corresponding “interval functions”, i.e., if the MLP is y = f(x) then the abstract one is [y] = F([x]), where “F” is the interval function corresponding to “f”.
2. We consider a “safety property”, i.e., if [x] is contained in the input range of f, then [y] will be contained in some interval [l,h] with l < h (the “safety bound”).
3. We prove the property on F using a decision procedure for Linear Arithmetic over reals, e.g., a Satisfiability Modulo Theory (SMT) solver. If the property holds for F, i.e., for all [x] in the domain of f we have that F([x]) is contained in [l,h], then this is the case also for f.

If a counterexample is found, then there is some interval [x] such that F([x]) is not contained in [l,h]. We consider any x contained in [x], and we try the corresponding f(x) to check whether f(x) is in [l,h]. If one is found, then f is faulty and must be fixed. Otherwise, the counterexample was spurious, so we restart from step (1) with a finer grained refinement.

In the end, if the MLP f passes our test, we are guaranteed that whatever input it will receive, it will never exceed its “specifications” in term of minimum and maximum expected output values.

---

**Luke**: In what ways do you think that formal methods helped to “explain” the MLP, and make it more transparent to human programmers?

---

**Armando**: The answer to this question largely depends on what we mean by “explanation”.

If by “explanation” we mean that, using our verification methodology, the neural net user is now able to understand clearly which “reasoning rule” is implemented by which neuron (or set of neurons), then our method does not provide any further insight in that direction.

However, if we look for “behavioral explanation” then, the techniques proposed in our paper, helps in at least two ways:

* the formal analysis of the original trained neural network enables the network user to establish whether its behaviors are in line with the specifications. When this is not the case, the user is given a (set of) counterexample(s) that will aid her in identifying the causes of misbehaviors. Indeed, by perusing the verification outputs, the user learns about the “points of failure” in the input/output space of the network, which, in turn, will enable him to improve his original design (e.g., by collecting more experimental evidence around those points)
* since a neural network is not “classical” software — albeit from a formal point of view we treat it exactly as such — it might be the case that the user cannot readily “fix” the network using counterexamples. Our “repair” procedure can help to fix the network in an automated way by performing a controlled training using the result of “spurious” counterexamples. If the system to be learned by the network is available for experimentation, our repair procedure can also leverage the results of actual counterexamples (a form of “active learning”).

Technically, repair exposes the network to the danger of “overfitting” input data, i.e., the variance in the network performance is very high when considering several different datasets. However, we controlled for that possibility, and it never occurred in our experiments.

---

**Luke**: Before our interview, you wrote to me that:

> I am quite skeptical that we will see any strongly autonomous robot around unless we are able to show (in a convincing way) that the robot is harmless for the environment…
>
>
> Since I do not expect to be able to achieve safe strong autonomy with “correct-by-construction” and “proven in use” philosophies alone (the ones we use to build airplanes), I expect formal analysis to be even more important in robots than in any other implement.
>
>

Were you saying that, despite the tremendous economic and military incentives to build ever-more-capable machines without waiting for safety-related capabilities to catch up, companies and governments *will in fact* not release strongly autonomous programs and robots “into the real world” without first making sure that we have the kind of confidence about their safety that we currently require for (e.g.) autopilot software?

---

**Armando**: Engineers know very well the kind of pressure your are talking about and, in fact, if we consider any catalog of “horror stories” – see e.g. [Software Horror Stories](http://www.cs.tau.ac.il/~nachumd/horror.html) for more than 100 cases related *only* to SW engineering – we cannot rule out the possibility of something bad happening when autonomous robots will hit the market.

For robots endowed with a relatively weak autonomy, or with a relatively low probability of causing serious harm to the surrounding environment, incidents may also be part of a trade-off wherein robot makers are not willing to invest huge amounts of effort to eliminate them completely. In these cases, I expect traditional risk analysis, and engineering best practices to prevail over a major shift to precise verification methods.

For robots endowed with strong autonomy, *and* a non-negligible potential to cause serious harms to the environment, I believe there will be a strong deterrent towards souping-up traditional engineering practices and adopting them in domains that they cannot really handle.

To make the discussion concrete, let us assume that the autonomous car made by Google is going to be sold as a product like, say, a mobile phone. A shipping company can now buy a fleet of such cars, and send them *driverless* to delivery parcels at the doors of their customers. Technologically this is feasible today, but we will not see this happening any time soon. The reasons are several, but let me mention a few that I consider important:

* Ethical. Do we really want to empower a machine to a level that it can *autonomously* decide, e.g., to run-down a pedestrian? Also an autopilot can fail and kill people, but it was clearly not the “will” of the autopilot to do so.
* Technical. Can we ensure that without an operator ready to press “emergency stop” the car will never behave in an hazardous way? Planes are operated by very-well trained officials, who can always “go manual” if anything goes wrong in the autopilot.
* Economical. Even if we had the technology to insure safety, would it be cost-effective? Right now, airplane makers can ensure, through standard engineering practices (e.g., redundancy, stress testing, preventive maintenance) that autopilots are cost effective within their operational requirements.
* Legal. Whose is the liability if an autonomous car causes an accident? The issue becomes especially bothersome if there are more legal entities involved in the product, e.g., the car maker, the system integrator, the “AI” supplier, the operator (assuming on-field configuration is available).

Going back to my initial statement about the importance of formal methods in robots, I expect that a crisp understanding of the phenomena governing the “brains” of an autonomous robot and its interactions with its “body” and the surrounding environment, are of paramount importance if we wish to address any of the above issues in a satisfactory way.

---

**Luke**: [Levitt (1999)](http://commonsenseatheism.com/wp-content/uploads/2014/01/Levitt-Robot-Ethics-Value-Systems-and-Decision-Theoretic-Behaviors.pdf) argued that as intelligent systems become increasingly autonomous, and operate in increasingly non-deterministic, unknown environments, it will not be possible to model their environment and possible behaviors entirely with tools like control theory:

> If robots are to put to more general uses, they will need to operate without human intervention, outdoors, on roads and in normal industrial and residential environments where unpredictable… events routinely occur. It will not be practical, or even safe, to halt robotic actions whenever the robot encounters an unexpected event or ambiguous [percept].
>
>
> Currently, commercial robots determine their actions mostly by control-theoretic feedback. Control-theoretic algorithms require the possibilities of what can happen in the world be represented in models embodied in software programs that allow the robot to pre-determine an appropriate action response to any task-relevant occurrence of visual events. When robots are used in open, uncontrolled environments, it will not be possible to provide the robot with *a priori* models of all the objects and dynamical events that might occur.
>
>
> In order to decide what actions to take in response to un-modeled, unexpected or ambiguously interpreted events in the world, robots will need to augment their processing beyond controlled feedback response, and engage in decision processes.
>
>

Thus, he argues, autonomous programs will eventually need to be decision-theoretic.

What do you think of that prediction? In general, what tools and methods do you think will be most relevant for achieving high safety assurances for the highly autonomous systems of the future?

---

**Armando**: I agree with Levitt when he says that “control theory” alone cannot deal with strongly autonomous robots. This is also what motivates the stream of research in Formal Methods dealing with Robotics and Automation (see, e.g., [Workshop on Formal Methods for Robotics and Automation](http://verifiablerobotics.com/RSS13/) for a recent event on the subject).

However, it might be the case that “engaging in decision theoretic processes” while the robot is performing its tasks raises challenging — to say the least — computational issues. Indeed, it is well known that even simple factual, i.e., Boolean, reasoning may require exponential time to be carried out. Since we expect reality to be much more complex than simple 0/1 evidence, reasoning in every “logic” that the robot is endowed with is going to be computationally hard.

Even assuming the availability of powerful heuristics and/or approximations, the problem is still made tough by real-time requirements in the decision process. Indeed, the robot may not take forever to plan its course of actions, and in many cases high variances in response time are not acceptable as well.

At the same time, making sure that a strongly autonomous robot stays safe using off-line verification only, is also probably not going to happen. This is because of computational issues (combinatorial state-space explosion, undecidability of expressive logics to deal with time/space/probabilities), but also because, as in control theory, most of the situations that will arise while the robot is running are hard to frame.

In my opinion, what we might see is a combination of several approaches, including:

* “Correct by construction” design and implementation
* Formal Verification and testing at the design and implementation stage
* Lightweight reasoning while the robot is executing

Reasoning during execution is probably going to be “self-reflective” by adopting e.g., efficient monitors, that can be automatically synthesized during the design process, and then deployed on the robot to ensure that certain safety conditions are met at all the times without the burden of “deep” logical reasoning.

Finally, actual robots will probably need to be “compliant” to humans, not only in their cognitive/reasoning abilities but also in their structure (see, e.g., [recent development in “Soft Robotics”](http://online.liebertpub.com/toc/soro/1/)). A “compliant” structure will improve security and/or will make resoning about security of the robot easier, since some “intrinsic safety” can be achieved by the use of innovative materials and electronics. In other words, we should not worry just about a “safe mind”, but we must retain the principle that a safer “mind” requires also a safer “body”.

---

**Luke**: Sometimes I like to distinguish “bottom up” vs. “top down” approaches to the challenge of long-term AI safety — that is, to the challenge of making sure that we can continue to achieve confident safety guarantees for AI systems as they grow in autonomy and capability.

“Bottom up” approaches to long-term AI safety build incrementally upon current methods for achieving confident safety guarantees about today’s limited systems. “Top down” approaches begin by trying to imagine the capabilities of the AI systems we’ll have several decades from now, and use those speculations to guide research today.

I would describe your response to my previous question as a “bottom up” perspective on long-term AI safety, and I think it’s the kind of perspective most people working on AI safety would give. To illustrate what I mean by “top down” approaches to long-term AI safety, I’ll provide three examples: (1) [Lampson (1979)](http://www.cs.umd.edu/%7Ejkatz/TEACHING/comp_sec_F04/downloads/confinement.pdf)‘s theoretical description of the “confinement problem,” which motivated the creation of the field of [covert channel communication](http://en.wikipedia.org/wiki/Covert_channel). (2) [Orseau & Ring (2012)](http://agi-conference.org/2012/wp-content/uploads/2012/12/paper_76.pdf)‘s work on the agent-environment boundary typically used as a simplifying assumption in reinforcement learning and other AI models, which I hope will inspire future research into more “naturalistic” AI models that will avoid problematic “Cartesian” mistakes in agent reasoning. (3) [Weaver (2013)](http://arxiv-web.arxiv.org/abs/1312.3626)‘s discussion of paradoxes in rational agency, which I hope will motivate later research into how future self-modifying agent-based AIs can reason coherently about the consequences of modifying their own decision-making code.

Even if you aren’t familiar with these examples already, do you have an opinion on “top down” approaches to long-term AI safety in general, and how they should (or shouldn’t) inform today’s research efforts?

---

**Armando**: Indeed, I was not familiar with the references you cited, and I can see that they are definitely much more “visionary” than what I have suggested previously. Since my background is in Engineering, I must acknowledge a bias towards “bottom-up” views as you define them, or simply “reductionist” and “pragmatic” as I would define them.

Your citations reminded my of a paper [by Weld and Etzioni](https://www.cs.auckland.ac.nz/~nickjhay/papersuni/RoughDraft200207-best_of_SASEMAS.pdf). I often quote and cite this paper in my works because, in my opinion, it had the merit to raise the level of concern of the AI community about “unleashing” strongly autonomous agents in the physical world. The contents of the paper, albeit slightly outdated, still can provide some useful guidelines and vocabulary for current research on safe AI (at least they did it for me). While it is not the kind of paper I would conceive, I believe this kind of speculation is useful to prepare the ground for more tangible achievements.

On the other hand, if we are to chase the current push for stronger autonomy in robots acting in the physical world, then we should probably be very pragmatic about what we can do with current technology in order to ensure safety at all times. While it is possible that future advances in technology will make a “decision theoretic” agent possible within the resource constraints of a mobile robot, it is current technology which is enabling, e.g., autonomous cars to drive around. Therefore, if we want this kind of robots to become products in the next few years, I believe we should focus research on making them trustworthy by leveraging current scientific and technological developments.

---

**Luke**: Are there any other readings you’d recommend, besides [Weld & Etzioni (1994)](https://www.cs.auckland.ac.nz/%7Enickjhay/papersuni/RoughDraft200207-best_of_SASEMAS.pdf), on the subject of long-term AI safety?

---

**Armando**: Weld and Etzioni returned on the subject of safe AI in their paper “[Intelligent agents on the internet: Fact, fiction, and forecast](http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.6.7045&rep=rep1&type=pdf)”

A similar theme is touched upon by Eichmann in his “[Ethical web agents](http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.57.3585&rep=rep1&type=pdf)“.

I have found out about the “Call to Arms” by Etzioni and Weld in a series of papers by Gordon, of which I would deem “[Asimovian Adaptive Agents](http://www.jair.org/media/720/live-720-1895-jair.pdf)” as the most relevant.

I had just stumbled upon (but still did not have a chance to read thoroughly) a relatively recent book by Wilks “Close Engagements with Artificial Companions: Key Social, Psychological, Ethical and Design Issues” ([see a review here](http://oldsite.aclweb.org/anthology-new/J/J11/J11-2006.pdf)). The book seems controversial at times, but it does contain many different views about artificial companions and it does raise some issues that we should probably keep well in mind when designing physically situated agents.

Finally, to have an idea about concrete projects going on in the field of robotics where safety is considered both a prominent value and an essential challenge, [this site](http://www.robotcompanions.eu/) lists many interesting recent developments and ideas for further research.

---

**Luke**: Thanks, Armando!

The post [Armando Tacchella on Safety in Future AI Systems](https://intelligence.org/2014/03/02/armando-tacchella/) appeared first on [Machine Intelligence Research Institute](https://intelligence.org).