Computational Models of Ethical Reasoning: Challenges, Initial Steps, and Future Directions

H ow can machines support or, even more significantly, replace humans in performing ethical reasoning? This question greatly interests machine ethics researchers. Imbuing a computer with the ability to reason about ethical problems and dilemmas is as difficult a task as there is for AI scientists and engineers. First, ethical reasoning is based on abstract principles that you can't easily apply in a formal, deductive fashion. So, the favorite tools of logicians and mathematicians, such as firstorder logic, aren't applicable. Second, throughout intellectual history, philosophers have proposed many theoretical frameworks, such as Aristotelian virtue theory, 1 the ethics of respect for persons, 2 act utilitarianism, 3 utilitarianism, 4 and prima facie duties, 5 and no universal agreement exists on which ethical theory or approach is the best. Furthermore, any of these theories or approaches could be the focus of inquiry, but all are difficult to make computational without relying on simplifying assumptions and subjective interpretation. Finally, ethical issues touch human beings profoundly and fundamentally. The premises, beliefs, and principles that humans use to make ethical decisions are quite varied, not fully understood, and often inextricably intertwined with religious beliefs. How do you take such uniquely human characteristics and distill them into a computer program? Undaunted by the challenge, scientists and engineers have, over the past 15 years, developed several computer programs that take initial steps in addressing these difficult problems. Here, I briefly describe a few of these programs and discuss in detail two programs that I developed, both of which employ techniques from the area of AI known as case-based reasoning and implement aspects of the ethical approach known as casuistry. One of these programs, Truth-Teller, accepts a pair of ethical dilemmas and describes the salient similarities and differences between them, from both an ethical and a pragmatic perspective. The other program, SIROCCO, accepts a single ethical dilemma and retrieves other cases and ethical principles that might be relevant. Neither program was designed to reach an ethical decision. I believe that reaching an ethical conclusion is, in the end, a human decision maker's obligation. Even if I believed these computational models were up to the task of autonomously reaching correct conclusions to ethical dilemmas, having a computer program propose decisions oversimplifies the obligations of human beings and makes assumptions about the "best" form of ethical reasoning. Rather, the aim of my work has been to develop programs that produce relevant information that can help humans as they struggle with difficult ethical decisions, as opposed to providing fully supported ethical arguments and conclusions. In other words, the programs are intended to stimulate the "moral imagination" 6 and help humans reach decisions. The difficulties in developing machines that can reason ethically present an intellectual and engineering challenge of the first order to the field of machine ethics. The long history of science and technology is rife with problems that have excited the innovative spirit of scientists, philosophers, and engineers, and the nascent field of machine ethics presents many new challenges. Even if I achieve my goal of creating a reliable "ethical assistant" but don't develop a fully autonomous ethical reasoner, it will be a significant step forward for machine ethics.
 Attempts to build programs that support or implement ethical reasoning Two of the earliest programs aimed at ethical reasoning, the Ethos System 7 and the Dax Cowart program, 8 were designed to help students work their way through thorny practical-ethics problems. Neither is an AI program, but each models aspects of ethical reasoning and acts as a pedagogical resource. Both programs feature an open, exploratory environment complete with video clips to provide a visceral experience of ethicalproblem solving. Donald Searing developed Ethos to accompany the engineering ethics textbook by Charles Harris, Michael Pritchard, and Michael Rabins. 6 Ethos provides a few prepackaged example dilemmas, including video clips and interviews, to help students explore real ethical dilemmas that arise in engineering. Ethos encourages rational, consistent ethical-problem solving in two ways. First, it provides a framework in which you can rationally apply moral beliefs. Second, it records the step-by-step decisions that an ethical decision maker takes when resolving a dilemma, so that those steps can later be reflected upon. The program decomposes moral decision making into three major steps: 1. framing the problem, 2. outlining the alternatives, and 3. evaluating those alternatives. The Dax Cowart program is an interactive multimedia program designed to explore the practical-ethics issue of a person's right to die. The program focuses on the real case of Dax Cowart, a victim of severe burns, crippling injuries, and blindness who insists on his right to die throughout enforced treatment for his condition. The central question is whether he should be allowed to die. The program presents video clips of interviews with Dax's doctor, lawyer, mother, nurses, and Dax himself to let the user experience the issue from different viewpoints. The program also presents clips of Dax's painful burn treatment to provide an intimate sense of his predicament. The program periodically asks the user to make judgments on whether Dax's request to die should be granted. Depending on how the user answers, the program branches to present information and viewpoints that might cause reconsideration of that judgment. Both Ethos and the Dax Cowart program are intended to instill a deep appreciation of the complexities of ethical decision making by letting the user interactively and iteratively engage with the various resources they provide. However, neither program involves any intelligent processing. All the steps and displays are effectively "canned," with deterministic feedback based on the user's actions. Research that has focused more specifically on the computational modeling of ethical reasoning includes that of Russell Robbins and William Wallace. 9 Their proposed computational model combines collaborative problem solving (that is, multiple human subjects discussing an ethical issue), the psychological theory of planned behavior, and the beliefdesire-intention model of agency. As a decision aid, this computational model is intended to take on multiple roles, including advisor, group facilitator, interaction coach, and forecaster for human subjects as they discuss and try to resolve ethical dilemmas. This system has only been conceptually designed, not implemented, and the authors might have overreached, in a practical sense, by trying to combine such a wide range of theories and technologies in a single model. But their ideas could serve as the foundation for future computational models of ethical reasoning. Earlier, Robbins, Wallace, and Bill Puka 10 implemented and experimented with a more modest system for supporting ethical-problem solving. They implemented this system as a series of Web pages, containing links to relevant ethical theories and principles, and a simple ethics "coach." The three researchers performed an empirical study in which the system's users were able to identify, for instance, more alternative ways to address a given ethical problem than subjects who used Web pages that didn't have the links or coaching. Their research is an excellent illustration of the difficulties confronting those who wish to build computational models of ethical reasoning: developing a relatively straightforward model, one that doesn't use AI or other advanced techniques, is within reach but is also limited in depth and fidelity to actual ethical reasoning. The more complex-yet more realistic-computational model that Robbins and Wallace conceived (but haven't imple-mented) will take considerable work to advance from concept to reality. Unlike my research and the research I've just described, Michael Anderson, Susan Anderson, and Chris Armen aim to develop programs that reason ethically and come to their own ethical conclusions. Susan Anderson states this goal most clearly. 11 Anderson, Anderson, and Armen have developed prototype computational models of ethical reasoning based on well-known theoretical frameworks. The first prototype they implemented was Jeremy, 12 based on Jeremy Bentham's theory of hedonistic act utilitarianism. 3 Bentham's utilitarianism proposes a "moral arithmetic" in which one calculates the pleasure and displeasure of those affected by every possible outcome in an ethical dilemma. The Jeremy program operationalizes moral arithmetic by computing "total net pleasure" for each alternative action, using this simple formula: Total Net Pleasure = Sum-Of (Intensity \* Duration \* Probability) for all affected individuals. Jeremy chooses as the correct action the action with the highest total net pleasure. Rough estimates must be provided of the intensity, duration, and probability, given a small set of possible values (for example, 0.8, 0.5, and 0.2 for probability estimates), for each action per individual. The three researchers claim that Jeremy is impartial and considers all actions. Anderson, Anderson, and Armen built a second prototype, W.D., 12 based on William D. Ross's seven prima facie duties 5 and John Rawls' theory of reflective equilibrium. 13 The general idea behind W.D. is that Ross's theory provides a comprehensive set of duties or principles relevant to ethical cases, such as justice, beneficence, and nonmaleficence, while Rawls' approach provides the foundation for a procedure to make ethical decisions given those duties. In particular, Rawls' approach inspired Anderson, Anderson, and Armen to imbue W.D. with a decision procedure that generalizes rules (or principles) from cases and tests the generalizations on further cases, with further iteration until the generated rules match ethical intuition. This procedure defines cases simply as an evaluation of a set of duties using integer estimates (ranging from -2 to 2) regarding how strongly each duty was violated or satisfied (for example, -2 represents a serious violation, and +2 is a maximal satisfaction of duty). Rawls' approach lends itself well to an AI machine learning algorithm. In particular, W.D. uses inductive-logic programming to learn Horn clause rules from each case, until the rules reach a "steady state" and can process subsequent cases without further learning. A third program that Anderson, Anderson, and Armen developed, MedEthEx, 14 is very similar to W.D., except that it's specific to medical ethics and uses Tom Beauchamp and James Childress's Principles of Biomedical Ethics 15 in place of Ross's prima facie duties. Like W.D., MedEthEx relies on reflective equilibrium and employs integer evaluation of principles, and it uses the same machine learning technique. This use of machine learning to support ethical reasoning is novel and quite promising. The natural fit between reflective equilibrium and inductive-logic programming is especially striking. On the other hand, their research might oversimplify the task of interpreting and evaluating ethical principles and duties. Reducing each principle and duty to an integer value on a five-point scale renders it almost trivial to apply machine learning to the resulting data, because the search space becomes drastically reduced. But can you really reduce principles such as beneficence or nonmaleficence to single values? Wouldn't people likely disagree on such simple dispositions of duties and principles? In my experience, and exemplified by the two computational models that I later discuss, perhaps the toughest problem in ethical reasoning is understanding and interpreting the subtleties and application of principles. Very-high-level principles such as beneficence and nonmaleficence, if applied to specific situations, naturally involve bridging a huge gap between the abstract and the specific. One potential way to bridge the gap is to use cases as exemplars and explanations of "open textured" principles, 16 not simply as a means to generalize rules and principles. (Open-textured terms are conditions, premises, or clauses that aren't precise, cover a wide range of specific facts, or are highly subject to interpretation and might even have different meanings in different contexts.) This is the tack that a different group of philosophers, the casuists, take.
 Truth-Teller I intended Truth-Teller as a first step in implementing a computational model of casuistic reasoning, in which someone makes a decision by comparing a problem to paradigmatic, real, or hypothetical cases. 17 Casuistry long ago fell out of favor with many philosophers and ethicists because they believed it to be too imprecise and based on moral intuitions. However, medical ethicists have recently been using it to help solve practical dilemmas. 18, 19 Unlike the approach that W.D. and MedEthEx use, casuistry (and hence Truth-Teller) focuses on the power of specific cases and case comparison, not on rules generalized from case evaluations. Truth-Teller compares pairs of given cases presenting ethical dilemmas about whether to tell the truth. 20, 21 The program marshals ethically relevant similarities and differences between the two cases from the perspective of the "truth teller" (the person facing the dilemma) and reports them to the user. In particular, it points out reasons for telling the truth (or not) that • apply to both cases, • apply more strongly in one case than another, or • apply to only one case. I adapted the dilemmas for Truth-Teller from Scruples, a party game in which participants challenge one another to resolve everyday ethical dilemmas. Figure 1 shows Truth-Teller's output in comparing two dilemmas. As you can see, these cases share similar themes, relationships, and structure. Truth-Teller recognizes the similarity and points this out in the first paragraph of its comparison text. The truth tellers in the two scenarios, Felicia and Kevin, essentially share the same reasons for telling the truth or not, and the second and third paragraphs of Truth-Teller's output detail this. There are no reasons for telling the truth (or not) that exist in one case but not
 Truth-Teller is comparing the following cases: CASE 1: Felicia is a young lawyer running her own business. A client, Henry, requires a complex legal transaction that Felicia has never done before. This type of transaction is rarely done by an inexperienced lawyer; usually attorneys handle many simper cases of the same type before handling such a complex case. In addition, if Felicia bungles the case, Henry and his family will go bankrupt. Should Felicia tell Henry about her inexperience in the matter? CASE 2: Kevin is a lawyer fresh out of law school. A client, Alida, requires a complex legal transaction that Kevin has never done before. However, Kevin was specifically trained in this type of transaction during law school and lawyers routinely accept this type of case fresh out of law school. Additionally, the consequences of the case, should it go badly, are minimal. Should Kevin tell the client about his inexperience in this matter?
 Truth-Teller's analysis: The decision makers, Felicia and Kevin, are confronted with very similar dilemmas because they share reasons both to tell the truth and not to tell the truth. The cases also share similar relationship contexts. The relationship between Felicia and Henry is identical to the relationship between Kevin and Alida; they are both 'is attorney of' relations. Felicia and Kevin share reasons to tell the truth. First, both protagonists share the reason to provide sales information so that a consumer can make an informed decision. In addition, Felicia and Kevin share the reason to disclose professional inexperience for, respectively, Henry and Alida. Third, both actors share the general reason to avoid harm. More specifically, Felicia has the reason to avoid a financial loss for Henrys family and Henry, while Kevin has the reason to avoid an unknown future harm for Alida. Finally, both actors share the reason to establish goodwill for future benefit. Felicia and Kevin also share reasons to not tell the truth. Both protagonists share the reason to enhance professional status and opportunities. Second, Felicia and Kevin share the reason to realize a financial gain for themselves. However, these quandaries are distinguishable. An argument can be made that Felicia has a stronger basis for telling the truth than Kevin. The reason 'to disclose professional inexperience,' a shared reason for telling the truth, is stronger in Felicia's case, since this type of complicated case is rarely done by an inexperienced lawyer. Additionally, the shared reason for telling the truth 'to avoid harm' is stronger in Felicia's case, because (1) Henry and his family will go bankrupt if the case is lost and (2) it is more acute ('One should protect oneself and others from serious harm.') the other, so Truth-Teller makes no comment on this. Finally, the last paragraph of Truth-Teller's comparison text points out each case's distinguishing features. Felicia has a greater obligation than Kevin to reveal her inexperience owing to established custom (that is, inexperienced lawyers rarely perform this transaction) and more severe consequences (that is, Henry and his family will go bankrupt if she fails). Figure 2 depicts Truth-Teller's semantic representation of Felicia's case in figure 1 . This representation served as input to the program to perform its reasoning. (That is, the Truth-Teller program doesn't "read" natural language input.) In this case, Felicia (the truth teller) can take one of two possible actions: tell Henry (the "truth receiver") the truth or remain silent about her inexperience. The truth teller might be able to take other actions in a scenario, such as trying to resolve a situation through a third party. Each possible action a protagonist can take has supporting reasons. For instance, two reasons for Felicia to tell the truth are • fairness (reason 2)-Felicia has an obligation to fairly disclose her inexperience-and • avoiding harm (reason 4)-Felicia might avoid financial harm to Henry and his family by telling the truth. Truth-Teller compares pairs of cases by aligning and comparing the reasons that support telling the truth or not in each case. More specifically, Truth-Teller's comparison method comprises four analysis phases: 1. Alignment builds a mapping between the reasons in the two cases. That is, it indicates the reasons that are the same and different across the two representations. 2. Qualification identifies special relationships among actors, actions, and reasons that augment or diminish the reasons' importance. For example, telling the truth to a family member is typically more important than telling the truth to a stranger. 3. Marshaling selects particular similar or differentiating reasons for arguing that (a) one case is as strong as or stronger than the other with respect to a conclusion, (b) the cases are only weakly comparable, or (c) the cases aren't comparable. 4. Interpretation generates prose that accurately presents the marshalled information so that a nontechnical human user can understand it. This phase generated the text labeled "Truth-Teller's Analysis" in figure 1 . To test Truth-Teller's ability to compare cases, I performed an evaluation in which five professional ethicists were asked to grade the program's output. The experts assessed its reasonableness (R), completeness (C), and context sensitivity (CS), on a scale of 1 (low) to 10 (high), for 20 case comparisons similar to the comparison in figure 1. The experts assigned mean scores of R = 6.3, C = 6.2, and CS = 6.1 across these comparisons. The evaluation also included two comparisons written by graduate students; not surprisingly, the ethicists graded these comparisons somewhat higher, at mean scores of R = 8.2, C = 7.7, and CS = 7.8. On the other hand, two of Truth-Teller's comparisons received higher scores than one of the human evaluations. These results indicate that Truth-Teller is moderately successful at comparing truthtelling dilemmas. Because I instructed the expert ethicists to "evaluate comparisons as you would evaluate short answers written by college undergraduates," it's quite encouraging that Truth-Teller performed as well as it did. However, two questions naturally arise: Why did the ethicists view Truth-Teller's comparisons as somewhat inferior to the humans'? How could Truth-Teller be brought closer to human performance? Several evaluators questioned Truth-Teller's lack of hypothetical analysis; the program makes fixed assumptions about the facts (that is, reasons, actions, and actors). One way to counter this might be to develop techniques that let Truth-Teller suggest hypothetical variations to problems, along the lines of the legal-reasoning program HYPO. 22 For instance, in the comparison in figure 1 , Truth-Teller might suggest that, if an (unstated and thus hypothetical) long-standing relationship between Felicia and Henry exists, there is additional onus on Felicia to reveal her inexperience. Another criticism of Truth-Teller involved the program's somewhat rigid approach of enumerating supporting, individual reasons and not relating one reason to another. Some form of reason aggregation might address this issue, by discussing the overall import of supporting reasons rather than focusing on individual reasons.
 SIROCCO I developed SIROCCO as a second step in exploring casuistry and how to realize it in a computational model. In particular, SIROCCO attempts to bridge the gap between general principles and concrete facts of cases. The program emulates how an ethical review board within the National Society of Professional Engineers decides cases by referring to, and balancing between, ethical codes and past cases. 23 Engineering-ethics principles, while more specific than general ethical duties, such as Ross's prima facie duties (for example, justice, beneficence, and nonmaleficence), still tend to be too general to decide cases. So, the NSPE review board often uses past cases to illuminate the reasoning behind principles and as precedent in deciding new cases. Consider, for example, the following ethical code from the NSPE: Code II.5.a. Engineers shall not falsify or permit misrepresentation of their ... academic or professional qualifications. They shall not misrepresent or exaggerate their degree of responsibility in or for the subject matter of prior assignments. Brochures or other presentations incident to the solicitation of employment shall not misrepresent pertinent facts concerning employers, employees, associates, joint ventures or past accomplishments with the intent and purpose of enhancing their qualifications and their work. This code specializes the more general principle of "honesty" in an engineering context. Each sentence deals with a different aspect of "misrepresentation of an engineer" and covers a wide range of possible circumstances. However, the code does not specifically state the precise circumstances that support application. Knowing whether this code applies to a particular fact situation requires that you recognize the applicability of and interpret open-textured terms and phrases in the code, such as "misrepresentation" and "intent and purpose of enhancing their qualifications." While these engineering-ethics codes are an example of abstract codes, they are by no means distinctive. Many principles and codes, generally applicable or domain specific, are abstract. Principles also typically conflict with one another in specific circumstances, with no clear resolution to that conflict. In the NSPE's analyses of over 500 engineering cases, it interprets principles such as II.5.a in the context of the facts of real cases, decides when one principle takes precedence over another, and provides a rich, extensional representation of principles. SIROCCO's goal, given a new case to analyze, is to provide the basic information with which a human reasoner-for instance, a member of the NSPE review board-could answer an ethical question and then build an argument or rationale for that conclusion. 24 Figure 3 shows an example of SIROCCO's output. First, the output displays the input case's facts and the question that the case raises. This particular case involves an engineering technician who discovers what he believes to be hazardous waste, suggesting a need to notify federal authorities. However, when the technician asks his boss, Engineer B, what to do with his finding, he's told not to mention his suspicions of hazardous waste to this important client, who might face cleanup expenses and legal ramifications from the finding. The question is whether it was ethical for Engineer B to give preference to his duty to his client over public safety. SIROCCO's analysis of the case consists of • a list of possibly relevant codes, • a list of possibly relevant past cases, and • a list of additional suggestions. You can run SIROCCO on over 200 ethical dilemmas and view analysis such as that in figure 3 by visiting http://sirocco.lrdc.pitt. edu/sirocco/index.html. SIROCCO accepts input, or target, cases in a detailed case-representation language called the Engineering Transcription Language. ETL represents a scenario's actions and events as a Fact Chronology of individual Technician A is a field technician employed by a consulting environmental engineering firm. At the direction of his supervisor Engineer B, Technician A samples the contents of drums located on the property of a client. Based on Technician A's past experience, it is his opinion that analysis of the sample would most likely determine that the drum contents would be classified as hazardous waste. If the material is hazardous waste, Technician A knows that certain steps would legally have to be taken to transport and properly dispose of the drum including notifying the proper federal and state authorities. Technician A asks his supervisor Engineer B what to do with the samples. Engineer B tells Technician A only to document the existence of the samples. Technician A is then told by Engineer B that since the client does other business with the firm, Engineer B will tell the client where the drums are located but do nothing else. Thereafter, Engineer B informs the client of the presence of drums containing "questionable material" and suggests that they be removed. The client contacts another firm and has the material removed.
 Question: Was it ethical for Engineer B not to inform his client that he suspected hazardous material? SIROCCO utilizes knowledge of past case analyses, including past retrieval of ethical codes and cases and the way past analyses utilized these knowledge elements, to support its retrieval and analysis in the target case. The program employs a two-stage graph-mapping algorithm to retrieve codes and cases. Stage 1 performs a "surface match" by retrieving all source cases from the program's database that share any fact with the target case. The cases in the database, which number over 400, are represented in EETL, an extended version of ETL. This stage then computes a score for all retrieved cases, based on fact matching between the target case and each source case, and outputs a list of ranked candidate source cases. Using A\* search (an AI technique), stage 2 attempts a structural mapping between the target case and each of the N top-ranking candidate source cases from stage 1. This search takes into account temporal relations and abstract matches. SIROCCO'S Analyzer module organizes and displays the top-rated structural mappings that the A\* search uncovered. The ethical codes cited in these top-rated source cases are also evaluated by the Analyzer for how well they apply to the target case. Figure 3 is an example of the Analyzer's output. I performed a formal experiment with SIROCCO to test how well it retrieved codes and cases compared to several other retrieval methods, including two full-text retrieval systems, MG (Managing Gigabytes) and Extended-MG. I also compared SIROCCO to three other methods: one that randomly selects codes and cases (Random), one that randomly selects codes and cases from the most frequently cited codes and cases (Informed-Random), and an ablated version of SIROCCO with operationalization function-ality excised (Non-Op SIROCCO). Using the F-Measure metric, I scored each method on the basis of how well its retrieved codes and cases overlapped with the humans' (that is, the NSPE review board's) retrieved codes and cases in evaluating the same cases. The F-measure is an information retrieval metric that combines precision and recall. 25, 26 I compared the methods on two dimensions: exact matching (the method and the humans retrieved precisely the same codes and cases) and inexact matching (the method and the humans retrieved closely related codes and cases). Figure 4 summarizes the results. The results showed that SIROCCO was significantly more accurate at retrieving relevant codes and cases than all the other methods except Extended-MG, to which it came close to being significantly more accurate (p = 0.057). Because these automated methods, particularly the information retrieval approaches MG and Extended-MG, are arguably the most competitive with SIROCCO, this experiment shows that SIROCCO is an able ethics-reasoning companion. On the other hand, as figure 4 shows, SIROCCO performed worse than the ethical review board (0.21 and 0.46 can be roughly interpreted as 21 percent and 46 percent overlapping with the board selections). At least some, if not most, of this discrepancy is because the inexact-matching metric doesn't fully capture correct selections. For example, in many instances SIROCCO actually selected a code or case that was arguably applicable to a case but that the board didn't select. In other words, using the review board as the "gold standard" has its flaws. Nevertheless, it's fair to say that although SIROCCO performs well, it doesn't perform quite at the level of an expert human reasoner.
 The relationship between Truth-Teller and SIROCCO Fundamentally, Truth-Teller and SIROCCO have different purposes. Truth-Teller is more useful in helping users compare cases. Although SIROCCO also compares cases, its results don't focus on case comparisons and presenting those comparisons to the user. Rather, SIROCCO is more useful for collecting a variety of relevant information that a user should consider in evaluating a new ethical dilemma. While Truth-Teller has a clear advantage in comparing cases and explaining those comparisons, it ignores the problem of how to identify potentially comparable cases in the first place. The program compares any pair of cases it's provided, no matter how different they might be. SIROCCO, on the other hand, determines which cases will most likely be relevant to a given target case and thus worth comparing. An interesting synthesis of the two programs would be to have SIROCCO retrieve comparable cases and have Truth-Teller compare them. For instance, the "algorithm" in figure 5 , adapted from Albert Jonsen and Stephen Toulmin, 17 represents the general approach a casuist would take in solving an ethical dilemma. First, given a new case, the casuistic reasoner would find cases (paradigms, hypotheticals, or real cases) that test the principles or policies in play in the new case. That is, the casuist would reach into its knowledge base of cases to find the past cases that might provide guidance in the new case. In effect, this is what SIROCCO does. Second, the reasoner would compare the new cases to the cases it retrieved. While SIROCCO does this to a limited extent, this is where Truth-Teller's capability to compare and contrast given cases, at a reasonably fine level of detail, would come in. Third, the casuist would argue how to resolve conflicting reasons. Both Truth-Teller and SIROCCO have at least a limited capability to perform this step. This is illustrated, for example, in Truth-Teller's example output, at the bottom of figure 1 , in which the program distinguishes the two cases by stating the reasons that apply more strongly in Felicia's case. SIROCCO does this by suggesting that one principle might override another in these particular circumstances (see the "Additional Suggestions" at the bottom of figure 3 ). Finally, in keeping with my vision of how to apply computational models to ethical decision making, a human would make the final decision about this ethical dilemma. To fully realize the casuistic problemsolving approach of figure 5 and combine the complementary capabilities of Truth-Teller and SIROCCO, the two programs would need common representational elements. SIROCCO uses primitives that closely model some of a fact situation's actions and events to represent cases as complex narratives. In this sense, SIROCCO's representational approach is more sophisticated and general than Truth-Teller's. On the other hand, SIROCCO's case comparisons aren't nearly as precise and issue-oriented as Truth-Teller's. Both Truth-Teller and SIROCCO focus and rely heavily on a knowledge representation of ethics, unlike, for instance, the programs of Anderson, Anderson, and Armen, which have little reliance on representation. The knowledge representation approach to building computational models of ethical reasoning has strengths and weaknesses. Its strength is its ability to represent cases and principles at a rather fine level of detail. For instance, a detailed engineering-ethics ontology supports SIROCCO, and a representation of reasons underlies Truth-Teller, as figure 2 shows. Such representation not only supports each model's reasoning approaches but also lets the models provide relatively rich explanations of their reasoning, as exemplified by the programs' output in figures 1 and 3. On the other hand, each model's representation is necessarily specific to its task and domain. So, Truth-Teller has a rich representation of truth-telling dilemmas-but not much else. SIROCCO has a deep representation of engineering-ethics principles and engineering scenarios, but no knowledge of more general ethical problem solving, such as the reasoning model embodied in W.D. and MedEthEx. So, another step required for unifying Truth-Teller and SIROCCO and implementing the casuistic approach of figure 5 would be a synthesis and generalization of their respective representational models.
 Lessons learned The primary lesson I've learned from the Truth-Teller and SIROCCO projects is that ethical reasoning is fundamentally different from reasoning in more formalized domains. In ethical reasoning, "inference rules" are available almost exclusively at an abstract level, in the form of principles. The difficulty of using formal logic to address and form arguments in such domains has long been recognized, 27 and some AI practitioners, particularly those interested in legal reasoning, have grappled with this issue. As Kevin Ashley pointed out, "The legal domain is harder to model than mathematical or scientific domains because deductive logic, one of the computer scientist's primary tools, does not work in it." 22 The ethical-reasoning domain, like the legal domain, can be viewed as a weak analytic domain where the given "rules" (that is, laws, codes, or principles) are available almost exclusively at a highly abstract, conceptual level. This means that the rules might contain open-textured terms. Also, in a weak analytic domain, abstract rules often conflict with one another in particular situations with no deductive or formal means of arbitrating such conflicts. That is, more than one rule might appear to apply to a given fact situation, but neither the abstract rules nor the general knowledge of the domain provides clear resolution. Another important lesson from the two projects is the sheer difficulty of imbuing a computer program with the sort of flexible intelligence required to perform ethical analysis. While both programs performed reasonably well in the studies I mentioned earlier, neither performed at an expert human's level. While the goal wasn't to emulate human ability, taking the task of ethical decision making away from humans, it's important that computational artifacts that purport to support ethical reasoning at least perform well enough to encourage humans to use the programs as aids in their own reasoning. As of this writing, only the Truth-Teller and SIROCCO computational models (and, perhaps to a lesser extent, Robbins, Wallace, and Puka's Web-based system) have been empirically tested in a way that might inspire faith in their performance. My contention that computer programs should act only as aids in ethical reasoning isn't due to a high regard for human ethical decision making. Of course, humans often make errors in ethical reasoning. Rather, my position is based, as I suggested earlier, on the existence of so many plausible, competing approaches to ethical problem solving. Which philosophical method can claim to be the "correct" approach to ethical reasoning, in the same sense that we accept calculus as a means of solving engineering problems and accept first-order logic for solving syllogisms? It's difficult to imagine that a single ethical-reasoning approach embodied in a single computer program could deliver even close to a definitive approach to ethical reasoning. Of course, many approaches might be considered "good enough" without being definitive. But the bar will likely be much higher for autonomous computer-based systems making decisions in an area as sensitive and personal to humans as ethical reasoning. Also, it's presumptuous to think that you could fully implement the subtleties of any of the well-known philosophical systems of ethics in a computer program. Any implementation of one of these theories is necessarily based on simplifying assumptions and subjective interpretation of that theory. For instance, as I mentioned before, W.D. simplifies the evaluation of Ross's prima facie duties by assigning each a score on a fivepoint scale. Both Truth-Teller and SIROCCO also make simplifying assumptions, such as Truth-Teller representing only reasons that support telling the truth or not, and not the circumstances leading to these reasons. Of course, making simplifying assumptions is a necessary starting point for gaining traction in the difficult area of ethical reasoning. My final reason for using computational models as only aids in ethical reasoning is my belief that humans simply won't accept autonomous computer agents making such decisions for them. But they might accept programs as advisors. G iven my view of the role of computational models and how they could (and should) support humans, a natural and fruitful next step is to use computational models of ethical reasoning as teaching aids. Ilya Goldin, Kevin Ashley, and Rosa Pinkus have taken steps in this direction. PETE (Professional Ethics Tutoring Environment) is a software tutor that leads a student step-by-step in preparing cases for class discussion. It encourages students to compare their answers to those of other students. 28 My most recent work and interest has also involved intelligent tutoring systems. 29, 30 As part of this focus, I've started to investigate whether case comparisons, such as those that Truth-Teller produces, could be the basis for an intelligent tutor. The idea is to explore whether Truth-Teller's comparison rules and procedures can • be improved and extended to cover the kinds of reasons involved in comparing more technically complex cases, such as those tackled by SIROCCO, and • serve as the basis of a Cognitive Tutor to help a student understand and perform Truth-Teller's phases. Cognitive Tutors are based on John Anderson's ACT-R (adaptive control of thoughtrational) theory, according to which humans use production rules, modular If-Then constructs, to perform problem-solving steps in a variety of domains. 31 Key concepts underlying Cognitive Tutors are learn by doing, helping students learn by engaging them in actual problem solving, and immediate feedback, providing guidance when students request a hint or make a mistake. For domains such as algebra, the production rules in a cognitive model indicate not only correct problem-solving steps a student might take but also plausible incorrect steps. The model provides feedback in the form of error messages, when the student takes a step anticipated by a "buggy rule," and hints, when the student asks for help. Developing a Cognitive Tutor for case comparison presents some stiff challenges. One particular challenge is that, unlike previous domains in which Cognitive Tutors have been used, such as algebra and programming, in practical ethics answers aren't always and easily identified as correct or incorrect. Also, the rules, as I discussed earlier, are more abstract and ill defined. As a result, while learn-by-doing fits ethics case comparison very well, the concept of immediate feedback needs adaptation. Unlike feedback in more technical domains, ethics feedback might be nuanced rather than sim-ply right or wrong, so the Cognitive Tutor approach must be adapted to this. The rules employed in Truth-Teller's first three phases, particularly the qualification phase, provide a core set of rules that can be improved and recast as a set of rules for comparing cases in a Cognitive Tutor framework. An empirical study of case comparisons, involving more technically complex ethics cases, will enable refinement and augmentation of these comparison rules. At the same time, the empirical study of humans comparing cases might reveal plausible misconceptions about the comparison process that can serve as buggy rules that present opportunities to correct the student. A related direction is exploring whether the priority rules of Ross's theory of prima facie duties, such as nonmaleficence normally overriding other duties and fidelity normally overriding beneficence, might benefit the Truth-Teller comparison method. At the very least it would ground Truth-Teller's approach in a more established philosophical theory (currently, priority rules are based loosely on Sissela Bok's research 32 ). Such an extension to Truth-Teller would also benefit the planned Cognitive Tutor, in that it could refer to Ross's theory to support explanations to students. Another important lesson from the two projects is the sheer difficulty of imbuing a computer program with the sort of flexible intelligence required to perform ethical analysis. Figure 1 . 1 Figure 1. Truth-Teller's output comparing two ethical dilemmas.
 M a c h i n e E t h i c s 32 wwwFigure 2 . 322 Figure 2. Truth-Teller's case representation for Felicia's case in figure 1.
 Figure 3 . 3 Figure 3. SIROCCO's output for an engineering ethical dilemma.
 \*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\* \*\*\* SIROCCO has the following suggestions \*\*\* for evaluating '92-6-2: Public Welfare -Hazardous Waste' \*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\* \*\*\* Possibly Relevant Codes: II-1-A: Primary Obligation is to Protect Public (Notify Authority if Judgment is Overruled). I-1: Safety, Health, and Welfare of Public is Paramount I-4: Act as a Faithful Agent or Trustee III-4: Do not Disclose Confidential Information Without Consent III-2-B: Do not Complete or Sign documents that are not Safe for Public II-1-C: Do not Reveal Confidential Information Without Consent II-3-A: Be Objective and Truthful in all Reports, Stmts, Testimony. \*\*\* Possibly Relevant Cases: 61-9-1: Responsibility for Public Safety \*\*\* Additional Suggestions: • The codes I-1 ('Safety, Health, and Welfare of Public is Paramount') and II-1-A (
 Figure 4 . 4 Figure 4. A comparison of SIROCCO with five other methods for retrieving ethical codes and cases. The F-Measure indicates how well each method performed in comparison to humans. In exact matching, the methods and humans retrieved precisely the same codes and cases. In inexact matching, the methods and humans retrieved closely related codes and cases.
 4 . 4 Evaluates the arguments to come to a decision.
 Figure 5 . 5 Figure 5. Truth-Teller's, SIROCCO's, and a human's potential role in casuistic problem solving.
 Selects a paradigm, a hypothetical, or past cases involving the principles or policies.2. Compares them to the problem to see if the reasons apply as stronglyin the problem as in the cases. SIROCCO Truth-Teller SIROCCO 1. 3. Argues how to resolve conflicting reasons in terms of criteria applied Truth-Teller in the cases. Human
 www.computer.org/intelligent IEEE INTELLIGENT SYSTEMS Authorized licensed use limited to: Carnegie Mellon Libraries. Downloaded on March 24,2022 at 01:37:25 UTC from IEEE Xplore. Restrictions apply.
 JULY/AUGUST 2006www.computer.org/intelligent
 Authorized licensed use limited to: Carnegie Mellon Libraries. Downloaded on March 24,2022 at 01:37:25 UTC from IEEE Xplore. Restrictions apply.
 www.computer.org/intelligent
 www.computer.org/intelligent IEEE INTELLIGENT SYSTEMS


Information gathering actions over human internal state

I. introduction Imagine driving on the highway. Another driver is in the lane next to you, and you need to switch lanes. Some drivers are aggressive and they will never brake to let you in. Others are more defensive and would gladly make space for you. You don't know what kind of driver this is, so you decide to gently nudge in towards the other lane to test their reaction. At an intersection, you might nudge in to test if the other driver is distracted and they might just let you go through (Fig. 1 bottom left). Our goal in this work is to give robots the capability to plan such actions as well. In general, human behavior is affected by internal states that a robot would not have direct access to: intentions, goals, preferences, objectives, driving style, etc. Work in robotics and perception has focused thus far on estimating these internal states by providing algorithms with observations of humans acting, be it intent prediction [23] , [13] , [6] , [3] , [4] , [16] , Inverse Reinforcement Learning [1] , [12] , [15] , [22] , [18] , driver style prediction [11] , affective state prediction [10] , or activity recognition [20] . Human state estimation has also been studied in the context of human-robot interaction tasks. Here, the robot's reward function depends (directly or indirectly) on the human internal state, e.g., on whether the robot is able to adapt to the human's plan or preferences. Work in assistive teleoperation or in human assistance has cast this problem as a Partially Observable Markov Decision Process, in which the robot does observe the physical state of the world but not the human internal state -that it has to estimate from human actions. Because POMDP solvers are not computationally efficient, the solutions proposed thus far use the current estimate Fig. 1 : We enable robots to generate actions that actively probe humans in order to find out their internal state. We apply this to autonomous driving. In this example, the robot car (yellow) decides to inch forward in order to test whether the human driver (white) is attentive. The robot expects drastically different reactions to this action (bottom right shows attentive driver reaction in light orange, and distracted driver reaction in dark orange). We conduct a user study in which we let drivers pay attention or distract them with cellphones in order to put this state estimation algorithm to the test. of the internal state to plan (either using the most likely estimate, or the entire current belief), and adjust this estimate at every step [8] , [7] , [5] . Although efficient, these approximations sacrifice an important aspect of POMDPs: the ability to actively gather information. Our key insight is that robots can leverage their own actions to help estimation of human internal state. Rather than relying on passive observations, robots can actually account for the fact that humans will react to their actions: they can use this knowledge to select actions that will trigger human reactions which in turn will clarify the internal state. We make two contributions: An Algorithm for Active Information Gathering over Human Internal State. We introduce an algorithm for planning robot actions that have high expected information gain. Our algorithm uses a rewardmaximization model of how humans plan their actions in response to those of the robot's [19] , and leverages the fact that different human internal states will lead to different human reactions to speed up estimation. Fig. 1 shows an example of the anticipated difference in reaction between a distracted and an attentive driver. Application to Driver Style Estimation. We apply our algorithm to estimating a human driver's style during the interaction of an autonomous vehicle with a human-driven vehicle. Results in simulation as well as from a user study suggest that our algorithm's ability to leverage robot actions for estimation leads to significantly higher accuracy in identifying the correct human internal state. The autonomous car plans actions like inching forward at an intersection (Fig. 1 ), nudging into another car's lane, or braking slightly in front of a human-driven car, all to estimate whether the human driver is attentive. Overall, we are excited to have taken a step towards giving robots the ability to actively probe end-users through their actions in order to better estimate their goals, preferences, styles, and so on. Even though we chose driving as our application domain in this paper, our algorithm is general across different domains and types of human internal state. We anticipate that applying it in the context of human goal inference during shared autonomy, for instance, will lead to the robot purposefully committing to a particular goal in order to trigger a reaction from the user, either positive or negative, in order to clarify the desired goal. Of course, further work is needed in order to evaluate how acceptant end-users are of different kinds of such probing actions.
 II. Information Gathering Actions We start with a general formulation of the problem of a robot needing to maximize its reward by acting in an environment where a human is also acting. The human is choosing its actions in a manner that is responsive to the robot's actions, and also influenced by some internal variables. While most methods that have addressed such problems have proposed approximations based on passively estimating the internal variable and exploiting that estimate, here we propose a method for active information gathering that enables the robot to purposefully take actions that probe the human. Finally, we discuss our implementation in practice, which trades off between exploration and exploitation.
 A. General Formulation We define a human-robot system in which the human's actions depend on some human internal state ϕ that the robot does not directly observe. In a driving scenario, ϕ might correspond to driving style: aggressive or timid, attentive or distracted. In collaborative manipulation scenarios, ϕ might correspond to the human's current goal, or their preference about the task. We let x ∈ X be a continuous physical state of our system. For our running example of autonomous cars, this includes position, velocity and heading of the autonomous and human driven vehicles. Let ϕ ∈ Φ be the hidden variable, e.g., human driver's driving style. We assume the robot observes the current physical state x t , but not the human internal state ϕ. The robot and human can apply continuous controls u R and u H . The dynamics of the system evolves as robot's and human's control inputs arrive at each step: x t+1 = f H ( f R (x t , u t R ), u t H ). (1) Here, f R and f H represent how the actions of the robot and of the human respectively affect the dynamics, and can be applied synchronously or asynchronously. We assume that while x changes via (1) based on the human and robot actions, ϕ does not. For instance, we assume that the human maintains their preferences or driving style throughout the interaction. The robot's reward function in the task depends on the current state, the robot's action, as well as the action that the human takes at that step in response, r R (x t , u t R , u t H ). If the robot has access to the human's policy π H (x, u R , ϕ), maximizing robot reward can be modeled as a POMDP [9] with states (x, ϕ), actions u R , and reward r R . In this POMDP, the dynamics model can be computed directly from (1) with u t H = π H (x t , u t R , ϕ). The human's actions can serve as observations of ϕ via some P(u H |x, u R ). In Sec. II-C we introduce a model for both π H and P(u H |x, u R , ϕ) based on the assumption that the human is maximizing their own reward function. If we were able to solve the POMDP, the robot would estimate ϕ based on the human's actions, and optimally trade off between exploiting its current belief over ϕ, and actively taking information gathering actions meant to cause human reactions that give the robot a better estimate of the hidden variable ϕ. Because POMDPs cannot be solved tractably, several approximations have been proposed for similar problem formulations [8] , [11] , [7] . These approximations are passively estimating the human internal state, and exploiting the belief to plan robot actions. 1 In this work, we take the opposite approach: we focus explicitly on active information gathering. We enable the robot to decide to actively probe the person to get a better estimate of ϕ. Our method can be leveraged in conjunction with exploitation methods, or be used alone when human state estimation is robot's primary objective.
 B. Reduction to Information Gathering At every step, the robot can update its belief over ϕ via: b t+1 (ϕ) ∝ b t (ϕ) • P(u H |x t , u R , ϕ). ( 2 ) To explicitly focus on taking actions to estimate ϕ, we redefine the robot's reward function to capture the information gain at every step: r R (x t , u R , u H ) = H(b t ) − H(b t+1 ) (3) with H(b) being the entropy over the belief: H(b) = − ∑ ϕ b(ϕ) log(b(ϕ)) ∑ ϕ b(ϕ) . ( 4 ) Optimizing expected reward now entails reasoning about the effects that the robot actions will have on what observations the robot will get, i.e., the actions that the human will take in response, and how useful these observations will be in shattering ambiguity about ϕ.
 C. Solution: Human Model & Model Predictive Control We solve the information gathering planning problem via Model Predictive Control (MPC) [14] . At every time step, we find the optimal actions of the robot u \* R by maximizing the expected reward function over a finite horizon. Notation. Let x 0 be the state at the current time step, i.e., at the beginning of the horizon. u R = (u 0 R , . . . , u N−1 R ) be a finite sequence of the robot's continuous actions, and u H = (u 0 H , . . . , u N−1 H ) be a finite sequence of human's continuous actions. Further, let R R (x 0 , u R , u H ) denote the reward over the finite horizon, if the agents started in x 0 and executed u R and u H , which can be computed via the dynamics in (1) . MPC Maximization Objective. At every time step, the robot is computing the best actions for the horizon: u \* R = arg max u R E ϕ R R (x 0 , u R , u \* ϕ H (x 0 , u R )) (5) where u \* ϕ H (x 0 , u R ) corresponds to the actions the human would take from state x 0 over the horizon of N steps if the robot executed actions u R . Here, the expectation is taken over the current belief over ϕ, b 0 . Simplifying (5) using the definition of reward from (3), we get: u \* R = arg max u R E ϕ H(b 0 ) − H(b N ) , (6) u \* R = arg max u R E ϕ − H(b N ) , (7) where the expectation remains with respect to b 0 . Human Model. We assume that the human maximizes their own reward function at every step. We let r ϕ H (x t , u t R , u t H ) represent human's reward function at time t, which is parametrized by the human internal state ϕ. Then, the sum of human rewards over horizon N is: R ϕ H (x 0 , u R , u H ) = N−1 ∑ t=0 r ϕ H (x t , u t R , u t H ) (8) Building on our previous work [19] , which showed how the robot can plan using such a reward function when there are no hidden variables, we compute u \* ϕ H (x 0 , u R ) through an approximation. We model the human as having access to u R a priori, and compute the finite horizon human actions that maximize the human's reward: u \* ϕ H (x 0 , u R ) = arg max u H R ϕ H (x 0 , u R , u H ) (9) One can find r ϕ H through Inverse Reinforcement Learning (IRL) [1] , [12] , [15] , [22] , by getting demonstrations of human behavior associated with a direct measurement of ϕ. The robot can use this reward in the dynamics model in order to compute human actions via (9) . To update the belief b and compute expected reward in (7) , we still need an observation model. We assume that actions with lower reward are exponentially less likely, building on the principle of maximum entropy [22] : P(u H |x, u R , ϕ) ∝ exp(ϕ H (x, u R , u H )) (10) Optimization Procedure. To solve (5) (or equivalently ( 7 )), we use a gradient descent optimization method, L-BFGS, designed for unconstrained nonlinear problems [2] . Therefore, we would like to find the gradient of the objective in equation ( 5 ) with respect to u R . Since the objective is the expectation of R R , we can reformulate this gradient as: ∂E ϕ R R (x 0 , u R , u \* ϕ H (x 0 , u R )) ∂u R = ∑ ϕ ∂R R (x 0 , u R , u \* ϕ H (x 0 , u R )) ∂u R • b 0 (ϕ) (11) Then, we only need to find ∂R R ∂u R , which is equivalent to: ∂R R (x 0 , u R , u \* ϕ H (x 0 , u R )) ∂u R = ∂R R (x 0 , u R , u H ) ∂u H ∂u \* ϕ H ∂u R + ∂R R (x 0 , u R , u H ) ∂u R | u H =u \* ϕ H (x 0 ,u R ) (12 ) Because R R , as indicated by (7) , simplifies to the negative entropy of the updated belief, we can compute both ∂R R (x 0 ,u R ,u H ) ∂u H and ∂R R (x 0 ,u R ,u H ) ∂u R | u H =u \* ϕ H (x 0 ,u R ) symbolically. This leaves ∂u \* ϕ H ∂u R . We use the fact that the gradient of R H will evaluate to zero at u \* ϕ H : ∂R H ∂u H (x 0 , u R , u \* ϕ H (x 0 , u R )) = 0 (13) Now, differentiating this expression with respect to u R will result in: ∂ 2 R H ∂u 2 H ∂u \* ϕ H ∂u R + ∂ 2 R H ∂u H ∂u R ∂u R ∂u R = 0 (14) Then, solving for ∂u \* ϕ H ∂u R enables us to find the following symbolic expression: ∂u \* ϕ H ∂u R = [− ∂ 2 R H ∂u H ∂u R ][ ∂ 2 R H ∂u 2 H ] −1 . ( 15 ) This expression allows finding a symbolic expression for the gradient in equation (11) .
 D. Explore-Exploit Trade-Off In practice, we use information gathering in conjunction with exploitation. We do not solely optimize the reward from Sec. II-B, but optimize it in conjunction with the robot's actual reward function assuming the current estimate of ϕ: r R (x t , u R , u H ) = H(b t ) − H(b t+1 ) ( 16 ) + λ • r goal (x t , u R , u H , b t ) At the very least, we do this as a measure of safety, e.g., we want an autonomous car to keep avoiding collisions even when it is actively probing a human driver to test their reactions. We choose λ experimentally, though existing techniques that can better adapt λ over time [21] . Despite optimizing this trade-off, we do not claim that our method as-is can better solve the general POMDP formulation from Sec. II-A: only that it can be used to get better estimates of human internal state. The next sections test this in simulation and in practice, in a user study, and future work will look at how to leverage this ability to better solve human-robot interaction problems.
 III. Simulation Results In this section, we show simulation results that use the method from the previous section to estimate human driver type in the interaction between an autonomous vehicle and a human-driven vehicle. In this section, we consider three different autonomous driving scenarios. In these scenarios, the human is either distracted or attentive during different driving experiments. The scenarios are shown in Fig. 2 , where the yellow car is the autonomous vehicle, and the white car is the human driven vehicle. Our goal is to plan to actively estimate the human's driving style in each one of these scenarios, by using the robot's actions.
 A. Attentive vs. Distracted Human Driver Models Our technique requires reward functions r ϕ H that model the human behavior for a particular internal state ϕ. We obtain a generic driver model via Continuous Inverse Optimal Control with Locally Optimal Examples [12] from demonstrated trajectories in a driving simulator in an environment with multiple autonomous cars, which followed precomputed routes. We parametrize the human reward function as a linear combination of features, and learn weights on the features. We use various features including features for bounds on the control inputs, features that keep the vehicles within the road boundaries and close to the center of their lanes. Further, we use quadratic functions of speed to capture reaching the goal, and Gaussians around other vehicles on the road to enforce collision avoidance as part of the feature set. We then adjust the learned weights to model attentive vs. distractive drivers. Specifically, we modify the weights of the collision avoidance features, so the distracted human model has less weight for these features. Therefore, the distracted driver is more likely to collide with the other cars while the attentive driver has high weights for the collision avoidance feature.
 B. Manipulated Factors We manipulate the reward function that the robot is optimizing. In the passive condition, the robot optimizes a simple reward function for collision avoidance based on the current belief estimate. It then updates this belief passively, by observing the outcomes of its actions at every time step. In the active condition, the robot trades off between this reward function and the Information Gain from (3) in order to explore the human's driving style. We also manipulate the human internal state to be attentive or distracted. The human is simulated to follow the ideal model of reward maximization for our two rewards.
 C. Driving Simulator We use a simple point-mass model for the dynamics of the vehicle, where x = x y θ v is the state of the vehicle. Here, x and y are the coordinates of the vehicle, θ is the heading, and v is the speed. Each vehicle has two control inputs u = u 1 u 2 , where u 1 is the steering input, and u 2 is acceleration. Further, we let α be a friction coefficient. Then, the dynamics of each vehicle is formalized as: [ ẋ ẏ θ v] = [v • cos(θ) v • sin(θ) v • u 1 u 2 − α • v]. (17)
 D. Scenarios and Qualitative Results Scenario 1: Nudging In to Explore on a Highway. In this scenario, we show an autonomous vehicle actively exploring the human's driving style in a highway driving setting. We contrast the two conditions in Fig. 2 (a). In the passive condition, the autonomous car drives on its own lane without interfering with the human throughout the experiment, and updates its belief based on passive observations gathered from the human car. However, in the active condition, the autonomous car actively probes the human by nudging into her lane in order to infer her driving style. An attentive human significantly slows down (timid driver) or speeds up (aggressive driver) to avoid the vehicle, while a distracted In this scenario, we consider the two vehicles at an intersection, where the autonomous car actively tries to explore human's driving style by nudging into the intersection. The initial conditions of the vehicles are shown in Fig. 2(c ). In the passive condition, the autonomous car stays at its position without probing the human, and only optimizes for collision avoidance. This provides limited observations from the human car resulting in a low confidence belief distribution. Autonomous Vehicle Human Driven Vehicle Passive Estimation Active Info Gathering In the active condition, the autonomous car nudges into the intersection to probe the driving style of the human. An attentive human would slow down to stay safe at the intersection while a distracted human will not slow down.
 E. Quantitative Results Throughout the remainder of the paper, we use a common color scheme to plot results for our experimental conditions. We show this common scheme in Fig. 3 : darker colors (black and red) correspond to attentive humans, and lighter colors (gray and orange) correspond to distracted humans. Further, the shades of orange correspond to active information gathering, while the shades of gray indicate passive information gathering. We also use solid lines for real users, and dotted lines for scenarios with an ideal user model learned through inverse reinforcement learning. This table is representative for the legends of Fig. 4 , Fig. ? ?, Fig. 5 , and Fig. 6 . Fig. 4 plots, using dotted lines, the beliefs over time for the attentive (left) and distracted (right) conditions, comparing in each the passive (dotted black and gray respectively) with the active method (dotted dark orange and light orange respectively). In every situation, the active method achieves a more accurate belief (higher values for attentive on the left, when the true ϕ is attentive, and lower values on the right, when the true ϕ is distracted). In fact, passive estimation sometimes incorrectly classifies drivers as attentive when they are distracted and vice-versa. The same figure also shows (in solid lines) results from our user study of what happens when the robot no longer interacts with an ideal model. We discuss these in the next section. Fig. 5 and Fig. 6 plot the corresponding robot and human trajectories for each scenario. The important takeaway from these figures is that there tends to be a larger gap between attentive and distracted human trajectories in the active condition (orange shades) than in the passive condition (gray shades), especially in scenarios 2 and 3. It is this difference that helps the robot better estimate ϕ: the robot in the active condition is purposefully choosing actions that will lead to large differences in human reactions, in order to more easily determine the human driving style.
 IV. User Study In the previous section, we explored planning for an autonomous vehicle that actively probes a human's driving style, by braking or nudging in and expecting to cause reactions from the human driver that would be different depending on their style. We showed that active exploration does significantly better at distinguishing between attentive and distracted drivers using simulated (ideal) models of drivers. Here, we show the results of a user study that evaluates this active exploration for attentive and distracted human drivers.
 A. Experimental Design We use the same three scenarios discussed in the previous section. Manipulated Factors. We manipulated the same two factors as in our simulation experiments: the reward function that the robot is optimizing (whether it is optimizing its reward through passive state estimation, or whether it is trading off with active information gathering), and the human internal state (whether the user is attentive or distracted). We asked our users to pay attention to the road and avoid collisions for the attentive case, and asked our users to play a game on a mobile phone during the distracted driving experiments. Dependent Measure. We measured the probability that the robot assigned along the way to the human internal state. Hypothesis. The active condition will lead to more accurate human internal state estimation, regardless of the true human internal state. Subject Allocation. We recruited 8 participants (2 female, 6 male) in the age range of 21-26 years old. All participants owned a valid driver license and had at least 2 years of driving experience. We ran the experiments using a 2D driving simulator with the steering input and acceleration input provided through a steering wheel and a pedals as shown in Fig. 1 . We used a within-subject experiment design with counterbalanced ordering of the four conditions.
 B. Analysis We ran a factorial repeated-measures ANOVA on the probability assigned to "attentive", using reward (active vs passive) and human internal state (attentive vs distracted) as factors, and time and scenario as covariates. As a manipulation check, attentive drivers had significantly higher estimated probability of "attentive" associated than distracted drivers (.66 vs .34, F = 3080.3, p < .0001). More importantly, there was a signifiant interaction effect between the factors (F = 1444.8, p < .000). We ran a post-hoc analysis with Tukey HSD corrections for multiple comparisons, which showed all four conditions to be significantly different from each other, all contrasts with p < .0001. In particular, the active information gathering did end up with higher probability mass on "attentive" than the passive estimation for the attentive users, and lower probability mass for the distracted user. This supports our hypothesis that our method works, and active information gathering is better at identifying the correct state. Fig. 4 compares passive (grays and blacks) and active (light and dark oranges) across scenarios and for attentive (left) and distracted (right) users. It plots the probability of attentive over time, and the shaded regions correspond to standard error. From the first column, we can see that our algorithm in all cases detects human's attentiveness with much higher probably than the passive information gathering technique shown in black. From the second column, we see that our algorithm places significantly lower probability on attentiveness, which is correct because those users were distracted users. These are in line with the statistical analysis, with active information gathering doing a better job estimating the true human internal state. Fig. 5 plots the robot trajectories for the active information gathering setting. Similar to Fig. 4 , the solid lines are the mean of robot trajectories and the shaded regions show the standard error. We plot a representative dimension of the robot trajectory (like position or speed) for attentive (dark orange) or distracted (light orange) cases. The active robot probed the user, but ended up taking different actions when the user was attentive vs. distracted in order to maintain safety. For example, in Scenario 1, the trajectories show the robot is nudging into the human's lane, but the robot decides to move back to its own lane when the human drivers are distracted (light orange) in order to stay safe. In Scenario 2, the robot brakes in front of the human, but it brakes less when the human is distracted. Finally, in Scenario 3, the robot inches forward, but again it stops when if the human is distracted, and even backs up to make space for her. Fig. 6 plots the user trajectories for both active information gathering (first row) and passive information gathering (second row) conditions. We compare the reactions of distracted (light shades) and attentive (dark shades) users. There are large differences directly observable, with user reactions tending to indeed cluster according to their internal state. These differences are much smaller in the passive case (second row, where distracted is light gray and attentive is black). For example, in Scenario 1 and 2, the attentive users (dark orange) keep a larger distance to the car that nudges in front of them or brakes in front of them, while the distracted drivers (light orange) tend to keep a smaller distance. In Scenario 3, the attentive drivers tend to slow down and do not cross the intersection, when the robot actively inches forward. None of these behaviors can be detected clearly in the passive information gathering case (second row). This is the core advantage of active information gathering: the actions are purposefully selected by the robot such that users would behave drastically differently depending on their internal state, clarifying to the robot what this state actually is. Overall, these results support our simulation findings, that our algorithm performs better at estimating the true human internal state by leveraging purposeful information gathering actions.
 V. discussion Summary. In this paper, we formalized the problem of active information gathering between robot and human agents, where the robot plans to actively explore and gather information about the human's internal state by leveraging the effects of its actions on the human actions. The generated strategy for the robot actively probes the human by taking actions that impact the human's action in such a way that they reveal her internal state. The robot generates strategies for interaction that we would normally need to hand-craft, like inching forward at a 4-way stop. We evaluated our method in simulation and through a user study for various autonomous driving scenarios. Our results suggest that robots are indeed able to construct a more accurate belief over the human's driving style with active exploration than with passive estimation. Limitations and Future Work. Our work is limited in many ways. First, state estimation is not the end goal, and finding how to trade off exploration and exploitation is still a challenge. Second, our optimization is close to real-time, but higher computational efficiency is still needed. Further, our work relies on a model (reward function) of the human for each ϕ, which might be difficult to acquire, and might not be accurate. Thus far, we have assumed a static ϕ, but in reality ϕ might change over time (e.g. the human adapts her preferences), or might even be influenced by the robot (e.g. a defensive driver becomes more aggressive when the robot probes her). We also have not tested the users' acceptance of information gathering actions. Although these actions are useful, people might not always react positively to being probed. Last but not least, exploring safely will be of crucial importance.
 Conclusion. We are encouraged by the fact that robots can generate useful behavior for interaction autonomously, and are excited to explore informationgathering actions on human state further, including beyond autonomous driving scenarios. Passive Robot stops to R through Fig. 6 : The user trajectories for each scenario. The gap between attentive and distracted drivers' actions is clear in the active information gathering case (first row). 1 :Fig. 2 : 12 Fig.2: Our three scenarios, along with a comparison of robot plans for passive estimation (gray) vs active information gathering (orange). In the active condition, the robot is purposefully nudging in or braking to test human driver attentiveness. The color of the autonomous car in the initial state is yellow, but changes to either gray or orange in cases of passive and active information gathering respectively. driver might not realize the autonomous actions and maintain their velocity, getting closer to the autonomous vehicle. It is this difference in reactions that enables the robot to better estimate ϕ. Scenario 2: Braking to Explore on a Highway. In the second scenario, we show the driving style can be explored by the autonomous car probing the human driver behind it. The two vehicles start in the same lane as shown in Fig.2(b), where the autonomous car is in the front. In the passive condition, the autonomous car drives straight without exploring or enforcing any interactions with the human driven vehicle. In the active condition, the robot slows down to actively probe the human and find out her driving style. An attentive human would slow down and avoid collisions while a distracted human will have a harder time to keep safe distance between the two cars. Scenario 3: Nudging In to Explore at an Intersection. In this scenario, we consider the two vehicles at an intersection, where the autonomous car actively tries to explore human's driving style by nudging into the intersection. The initial conditions of the vehicles are shown in Fig.2(c). In the passive condition, the autonomous car stays at its position without probing the human, and only optimizes for collision avoidance. This provides limited observations from the human car resulting in a low confidence belief distribution. In the active condition, the autonomous car nudges into the intersection to probe the driving style of the human. An attentive human would slow down to stay safe at the intersection while a distracted human will not slow down.
 Fig. 3 : 3 Fig. 3: Legends indicating active/passive robots, attentive/distracted humans, and real user/ideal model used for Fig.4, Fig.??, Fig.5, and Fig.6.
 Fig. 4 : 4 Fig. 4:The probability that the robot assigns to attentive as a function of time, for the attentive (left) and distracted (right). Each plot compares the active algorithm to passive estimation, showing that active information gathering leads to more accurate state estimation, in simulation and with real users.
 Fig. 5 : 5 Fig.5: Robot trajectories for each scenario in the active information gathering condition. The robot acts differently when the human is attentive (dark orange) vs. when the human is distracted (light orange) due to the trade-off with safety.
 One exception is Nikolaidis et al. [17] , who propose to solve the full POMDP, albeit for discrete and not continuous state and action spaces.
 Authorized licensed use limited to: Carnegie Mellon Libraries. Downloaded on March 24,2022 at 00:30:44 UTC from IEEE Xplore. Restrictions apply.

Dorsa Sadigh
S. Shankar Sastry
Sanjit A. Seshia
Anca Dragan