Epistemic Therapy for Bias in Automated Decision-Making

diagnose the epistemic and ethical challenges that may arise in AI system design. Since much of the discussion on anthropomorphic biases has been conducted through either anthropomorphizing AI models or relegating all responsibility for bias onto problematic data, our goal is to move away from these ambiguous analogies and confront the nuance of bias in automated decision-making. Our discussion is motivated by several recent cases of AI system implementation and research that have already been flagged as problematic. We first examine systems where data was assumed to be representative of the real world but in fact encoded human biases that were later amplified by AI systems [2, 30] . We then consider cases in the machine learning literature where common notions of bias and equity are rationalized away by researchers using predictions provided by AI models [11, 34] . Finally, we consider cases where researchers attempt to directly address anthropomorphic biases they perceive in their model prediction, leading to researchers instead encoding their own implicit biases into AI systems [7, 9] . These three motivating examples show that the nature of anthropomorphic bias in AI is paradoxical and counterintuitive, while the use of direct analogies to human bias actually erase the specific contexts that cause anthropomorphic bias to manifest in the first place. In order to diagnose and work through these ambiguities, we provide two major contributions to the discussion of bias in AI systems. First, we characterize the kinds of biases that arise from complex interactions between engineers, algorithms, and data. To do this, we draw on previous work in the philosophy of mind literature that characterizes human bias as a set of aliefs [17] , belief-like dispositions that also contain an affective component, and introduce the concept of alief-discordant beliefs. In our discussion, we demonstrate that these alief-discordant beliefs provide the framework necessary to understand the trade-off between the moral dispositions of system engineers and the explicit relations encoded by AI models. Second, we apply the notion of alief-discordant beliefs to craft design principles for AI in light of human biases. We conclude by illustrating how these principles can be used to diagnose the ethical concerns that motivate practitioners, performing a type of "epistemic therapy" for the benefit of the machine learning community. Much as psychotherapy is used to confront subconscious issues through a process of dynamic interpersonal conversation, we propose "epistemic therapy" for automated decision-making as the process by which alief-discordant beliefs are identified and confronted by engineers. The design principles we outline in this paper support this "epistemic therapy" by aiding in the implementation of AI systems and navigating to avoid biased outcomes.
 EPISTEMIC ETHICS IN FAIR MACHINE LEARNING The machine learning literature has explored predictive bias as a barrier to fair outcomes. In particular, many papers have suggested algorithmic means of curbing bias in predictions, proposing the notion of "optimal" fairness as solutions to optimization problems [11] . Several variations of optimal fairness have been proposed, including equalizing prediction metrics (e.g. TPR, FPR, accuracy) across protected classes [1, 10, 13, 20] , and producing models whose predictions are independent of protected features [7, 9, 28, 35] . There have been some technical critiques leveled against these notions of optimal fairness [11, 25] , as well as several impossibility results that show it is difficult to satisfy all of these notions of fairness simultaneously [14, 21] . We complement this literature by providing a framework of epistemic ethics that could be used to justify the necessary engineering design choices made in implementing certain methods of optimal fairness. Specifically, our framework resolves some of the skepticism in using these methods and addresses what technical knowledge and data engineers ought to have in order to manage and avoid anthropomorphic bias in AI systems they design. A related stream of social science literature has discussed the nature of bias in machine learning [4, 6, 12, 26] , enumerating potential ethical concerns and discussing whether decision-making can be automated without compromising human dignity responsibility. In contrast, our discussion emphasizes how the perception of different kinds of bias in AI systems is responsible for their supposed immorality, which implies they are better understood within the realm of philosophy of mind than ethics proper. Of particular relevance to this paper is the discussion proposed by Binns [6] , which analyzes anthropomorphic bias by contrasting human mental states with the mechanisms of automated decision-making. We will present a complementary dissolution of anthropomorphic bias through the notion of alief-discordant beliefs, directly influenced by the work of Tamar Gendler on characterizing human biases [17] .
 CASE STUDIES We will first build intuition on how "anthropomorphic bias" can affect researchers and engineers through reference to prominent existing case studies. In particular, we consider cases where anthropomorphic bias may manifest in how 1) the values of engineers inform the techniques of AI system creation; 2) the epistemology of the data and encoding characterize the resulting AI system; 3) allocation of moral culpability labels interactions between data and system design throughout the deployment phase. Each of these examples illustrates how anthropomorphic bias can be attributed to an underlying discordance between the distinctive epistemic frames of engineers, data, and system design. We discuss three major causes of this discordance, each with its own mixing of ethical culpability between data, algorithms, and engineers. These are: (i) AI systems that are deployed as vehicles for moral dispositions, either encoded by engineers or sedimented within data; (ii) engineers that outsource their own moral compasses to the labels generated by AI systems; (iii) dogmatic reconciliation of inherited dispositions and generated propositions that result in adverse effects.
 AI systems as Vehicles for Moral Dispositions First we consider cases where AI systems are deployed as vehicles for moral dispositions implicitly harbored by engineers or encoded in training data. Two famous cases of such discordance discussed in FAT literature include the COMPAS system which was used to predict criminal recidivism [2, 22] , and Tay the Microsoft-deployed chat bot that was hijacked by white supremacists [5, 29, 30] . 3.1.1 COMPAS. The COMPAS system was initially developed to predict a recidivism risk score for arrested individuals and was deployed to several states [2] . However, even though explicit racial detail was not inserted into the system input, it was shown that COMPAS would predict lower risk scores for white individuals and higher scores for people of color. Moreover, analysis of the system output showed that the false positive rate for people of color was significantly higher than that of white individuals [22] . From a technical perspective, it is generally agreed that the system produced biased output despite not taking in explicit racial data, since race is correlated with other features that were used as input (e.g. education, residence address, income) and labels used for training were obtained from historical arrest records that contain a documented inherent bias against certain communities [22] . In essence, the way the COMPAS system was designed and implemented transformed social biases extant in the justice system and reified them through automated classification. However, it is difficult to confer responsibility for this to a specific agent-while the data's encoded bias should have been made explicit by those who collected it, system engineers also failed to properly account for this bias in the data. What the engineers ought to have done in this particular case is critically examine the source of the data and any bias that it may convey, and use a proper training technique for their model that could account for the problematic predictions. 3.1.2 Microsoft Tay. Tay was a chatbot developed by Microsoft research to interact with the greater public on social media and mimic the language patterns of a 19 year old girl [5] . While in closed company testing, it was reported that Tay was performing extremely well without significant incident. Within 24 hours of being deployed online, and to the surprise of the research team, Tay was re-tweeting white supremacist propaganda due to a loosely coordinated attack by certain forum users [29, 30] . Much like COMPAS, the deployment of Tay also suffered from becoming a passive vehicle for unquestioned moral associations. However, the associations in question are not necessarily the ones harbored by the data but by the engineers that designed the system. In particular, the engineers were not appropriately skeptical of the reaction of the internet community at large and assumed that they would behave in a similar manner to the corporate testers that interacted with Tay in house. The fallout from this situation could have been reduced had the engineers confronted their own underlying assumptions and properly designed the training methods of the model to not accept all input data equally.
 Automatic Alief Falsification Another cause of the discordance can result from a 'skeptical' overcorrection based on the system pipeline. Two of the principle tenets Spotlight 1: Normative Perspectives AIES'19, January 27-28, 2019, Honolulu, HI, USA of data science include the belief that all rational explanations (i.e. those that rely on mathematical reasoning) are superior to all other forms of disposition generation, and that sufficient information about the state of the world can be extracted from data. This moral disposition is conveyed by the recent work of Corbett-Davies and Goel [11] on fairness measures, and Wang and Kosinski's [34] work on detecting sexual orientation with pictures . While we strive to identify ethical culpability in this section, we do not ascribe malicious intentions to the engineers that design AI systems. In particular, we believe both sets of authors that we discuss here did have good intentions when creating their work, but that their results are problematic when examined in a critical context. The key problematic aspect of these discordances is that instead of attempting to harmonize their own moral dispositions with the propositions generated by models, engineers are too eager to dispose of their initial presuppositions.
 3.2.1 The Measure and Mismeasure of Fairness. Corbett-Davies and Goel [11] have presented several leading methods of optimal fairness in the FATML literature and critique each family of methods to show how they may violate certain notions of fairness. The authors rely on working definitions of fairness from parts of the economics and legal literature and show how these can be incongruent with mathematical notions of fairness. When analyzing the notion of fairness through classification parity, the authors note that different populations of individuals will by nature have different means and variances that could account for lack of parity. As a real world example of this, the authors used the example of the COMPAS model we previously discussed. They note that since black individuals had higher recidivism rates than white individuals, this group was in fact accurately predicted as having a higher risk to society than whites. Although the authors do note that this difference in rate is caused by both historic and systemic factors, they argue that these are not crucial to examine when making policy decisions. In particular, since the authors assume the prediction of individual risk is accurate, they claim that policy actions to ensure prediction parity between the populations would result in an unfairly harsher prediction rate against the white population. They further argue that such actions would harm the black population with an inappropriately low predicted rate of recidivism [11] . Thus, the authors arrive at a conclusion that is in contrast to their initially stated assumptions ("demographic parity is important"), due to their failure to critically engage with the context of the data (namely its internal generation by a system with known bias [22] ).
 Detecting Sexual Orientation with AI. A similar tension can be found in Wang and Kosinski's paper [34] , which describes an artificial neural network model that processes facial images to predict an individual's sexual orientation. In particular, the authors used face photos scraped from dating websites that they classified as heterosexual or homosexual using the user's dating profile, and showed that a model trained on these photos has good accuracy when predicting sexual orientation. The authors' main claim in this paper is that the predictive power of AI models can be harnessed to encode complex patterns in facial features that could indicate an individual's sexual orientation. This research has received a lot of backlash for suggesting a new form of digital physiognomy [23, 27, 33] . While the researchers may have started with the assumption that physiognomy is pseudoscience, they readily discarded this in favor of the view generated by their model that facial features can predict sexual orientation. Kosinski himself has defended the study as revealing both the "huge promise" of big data as well as the risks due to loss of privacy [31] . Meanwhile, critics have argued that this attempt to subject sexual orientation to objective measurement, while an interesting exercise in classification that reveals unexpected correlations with high accuracy, is erroneous as it fails to account for the subjectivity of social context [16] and reifies social stereotypes [24] . However, there is a danger of repeating the study's mistake by assuming that automated systems can only reify existing gender ontologies. It is, for example, possible that previously invisible correlations between bone structure and sexual preference really do exist, encouraging future work to explore and falsify new hypotheses. Meanwhile, a more critical analysis of the model might suggest that the contextual purpose of dating profile pictures is to broadcast sexual orientation to potential partners, rather than neutrally reflect how facial features predict sexual preferences. Since the engineers did not seriously consider this, they propagated a questionable conclusion based on the model output. We will later suggest interpreting such research studies as generating authentic discordances between our intuitions about the social world and novel beliefs about it that must be examined and deliberated, rather than summarily dismissed.
 Dogmatic Reconciliation Finally, we consider a class of discordances arising when the system engineers do attempt to harmonize their moral dispositions with system-generated propositions, but unfortunately use blunt methods to make systems comply with the former. The failure occurs when engineers, despite using values-based design, still encode their implicit biases into AI models through training and formulation instead of explicating and confronting their own assumptions. We consider two recent papers from the FATML literature that focus on the problem of biased predictions resulting from natural language processing: "Women also Snowboard" [9] and "Man is to Computer Programmer as Woman is to Homemaker" [7] . In both of these papers, the authors seek to correct gender related bias in various downstream tasks that occurs using state-of-the-art word vector embeddings for natural language processing. In "Women also Snowboard" [9] the authors address gender bias that occurs when developing AI systems for automatic image captioning. Specifically, the authors note that given certain contexts (e.g. sports equipment, computers, purses), image captioning systems tend to give incorrect predictions that fit common gender expression stereotypes. For instance, a captioning model might predict the caption of a picture of a woman snowboarding as a man snowboarding, since men are more associated with sports contexts. The solution that is introduced to curb this problem involves creating two classes of words ("male" vs. "female") and formulating a loss function to be used in model training that actually incentivizes confusion between these classes if insufficient evidence is found to make a gendered inference. Likewise, in "Man is to Computer Programmer as Woman is to Homemaker" [7] , the authors attempt to address gender bias that can be observed when performing analogy tasks using word vectors. While vector embedding in words generally yields useful semantic analogies (e.g. man is to king as woman is to queen), the authors note that certain problematic analogies are also picked up by these embeddings given certain corpora. To curtail these problematic analogies, the authors propose a method in which they compute a "Man to Woman" subspace of the embedded vector space, and formulate a way to reduce the the projection of non-gendered words on this subspace, thus removing much of the unintentional gender encoding that could have been contained by those words. Both of these works confront a discordance between model and social understandings of gender, namely that it should not be informative for certain aspects of an individual (women can also be snowboarders and computer programmers), and attempt to harmonize this view with the generated propositions of these given models. That said, the solutions proposed by the authors do not directly correct for the notion that gender expression should be uninformative on certain predictions, and instead address the problem of women being underrepresented in data and should be predicted with equal probability to men in certain contexts. This is a subtle distinction, but to illustrate it fully, we note that both solutions presented in these papers assume some kind of distinct "male" to "female" distinction and not more contextuallynuanced forms of gender identity. Essentially, by attempting to produce a solution to the system's discordant moral claims, the authors have hard-coded a cis-gender understanding of human sexuality into the models. The authors thus do not directly engage with the root discordance they seek to address, and instead provide a potentially problematic stop-gap solution that reacts only to their morally-charged dispositions. One way of mitigating these effects would have been a more thorough examination of the social-scientific literature on gender expression, and broadening the gender diversity of the research teams involved. In particular, since the methods proposed by the authors involve a rigid encoding of expression, they produce training techniques that are incompatible with intersectional gender identities. For instance, a correction agnostic to the intersection of gender expression and race might result in false corrections due to its rigid coding.
 THE CONTEXT OF ANTHROPOMORPHIC BIAS The questions surrounding anthropomorphic bias -is it always traceable to some original context of human bias? is it original to the statistical compromises that accompany automated decisionmaking? can it be avoided entirely?-demand a deeper philosophical analysis. In this section we proceed to conduct this analysis in two parts.
 Gendler on Belief-Discordant Alief Some form of "bias", however it is defined, is inevitable when any small team of humans derives actuarial interventions for broad populations. Trade-offs between accuracy and variance or false positive vs. false negative rates have been a hallmark of statistical inference since the discipline's birth, and while AI has considerably increased the scale and speed of such inferences in deployment, they have not fundamentally changed the rules of this game. Instead we should aim for a principled trade-off between the limits of inference, given messy data sets, imperfect model choices, or limited training time. But what might such a principled trade-off look like for anthropomorphic bias, which combines affect-laden human intuitions with machines' capacity for semantically-arbitrary classification? We appeal to Gendler's [17] work on the complex and codependent relationship between belief and "alief", which defines the latter as "a mental state with associatively-linked content that is representational, affective and behavioral, and that is activatedconsciously or nonconsciously -by features of the subject's internal or ambient environment. Aliefs may be either occurrent or dispositional." Gendler illustrates this phenomenon through the concept of belief-discordant alief, which accounts for scenarios in which people are afraid of walking on an open skywalk despite its structural safety, won't touch an object for fear of "cooties, " or automatically reach for wallets despite knowing you left it at home. Belief-discordant alief is the triggering of affective response patterns and automatic motor routines opposed to "explicit, conscious, vivid, occurrent belief" [17] . That is, it arises when we enter a situation that triggers us into a cognitive state that counteracts what our 'better' judgment knows not to be the case. There is some semantic risk in applying such technical philosophical concepts to a problem as provocative and wide-ranging as anthropomorphic bias, which is already the subject of a rapidly growing empirical literature. However, we feel Gendler's language is not just relevant but necessary for diagnosing the problem in machine learning, for two reasons. First, Gendler's examples and qualifiers succeed in contrasting belief with alief by defining the former in a strongly computational sense: belief is an explicit proposition whose content is discrete (not associative), is universally held (not situationally triggered), and refers conclusively to external reality (not emotions or habits of mind). We shall see that these technical descriptors of belief and alief are extremely useful for diagnosing the specific epistemic tensions within AI systems and the moral dispositions of those who design or interpret them. Second, Gendler's terminology emphasizes the discordance between different kinds of bias as the source of the real problem, not implicit bias in isolation. She holds, following Hume, that the hallmark of alief is a kind of association by which semantic, emotional, and behavioral dimensions are crystallized over time. In belief-discordant alief, there is something about an environment's psychological effects that trigger one to automatically respond in a way opposed to one's beliefs about it. The associative content of alief is highly arbitrary, just as one's explicit beliefs may be fundamentally prejudiced. This is key for grasping the confusions surrounding anthropomorphic bias in machine learning: AI systems aren't conscious, yet classify social artifacts much as we do; system designers strive for formal accuracy, yet display strong moral affect in response to automated claims. The biases of both systems and designers play a role in generating the discordant environments in which a machine's classifications feel inappropriate or morally wrong, and we can make sense of this by recognizing the ontological primacy of belief-alief discordance over the isolated prejudices of humans and machines.
 Alief-Discordant Beliefs in Machine Learning To properly apply Gendler's insights, we propose the concept of alief-discordant belief to describe the origin, form, and consequence of anthropomorphic bias in automated systems. When deployed, these systems (e.g. image captioning) transpose human-generated forms of alief by computationally remaking the context within which our aliefs typically operate. In Gendler's terms, they generate beliefs that violate the habitual associations between semantic meaning (e.g. these variables are related given a specific parameter space) and moral dispositions (snowboarding is something anyone can do), producing a visceral reaction from the designers ("women also snowboard") that demands reconciliation [9] . Consequently, the "bias" of automated systems refers to the uncanny semantic associations that arise from applying our aliefs to a purely datadriven setting, violating the contextual ties between disposition, affect, and representation that underpin our aliefs. When confronted with examples of an image classifier that offend us, we may have an automatic affective response that counteracts the beliefs that are either encoded into the algorithm's learning procedure (the belief that the classifier can learn semantic associations in an objective, impartial manner and arrive at ground truth) or generated by it (only men snowboard, certain faces are gay). In other words, the classifier crystallizes propositions about its own learning procedure as well as its predictive outputs, either of which can give rise to alief-discordant belief. Although such a classifier is just a computational function, there is a tendency for designers to anthropomorphize it as if it had autonomous beliefs, leading to a search for where these beliefs come from or who is to blame. This can leave engineers in the position of apologizing for the very data that is needed for the model learn anything useful (much like the case of MS Tay learning English for Nazi posts). We suggest interpreting such systems as automatic belief generators that compel us to reinterpret our own aliefs in the deployment context. Rather than claiming an algorithm is "biased," we should confront the tension it creates between beliefs and aliefs we have about the social world. We must avoid compounding this tension by manually patching in solutions to our most violated or dogmatically-held aliefs (e.g. we calibrate a classifier generate all outcomes independently of protected attributes). Instead, we must address the discordance that is making us feel uncomfortable (such as in the cases of [9] and [7] ). Rather than blaming the data or its labelers as biased, system designers are responsible for sorting out these discordances as they arise by harmonizing the generated beliefs with their own newly-challenged, inherited aliefs. The goal is thus not to try to make automated systems unbiased, but to interrogate the beliefs generated, the procedure for that generation, and the relation of both to our own aliefs. These factors can be made to map onto specific components of the AI's socio-technical infrastructure: the system engineer, the chosen model and training methods, and the data used. The onus is therefore on figuring out where in this pipeline our aliefs are being violated, how each contributes to this violation, and which of these components is most responsible.
 TOWARDS A TECHNO-SOCIETAL INFRASTRUCTURE Intersectional identities are a key hurdle in identifying the contexts that lead to alief-belief discordance. From the engineers' standpoint, this can result when either data does not contain sufficient examples of them, or they cannot be effectively encoded for the purposes of inference. And from the systems' perspective, intersectionality cannot be fully captured without generating a combinatorial explosion, since the interplay of relevant social variables (e.g. race, gender) will expand exponentially as more are deemed relevant for a given context. Confronting both sources requires interventions on the decision-making pipeline itself, and discordances are managed most readily if each step of the pipeline is designed to maintain one analytic component of alief-discordant belief. Prereflective moral dispositions necessarily inform the application of this pipeline through continuous attention to context, but the mechanics themselves are tied to conditions of explicit knowledge representation in the form of system-generated propositions. A significant component of "fair" machine learning is the integrity and documentation of this techno-societal pipeline, such that bias can be managed well beneath the psychological threshold of moral outrage that has regrettably defined the public reception of prominent case studies (see for example [32] ). This implies that much of the burden for moral responsibility gets shifted from the data to model training and finally to engineers-if practitioners find themselves ignorant of the context of what they are working on, it is inappropriate to shift blame onto your tools.
 Data The collection of human data requires a compromise between alief (whatever moral compulsion(s) was felt in the leadup to collecting it) and belief (whatever data structure and type was determined to best represent assumptions about reality). There is always a moral context that informs the data content, just as there is an explicit frame of representation that accounts for its form. This original compromise is often invisible and unacknowledged in a given dataset. While troubling, the main problem here is how to deal with this crystallization to ensure it can be accounted for and is traceable. Here the following guidelines are necessary: Data as context-specific: data must be documented with the original priorities of those who collected it, and the relevant case law that informed its collection. These "datasheets for datasets" help ensure that transparency and accountability are baked into the pipeline from the start [15] . This will provide a traceable baseline for the alief(s) lying behind the original data even after it has been used to generate classification regimes. Data as explicit: data should be annotated so that the assumptions behind its collection are clearly documented, rather than left implicit. This is meant to account for the beliefs or prior evidence that informed why the data was collected and organized in a certain manner, rather than another. Data as contestable: data should be publicly available so that its alief-belief crystallization, however messy or regrettable, can be challenged by those most subject to its labels or classification. This affirms that the alief-belief matrix behind its collection is a product of compromise worthy of continual reflection and deliberation.
 Model and Training As a machine learning model is being trained, data-and with it a complex web of social relations, moral drives, and unstated representational axioms-are reified as generated beliefs about the macro-environment in question. This is the great benefit and cost of machine learning at scale, to wring more intuition out of a dataset than existed anywhere in the minds of those who produced it. The problem is to ensure the generated beliefs accord with the inherited beliefs that defined that data, or at least the beliefs of system engineers: Models as interpretable: models should be easy to understand by qualified humans so that classifications have a clear semantic context, requiring explainability. If this is not the case, as can happen in deep learning [18] , it will make the work of identifying bias difficult, because the model itself cannot be consulted to resolve the discrepancy or point to likely solutions. This "discordance slack" should instead be reined in as early as possible. Models as intuitive: model assumptions should be documented so they are easily altered, without an unnecessary amount of work going into why these assumptions were chosen. This is necessary to ensure that the belief generation of the model is kept distinct from the belief testing of its trainers, rather than the two becoming conflated. Models as corrigible: assumptions should be transparent and available to the wider machine learning community so that they can be challenged. This is a redundancy check against the psychological biases of the model trainers and helps bolster the accordance between humans' and the model's beliefs.
 Engineers Engineers are the ultimate source of aliefs that are discordant with the model's generated beliefs. As those who are using the model to classify new data in contexts other than what the model was trained on, they will often bear the responsibility for its failures and specifically for the discordance between their own moral dispositions and the model's classifications. To review what has been stated already, where the data is dirty and the model is a black box, alief-discordant belief is almost inevitable. This is the scenario we are trying to anticipate by making the discordance manageable. Discordance as discoverable: engineers should be trained about psychological bias to better identify discordances where they are subtle or hidden in edge cases for the model in question. The goal is to make note of discordance before the public discovers it in deployment, which could cause social harm and also make the model less trustworthy. Discordance as tractable: engineers should be expertly informed about the likely problems with data and model training, i.e. have a general grasp of the pipeline and its context to better confront discordances when or if they arise. The goal here is to know how to manage the discordance, rather than simply to flag it. Discordance as contextual: engineers should be trained to maintain healthy work environments, have access to legal consultation, and cultivate emotional intelligence to better process discordances when they themselves feel them. Thanks to Three Mile Island and other man-made disasters, we have learned that fearmongering, societal distrust, and lasting damage can be avoided if engineers respond appropriately to a crisis rather than misrepresent the nature of the problem either to themselves or the public.
 CONCLUSION We have suggested alief-discordant belief to avoid the shoals of anthropomorphic bias in an automated context. Epistemically, aliefdiscordant belief accounts for the subtle ways in which human cognitive bias enters a machine learning pipeline, first through the dataset, then through model and training specification, finally in the dispositions of system engineers. Ethically, we have suggested that machine learning practitioners should work to maintain the integrity of this pipeline so that alief-discordant belief, once generated by engineers interacting with the model, is manageable with respect to the actual stakes of the social context in question. Part of the wider project of realizing fairness through machine learning is for engineers to interpret themselves as part of this context, which includes the wider machine learning community as well as potentially-vulnerable populations of protected social categories. In light of this therapy, negotiating the discord between human aliefs and machine-generated beliefs may depend on crafting a wholly new context for automated decision-making, in which we get better at designing machines that supply us with beliefs that we are more critically prepared to adopt. Spotlight 1 : 1 Normative Perspectives AIES'19, January 27-28, 2019, Honolulu, HI, USA


Exploring AI Futures Through Role Play

Introduction Artificial intelligence (AI) has an enormous potential to have a radical effect on our future, and we all have a vested interest in trying to ensure that AI is beneficial for humanity. However, there is currently no consensus in the AI research community on what the long-term impacts of AI, potentially including general artificial intelligence, will be, or the length of time over which such changes will take place. Moreover, there are many nearand mid-term concerns that have the potential for transformative societal change [5, 14] . The vast uncertainty around AI futures, especially ones that involve transformative societal changes, has precipitated the creation of an AI strategy research community that seeks to ensure the development of AI for the social good [8] . In collaboration with researchers in this community, we have developed an AI futures role-play game, which we present here, to complement the research being done and to disseminate key questions and findings to individuals and organizations engaged with, or seeking to engage with, AI strategy and policy. A role-play game involves a collection of people assuming roles to enact a situation in a realistic manner so as to predict a possible future if certain strategies are employed. Role-plays have a track record of high forecasting, strategic and educational utility in a wide variety of fields, including business [15, 21] , medicine [18] , politics [16] and education [6, 21] . In the domain of AI strategy, we believe that role-plays can illuminate plausible futures, identify critical decision points, educate, raise awareness, highlight gaps in research knowledge and promote shared visions for beneficial AI. In this paper we first review the wargaming literature to show where role-plays have been successfully used in other domains, before next looking at why role-plays offer particular benefits within the AI strategy space. We then describe the current structure of the game (still under active development), and we highlight some of the lessons learned from the 30 game events to date. We conclude with a discussion of the benefits and shortcomings of this AI role-play game, our plans for further development and dissemination, and a call for others to try roleplay as a methodology for exploring AI futures. Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for third-party components of this work must be honored. For all other uses, contact the Owner/Author. AIES '20, February 7-8, 2020 , New York, NY, USA © 2020 Copyright is held by the owner/author(s). ACM ISBN 978-1-4503-7110-0/20/02. https://doi.org/10.1145/3375627.3375817 2 Literature Review "Serious" role-plays are most well known in their military form, wargames, where they have been used, since their Prussian development in the late 18th century, to teach military strategy in a safe but dynamic manner. Wilkinson [24] championed the introduction of wargames into the British army in the late 1800's and claimed that "probably no form of military study is more useful if properly conducted". Original wargames had two militaries (teams of players) fighting over some well-defined territory (the board). More recently, emerging technologies and distributed warfare necessitated wargames to shift focus from territory and troop movement to also include the diplomacy of international relations [22, 23] , nuclear conflict escalation [20] and cyber operations [1] . The potential of role-plays to provide exploration and training in uncertain and high stakes situations has been recognized in other fields. In business role-plays, for example, the conflict arises between a company and its competitors vying for market share and profit. Hamel and Prahalad [15] explain the popularity of role-plays in business as due to their ability to develop foresight, an essential skill for competitive survival: "an organization which does not develop a picture of the future, which does not develop foresight, is most likely not involved in the future." In education [6, 21] , medicine [18] and politics [16] , role-plays are used for formulating strategy, developing foresight, crisis response preparation, and for training and recruitment exercises. Nevertheless, despite their proven utility in a variety of different fields, role-plays have so far not been used in AI futures studies [3] .
 When are role plays suitable? Parson [19] claims that role-plays are most appropriate for situations involving high enough stakes to merit substantial investment in knowledge about decision-making, but whose complexity imposes limits on the usefulness of standard decision-making procedures. Parson [19] gives three suitability conditions for role-plays: 1. Key outcomes depend on the interacting decisions of multiple agents. 2. Decision choice sets are ambiguous or poorly known. 3. Large numbers of complex organizations may be required to work together. The existence of multiple interacting complex agents compounded with a huge set of possible decisions creates an enormous potential scenario space. It is too difficult for one expert alone to think beyond linear views of reality, but roleplays allow non-linear effects to be explored. For instance, one can look for convergent outcomes arising from different strategies or identify the decisions pivotal in shaping the scenario's end state. Armstrong [2] compares and contrasts different forecasting techniques and adds that a role-play needs: 4. Large changes expected. 5. Conflict between the decision makers. If only small changes are expected, forecasters can extrapolate from current or historical examples. However, large technological step-changes -such as the discovery of electricity or the invention of the world wide web -entirely transform history's trajectory and are much harder to predict. This observation has been captured in Roy Amara's law [17] : "we tend to overestimate the effect of a technology in the short run and underestimate the effect in the long run." Finally, conflict generates oppositional friction which catalyzes the generation of creative ideas. Without the competitive nature of an adversarial interaction, it is hard to generate the surprising edge-case scenarios which are exactly the ones hoped to be reached by role-plays.
 Do They Work? There is empirical evidence substantiating role-playing's greater forecasting accuracy over unaided judgements in certain situations. Green, Graefe and Armstrong [11] showed that both experts and novices were terrible at forecasting conflict decisions if they were simply asked to "put yourself in the other person's shoes". This is because, regardless of expertise, it is too difficult to individually think through the interactions of competing characters with divergent goals in complex situations. However, if the subjects adopted the roles of the protagonists in the conflicts and interacted with each other, their group decisions predicted the actual decisions with much greater accuracy -the greater realism afforded by role-playing enabled superior forecasts to be made. Role-plays can also mitigate against myopia by helping overcome various different psychological biases. Cyert, March and Starbuck [7] found that subjects presented with the same data produced significantly different forecasts depending on which role they were told to act out and Babcock et al. [4] evidenced the tendency of role-players to make biased judgements and interpret the briefing material differently according to the role they were assigned. This explains why roleplaying benefits from larger groups and greater diversity. Duke and Geurts [10] found that role-playing activities in a political setting can mitigate the danger of 'groupthink', where only readily politically feasible or easy-to-implement strategies are discussed. It is not guaranteed that every role-play will provide novel insights and any insights that do arise might not be relevant or applicable. However, even end-game states that have ostensibly departed significantly from common sense are worth exploring. DeWeerd [9] found that when predicting the future, readers tended to reject all predictions which didn't match their own and yet were willing to accept at the hand of history what they would instantly reject if predicted in advance by someone else (examples of "unthinkable" outcomes may include the result of the Brexit referendum, the election of Donald Trump, or the attack on Pearl Harbor). Edge-case scenarios are a key virtue of role-plays since they throw up black-swan eventsunprecedented and low probability, but high impact events which are invisible to canonical forecasting techniques.
 Methods The first task when planning a game is to identify the objective (i.e. training, educational, exploring futures or a mix of these). The project to date has explored two different methodological approaches to the game, with slightly different objectives: unstructured and semi-structured games. In both cases, the aim of the game was to allow players to experience from a firstperson perspective significant changes that could be brought by the development and deployment of AI technologies, and to have players face strategically challenging decisions from a position of power, be it governmental, corporate, military, societal or other.
 Unstructured role-play For the first two years of the project, we explored various forms of unstructured role-play. Players were drawn from a variety of different backgrounds with varying AI and geopolitical expertise. The players choose their own roles and took unconstrained actions, limited only by the time available and their own creativity. Resolution of actions thus depended heavily on the "world model" of the game's facilitator, which we aimed to address through regular engagement with the AI research literature and the AI strategy research field. The overall aim of these role-plays was to trial out the methodology, and to give players a space in which to explore strategic questions that they found interesting. Whenever it was possible, early discussions with the players or a coordinator helped to determine the initial frame for the scenario and roles adopted by the players. If this was not possible, these would be chosen by the facilitator or in collaboration with the players at the beginning of the game. We experimented with different ways to brief the players, including the facilitator meeting with each player ahead of the session (for 15-30 minutes each) to discuss the strategic questions they are interested in and to flesh out the role for that player. In an alternative approach, the facilitator would tell players the available roles and provide the general framing of the game, before asking the players to do their own background reading on those roles. Which organizations and roles will matter for the future of AI is an open research question: as time goes by, the number of identified stakeholders only increases, from governments to corporations, militaries to NGOs, artists, environmentalists, criminals and citizens, AI will shape, and be shaped by, a vast number of stakeholders. Nonetheless, the power distribution amongst these actors is not equal, and in varying the organizations and roles in the unstructured role-play we were able to explore different narratives that emerge when different groups and types of power are at play. Over 30 games we trialed many different roles. These broadly clustered around: During the game, the participants would "assume" their chosen/assigned role and take actions according to their model of that actor. Participants are instructed to deliberately try to bring about a world that is in line with the goals of the role they are playing, but they should be mindful of real-world constraints. This is because the more closely the simulation captures essential behavioral features of the real system, the more likely insights are to be relevant. On the other hand, a key strength of role-plays is in exploring edge-case scenarios. Thus, as a general rule of thumb, actions should be realistic but not overly conservative. For example, one in ten actions should be an action that the participant assigns 10% likelihood of being taken. Participants are encouraged to "flesh out" the world model by adding details that complement the details provided by the facilitator.  Government organizations During previous games, participants tried out a wide range of actions, including: Games typically involve 4-8 turns, with each turn representing one or two years, thus covering a period of 4-16 years, with the starting point set in the foreseeable future (as near as 2020 or as far as 2028). The game proceeds by alternating between negotiation phases, private action phases, public actions phases, and "world updates" from the facilitator. Outcomes of actions are determined by the facilitator, sometimes assisted by a random number generator, and relayed either to teams directly or in the "world updates" at the beginning of the next turn. Figure 1 depicts the final state of the world for a recent game held in Cambridge. Immediately at the end of the scenario role-play there is a debrief where any hidden state of the world is revealed, and participants evaluate how well their assigned role performed relative to their goal. Participants are asked to reflect on the final state of the world, on actions or developments that they found surprising, and to record any disagreements with how actions were chosen or resolved. This phase of gleaning participants' insights is crucial for challenging the facilitator's own subjective biases or knowledge gaps whilst bolstering the participants' internalization of what they have learned. 
 Semi-structured role-play Based on our experience with unstructured role-play, we started developing a semi-structured game that involves a predefined game system for AI R&D, and a pre-defined set of organizations and characters, to create an experience that explores several questions we have found to be particularly pertinent, including: To explore these questions, we designed an eight-player game with the following roles: We also created organization sheets and character sheets with key information to familiarize players with their roles. As in the unstructured game, players were asked to spend each turn negotiating with each other before deciding upon a set of private and public actions. However, we made AI R&D actions structured around a technology tree (Figure 2 ) and a set of products and capabilities (Figure 3 ).
 State of the World: The unemployment rate is now 17.1%. However, people are content for the most part because of social services such as Universal Basic Income. Multiple sources have confirmed to the New York Times that AGI has been developed. Sources range in assessing the system's safety, from 25% to 75% confidence. China is preparing to establish a small city on Mars. The first 10,000 citizens will arrive during 2035.
 Paper Presentation AIES '20, February 7-8, 2020, New York, NY, USA Each turn, the different organizations have a limited amount of AI "talent" they can allocate towards pursuing technological advancement (research) or products and capabilities (development). This can be pursued in public or in private. R&D success is determined by dice rolls, with one dice per "talent" allocated to a project. The players' action can affect the distribution of "talent" in the "world". This resource management mechanic was added to explore conditions of competition and scarcity in the input to AI R&D. This is one such factor that generates an adversarial tension which drives players to seek more creative solutions. Another mechanic we added to the game was "world chaos", a catch-all parameter for global instability, whereby we capture various negative externalities of AI, including the potential of AI "hype" to distract states from addressing issues such as inequality or climate change.
 Preliminary Results The unstructured game format has allowed the exploration of many different scenarios, but consequently it is difficult to identify patterns in the scenarios played. Overall, participant satisfaction appears high, with most players indicating their interest in playing again, and many interested in reading more about different organizations, roles, actions or strategic analyses. The following range of topics have come up in at least one of the scenarios that emerged from unstructured role-play (this list is not exhaustive): It would often be the case that five or more of the above topics would be touched upon in a single game.  The semi-structured game is more constrained, and therefore provides a better opportunity to detect patterns of gameplay and decision-making. While the number of games played with the new system to date is small, and the game system is still under active development, a few patterns are beginning to emerge. First, and perhaps not surprisingly, the familiarity of players with the roles assigned to them varies greatly -while many have at least some idea of what the President of the United States cares about or what they can do, that is not the case for tech CEOs or heads of AI labs; while a good facilitator can use the game to provide some basic information about the role and some of the decisions they face (as they relate to AI), the game is more useful for all participants when the players start the game with more background knowledge, both about AI and about the contexts in which decisions about AI are made. Second, the rapid growth of options that comes with the proliferation of available AI capabilities can be overwhelming to players, who respond to the large number of available R&D avenues by either choosing randomly, picking quickly a "good enough" target, or aiming for a specific long-term technology. While this is to some extent an artefact of the game mechanics, we think it may capture a real issue, especially if AI progress lends itself to faster AI R&D and if progress remains stochastic and uncertain. Third, cooperation between actors on AI safety and (some) restriction on destabilizing uses of AI seem to both be robustly beneficial, and often necessary, for all players to achieve a good outcome. Such agreements are easier to establish if players are biased towards pro-social strategies, or when cooperation begins early in the game and is maintained and expanded, or when players recognize a common threat (e.g. an unsafe or destabilizing technology nearing completion or after a major accident). Competitive behavior, including adversarial actions (espionage, sabotage, poaching, tariffs), can sometimes be excused when a common threat emerges and the need for cooperation becomes apparent. Fourth, in almost all games players will continue pursuing ever-more-advanced technologies and capabilities, with all the resources they can afford to spend on R&D, even if the potential value of these advances is unclear, and even when earlier advances have proven to be destabilizing or risky. Fifth, while the game has a built-in system for AI R&D, some players are happy to innovate "around" this system, proposing new products or establishing new organizations during the game. Finally, different sets of possible coalitions and alliances are explored and can prove stable, including between government and corporate entities within a state, between governments, between organizations, and between sub-roles within organizations (e.g. between heads of AI labs).
 Discussion and Conclusions The AI role-play game described here presents an innovative methodology for studying and teaching the impacts of AI. It can educate, raise awareness and help illuminate the positive and negative effects of certain developments, strategic decisions, and Paper Presentation AIES '20, February 7-8, 2020, New York, NY, USA inter-stakeholder interactions thereby potentially improving the outcomes of deploying current and future AI technologies. The versatility of the technique, including its ability to be adapted to different problems throughout organizations to address the near-, mid-and long-term impacts of AI, is well-suited to the large variety of challenges encountered in the development and deployment of AI technologies across all aspects of human life. Many of the other benefits of role-play games discussed in the literature review are also applicable. The games we have conducted to date, and especially the semi-structured games, focused on a set of questions and insights that we find particularly salient. These include the interaction between near-, mid-and long-term risks, challenges of cooperation and competition between powerful actors across states of public/private lines, trade-offs between public and private research and between research and product development, and the experience of making potentially transformative decisions from a first-person (powerful) perspective under conditions of uncertainty and time pressure. The feedback from players to date suggests that these aspects of the game indeed appear salient to players, and lead to greater engagement with these questions. Based on players' feedback, and from design constraints of the game itself, we also note several shortcomings and limitations we have not yet been able to address. For example, in our focus on strategic decision making, we have focused on roleplay from the perspective of powerful actors, rather than actors with marginalized standpoints, or more detached viewpoints that highlight systemic and emergent drivers; the literature suggests that role-plays and games can be adapted for these perspectives, and have been powerful in building empathy or systems thinking when done well, and this is a potential direction for expansion. Another limitation is the sped-up timeline, that imagines dramatic, and occasionally transformative, technological shifts within an almost-foreseeable political horizon. This reflects our limited ability to conduct a realistic role-play that takes place in the political reality of 2050 or 2100, but it does mean we adopt an assumption of fairly rapid technology progress, which is in no way certain -experts significantly disagree about timelines to transformative AI capabilities and impacts [12, 13] . This design decision was made in part to play to the strength of role-plays, which shine in exploring large and uncertain effects, partly to create a more engaging and dramatic setting, and partly because if transformative impacts happen in a matter of decades, it becomes of paramount importance to prepare decision-makers to handle the decisions they will need to face. We also stress the communication of this bias in every game debrief, but we see how this sped-up timeline may be less appealing to some, and are exploring versions of the game that focus on less dramatic changes while exploring in greater nuance short-and mediumterm impacts of technologies that are already making real research progress. Moving forward, we will continue to develop the game to increase realism, reflect new strategic research questions as they emerge, and cover more scenarios of interest to a broader set of stakeholders. We aim to expand the context of the game, through different unstructured and semi-structured scenarios and mechanics, to cover an increasing range of actors, actions, technologies and impacts of AI. This necessitates staying in constant contact and mutual feedback with the developing fields of AI futures and AI strategy as the impacts of AI become more apparent in every corner of life. In addition, we hope to continue to develop the educational potential of the game into something that can be used to raise awareness and train a variety of audiences and publics, including younger members of the public, through the development of a digital platform and a digital version of the game. Readers interested in exploring the game are invited to register their interest, 1 and those who wish to join the development of the game are encouraged to contact the authors. By allowing stakeholders to explore alternative AI futures, prepare for surprises and uncertainty, and learn from "simulated" mistakes, we hope that the methodology we propose will help win the race between the capability of the technologies we develop and our wisdom in deploying and governing them. For these reasons we see the method's impact on the use of AI for social good as being far reaching and we invite members of the community to join us in exploring the future of AI through role-play. Figure 1 : 1 Figure 1: Final state of the world from a recent unstructured game. The game involved three players playing the roles of presidents for the USA, PRC and EU. Technological progress and impacts of players' actions were determined by the facilitator.
  How do the geopolitical relations between the two leading AI powers, US and China, affect the kinds of technologies developed and the way they are deployed?  How do state-corporate relationships play out as AI technologies become more advanced and allow for a greater generation and consolidation of wealth?  How do states make decisions about the deployment of AI capabilities in strategic domains such as military and social control?  How do corporations balance the pursuit of risky basic research that could lead to transformative capabilities with nearer term, profit generating products and services?  How, and to what extent, are states willing to "pick up the tab" for negative externalities created by their corporate champions?  What factors contribute to a decision to make a research project private/secret, or conversely to publish results and share capabilities? How do actors respond when they suspect other actors of pursuing advanced research in secret?  How do decisions and impacts relating to near-term AI technologies affect the landscape in which decisions are made about mid-term AI technologies, and similarly how do those affect the setting for decisions about long-term AI?
  PRC government: president and minister of national defense  US government: president and secretary of defense  Tencent: CEO and head of AI lab  Alphabet: CEO and head of AI lab
 Figure 2 : 2 Figure 2: AI Technology Tree used in the game. Each box represents a technology that can be pursued by the players, with associated products and capabilities. Technologies advance from left to right, with progress on earlier technologies required before later technologies are "unlocked".
 Figure 3 : 3 Figure 3: An example of AI products a player can develop in the game. The required technology and the effect on game parameters are listed at the bottom of the card.
 Non-governmental actors, including academia, independent research organizations, advocacy groups and think tanks.  Supra-national actors, e.g. the EU or the UN. and roles in different countries, including the executive, the military, diplomatic services, health, and government R&D functions.  Corporate actors in different geographies and sectors, including tech (based on business models we see around the world, e.g. advertising, e-commerce or tech infrastructure), robotics, weapons, health. 
 To register your interest in participation, please fill out the form at https://sites.google.com/view/intelligence-rising/home.

Shahar Avin
Ross Gruetzemacher
James Fox