Value of Global Catastrophic Risk (GCR) Information: Cost-Effectiveness-Based Approach for GCR Reduction

Introduction Global catastrophic risks (GCRs) are risks of events that could significantly harm or even destroy human civilization at the global scale (Hempsell 2004 , Baum 2010 . GCRs presently posing hazards to humanity include nuclear war (Sagan 1983 , Turco et al. 1983 , Robock et al. 2007 , Cirincione 2008 , Hellman 2008 , Barrett et al. 2013 ) and pandemic diseases (Nouri and Chyba 2008) . In the near to longer-term future, GCRs could include climate change (Weitzman 2009 , Travis 2010 ) and misuse or accidents involving technological developments in areas such as artificial intelligence (Yudkowsky 2008 , Chalmers 2010 , Sotala 2010 ) and nanotechnology (Phoenix and Treder 2008) . Proposed interventions to reduce GCR include nuclear disarmament (Robock et al. 2007 ), development and distribution of vaccines and antiviral medications (Osterholm 2005) , reducing greenhouse gas emissions through public policies (Aldy et al. 2003) and various individual behaviors (Dietz et al. 2009) , and abstaining from developing certain technologies (Joy 2000) . A growing body of work makes the case that reducing GCR, or certain types of GCR, is of very high value and thus should be one of the highest objectives for society (Ng 1991 , Bostrom 2002 , Posner 2004 , Matheny 2007 , Tonn 2009 , Ćirković et al. 2010 , Beckstead 2013 . Published estimates of the value of preventing global catastrophe range vary wildly, from $10 billion (Bostrom and Cirkovic 2008) to infinity (Weitzman 2009 , Baum 2010 , depending partly on the definition used for "global catastrophe." Even the low end of this suggests a large allocation of resources toward GCR reduction. However, setting GCR reduction as a high priority is not a sufficient guide for action: there are many open questions regarding how best to allocate resources for GCR reduction. One basic question is how much to allocate toward direct risk-reducing interventions and how much to allocate to research to inform these interventions. The decision analysis concept of expected value of information (Clemen and Reilly 2001 , Keisler 2004 , Bhattacharjya et al. 2013 can inform decisions about how much to spend on information (i.e., reduce Decision Analysis, 2017 , vol. 14, no. 3, pp. 187-203, © 2017 uncertainties) prior to making other resource allocation decisions. Usually in value-of-information calculations, decision options are evaluated using utility functions, money, or functionally similar metrics that have implicit commensurability between option trade-offs, e.g., lives saved versus dollars spent. However, equating lives and dollars, e.g., using a typical value of statistical life (VSL) saved, may be inappropriate given the potentially vast scale of GCRs. (Moreover, quantifying total event consequences of global catastrophe in conventional benefit-cost terms would be complicated by uncertainties about direct event impacts, indirect impact factors such as public behavioral responses, and the levels of such impacts that could be borne before reaching civilizational-collapse tipping points.) We take a different, cost-effectiveness-based approach in this paper instead. A cost-effectiveness-based equation for value of information also may be useful in other domains where typical VSLs would not be appropriate. In this paper, we argue that value of information based on cost-effectiveness is a useful tool for analysis of GCR to inform risk-reduction decisions, and we show that it can be defined in a practical manner. We argue that such an approach would be most valuable if applied in a comprehensive, integrated fashion to all major types of GCR, rather than one at a time. We describe a number of challenges that would arise in such efforts, and argue that these challenges can be addressed. We also provide an illustrative, though highly idealized, example that shows how a practical value of information calculation can work. It also provides support for our argument that such calculations can have considerable value, and it provides further support for our argument that value of information can provide additional insight when more than one GCR is under consideration. In Section 2 of this paper, we give a brief overview of the basics of the approach and how to apply it to GCRs and risk-reduction interventions in a comprehensive, integrated fashion. In Section 3, we discuss key challenges in real-world implementation of this paper's framework and argue that these challenges can be addressed. In Section 4, we illustrate the basic framework using a simple notional model of GCR from two types of near-Earth objects (NEOs; i.e., asteroids and extinct comets) as well as nuclear war, and consideration of two related risk-reduction measures. The illustrative example shows that such calculations can have considerable value, especially when considering multiple GCRs. We conclude in Section 5. (In the appendix, we provide a detailed derivation of our formula for the expected value of information in terms of the costeffectiveness of GCR reduction.)
 Overview of Framework for Value of GCR Information In this section, we briefly discuss ways to approach three linked sets of quantitative issues: first, representing the probabilities of multiple GCRs; second, assessing the overall cost-effectiveness of GCR-reduction measures and calculating the value of information for GCR reduction; third, contrasting perfect and imperfect information. More details of our approaches and assumptions are given in the following sections.
 GCR Probabilities Figure 1 is a fault tree or logic tree illustrating that there are multiple types of global catastrophic risks, and occurrence of each is assumed to be causally independent of the others, at least at the level of detail used in the fault tree (e.g., nuclear war does not cause asteroid impact). The event "Global Catastrophe" is the top event, with round-corner nodes for a series of GCR types branching out below, all connected by an OR gate. The fault tree graphically indicates that a global catastrophe will occur if any of the following types of event occur with global catastrophe-level consequences: a large NEO impact (either an asteroid or a comet impact), large nuclear war, or a combination of smaller events (small NEO impact plus small nuclear war), etc. In addition, Figure 1 includes square-corner decision node risk management for two types of GCR-reduction options (i.e., NEO redirection, and food stockpiling) that could reduce the probabilities of global catastrophe-level outcomes. Grey arrows from the square-corner decision nodes to the roundcorner fault tree nodes indicate that the risk management decisions can influence the risks of global catastrophe-level events. Figure 1 also illustrates that some risk-reduction measures, e.g., food stockpiling, can have benefits in reducing multiple types of GCR. Although the fault tree portion of Figure 1 is quite simple, it is intended to underline the main motivation for considering GCRs as a whole, and not just individual types such as asteroids, comets, or nuclear
 Food stockpiling Combination of small nuclear war and small asteroid impact GC Small nuclear war Small asteroid impact AND war: to assess and reduce the total probability of global catastrophic risk, ideally we would assess all types of GCRs and GCR-reduction measures in a comprehensive way. The framework also can account for interactions between GCR events, such as when occurrence of one type of event reduces society's resilience to or even causes another type of event (Baum et al. 2013) . Such interactions between GCRs could be represented using larger, more detailed fault trees (e.g., by adding branches for scenarios in which both NEO impact and nuclear war events occur around the same time, either just by chance of timing or because an NEO impact somehow causes nuclear war), though it could be difficult to explicitly account for many GCR-interaction scenarios, and important uncertainties could remain about unmodeled GCR-interaction dynamics. Figure 2 is a generic consequence exceedance probability plot for some type of event (e.g., NEO impacts), with curves showing relationships between event consequence and the probability of events with consequence exceeding that level, for both initial and reduced event risks. The figure illustrates that reduction in probability of global catastrophe can be achieved either by reduction of probability of events or by reduction of consequences. Starting from the upper right of the figure, the point where the initial event probabilityconsequence curve intersects with the global catastrophe consequence threshold indicates the initial probability of global catastrophe. The figure also includes two reduced-risk curves, one for reduced event probability and another for reduced event consequence. The curves for reduced probability and reduced consequence have been placed where they result in the same reduction in probability of global catastrophe, partly to keep the figure simple and partly to emphasize the idea that GCR reduction can be achieved by reducing either event probability or consequence. For example, NEO impact risk-reduction measures could reduce the probabilities of global catastrophelevel outcomes either by shifting the curve downward with reduced NEO impact probabilities (e.g., via NEO redirection) or by shifting the curve leftward with reduced NEO impact consequences (e.g., increasing societal resilience to NEO impact via food stockpiling). Thus, probabilities of global catastrophe for a particular GCR event type could be calculated as a function of Decision Analysis, 2017 Analysis, , vol. 14, no. 3, pp. 187-203, © 2017 global catastrophe consequence threshold, using consequence exceedance probability models for that event type. Of course, development of appropriate consequence exceedance probability models would often require substantial research, especially when focusing on rare or unprecedented events, for which a lack of data often leads to substantial uncertainties and biases (Taylor 2008) . In many cases, there would be large uncertainties for both the direct consequences of an event (e.g., in terms of atmospheric soot loading from various nuclear war or NEO impact scenarios) and what threshold level of consequences would result in global catastrophe (e.g., in terms of the effects of atmospheric soot loading on agricultural productivity and other indirect effects on human society, which could be highly nonlinear if stresses could reach civilizational resilience-exceedance tipping points). Such uncertainties could be modeled using probability distributions for the global catastrophe consequence threshold and exceedance probability function. One way to represent uncertainties is to display 5th and 95th percentile value lines in addition to the mean value lines (Garrick 2008) , as shown in Figure 3 . Given the previously mentioned assumption of causal independence, Equation (1) gives the total probability of a global catastrophe-level event within some time period, p total , as a function of the independent probabilities p j of catastrophe events of each GCR type, j, for a total of y GCR types. Equation (1) is mathematically consistent with the previous statement that a global catastrophe will occur if a type of global catastrophe involving a large asteroid impact, a comet impact, or nuclear war, etc., occurs: p total 1 − y j (1 − p j ). (1)
 Cost-Effectiveness and Value of Information for GCR Reduction Figure 4 is a high-level decision tree, consistent with typical trees used in calculating value of information. (The tree does not include specific quantitative values for probabilities, costs, and benefits, but it does indicate the general sequence of decisions and events.) The leftmost square decision node represents a decision to be made on whether to invest in research to inform decisions on risk-reduction measures; other decision nodes represent addition to a basic decision on whether to measures to reduce risks. In Figure 4 , the research decision is simple: conduct research to better understand whether risks are currently high or low, or do not conduct such research. The risk-reduction decision options are also simple: invest to reduce risks or do not invest to reduce risks. The decision on whether to conduct research is made before the decision on whether to invest in reducing risks. If the decision maker chooses not to conduct research, then they make the riskreduction decision with some amount of uncertainty about whether risks begin as high or low. (That uncertainty is represented by circular chance nodes, and the outcomes of chance nodes are represented by diamonds.) If the decision maker does choose to conduct research, then they have more information and less uncertainty about whether risks begin as high or low, and the decision maker can use that information when making their decision on whether to invest in reducing risks. A full valuation of GCR-reduction interventions, including research to gain information, requires some evaluative metric. Typically, decision options are evaluated using utility functions or functionally similar metrics. 1 Such metrics have implicit commensurability between option trade-offs, e.g., lives saved versus dollars spent. Use of such approaches allows for a relatively simple equation for expected value of options with various attributes (Clemen and Reilly 2001) , including trade-offs between GCR reduction and other objectives. In this paper, we avoid full valuations and instead conduct partial valuations in terms of costeffectiveness, measured in GCR reduction per unit cost. We focus on cost-effectiveness for two reasons. First, a full valuation for GCR is complicated by the widely varying estimates for the value of preventing global catastrophe; which can range from $10 billion to infinity, as mentioned in Section 1. Second, many GCRreduction decisions involve allocating resources, such as money. However, equating lives and dollars, e.g., using a value of statistical life saved, may be inappropriate given the scale of GCRs. Therefore, our equation for calculating value of information is based on riskreduction cost-effectiveness, which incorporates estimates of the performance and costs of risk-reduction options without use of VSLs. Our cost-effectivenessbased equation for value of information may be useful in other domains where VSLs would not be appropriate. We assume that there are one or more decisions to be made about the allocation of resources to some combination of options for risk reduction and options for research, and that the decision rule is to choose whatever combination of options has best overall expected GCR-reduction cost-effectiveness among options considered in the analysis. (Such considerations could occur in a series of risk-reduction decisions, in which case the goal could be to identify the most cost-effective interventions first, and then the second most, and so on, until a risk-reduction budget or target has been reached.) Then, in such decisions, the decision maker should buy as much risk reduction (and risk research enabling better risk-reduction decisions) as they can at whatever total cost, as long as that results in the greatest cost-effectiveness. Such decisions can arise when considering public policies, as well as the actions of individuals and other nongovernmental organizations. (We assume that budgets are not an issue in the context of the risk-reduction and research options under consideration, and we do not explicitly account for potential budget constraints in the following. However, consideration of budget constraints could be addressed as an extension of the approach used in the following.) For the purposes of this analysis, we ignore actual costs of research and focus on the amount of resources the decision maker ought to be willing to pay for Decision Analysis, 2017 Analysis, , vol. 14, no. 3, pp. 187-203, © 2017 the value added by the research in the context of the decision the research could inform. In other words, we focus on finding the maximal potential benefits of research. We assume that research ought to be invested in up to the point where a funder would obtain no further benefit from investing in additional research (because up to that point, they would get a better overall cost-effectiveness by investing in additional research). At that point, the expected cost-effectiveness of the best risk-reduction option before research is equal to the expected cost-effectiveness of the best riskreduction option after research, including the cost of research. Equation (2) gives the value of research as the costeffectiveness-based expected value of perfect information, CEEVPI (see the appendix for derivation): CEEVPI E c b s (p a 0 − p a s ) p b 0 − p b s − c a s . (2) The equation assumes the following: There exists a set of n available risk-reduction options numbered 0, 1, . . . , i, . . . , n. Option number 0 is the status quo case, where no new (or non-"business-as-usual") riskreduction option is implemented. The cost of implementing risk-reduction i is c i . (It costs nothing to do nothing, so c 0 0.) The annualized total probability of global catastrophe if implementing option i is p i . (We make the simplifying assumption that p i values are static, or unchanging over the relevant time period. Consideration of dynamic, or time-varying, p i values could be addressed as an extension of the approach used in the following.) Each c i is treated as a random variable with some probability distribution reflecting uncertainty about the true cost of implementing intervention i. Each p i is also treated as a random variable, with a probability distribution reflecting plausible estimates of the annual probability of global catastrophe given intervention i. 2 Computationally, the uncertainty is represented using Monte Carlo simulation, where in Monte Carlo simulation iteration m there are sampled values c im and p im . The risk-reduction option s is the option with the "best" or highest risk-reduction cost-effectiveness in Monte Carlo iteration m. In addition to decisions on which risk-reduction option to choose, there are also decisions on whether to first spend some resources on research to reduce uncertainties (and to more accurately identify which riskreduction option would be most cost-effective) before making decisions on risk reduction options. We denote whether research is conducted to reduce uncertainty on a particular factor using superscript b for "before" research, or without information from research, and superscript a for "after" research, or with information from research. Generally, research will have the greatest expected value if it has substantial possibility of informing a decision, i.e., a choice between risk-reduction options. However, the CEEVPI formula also implies that if it is expected that the best option after research is the same as the best option before research (i.e., if s a s b ), then the research still can have positive expected value if it is inexpensive enough and also provides sufficient reduction of uncertainties in p and c factors. We provide an example, calculating CEEVPI for illustrative catastrophic NEO impact risks and riskreduction options, in Section 4. The example suggests that the value of GCR information could be quite substantial.
 Perfect and Imperfect Information In the context of a decision analytic model, the value of information is based on the extent to which information reduces the uncertainty about the value of a particular parameter in the model. Perfect information eliminates that uncertainty. The expected value of perfect information (EVPI) is the difference between the expected value of a decision with perfect information (where the new information influences the decision we make) and without additional information (where we make the decision with our initial level of uncertainty; Clemen and Reilly 2001). We do not expect real-world GCR research to yield perfect information in the sense of eliminating all uncertainties. In general, EVPI calculations are used to set an upper limit to how much should be spent on reducing uncertainty. On their own, EVPI calculations cannot predict how valuable specific research will be in reducing uncertainty. However, even imperfect information can have great value in reducing decision model parameter uncertainties by some amount. Straightforward extensions of the approach to EVPI calculations used in this paper (based on costeffectiveness calculations) could provide methods to assess the expected value of imperfect information (Clemen and Reilly 2001) and expected value of including uncertainty (Morgan and Henrion 1990) .
 Key Challenges of Integrated Assessment of GCR In this section, we discuss important challenges for the implementation of our framework for calculating value of information, and for comprehensive, integrated assessment of GCR to inform risk-management decisions. We have already mentioned some of these challenges, which we discuss further here. We also discuss others that we have not mentioned previously. One challenge is that in the real world, there would often be complex interactions between GCRs, not all of which could be modeled. As previously mentioned, one important simplification of our approach is the assumption of independence of GCRs except where indicated in the model. In principle, many types of interactions could be accounted for by building them into fault trees or other model components, but that could require substantial efforts. As with modeling of any complex system, there would be large uncertainties about how much of the real-world dynamics would remain unmodeled. A similar set of challenges (and irreducible uncertainties) would be encountered in attempting to define global catastrophe consequence thresholds. Another challenge would be in setting appropriate thresholds for catastrophe. An important simplification of our approach is that we use a binary threshold for catastrophe (i.e., an event is only regarded as a global catastrophe if the event's consequences exceed the global catastrophe consequence threshold, however that is defined). In reality, events of a range of magnitudes could be regarded as global catastrophes, either because different stakeholders have different definitions of what constitutes a global catastrophe, or because of uncertainties about what levels of Decision Analysis, 2017 , vol. 14, no. 3, pp. 187-203, © 2017 direct effects from catastrophe events would reach civilizational tipping points. (Those uncertainties would stem partly from the difficulty of predicting indirect effects of catastrophe events, which involve complex factors such as the behavioral responses of large human populations. However, the analytic challenges and uncertainties would be even greater if the aim were to quantify total event consequences in conventional benefit-cost terms, which is another reason to use a simpler cost-effectiveness approach.) Differences between global catastrophe thresholds can have important implications for decision making. 3 Decisions should favor preventing higher-magnitude global catastrophes or decreasing the severity of any given global catastrophe. Furthermore, ideally, decisions would be robust (not highly sensitive) to placement of the global catastrophe threshold. As always, sensitivity analysis can usefully examine the decision implications of varying global catastrophe thresholds, and uncertainty analysis can suggest ranges to use in sensitivity analysis. There also would be challenges in defining what decision procedures to actually use, and how to incorporate considerations such as budget constraints and timing decisions. It seems unwise to take a perfectionist approach to assessing risks and risk-reduction optimality, because the complexity and scale of all potential risks and intervention options (including all interactions and combinations) could make that approach intractable. A more practical approach could be to make a series of risk-reduction decisions, either at regular or irregular intervals, that would first implement the most cost-effective interventions (or combination of interventions), then the second most, and so on, until a risk-reduction budget or target risk level was reached (essentially a greedy algorithm solution to a knapsack problem, in operations research terms). We believe the latter approach would be roughly consistent with our basic framework, though our current framework does not attempt to explicitly account for budget constraints, nor decision sequentiality. It also should be noted that our basic approach implicitly assumes the goal is zero probability of global catastrophe, but other targets could be used; for example, Tonn (2009) suggests a 10 −20 annual probability of global catastrophe as an "acceptable risk" target. Finally, accounting for timing of events and interventions could present substantial complications. For some issues, it could be important to account for decisions of exactly when to research, when to implement measures, and in what sequence; the urgency of implementing various measures also could be important. Although time dependencies are not explicitly reflected in the level of detail given in this paper, implicitly they could be incorporated into the model parameter values for effects of the risk-reduction measures. (For example, if considering implementation of an intervention today, versus some years from now, and if the GCR minimization objective is to minimize the probability of global catastrophe over the next century, then for many GCRs types such as NEO impact, presumably analysis would show greater GCR-reduction benefits from implementing interventions sooner rather than later.) At least in principle, time dependencies could be accounted for in modules whose outputs are fed to the model structure shown in this paper. Another challenge is that in the real world, there is not a single very well-funded actor whose prime objective is to reducing GCR cost-effectively. Instead, there are many potentially important decision makers, each with limited budgets and responsibility for GCR factors, and with various objectives that compete with GCR reduction. Potentially important decisionmaking entities include government agencies, such as the U.S. National Aeronautics and Space Administration (NASA), which have programs to address specific categories of GCR such as NEO impact risks; nongovernmental organizations such as the Open Philanthropy Project, which have programs to address either specific categories of GCR or all GCR broadly; corporations such as Walmart, whose product management decisions can have implications for societal resilience, emerging technologies, and other GCR factors; and individuals such as researchers, whose work can improve understanding of GCR factors and that have decisions to make about where to focus their own research efforts. Nevertheless, if credible integrated assessment has identified some GCR-reduction options as clearly being more cost-effective than others, that could influence decisions by various means, especially where actors already have some incentives to reduce societal risks. For government agencies, integrated assessment could inform budget reallocations, e.g., taking funding from low-value areas to fund highervalue risk-reduction programs, and incentives could be provided via government rules that encourage costeffective risk-reduction benefits to society. At the other end of the size spectrum, for individual researchers, integrated assessment could suggest which kinds of research could best lead to risk-reduction societal impacts, which are encouraged by both formal funding reviews and informal norms. Nongovernmental organizations and corporations also often combine efforts on voluntary stewardship initiatives and other programs to reduce societal risks, and thereby gain reputational rewards. Some of the most important challenges concern the scope of analysis, such as what GCRs and riskreduction measures to consider initially (given that starting-point estimates or at least bounding ranges would be needed for all associated modeling parameter values). 4 One approach that should be relatively tractable is breadth first: Begin by taking a broad but shallow approach to modeling GCRs and riskreduction options relatively comprehensively, but with little detail, and with quantitative parameter estimates aimed only at bounding ranges of uncertainties. Then, a series of subsequent, repeated model-improvement steps could iteratively add depth (i.e., to add detail and better quantitative estimates using the best available empirical data, expert judgment, etc.), and decisions on where to focus model-improvement efforts via research could be guided by value-of-information calculations.
 Illustrative Example: Notional Model of NEO Impact Risk and Mitigation In this section, we illustrate our concepts using information in the literature on impact risks posed by two types of near-Earth objects as well as nuclear war. We also provide illustrative modeling of two types of impact risk-reduction measures (i.e., NEO redirection and food stockpiling) that could reduce the probabilities of global catastrophe-level outcomes. These are very simple, notional models of risks and decisions, intended only to illustrate our value-of-information concepts. The example does not attempt to reflect all the latest references, such as the information on asteroid and comet impact risks yielded by the Wide-field Infrared Survey Explorer (WISE) and Near-Earth Object WISE (NEOWISE) survey programs. (Such research has often resulted in downward revisions in catastrophe probability estimates for both high-and low-albedo NEOs.) The example also does not attempt to estimate the risks or risk-reduction benefits related to GCRs besides NEOs and nuclear war, although considering those would affect overall GCR-reduction cost-effectiveness estimates. For example, food stockpiling could have benefits in reducing the effective consequences of pandemics, which is a category of GCR that is not considered in this illustrative example.
 Illustrative Model GCRs, Risk-Reduction Measures, and Assumptions The first type of NEO we model is "bright" or easily visible asteroids/comets, which can be observed and tracked long before impact using current astronomical capabilities. The Spaceguard Survey is believed to have detected most such NEOs with greater than 1 km diameter (National Research Council 2010). The second type of NEO we consider is "dark," or lowreflectivity damocloids, which current identification and tracking systems may not see until the objects are already headed directly toward impact. Partly because of the difficulty of observing damocloids using optical telescopes, there are large uncertainties about the frequencies of damocloid impact (Napier 2008 , National Research Council 2010 . Before the Spaceguard survey, such objects were thought to be a small risk relative to other NEOs, but damocloids and long-orbit objects have more recently been viewed as potentially posing the majority of remaining impact risk (National Research Council 2010). The NEOWISE survey program has been using infrared technology to better identify damocloids. Presumably, additional investments in research using infrared, radar, or other technologies could provide better observations of damocloids. Perhaps such damocloid observation systems would be deployed as some combination of Earth-based systems, satellites, and probes. We also model two types of impact event risk-reduction measures. First are NEO orbit redirection measures that offer good and relatively inexpensive reduction of risks of asteroids that are identified and thought to impact years or decades away (Matheny 2007) . The NEO redirection measures would reduce the probability of impact of a large asteroid. However, we assume Decision Analysis, 2017 , vol. 14, no. 3, pp. 187-203, © 2017 that they would not reduce the probability of impact of damocloids (at least, not without additional investments to identify damocloids, which is beyond the scope of this illustrative example). The second type of risk-reduction measure is food stockpiling to provide significant food reserves for a large number of people in case of a period of reduced food production (Rampino 2008) . The impact effects of large asteroids and comets could be broadly similar to nuclear winter and supervolcanism in their negative impacts on global food production. Food stockpiles may help humanity to survive either event. Rampino (2008) mentions that one potential supervolcanism survival strategy would be to stockpile enough food (e.g., grain) to last several years until agricultural productivity goes back up. Rampino (2008) notes that current inventories are only equivalent to about two months' consumption. While difficult to accomplish in many parts of the world, it still might be relatively feasible without advanced technology, should be relatively uncontroversial (especially if production is handled in a way that does not drive up global food prices very much), and could have some value across a number of GCR hazards including war, quarantine after pandemic, etc. In addition, unlike some other GCR mitigation measures, stockpiled food should retain near its purchase value in normal usage even in time periods where no GCR scenarios arise (i.e., if no emergencies arise before the stored food expires, the food can be eaten when rotated out of the stockpile and replaced with new reserves). We implement calculations for the illustrative example in a computational model using the software package Analytica by Lumina Decision Systems. The computational model incorporates all the defined equations and parameters. To estimate probability distributions of outputs, the model performs Latin hypercube sampling, with a model sample size of 10,000 iterations. The model varies continuous-valued inputs according to the previously given probability distributions, and the model produces probabilistic values of its outputs. For more on relevant distributions, e.g., uniform and triangular distributions, see Morgan and Henrion (1990) or the Analytica user guide (Chrisman et al. 2007) .
 Assumptions for Baseline P(Global Catastrophe). Our estimation of the probabilities of bright object and dark object impact risks is based partly on first estimating the total bright object asteroid impact risk, and then estimating how large the dark object comet impact risk is in comparison specifically to bright objects. We estimate the total probability of impacts of asteroids at least 1 km in size as corresponding to a frequency of one in 3 × 10 5 years. That is based on Figure 2 .4 on p. 8 of the National Research Council (2010) or the equivalent on p. 19 of the National Research Council (2010), which indicate a 1 km object impacts approximately once every 4 × 10 5 years. We assume that only 15% of the total population of bright NEOs remain undiscovered (National Research Council 2010) . For bright NEOs that have already been discovered, we also assume negligible impact risk: "none of those detected objects has a significant chance of impacting Earth in the next century" (National Research Council 2010, p. 19). The simple way we reflect that in the model is to say the impact risk from visible/ bright NEOs is 0.15 × (1/(3 × 10 5 )). We assume that the impact frequency of damocloids has a probability distribution of Uniform(0, 4) × (1/(3 × 10 5 )), based on a statement by (Napier 2008, p. 229 ) that the hazard from damocloids of 1 km diameter "is unknown; it could be negligible, or could more than double the risk assessments based on the objects we see." Some corroboration is provided by the statement by (Napier 2008, p. 226 ) that at the time of his writing, for 1 km objects, there was an "expected impact frequency of about one such body every 500,000 years." Once every 500,000 years is about the same as the once every 3 × 10 5 years we assume for over-1-km visible objects, but for better consistency and comparability with bright NEOs, we use 3 × 10 5 instead of once every 500,000. Napier (2008, p. 225 ) also observes the following: Estimates based on the mean impact cratering rate indicate that, on the long-term, a 1 km impactor might be expected every half a million years or so. Again, modeling uncertainties to do with both excavation mechanics and the erratic replenishment of the near-Earth object (NEO) population yield an overall uncertainty factor of a few. A rate of one such impact every 100,000 years cannot be excluded by the cratering evidence. All of the above numbers also have additional uncertainty factors (coefficients) of Triangular(0.5, 1, 2), which is loosely based on the statement by the (National Research Council 2010, p. 8) that the uncertainties in intervals between impacts are "on the order of With the completion of the Spaceguard Survey (that is, the detection of 90 percent of NEOs greater than 1 kilometer in diameter), long-period comets will no longer be a negligible fraction of the remaining statistical risk, and with the completion of the George E. Brown, Jr. Near-Earth Object Survey (for the detection of 90 percent of NEOs greater than 140 meters in diameter), longperiod comets may dominate the remaining unknown impact threat. Finally, for an extremely simple estimate of the annual probability of nuclear war, based loosely on estimates given in the literature (Hellman 2008 , Barrett et al. 2013 , Lundgren 2013 , we simply use Triangular(0, 0.0001, 0.001). It seems likely that the annual probabilities of global catastrophe events are orders of magnitude higher for large-scale nuclear war than for large NEO/comet impacts. In our calculations, we use a simplifying approximation of annual probability as being equivalent to annual frequency (e.g., a frequency of one event in 500,000 years implies an annual probability of 1/500,000). Table 1 contains summaries of the assumed annual probabilities of global catastrophe-level events from each considered GCR type. The expressions reflect the substantial uncertainties.
 Assumptions for Reduction in P(Global Catastrophe) if Implementing Each GCR-Reduction Mea- sure. Although the lower bound of effectiveness of NEO detection and redirection might seem to be quite low, it is based partly on the idea that NEOs that have not already been discovered might be significantly more difficult to detect than the ones that have already been detected. This was suggested by Napier (2008, p. 226) regarding the success of NEO detection efforts to date: "There is a caveat: extremely dark objects would go undiscovered and not be entered in the inventory of global hazards." For food stockpiling, the assumed probability distribution for the reduction in probability of global catastrophe-level NEO/comet impacts is Uniform(0.1, 0.9). It assumes that the stockpile would be comprised of extremely inexpensive sources of calories and nutrients (see below on cost assumptions), for which there would be large uncertainties about riskreduction performance. Table 2 contains summaries of the assumed effects and costs of GCR-reduction measures. (A status quo option, which adds no cost and does not reduce GCR, is omitted from the table but is an additional option in the model.) The effects of the measures are given in terms of their assumed reduction in probability of global catastrophe from each GCR type. The costs of the measures are given in terms of the present value of their costs in 2012 dollars.
 Assumptions for Costs of GCR-Reduction Mea- sures. The cost estimate for food stockpiling assumes a world population of 7 billion, a one-year stockpile, and a per-person-year stockpile cost based on the food expenditures of the world's poorest people, which is approximately $0.70 per day (GiveWell 2013). The cost for tracking and redirection capability assumes 30 years of costs, with $250 million annual costs (National Research Council 2010).
 Example Results In this section, we give results from the computational model for the illustrative example, using the previously stated assumptions. Figure 5 gives the probability density function (PDF) of the base-case annual probability of global catastrophe from both visible and dark NEO impacts. (On the horizontal axis, "u" is "mu," or micro, i.e., 10 −6 .) Contemplating probabilities of probabilities can be confusing, but it is easy to see in PDF figures where there are broad spreads of probability (corresponding to great uncertainties) or narrow spreads (for less uncertainty). The figure shows that there are substantial uncertainties about dark-object damocloid risks and even greater uncertainties about Decision Analysis, 2017 Analysis, , vol. 14, no. 3, pp. 187-203, © 2017 NEO tracking and redirection measures Uniform(0.1, 0.9) 0 0 7.5 Food stockpiling for all of humanity Uniform(0.1, 0.9) Uniform(0.1, 0.9) Uniform(0.1, 0.9) 1,800 nuclear war risks, both of which could be much greater than visible-NEO impact risks. Table 3 gives the mean cost-effectiveness of GCRreduction measures (without research to reduce uncertainties) in terms of how much the measure reduces the average total probability of global catastrophe per dollar spent on the measure. (Recall that in these terms, a high number for cost-effectiveness is desirable, because it indicates a large reduction in global catastrophe probability for the dollars spent. The calculations incorporate the global catastrophe probability distributions shown in Figure 5 , which showed that nuclear war risks could be much greater than NEO impact risks.) 3 's mean cost-effectiveness comparison would seem to suggest spending on food stockpiling if nuclear war risk is included in the scope of analysis, but instead would suggest spending on NEO tracking if nuclear war risk is not included in the scope of analysis. Moreover, as mentioned previously, there are substantial uncertainties about the risks and costeffectiveness, and food stockpiles might actually be more cost-effective than NEO tracking even if nuclear war risk is not included in the scope of analysis. Figures 6 and 7 give the PDF of the cost-effectiveness for each GCR-reduction measure, if nuclear war risk is or is not included in analysis, respectively. (The status quo option has a cost-effectiveness of zero because it does not change GCR probability.) The figures indicate the overlapping ranges of the probability distributions of cost-effectiveness of food stockpiling and NEO redirection measures. According to the assumptions used in the Monte Carlo model, if nuclear war risk is included in the scope of analysis, there is a 0.8 probability that food stockpiling will be the most cost-effective measure, and there is a 0.2 probability NEO tracking and redirection will be most cost-effective. Conversely, if nuclear war risk is not included in the scope of analysis, there is a 0.999 probability that NEO tracking and redirection will be the most cost-effective measure, and there is a 0.001 probability that global food stockpiling will be most cost-effective. Further research could reduce uncertainties to better determine which risk-reduction measure would really be more cost-effective. According to the Monte Carlo model's assumptions and use of Equation (A.3), the cost-effectiveness-based expected value of perfect information (CEEVPI) in the illustrative examples in this paper is $2 billion if nuclear war is included in the scope of analysis, and $400 million if nuclear war is not included in the scope of analysis. In this illustrative example, research on the risks and risk-reduction effectiveness would have a substantial expected value, largely because of the huge uncertainties about the baseline risks and about the effectiveness of risk-reduction measures. The example also supports the argument that we can learn something valuable by doing the analysis for more than one type of GCR at a time.
 Conclusion In this paper, we argue that value of information based on cost-effectiveness is a useful tool for analysis of GCR to inform risk-reduction decisions and show how to apply it to GCRs and risk-reduction interventions in a comprehensive, integrated fashion. We discuss key challenges in real-world implementation of this paper's framework and argue that these challenges can be addressed. We then illustrate these concepts with simple example models of impact risks from both visible and "dark" near-Earth objects as well as nuclear war effects and consideration of related risk-reduction measures. The illustrative example shows that such calculations can have considerable value, and also supports looking at more than one GCR at a time. Unlike most value of information approaches, our approach for calculating value of information is based on risk-reduction cost-effectiveness, to avoid implicitly equating lives and dollars, e.g., using a VSL, which may be inappropriate given the scale of GCRs. Our equation for value of information may be useful in other domains where VSLs would not be appropriate. Our suggested approach could be used generally to work toward a comprehensive rigorous assessment of GCRs and risk-reduction options. A useful step could be to expand and update this paper's illustrative model (e.g., to reflect more recent NEO research 5 and other NEO risk-management options 6 ). However, it would be more valuable to work toward a broader agenda for integrated assessment to inform GCR-reduction decisions. Ideally, the scope of such assessment would address all important GCRs over key time periods (e.g., the next century) and also key risk-reduction options of relevant stakeholders (including, but not limited to, public policy options of governments). This paper's framework could help guide steps in such assessment by prioritizing pieces of research in terms of value of information for reducing the total probability of GCRs. While real-world GCR research would not result in perfect information, even imperfect information could have significant value in informing GCR-reduction resource allocation decisions. Our approach could have great value in comprehensively, rigorously assessing GCR and risk-reduction options. Prior GCR research is of only limited value to informing GCR-reduction decisions. Much of the work to date has focused on specific GCRs, leaving great Decision Analysis, 2017 , vol. 14, no. 3, pp. 187-203, © 2017 uncertainty about which GCRs are most important to focus on. Notable exceptions include research findings that GCRs from cosmic events are small relative to GCRs from human actions (Tegmark and Bostrom 2005) , an informal survey of GCR researchers providing estimates of the probabilities of human extinction from a small number of GCR types (Sandberg and Bostrom 2008) , analyses of interacting sequences of GCRs (Tonn and MacGregor 2009, Baum et al. 2013) , and several largely qualitative surveys (Bostrom 2002 , Rees 2003 , Posner 2004 , Smil 2008 , Cotton-Barratt et al. 2016 . These studies are insightful but do not provide rigorous quantitative recommendations for riskreduction resource allocations. We are aware of only one study, that of Leggett (2006) , that attempts to quantitatively evaluate GCR-reduction measures across a broad space of GCR, but that study has shortcomings such as not considering all GCR categories nor all potentially valuable GCR-reduction measures. The modest literature available does not come close to resolving the large uncertainties surrounding both the GCRs themselves and the effectiveness of possible riskreducing interventions. Our work suggests that comprehensive, integrated assessment of GCRs could be quite valuable for informing GCR-reduction decisions, and tools can be developed for making comprehensive, integrated assessments for informing GCR-reduction decisions.
 Appendix. Derivation of Cost-Effectiveness-Based Formula for Expected Value of Information In this appendix, we provide the detail on our derivation of the CEEVPI formula. Our following calculations are aided by two simplifying assumptions, as discussed previously: a binary threshold for global catastrophes and independence of different GCRs (except to the extent that GCR event interactions and dependencies are accounted for in the fault trees or other model components). Then let X j equal the value loss due to global catastrophe j. Because of the binary threshold assumption, X j is the same for all j. Let R(t) be the risk of an event occurring during time period t, with R probability × magnitude. Then, given the independence of different GCRs, the total risk for y GCRs is R tot (t) p tot (t)X 1 − y j (1 − p j (t)) X. (A.1) We further assume that all GCRs have sufficiently low probabilities per time period p j (t) that the total probability of global catastrophe in that time period can be approximated as the sum of the independent probabilities, such that R tot (t) ≈ y j p j (t) X. (A.2) In this paper, we evaluate possible GCR-reducing interventions in terms of their cost-effectiveness, i.e., their reduction in GCR per unit cost. We favor cost-effectiveness for two reasons. First, cost-benefit analysis is hampered by the challenge of quantifying the value loss due to global catastrophe, X. The benefit of interventions is the reduction in risk, which also depends on X. While X is generally believed to be very large, quantitative estimates span a huge range, as mentioned previously. In contrast, cost-effectiveness analysis does not depend on X. Let c i and CE i be the cost and cost-effectiveness of intervention i. Then, CE i R 0, tot (t) − R i, tot (t) c i p 0, tot (t) − p i, tot (t) c i X. (A.3) Since X is equal for all global catastrophes, comparisons of the cost-effectiveness of different interventions are the same regardless of the value of X. We assume that there is a decision to be made about allocation of resources to some combination of direct risk reduction and research, and that the main decision rule is to choose whatever combination of options has best overall expected GCR-reduction cost-effectiveness among options considered in the analysis. Then the decision maker should buy as much risk reduction (and risk research enabling better risk-reduction decisions) as they can at whatever total cost, as long as that results in the greatest cost-effectiveness. (We assume that budgets are not an issue in the context of the risk-reduction and research options under consideration, and we do not explicitly account for potential budget constraints in the following. This implicitly assumes that sufficient total resources are either being provided by a single entity or are coordinated in some fashion.) If the information's expected effect and cost are such that even with the research cost included it would achieve a better cost-effectiveness than whatever would have been the optimal investment before the research based on expected values, We denote cases whether research is conducted to reduce uncertainty on a particular factor using superscript b for "before" research, or without information from research, and superscript a for "after" research, or with information from research. (Thus, before research is conducted on p i , it is p b i , and after research is conducted, it is p a i .) Again, we ignore actual costs of research and focus on the amount of resources the decision maker ought to be willing to pay for the total value added by the research. We use the term w to denote the amount of resources the decision maker ought to be willing to pay for the total added value of conducting research. (It adds no value to do no research.) For the purposes of this derivation, we do not provide more detailed breakdowns of the amount of resources the decision maker ought to be willing to pay for the value added by performing specific pieces of research that comprise total value added by research v, which actually could consist of separate pieces of research on different uncertain factors. (The amount of resources the decision maker ought to be willing to pay for the value added by each piece of research could be assessed using an extension of the derivation provided here.) Note that the best option after research, s a (which has costeffectiveness in Monte Carlo iteration m of (p a 0m − p a sm )/c a sm ) is not necessarily the same as the best option before research, s b (which has cost-effectiveness in Monte Carlo iteration m of (p b 0m − p b sm )/c b sm ). Research that reduces but does not eliminate uncertainty about a factor yields imperfect information. In a case where research produces perfect information about a factor, all uncertainty is eliminated about the factor after research. In terms of Monte Carlo iterations, after perfect information, one of the Monte Carlo iterations will have randomly sampled factor values whose are closest to the actual real-world factor values. As long as doing more research adds more value, and if we ignore the actual costs of performing the research, we assume that resources for research ought to be invested in up to the point where research would be so expensive that a funder would obtain no further benefit from investing in additional research (because up to that point, they would get a better overall cost-effectiveness by investing in additional research). At that point, the expected cost-effectiveness of the best riskreduction option before research is equal to the expected costeffectiveness of the best risk-reduction option after research, including the amount of resources the decision maker ought to be willing to pay for the total value added by research: The E[ • ] terms can be distributed and regathered because all the relevant calculations (i.e., both the expected-value calculations and the cost-effectiveness calculations) involve linear operations and because p and c variables are assumed Decision Analysis, 2017 , vol. 14, no. 3, pp. 187-203, © 2017 to be uncorrelated. (To be more specific, manipulation of the numerator and denominator are allowed because they are linear operations, and multiplicative operations are allowed because the covariance is assumed to be 0.) Distributing and rearranging the terms to solve for the expected value of the amount of resources the decision maker ought to be willing to pay for the total value added by research, (A.11) This formula for CEEVI actually applies to both perfect information and imperfect information cases. However, our focus in this derivation is on the limiting case where the research yields perfect information, which provides the upper limit to the value of research, i.e., the costeffectiveness-based expected value of perfect information, CEEVPI. It turns out that when used in Analytica software by Lumina Decision Systems, the above CEEVI formula can be used in a straightforward fashion to set up the Monte Carlo simulation computations for CEEVPI (by directly using each factor's Monte Carlo sampling values in each Monte Carlo iteration), and that is what we use in the illustrative Analytica model accompanying this paper. (Computation of the cost-effectiveness-based expected value of imperfect information, CEEVII, would require an extra step to simulate after-research imperfect-information probability distributions for each factor, instead of after-research perfectinformation point values.) Endnotes 1 For an example of a canonical utility function based decision analysis framework for one GCR category, asteroid and comet impact risk, see Lee et al. (2014) . 2 For an illustrative example of a probability distribution reflecting uncertainty about an annualized global catastrophe probability, see Figure 5 in Section 4.2. For more on such probability distributions, see Chapters 4 and 5 of Morgan and Henrion (1990) . 3 For some GCR types, it may not be most useful to think in terms of consequence exceedance thresholds, but in terms of probabilities of various possibilities, such as in future "artificial superintelligent catastrophe" scenarios. However, modeling approaches such as fault trees could be useful for some such scenarios (Barrett and Baum 2017 ). 4 There are also related challenges in the selection of metrics, such as for event consequences: whether to focus on estimated fatalities over some specific time scale, or to also consider economic impact, etc. Even choosing cost metrics for use in cost-effectiveness analysis presents challenges. In this paper, we assume cost is defined in monetary (dollar) terms, but those have limitations (Baum 2012) , and scarcities exist for other resources such as labor capacity. 5 See, for example, Reinhardt et al. (2015) . 6 For example, there are a number of options for alternative food sources during a crop-failure crisis (Denkenberger and Pearce 2015) . Those potentially could be more cost-effective than food stockpiling, but we believe their effectiveness also would have greater uncertainty because of complexity, etc. Figure 1 . 1 Figure 1. High-Level Global Catastrophe Fault Tree and Risk Management Decision Influence DiagramGlobal catastrophe (GC)
 Figure 3 . 3 Figure 3. Uncertainties in Global Catastrophe Probability Modeling Event consequence exceedance probability
 Figure Figure 5. (Color online) PDF of Baseline Annual Probabilities of Global Catastrophe
 Figure 6 . 6 Figure 6. (Color online) PDF of Cost-Effectiveness of GCR-Reduction Measures if Nuclear War Risk Is Included in Scope (Status quo)
 previous expression for E[w] gives the value of research as the cost-effectiveness-based expected value of information,
 INFORMS Figure 2. Global Catastrophe Probability as Function of Event Consequence and Exceedance Probability Event Reduced consequence exceedance probability event consequence threshold (CT) Global catastrophe (GC) consequence Initial P(GC) Reduced P(GC) Reduced event probability Event consequence
 INFORMS Figure 4. High-Level Decision Tree for Research and Risk-Reduction Decisions Do not invest to Risk is high Risk reduce risks Risk is low reduction decision Do not conduct without research Invest to reduce risks Risk was high, and is being reduced Risk was low, and is being reduced research Research decision Do not invest to Conduct research Risk is high Risk reduction risk is high decision after research when reduce risks Invest to Risk was high, and is being reduced Risk is high reduce risks Do not invest to Risk reduction reduce risks Risk is low decision after risk is low research when Invest to Risk was low, and is being reduced Risk is low reduce risks
 Table 1 . 1 Expressions for Assumed Baseline Annual Probabilities of Global Catastrophe GCR types Baseline P(Global Catastrophe) Visible near-Earth objects Triangular(0.5, 1, 2) • 0.15 • (1/(3 • 10 5 )) Long-period comets Triangular(0.5, 1, 2) (damocloids) • Uniform(0, 4) • (1/(3 • 10 5 )) Nuclear war Triangular(0, 0.0001, 0.001) a factor of two." We assume that these uncertainties in the visible/bright NEO frequencies are uncorrelated with the uncertainties in the damocloid frequencies. Some corroboration of the relative risks of bright ver- sus dark objects, and associated uncertainties, is pro- vided by the (National Research Council 2010, p. 22):
 Table 2 . 2 Assumed Effects and Costs of Risk-Reduction MeasuresReduction in P(Global Catastrophe) from each GCR type Visible near- Long-period comets GCR-reduction measures Earth objects (damocloids) Nuclear war Costs ($Billion)
 Table 3 . 3 Mean Cost-Effectiveness of GCR-Reduction Measures (Reduction in Total Global Catastrophe Probability per Dollar) Food Both food Include NEO tracking stockpiling stockpiling nuclear and redirection for all of and NEO war in scope? measures humanity tracking/redirection Yes 4 × 10 −17 1 × 10 −16 1 × 10 −16 No 4 × 10 −17 2 × 10 −18 2 × 10 −18
 Table


Robust Multi-Agent Reinforcement Learning via Minimax Deep Deterministic Policy Gradient

Introduction Most real-world problems involve interactions between multiple agents and the complexity of problem increases significantly when the agents co-evolve together. Thanks to the recent advances of deep reinforcement learing (DRL) on single agent scenarios, which led to successes in playing Atari game (Mnih et al. 2015) , playing go (Silver et al. 2016 ) and robotics control (Levine et al. 2016) , it has been a rising trend to adapt single agent DRL algorithms to multi-agent learning scenarios and many works have shown great successes on a variety of problems, including automatic discovery of communication and language (Sukhbaatar, Fergus, and others 2016; Mordatch and Abbeel 2017) , multiplayer games (Peng et al. 2017a; OpenAI 2018) , traffic control (Wu et al. 2017 ) and the analysis of social dilemmas (Leibo et al. 2017) . The critical challenge when adapting classical single agent DRL algorithms to multi-agent setting is the training instabil-ity issue: as training progresses, each agent's policy is changing and therefore the environment becomes non-stationary from the perspective of any individual agent (in a way that is not explainable by changes in the agent's own policy). This non-stationary problem can cause significant problems when directly applying the single agent DRL algorithms, for example, the variance of the policy gradient can be exponentially large when the number of agents increases (Lowe et al. 2017 ). To handle this instability issue, recent works, such as the counterfactual multi-agent policy gradients (Foerster et al. 2017 ) and the Multi-Agent Deep Deterministic Policy Gradient (MADDPG) (Lowe et al. 2017) , proposed to utilized a centralized critic within the actor-critic learning framework to reduce the variance of policy gradient. Despite the fact that using a centralized critic stabilizes training, the learned policies can still be brittle and sensitive to its training partners and converge to a poor local mode. This is particularly severe for competitive environments: when the opponents alter their policies during testing, the performance of the learned policies can be drastically worse (Lazaridou, Peysakhovich, and Baroni 2016) . Hence, a robust policy becomes desirable in multi-agent setting: a well-trained agent should be able behave well in testing when competing against opponents even with strategies different from its training partners. In this work, we focus on robust multi-agent reinforcement learning with continuous action spaces and propose a novel algorithm, MiniMax Multi-agent Deep Deterministic Policy Gradient (M3DDPG). M3DDPG is a minimax extension 1 of the MADDPG algorithm (Lowe et al. 2017) . Its core idea is that during training, we force each agent to behave well even when its training opponents response in the worst way. Our major contributions are summarized as follow: • We introduce the minimax approach to robust multi-agent DRL and propose a novel minimax learning objective based on the MADDPG algorithm; • In order to efficiently optimize the minimax learning objective, we propose an end-to-end learning approach, Multiagent Adversarial Learning (MAAL), which is inspired by the adversarial training (Goodfellow, Shlens, and Szegedy 2014) technique. • We empirically evaluate our proposed M3DDPG algorithm on four mixed cooperative and competitive environments and the agents trained by M3DDPG outperform baseline policies on all these environments. In the rest of the paper, we will firstly present related works in section 2. Notations and standard algorithms are described in section 3. Our main algorithm, M3DDPG, is introduced in section 4. Experimental results are in section 5.
 Related Work Multi-agent reinforcement learning (Littman 1994 ) has been a long-standing field in AI (Hu, Wellman, and others 1998; Busoniu, Babuska, and De Schutter 2008) . Recent works in DRL use deep neural networks to approximately represent policy and value functions. Inspired by the success of DRL in single-agent settings, many DRL-based multi-agent learning algorithms have been proposed. Forester et al. (2016b) and He et al. (2016) extended the deep Q-learning to multiagent setting; Peng et al. (2017a) proposed a centralized policy learning algorithm based on actor-critic policy gradient; Forester et al. (2016a) developed a decentralized multi-agent policy gradient algorithm with centralized baseline; Lowe et al. (2017) extended DDPG to multi-agent setting with a centralized Q function; Wei et al. (2018) and Grau-Moya (2018) proposed multi-agent variants of the soft-Q-learning algorithm (Haarnoja et al. 2017) ; Yang et al. (2018) focused on multi-agent reinforcement learning on a very large population of agents. Our M3DDPG algorithm is built on top of MAD-DPG and inherits the decentralized policy and centralized critic framework. Minimax is a fundamental concept in game theory and can be applied to general decision-making under uncertainty, prescribing a strategy that minimizes the possible loss for a worst case scenario (Osborne and others 2004) . Minimax was firstly introduced to multi-agent reinforcement learning as minimax Q-learning by Littman (1994) . More recently, some works combine the minimax framework and the DRL techniques to find Nash equilibrium in two player zero-sum games (Foerster et al. 2018; Pérolat et al. 2016; Grau-Moya, Leibfried, and Bou-Ammar 2018) . In our work, we utilize the minimax idea for the purpose of robust policy learning. Robust reinforcement learning was originally introduced by Morimoto et al. (2005) considering the generalization ability of the learned policy in the single-agent setting. This problem is also studied recently with deep neural networks, such as adding random noise to input (Tobin et al. 2017) or dynamics (Peng et al. 2017b) during training. Besides adding random noise, some other works implicitly adopt the minimax idea by utilizing the "worst noise" (Pinto et al. 2017; Mandlekar et al. 2017) . These works force the learned policy to work well even under the worst case perturbations and are typically under the name of "adversarial reinforcement learning", despite the fact that the original adversarial reinforcement learning problem was introduced in the setting of multi-agent learning (Uther and Veloso 1997) . In our M3DDPG algorithm, we focus on the problem of learning polices that is robust to opponents with different strategies. Within the minimax framework, finding the worst case scenario is a critical component. Lanctot et al. (2017) proposed an iterative approach that alternatively computes the best response policy while fixes the other. Gao et al. (Gao, Mueller, and Hayward 2018) replace "mean" in the temporal difference learning rule with "minimum". In our work, we proposed MAAL, which is a general, efficient and fully end-to-end learning approach. MAAL is motivated by adversarial training (Goodfellow, Shlens, and Szegedy 2014) and suitable for arbitrary number of agents. The core idea of MAAL is approximating the minimization in our minmax objective by a single gradient descent step. The idea of one-step-gradient approximation was also explored in metalearning (Finn, Abbeel, and Levine 2017) .
 Background and Preliminary In this section, we describe our problem setting and the standard algorithms. Most of the definitions and notations follow the original MADDPG paper (Lowe et al. 2017) .
 Markov Games We consider a multi-agent extension of Markov decision processes (MDPs) called partially observable Markov games (Littman 1994) . A Markov game for N agents is defined by a set of states S describing the possible configurations of all agents, a set of actions A 1 , ..., A N and a set of observations O 1 , ..., O N for each agent. To choose actions, each agent i uses a stochastic policy π π π θi : O i × A i → [0, 1] parameterized by θ i , which produces the next state according to the state transition function T : S × A 1 × ... × A N → S. Each agent i obtains rewards as a function of the state and agent's action r i : S × A i → R, and receives a private observation correlated with the state o i : S → O i . The initial states are determined by a distribution ρ : S → [0, 1]. Each agent i aims to maximize its own total expected return R i = T t=0 γ t r t i where γ is a discount factor and T is the time horizon. To minimize notation, in the following discussion we will often omit θ from the subscript of π π π.
 Q-Learning and Deep Q-Networks (DQN) Q-Learning and DQN (Mnih et al. 2015) are popular methods in reinforcement learning and have been previously applied to multi-agent settings (Foerster et al. 2016a; Tesauro 2004 ). Q-Learning makes use of an action-value function for policy π π π as Q π π π (s, a) = E[R|s t = s, a t = a]. This Q function can be recursively rewritten as Q π π π (s, a) = E s [r(s, a) + γE a ∼π π π [Q π π π (s , a )]] . DQN learns the action-value function Q \* corresponding to the optimal policy by minimizing: L(θ) = E s,a,r,s [(Q \* (s, a|θ) − y) 2 ], (1) where y = r + γ max a Q \* (s , a ). Q is a target Q function whose parameters are periodically updated with the most recent θ, which helps stabilize learning. Another crucial component of stabilizing DQN is the use of an experience replay buffer D containing tuples (s, a, r, s ). Q-learning algorithm is most suitable for DRL agents with discrete action spaces.
 Policy Gradient (PG) Algorithms Policy gradient methods is another popular choice for a variety of RL tasks. Let ρ π denote discounted state visitation distribution for a policy π. The main idea of PG is to directly adjust the parameters θ of the policy in order to maximize the objective J(θ) = E s∼ρ π π π ,a∼π π π θ [R] by taking steps in the direction of ∇ θ J(θ). Using the Q function defined previously, the gradient of the policy can be written as (Sutton et al. 2000) : ∇ θ J(θ) = E s∼ρ π π π ,a∼π π π θ [∇ θ log π π π θ (a|s)Q π π π (s, a)], (2) where p π π π is the state distribution. The policy gradient theorem has given rise to several practical algorithms, which often differ in how they estimate Q π π π . For example, one can simply use a sample return R t = T i=t γ i−t r i , which leads to the REINFORCE algorithm (Williams 1992 ). Alternatively, one could learn an approximation of the true action-value function Q π π π (s, a) called the critic and leads to a variety of actor-critic algorithms (Sutton and Barto 1998) .
 Deterministic Policy Gradient (DPG) Algorithms DPG algorithms extends the policy gradient algorithm to deterministic policies µ µ µ θ : S → A (Silver et al. 2014) . In particular, under certain conditions we can write the gradient of the objective J(θ) = E s∼ρ µ µ µ [R(s, a)] as: ∇ θ J(θ) = E s∼D [∇ θ µ µ µ θ (s)∇ a Q µ µ µ (s, a)| a=µ µ µ θ (s) ], (3) where D is the replay buffer. Since this theorem relies on ∇ a Q µ µ µ (s, a), it requires the action space A (and thus the policy µ µ µ) be continuous. Deep deterministic policy gradient (DDPG) (Lillicrap et al. 2015 ) is a variant of DPG where the policy µ µ µ and critic Q µ µ µ are approximated with deep neural networks. DDPG is an off-policy algorithm, and samples trajectories from a replay buffer of experiences that are stored throughout training. DDPG also makes use of a target network, as in DQN (Mnih et al. 2015) .
 Multi-Agent Deep Deterministic Policy Gradient Directly applying single-agent RL algorithms to the multiagent setting by treating other agents as part of the environment is problematic as the environment appears nonstationary from the view of any one agent, violating Markov assumptions required for convergence. This non-stationary issue is more severe in the case of DRL with neural networks as function approximators. The core idea of the MADDPG algorithm (Lowe et al. 2017 ) is learning a centralized Q function for each agent which conditions on global information to alleviate the non-stationary problem and stabilize training. More concretely, consider a game with N agents with policies parameterized by θ θ θ = {θ 1 , ..., θ N }, and let µ µ µ = {µ µ µ 1 , ..., µ µ µ N } be the set of all agents' policies. Then we can write the gradient of the expected return for agent i with policy µ µ µ i , J(θ i ) = E[R i ] as: ∇ θi J(θ i ) = E x,a∼D [∇ θi µ µ µ i (o i )∇ ai Q µ µ µ i (x, a 1 , ..., a N )| ai=µ µ µi(oi) ] , (4) Here Q π π π i (x, a 1 , ..., a N ) is a centralized action-value function that takes as input the actions of all agents, a 1 , . . . , a N , in addition to some state information x (i.e., x = (o 1 , ..., o N )) , and outputs the Q-value for agent i. Let x denote the next state from x after taking actions a 1 , . . . , a N . The experience replay buffer D contains the tuples (x, x , a 1 , . . . , a N , r 1 , . . . , r N ), recording experiences of all agents. The centralized action-value function Q µ µ µ i is updated as: oj ) , where µ µ µ = {µ µ µ θ 1 , ..., µ µ µ θ N } is the set of target policies with delayed parameters θ i . L(θ i ) = E x,a,r,x [(Q µ µ µ i (x, a 1 , . . . , a N ) − y) 2 ], (5) y = r i + γ Q µ µ µ i (x , a 1 , . . . , a N ) a j =µ µ µ j ( Note that the centralized Q function is only used during training. During decentralized execution, each policy µ µ µ θi only takes local information o i to produce an action.
 Minimax Multi-Agent Deep Deterministic Policy Gradient (M3DDPG) In this section, we introduce our proposed new algorithm, Minimax Multi-agent Deep Deterministic Policy Gradient (M3DDPG), which is built on top of the MADDPG algorithm and particularly designed to improve the robustness of learned policies. Our M3DDPG algorithm contains two major novel components: Minimax Optimization Motivated by the minimax concept in game theory, we introduce minimax optimization into the learning objective; Multi-Agent Adversarial Learning The continuous action space results in computational intractability issue when optimizing our proposed minimax objective. Hence, we propose Multi-Agent Adversarial Learning (MAAL) to solve this optimization problem.
 Minimax Optimization In multi-agent RL, the agents' policies can be very sensitive to their learning partner's policy. Particularly in competitive environments, the learned policies can be brittle when the opponents alter their strategies. For the purpose of learning robust policies, we propose to update policies considering the worst situation: during training, we optimize the accumulative reward for each agent i under the assumption that all other agents acts adversarially. This yields the minimax learning objective max θi J M (θ i ) where J M (θ i ) = E s∼ρ µ µ µ [R i ] = min a t j =i E s∼ρ µ µ µ T t=0 γ t r i (s t , a t 1 , . . . , a t N ) a t i =µ µ µ(o t i ) (6) = E s 0 ∼ρ min a 0 j =i Q µ µ µ M,i (s 0 , a 0 1 , . . . , a 0 N ) a 0 i =µ µ µ(o 0 i ) . (7) Critically, in Eq. 6, state s t+1 at time t + 1 depends not only on the dynamics ρ µ µ µ and the action µ µ µ i (o t i ) but also on all the previous adversarial actions a t j =i with t ≤ t. In Eq. 7, we derive the modified Q function Q µ µ µ M (s, a 1 , . . . , a N ), which is naturally centralized and can be rewritten in a recursive form Q µ µ µ M,i (s, a 1 , . . . , a N ) = r i (s, a 1 , . . . , a N )+ γE s min a j =i Q µ µ µ M,i (s , a 1 , . . . , a N ) a i =µ µ µi(s ) . Importantly, Q µ µ µ M (s, a 1 , . . . , a N ) conditions on the current state s as well as the current actions a 1 , . . . , a N and represents the current reward plus the discounted worst case future return starting from the next state, s . This definition brings the benefits that we can naturally apply off-policy temporal difference learning later to derive the update rule for Q µ µ µ M . Note that for each agent i, none of the adversarial actions depend on its parameter θ i , so we can directly apply the deterministic policy gradient theorem to compute ∇ θi J M (θ i ) and use off-policy temporal difference to update the Q function. Thanks to the centralized Q function in MADDPG (Eq. 4), which takes in the actions from all the agents, our derivation naturally applies and is perfectly aligned with the MADDPG formulation (Eq. 4) by injecting a minimization over other agents' actions as follows: ∇ θ i J M (θ i ) = E x∼D   ∇ θ i µ µ µ i (o i )∇a i Q µ µ µ M,i (x, a 1 , . . . , a i , . . . a N ) a i = µ µ µ i (o i ) a j =i = arg mina j =i Q µ µ µ M,i (x, a 1 , . . . , a N )   , (9) where D denotes the replay buffer and x denotes the state information. Correspondingly, we obtain the new Q function update rule by adding another minimization to Eq. 5 when computing the target Q value: L(θ i ) = E x,a,r,x ∼D [(Q µ µ µ M,i (x, a 1 , . . . , a N ) − y) 2 ], (10) y = r i + γ Q µ µ µ M,i (x , a 1 , . . . , a i , . . . , a N ) a i = µ µ µ i (o i ), a j =i = arg min a j =i Q µ µ µ M,i (x , a 1 , . . . , a N ), where µ µ µ i denotes the target policy of agent i with delayed parameters θ i , and Q µ µ µ M,i denotes the target Q network for agent i. Combining Eq. 9 and Eq. 10 yields our proposed minimax learning framework.
 Multi-Agent Adversarial Learning The critical challenge in our proposed minimax learning framework is how to handle the embedded minimization in Eq. 9 and Eq. 10. Due to the continuous action space as well as the non-linearity of Q function, directly optimizing the minimization problem is computationally intractable. A naive approximate solution can be performing an inner-loop gradient descent whenever performing an update step of Eq. 9 or Eq. 10, but this is too computationally expensive for practical use. Here we introduce an efficient and end-to-end solution, multi-agent adversarial learning (MAAL). The main ideas of MAAL can be summarized in two steps: (1) approximate the non-linear Q function by a locally linear function; (2) replace the inner-loop minimization with a 1-step gradient descent. Note the core idea of MAAL, locally linearizing the Q function, is adapted from the recent adversarial training technique originally developed for supervised learning. We will discuss the connection between adversarial training and MAAL in the end of this section. For conciseness, we first consider Eq. 10 and rewrite it into the following form with auxiliary variables : y = r i + γ Q µ µ µ M,i (x , a 1 , . . . , a i , . . . , a N ) (11) a k = µ µ µ k (o k ), ∀1 ≤ k ≤ N a j = a j + j , ∀j = i j =i = arg min j =i Q µ µ µ M,i (x , a 1 + 1 , . . . , a i , . . . , a N + N ). Eq. 11 can be interpreted as we are now seeking for a set of perturbations such that the perturbed actions a decrease Q value the most. By linearizing the Q function at Q µ µ µ M,i (x, a 1 , . . . , a N ), the desired perturbation j can be locally approximated by the gradient direction at Q µ µ µ M,i (x, a 1 , . . . , a N ) w.r.t. a j . Then we use this local to derive an approximation ˆ j to the worst case perturbation by taking a small gradient step: ∀j = i, ˆ j = −α∇ aj Q µ µ µ M,i (x , a 1 , . . . , a j , . . . , a N ), (12) where α is a tunable coefficient representing the perturbation rate. It can be also interpreted as the step size of the gradient descent step: when α is too small, the local approximation error will be small but due to the small perturbation, the learned policy can be far from the optimal solution of the minimax objective we proposed; when α is too large, the approximation error may incur too much trouble for the overall learning process and the agents may fail to learn good policies. We can apply this technique to Eq. 9 as well and eventually derive the following formulation: ∇ θ i J(θ i ) = E x,a∼D     ∇ θ i µ µ µ i (o i )∇a i Q µ µ µ M,i (x, a 1 , . . . , a i , . . . a N ) a i = µ µ µ i (o i ) a j = a j + ˆ j , ∀j = i ˆ j = −α j ∇a j Q µ µ µ M,i (x, a 1 , . . . , a N )     , ( 13 ) and L(θ i ) = E x,a,r,x [(Q µ µ µ M,i (x, a 1 , . . . , a N ) − y) 2 ], (14) y = r i + γ Q µ µ µ M,i (x , a 1 , . . . , a i , . . . , a N ) a k = µ µ µ k (o k ), ∀1 ≤ k ≤ N a j = a j + ˆ j , ∀j = i ˆ j = −α j ∇ a j Q µ µ µ M,i (x, a 1 , . . . , a N ), where α 1 , . . . , α N are additional parameters. MAAL only requires one additional gradient computation, and can be executed in a fully end-to-end fashion. Finally, combining Eq. 13 and Eq. 14 completes MAAL. The overall algorithm, M3DDPG, is summarized as Algo. 1. Algorithm 1: Minimax Multi-Agent Deep Deterministic Policy Gradient (M3DDPG) for N agents for episode = 1 to M do Initialize a random process N for action exploration, and receive initial state information x for t = 1 to max-episode-length do for each agent i, select action a i = µ µ µ θi (o i ) + N t w.r.t. the current policy and exploration Execute actions a = (a 1 , . . . , a N ) and observe reward r and new state information x Store (x, a, r, x ) in replay buffer D, and set x ← x for agent i = 1 to N do Sample a random minibatch of S samples (x k , a k , r k , x k ) from D Set y k = r k i + γ Q µ µ µ M,i (x k , a 1 , . . . , a N )| a i =µ µ µ i (o k i ),a j =i =µ µ µ j (o k j )+ˆ j with ˆ j defined in Eq. 14 Update critic by minimizing the loss L(θ i ) = 1 S k y k − Q µ µ µ M,i (x k , a k 1 , . . . , a k N ) 2 Update actor using the sampled policy gradient with ˆ j defined in Eq. 13: ∇ θi J ≈ 1 S k ∇ θi µ µ µ i (o k i )∇ ai Q µ µ µ M,i (x k , a 1 , . . . , a i , . . . , a N ) ai=µ µ µi(o k i ),a j =i =a k j +ˆ j end for Update target network parameters for each agent i: θ i ← τ θ i + (1 − τ )θ i end for end for
 Discussion Connection to Adversarial Training Adversarial training is a robust training approach for deep neural networks on supervised learning . The core idea is to force the classifier to predict correctly even when given adversarial examples, which are obtained by adding a small adversarial perturbation to the original input data such that the classification loss can be decreased the most. Formally, suppose the classification loss function is L(θ) = E x,y [f θ (x; y)] with input data x and label y. Adversarial training aims to optimize the following adversarial loss instead L adv (θ) = E x,y [f θ (x + ; y)] (15) = arg max ≤α f θ (x + ; y). The core technique to efficiently optimize L adv (θ) is to locally linearize the loss function at f θ (x; y) and approximate by the scaled gradient. Thanks to the centralized Q function, which takes the actions from all the agents as part of the input, we are able to easily inject the minimax optimization (Eq. 11) and represent it in a similar way to adversarial training (Eq. 15) so that we can adopt the similar technique to effectively solve our minimax optimization in a fully end-to-end fashion. Connection to Single Agent Robust RL M3DDPG with MAAL can be also viewed as the special case of robust reinforcement learning (RRL) (Morimoto and Doya 2005) in the single agent setting, which aims to bridge the gap between training in simulation and testing in the real world by adding adversarial perturbations to the transition dynamics during training. Here, we consider the multi-agent setting and add worst case perturbations to actions of opponent agents during training. Note that in the perspective of a single agent, perturbations on opponents' actions can be also considered as a special adversarial noise on the dynamics. Choice of α In the extreme case of α = 0, M3DDPG degenerates to the original MADDPG algorithm while as α increases, the policy learning tends to be more robust but the optimization becomes harder. In practice, using a fixed α throughout training can lead to very unstable learning behavior due to the changing scale of the gradients. The original adversarial training paper (Goodfellow, Shlens, and Szegedy 2014) suggests to compute with a fixed norm, namely g = ∇ x f θ (x; y), ˆ = α g g , where x denotes the input data to the classifier and y denotes the label. In M3DDPG, we can adaptively compute the perturbation ˆ j by g = ∇ aj Q µ µ µ M,i (x, a 1 , . . . , a N ), ˆ j = −α j g g . (16) Eq. 16 generally works fine in practice but in some hard multi-agent learning environments, unstable training behavior can be still observed. We suspect that it is because the changing norm of actions in these situations. Different from the supervised learning setting where the norm of the input data x is typically stable, in reinforcement learning the norm of actions can drastically change even in a single episode. Therefore, it is possible to see cases that even a perturbation with a small fixed norm overwhelms the action a j , which may potentially lead to computational stability issue. Therefore, we also introduce the following alternative for adaptive perturbation computation: g = ∇ aj Q µ µ µ M,i (x, a 1 , . . . , a N ), ˆ j = −α j a j g g . (17) Lastly, note that in a mixed cooperative and competitive environment, ideally we only need to add adversarial perturbations to competitors. But empirically we observe that also adding (smaller) perturbations to collaborators can further improve the quality of learned policies.
 Experiments We adopt the same particle-world environments as the MAD-DPG paper (Lowe et al. 2017) as well as the training configurations. α is selected from a grid search over 0.1, 0.01 and 0.001. For testing, we generate a fixed set of 2500 environment configurations (i.e., landmarks and birthplaces) and evaluate on this fixed set for a fair comparison.
 Environments The particle world environment consists of N cooperative agents, M adversarial agents and L landmarks in a twodimensional world with continuous space. We focus on the four mixed cooperative and competitive scenarios to best examine the effectiveness of our minimax formulation. Covert communication This is an adversarial communication environment, where a speaker agent ('Alice') must communicate a message to a listener agent ('Bob') (N = 2), who must reconstruct the message at the other end. However, an adversarial agent ('Eve') (M = 1) is also observing the channel, and wants to reconstruct the message -Alice and Bob are penalized based on Eve's reconstruction, and thus Alice must encode her message using a randomly generated key, known only to Alice and Bob. Keep-away This scenario consists of L = 1 target landmark, N = 2 cooperative agents and M = 1 adversarial agent. Cooperating agents need to reach the landmark and keep the adversarial agent away from the landmark by pushing it while the adversarial agent must stay at the landmark to occupy it. Physical deception Here, N = 2 agents cooperate to reach a single target landmark from a total of L = 2 landmarks. They are rewarded based on the minimum distance of any agent to the target (so only one agent needs to reach the target landmark). However, a lone adversary (M = 1) also desires to reach the target landmark; the catch is that the adversary does not know which of the landmarks is the correct one. Thus the cooperating agents, who are penalized based on the adversary distance to the target, learn to spread out and cover all landmarks so as to deceive the adversary. Predator-prey In this variant of the classic predator-prey game, N = 3 slower cooperating agents must chase the faster adversary (M = 1) around a randomly generated environment with L = 2 large landmarks impeding the way. Each time the cooperative agents collide with an adversary, the agents are rewarded while the adversary is penalized.
 Comparison to MADDPG To evaluate the quality of learned policies trained by different algorithms in competitive scenarios, we measure the performance of agents trained by our M3DDPG and agents by classical MADDPG in the roles of both normal agent and adversary in each environment. The results are in Figure 2 , where we measure the rewards of the normal agents in different scenarios and normalize them to 0-1. We notice that in all the environments, the highest score is achieved when the M3DDPG agents play as the normal agents against the MADDPG adversary (Minimax vs MA); while the lowest score is when the MADDPG agents act as normal agents against the M3DDPG adversary (MA vs Minimax). This indicates that policies trained by M3DDPG have much higher quality than original MADDPG.
 Evaluation with Disruptive Adversaries Instead of M3DDPG and MADDPG directly competing against each other, now we consider their performances in the worst possible situations through their individual competitions against disruptive adversaries. We construct disruptive adversaries by (1) fixing the trained normal agents (M3DDPG or MADDPG); (2) setting the reward of the disruptive adversary as the negative value of normal agents' reward (so that the problem becomes zero-sum); and then (3) using DDPG to train the disruptive adversary from scratch under the zerosum reward setting. In the particle world environment, the competitive scenarios are generally not zero-sum, which implies that according to the default reward function, the adversaries may have different purposes rather than directly disrupting the normal agents. So, in order to evaluate the effectiveness our minimax optimization in the worst situation, we convert every problems into a zero-sum form and compare the performances of our M3DDPG agents as well as the MADDPG agents against this artificially constructed adversaries. Moreover, since each of our four environments has only 1 adversary, after fixing the normal agents, the learning problem degenerates to the single agent setting and classical DDPG is sufficient to stably train a disruptive adversary. The results are shown in Figure 3 , where we plot the reward of the fixed normal agents of different algorithms as the training of the disruptive adversaries progresses until convergence. Note that due to the different environment designs, the difficulty for the disruptive agents to break the strategy of normal agents varies: for example, in convert communication, since the private key is not accessible to the adversary agent, breaking the encrypted communication will be very hard; while in physical deception, since we do not allow communication and fix the normal agents, a smart enough adversary may easily infer the target landmark by observing the initial behavior of the two cooperative agents. Nevertheless, despite these intrinsic properties, the M3DDPG agents (Minimax) achieves higher reward in all the scenarios, which implies better robustness even in the worst situation.
 Conclusion In this paper, we propose a novel algorithm, minimax mulitagent deep deterministic policy gradient (M3DDPG), for robust multi-agent reinforcement learning, which leverages the minimax concept and introduces a minimax learning objective. To efficiently optimize the minimax objective, we propose MAAL, which approximates the inner-loop minimization by a single gradient descent step. Empirically, M3DDPG outperforms the benchmark methods on four mixed cooperative and competitive scenarios. Nevertheless, due to the single step gradient approximation in MAAL, which is efficient in computation, an M3DDPG agent can only explore locally worst situation during the evolving process at training, which can still lead to unsatisfying behavior when testing opponents have drastically different strategies. It will be an interesting direction to re-examine the robustness-efficiency trade-off in MAAL and further improve policy learning by placing more computations on the minimax optimization. We leave this as our future work. Figure 1 : 1 Figure 1: Illustrations of some environments we consider, including Physical Deception (left) and Predator-Prey (right).
 Figure 2 : 2 Figure2: Comparison between M3DDPG (Minimax) and classical MADDPG (MA) on the four mixed competitive environments. Each bar cluster shows the 0-1 normalized score for a set of competing policies in different roles (agent vs adversary), where a higher score is better for the agent. In all cases, M3DDPG outperforms MADDPG when directly pitted against it.
 Figure 3 : 3 Figure 3: Performances of M3DDPG (Minimax, red) and MADDPG (MA, blue) under the worst situation, i.e., against the disruptive adversaries, on convert communication, keep-away, physical deception and predator-pray from left to right. The y-axis denotes the reward of normal agents (fixed) and x-axis denotes the training episodes performed of the disruptive adversaries. Higher reward implies a more robust policy. Agents trained by M3DDPG (Minimax) perform better on all the scenarios.
 In fact, we are dealing with gains, i.e., maximizing each agent's accumulative reward, so the "minimax" here is essentially "maximin". We keep the term "minimax" to be consistent with literature.

Shihui Li
Yi Wu
Xinyue Cui
Honghua Dong
Fei Fang
Stuart Russell