If multi-agent learning is the answer, what is the question?

Introduction The topic of learning in multi-agent systems, or multi-agent learning (MAL henceforth), has a long history in game theory, almost as long as the history of game theory itself. 2 As early as 1951, fictitious play [10] was proposed as a learning algorithm for computing equilibria in games and there have been proposals for how to evaluate the success of learning rules going back to [23] and [5] . Since that time hundreds, if not thousands, of articles have been published on the topic, and at least two books ( [20] and [54] ). In Artificial Intelligence (AI) the history of single-agent learning is as rich if not richer, with thousands of articles, many books, and some very compelling applications in a variety of fields (for some examples see [29, 40] , or [50] ). While it is only in recent years that AI has branched into the multi-agent aspects of learning, it has done so with something of a vengeance. If in 2003 one could describe the AI literature on MAL by enumerating the relevant articles, today this is no longer possible. The leading conferences routinely feature articles on MAL, as do the journals. 3 While the AI literature maintains a certain flavor that distinguishes it from the game theoretic literature, the commonalities are greater than the differences. Indeed, alongside the area of mechanism design, and perhaps the computational questions surrounding solution concepts such as the Nash equilibrium, MAL is today arguably one of the most fertile interaction grounds between computer science and game theory. The MAL research in both fields has produced some inspiring results. We will not repeat them here, since we cannot be comprehensive in this article, but nothing we say subsequently should be interpreted as belittling the achievements in the area. Yet alongside these successes there are some indications that it could be useful to take a step back and ask a few basic questions about the area of MAL. One surface indication is the presence of quite a number of frustrating dead ends. For example, the AI literature attempting to extend Bellman-style single-agent reinforcement learning techniques (in particular, Q-learning [53] ) to the multi-agent setting, has fared well in zero-sum repeated games (e.g., [36] and [38] ) as well as common-payoff (or 'team') repeated games (e.g., [14, 31, 52] ), but less well in general-sum stochastic games (e.g., [21, 26, 37] ) (for the reader unfamiliar with this line of work, we cover it briefly in Section 4). Indeed, upon close examination, it becomes clear that the very foundations of MAL could benefit from explicit discussion. What exact question or questions is MAL addressing? What are the yardsticks by which to measure answers to these questions? The present article focuses on these foundational questions. To start with the punch line, following an extensive look at the literature we have reached two conclusions: • There are several different agendas being pursued in the MAL literature. They are often left implicit and conflated; the result is that it is hard to evaluate and compare results. • We ourselves can identify and make sense of five distinct research agendas. Not all work in the field falls into one of the five agendas we identify. This is not necessarily a critique of work that doesn't; it simply means that one must identify yet other well-motivated and well defined problems addressed by that work. We expect that as a result of our throwing down the gauntlet additional such problems will be defined, but also that some past work will be re-evaluated and reconstructed. Certainly we hope that future work will always be conducted and evaluated against well-defined criteria, guided by this article and the discussion engendered by it among our colleagues in AI and game theory. In general we view this article not as a final statement but as the start of a discussion. In order to get to the punch line outlined above, we proceed as follows. In the next section we define the formal setting on which we focus. In Section 3 we illustrate why the question of learning in multi-agent settings is inherently more complex than in the single-agent setting, and why it places a stress on basic game theoretic notions. In Section 4 we provide some concrete examples of MAL approaches from both game theory and AI. This is anything but a comprehensive coverage of the area, and the selection is not a value judgment. Our intention is to anchor the discussion in something concrete for the benefit of the reader who is not familiar with the area, and-within the formal confines we discuss in Section 2-the examples span the space of MAL reasonably well. In Section 5 we identify five different agendas that we see (usually) implicit in the literature, and which we argue should be made explicit and teased apart. We end in Section 6 with a summary of the main points made in this article. A final remark is in order. The reader may find some of the material in the next three sections basic or obvious; different readers will probably find different parts so. We don't mean to insult anyone's intelligence, but we err on the side of explicitness for two reasons. First, this article is addressed to at least two different communities with somewhat different backgrounds. Second, our goal is to contribute to the clarification of foundational issues; we don't want to be guilty of vagueness ourselves.
 The formal setting We will couch our discussion in the formal setting of stochastic games (aka Markov games). Most of the MAL literature adopts this setting, and indeed most of it focuses on the even more narrow class of repeated games. Furthermore, stochastic games also generalize Markov Decision Problems (MDPs), the setting from which much of the relevant learning literature in AI originates. These are defined as follows. A stochastic game can be represented as a tuple: (N, S, A, R, T ). N is a set of agents indexed 1, . . . , n. S is a set of n-agent stage games. A = A 1 , . . . , A n , with A i the set of actions (or pure strategies) of agent i (note that we assume the agent has the same strategy space in all games; this is a notational convenience, but not a substantive restriction). R = R 1 , . . . , R n , with R i : S × A → R giving the immediate reward function of agent i for stage game S. T : S × A → Π(S) is a stochastic transition function, specifying the probability of the next stage game to be played based on the game just played and the actions taken in it. We also need to define a way for each agent to aggregate the set of immediate rewards received in each state. For finitely repeated games we can simply use the sum or average, while for infinite games the most common approaches are to use either the limit average or the sum of discounted awards ∞ t=1 δ t r t , where r t is the reward received at time t. A repeated game is a stochastic game with only one stage game, while an MDP is a stochastic game with only one agent. While most of the MAL literature lives happily in this setting, we would be remiss not to acknowledge the literature that does not. Certainly one could discuss learning in the context of extensive-form games of incomplete and/or imperfect information (cf. [28] ). We don't dwell on those since it would distract from the main discussion, and since the lessons we draw from our setting will apply there as well. Although we will not specifically include them, we also intend our comments to apply at a general level to large population games and evolutionary models, and particularly replicator dynamics (RD) [47] and evolutionary stable strategies (ESS) [49] . These are defined as follows. The replicator dynamic model assumes a population of homogeneous agents each of which continuously plays a two-player game against every other agent. Formally the setting can be expressed as a tuple (A, P 0 , R). A is the set of possible pure strategies/actions for the agents indexed 1, . . . , m. P 0 is the initial distribution of agents across possible strategies, m i=1 P 0 (i) = 1. R : A × A → R is the immediate reward function for each agent with R(a, a ) giving the reward for an agent playing strategy a against another agent playing strategy a . The population then changes proportions according to how the reward for each strategy compares to the average reward: dt (P t (a)) = P t (a)[u t (a) − u \* t ], where u t (a) = a P t (a )R(a, a ) and u \* t = a P t (a)u t (a). A strategy a is then defined to be an evolutionary stable strategy if and only if for some > 0 and for all other strategies a , R(a, (1 − )a + a ) > R(a , (1 − )a + a ). As the names suggest, one way to interpret these settings is as building on population genetics, that is, as representing a large population undergoing frequent pairwise interactions. An alternative interpretation however is as a repeated game between two agents, with the distribution of strategies in the population representing the agent's mixed strategy (in the homogeneous definition above the two agents have the same mixed strategy, but there exist more general definitions with more than two agents and with non-identical strategies). The second interpretation reduces the setting to the one we discuss. The first bears more discussion, and we do it briefly in Section 4. And so we stay with the framework of stochastic games. What is there to learn in these games? Here we need to be explicit about some aspects of stochastic games that were glossed over so far. Do the agents know the stochastic game, including the stage games and the transition probabilities? If not, do they at least know the specific game being played at each stage, or only the actions available to them? What do they see after each stage game has been played-only their own rewards, or also the actions played by the other agent(s)? Do they perhaps magically see the other agent(s)' mixed strategy in the stage game? And so on. In general, games may be known or not, play may be observable or not, and so on. We will focus on known, fully observable games, where the other agent's strategy (or agents' strategies) is not known a priori (though in some case there is a prior distribution over it). In our restricted setting there two possible things to learn. First, the agent can learn the opponent's (or opponents') strategy (or strategies), so that the agent can then devise a best (or at least a good) response. Alternatively, the agent can learn a strategy of his own that does well against the opponents, without explicitly learning the opponent's strategy. The first is sometimes called model-based learning, and the second modelfree learning. In broader settings there is more to learn. In particular, with unknown games, one can learn the game itself. Some will argue the restricted setting is not a true learning setting, but (a) much of the current work on MAL, particularly in game theory, takes place in this setting, and (b) the foundational issues we wish to tackle surface already here. In particular, our comments are intended to also apply to the work in the AI literature on games with unknown payoffs, work which builds on the success of learning in unknown MDPs. We will have more to say about the nature of 'learning' in the setting of stochastic games in the following sections.
 On some special characteristics of multi-agent learning Before launching into specifics, we wish to highlight the special nature of MAL. There are two messages we would like to get across, one aimed at AI researchers specifically and one more broadly. Both lessons can be gleaned from simple and well-known examples. Consider the game described in Fig. 1 . In this game the row player has a strictly dominant strategy, Down, and so seemingly there is not much more to say about this game. But now imagine a repeated version of this game. If the row player indeed repeatedly plays Down, assuming the column player is paying any attention, he (the column player) will start responding with Left, and the two will end up with a repeated (Down, Left) play. If, however, the row player starts repeatedly playing Up, and again assuming the column player is awake, he may instead start responding by playing Right, and the two players will end up with a repeated (Up, Right) play. The lesson from this is simple yet profound: In a multi-agent setting one cannot separate learning from teaching. In this example, by playing his dominated strategy, the row player taught the column player to play in a way that benefits both. Indeed, for this reason it might be more appropriate to speak more neutrally about multi-agent adaptation rather than learning. We will not fight this linguistic battle, but the point remains important, especially for computer scientists who are less accustomed to thinking about interactive considerations than game theorists. In particular, it follows there is no a priori reason to expect that machine learning techniques that have proved successful in AI for single-agent settings will also prove relevant in the multi-agent setting. The second lesson we draw from the well-known game of Rochambeau, or Rock-Paper-Scissors, given in Fig. 2 . As is well known, this zero-sum game has a unique Nash equilibrium in which each player randomizes uniformly among the three strategies. One could conclude that there is not much more to say about the game. But suppose you entered a Rochambeau tournament. Would you simply adopt the equilibrium strategy? If you did, you would not win the competition. This is no idle speculation; such competitions take place routinely. For example, starting in 2002, the World Rock Papers Scissors Society (WRPS) standardized a set of rules for international play and has overseen annual International World Championships as well as many regional and national events throughout the year. These championships have been attended by players from around the world and have attracted widespread international media attention. The winners are never equilibrium players. For example, on October 25th, 2005, 495 people entered the competition in Toronto from countries as diverse as Norway, Northern Ireland, the Cayman Islands, Australia, New Zealand and the UK. The winner was Toronto Lawyer Andrew Bergel, who beat These tournaments of course are not a perfect match with the formal model of repeated games. However, we include the example not only for entertainment value. The rules of the RPS tournaments call for 'matches' between players, each match consisting of several 'games', where a game is a single play of RPS. The early matches adopted a "best of three of three" format, meaning that the player who wins a best of three set garners one point and requires two points to take the match. The Semi-Finals and the Final Match used the "best of three of five" format, meaning that the player who wins a best of three set garners one point and requires three points to take the match. And so the competition really consisted of a series of repeated games, some of them longer than others. 4 Entertainment aside, what do we learn from this? We believe that this is a cautioning tale regarding the predictive or prescriptive role of equilibria in complex games, and in particular in repeated games. There are many examples of games with complex strategy spaces, in which equilibrium analysis plays little or no role-including familiar parlor games, or the Trading Agent Competition (TAC), a computerized trading competition. 5 The strategy space in a repeated game (or more generally a stochastic game) is immense-all mappings from past history to mixed strategies in the stage game. In such complex games it is not reasonable to expect that players contemplate the entire strategy space-their own or that of the opponent(s). Thus, (e.g., Nash) equilibria don't play here as great a predictive or prescriptive role. Our cautioning words should be viewed as countering the default blind adoption of equilibria as the driving concept in complex games, but not as a sweeping statement against the relevance of equilibria in some cases. The simpler the stage game, and the longer its repetition, the more instructive are the equilibria. Indeed, despite our example above, we do believe that if only two players play a repeated RPS game for long enough, they will tend to converge to the equilibrium strategy (this is particularly true of computer programs, that don't share the human difficulty with throwing a mental die). Even in more complex games there are examples where computer calculation of approximate equilibria within a restricted strategy space provided valuable guidance in constructing effective strategies. This includes the game of Poker ( [33] and [4] ), and even, as an exception to the general rule we mentioned, one program that competed in the Trading Agent Competition [13] . Our point has only been that in the context of complex games, so-called "bounded rationality", or the deviation from the ideal behavior of omniscient agents, is not an esoteric phenomenon to be brushed aside.
 A (very partial) sample of MAL work To make the discussion concrete, it is useful to look at MAL work over the years. The selection that follows is representative but very partial; no value judgment or other bias are intended by this selection. The reader familiar with the literature may wish to skip to Section 4.3, where we make some general subjective comments. Unless we indicate otherwise, our examples are drawn from the special case of repeated, two-person games (as opposed to stochastic, n-player games). We do this both for ease of exposition, and because the bulk of the literature indeed focuses on this special case. We divide the coverage into three parts: techniques, results, and commentary.
 Some MAL techniques We will discuss three classes of techniques-one representative of work in game theory, one more typical of work in AI, and one that seems to have drawn equal attention from both communities. entertaining reading; of note is the restriction of the strategy space to Rock, Paper and Scissors, and explicitly ruling out others: "Any use of Dynamite, Bird, Well, Spock, Water, Match, Fire, God, Lightning, Bomb, Texas Longhorn, or other non-sanctioned throws, will result in automatic disqualification". The overview of the RPS society and its tournaments is adapted from the inimitable Wikipedia, the collaborative online encyclopedia, as available on January 2, 2006. Wikipedia goes on to list the champions since 2002; we note without comment that they are all male Torontonians. The results of the specific competition cited are drawn from the online edition of the Boise Weekly dated November 2, 2005. The Boise Weekly starts the piece with "If it weren't true, we wouldn't report on it". 5 http://tac.eecs.umich.edu.
 Model-based approaches The first approach to learning we discuss, which is common in the game theory literature, is the model-based one. It adopts the following general scheme: 1. Start with some model of the opponent's strategy. 2. Compute and play the best response. 3. Observe the opponent's play and update your model of her strategy. 4. Goto step 2. Among the earliest, and probably the best-known, instance of this scheme is fictitious play [10] . The model is simply a count of the plays by the opponent in the past. The opponent is assumed to be playing a stationary strategy, and the observed frequencies are taken to represent the opponent's mixed strategy. Thus after five repetitions of the Rochambeau game in which the opponent played (R, S, P , R, P ), the current model of her mixed strategy is (R = 0.4, P = 0.4, S = 0.2). There exist many variants of the general scheme, for example those in which one does not play the exact best response in step 2. This is typically accomplished by assigning a probability of playing each pure strategy, assigning the best response the highest probability, but allowing some chance of playing any of the strategies. A number of proposals have been made of different ways to assign these probabilities such as smooth fictitious play [18] and exponential fictitious play [19] . A more sophisticated version of the same scheme is seen in rational learning [30] . The model is a distribution over the repeated-game strategies. One starts with some prior distribution; for example, in a repeated Rochambeau game, the prior could state that with probability 0.5 the opponent repeatedly plays the equilibrium strategy of the stage game, and, for all k > 1, with probability 2 −k she plays R k times and then reverts to the repeated equilibrium strategy. After each play, the model is updated to be the posterior obtained by Bayesian conditioning of the previous model. For instance, in our example, after the first non-R play of the opponent, the posterior places probability 1 on the repeated equilibrium play.
 Model-free approaches An entirely different approach that has been commonly pursued in the AI literature [29] , is the model-free one, which avoids building an explicit model of the opponent's strategy. Instead, over time one learns how well one's own various possible actions fare. This work takes place under the general heading of reinforcement learning, 6 and most approaches have their roots in the Bellman equations [3] . The basic algorithm for solving for the best policy in a known MDP starts by initializing a value function, V 0 : S → R, with a value for each state in the MDP. The value function can then be iteratively updated using the Bellman equation: V k+1 ← R(s) + γ max a s T (s, a, s )V k (s ) The optimal policy can then be obtained by selecting the action, a, at each state, s, that maximizes the expected value: s T (s, a, s )V k (s ). Much of the work in AI has focused strategies for rapid convergence, on very large MDPs, and in particular on unknown and partially observable MDPs. While this is not our focus, we do briefly discuss the unknown case, since this is where the literature leading to many of the current approaches for stochastic games originated. For MDPs with unknown reward and transition functions, the Q-learning algorithm [53] can be used to compute an optimal policy. Q(s, a) ← (1 − α t )Q(s, a) + α t R(s, a) + γ V (s ) V (s) ← max a∈A Q(s, a) As is well known, with certain assumptions about the way in which actions are selected at each state over time and constraints on the learning rate schedule, α t , Q-learning can be shown to converge to the optimal value function V \* . The Q-learning algorithm can be extended to the multi-agent stochastic game setting by having each agent simply ignore the other agents and pretend that the environment is passive: Q i (s, a i ) ← (1 − α t )Q i (s, a i ) + α t R i (s, a) + γ V i (s ) V i (s) ← max a i ∈A i Q i (s, a i ) Several authors have tested variations of the basic Q-learning algorithm for MAL (e.g., [48] ). However, this approach ignores the multi-agent nature of the setting entirely. The Q-values are updated without regard for the actions selected by the other agents. While this can be justified when the opponents' distributions of actions are stationary, it can fail when an opponent may adapt its choice of actions based on the past history of the game. A first step in addressing this problem is to define the Q-values as a function of all the agents' actions: Q i (s, a) ← (1 − α)Q i (s, a) + α R i (s, a) + γ V i (s ) We are however left with the question of how to update V , given the more complex nature of the Q-values. For (by definition, two-player) zero-sum SGs, Littman suggests the minimax-Q learning algorithm, in which V is updated with the minimax of the Q values [36] : V 1 (s) ← max P 1 ∈Π(A 1 ) min a 2 ∈A 2 a 1 ∈A 1 P 1 (a 1 )Q 1 s, (a 1 , a 2 ) Later work (such as the joint-action learners in [14] and the Friend-or-Foe Q algorithm in [37] ) proposed other update rules for the Q and V functions focusing on the special case of common-payoff (or 'team') games. A stage game is common-payoff if at each outcome all agents receive the same payoff. The payoff is in general different in different outcomes, and thus the agents' problem is that of coordination; indeed these are also called games of pure coordination. The work on zero-sum and common-payoff games continues to be refined and extended (e.g., [8, 32, 34, 52] ). Much of this work has concentrated on provably optimal tradeoffs between exploration and exploitation in unknown, zerosum games; this is a fascinating topic, but not germane to our focus. More relevant are the most recent efforts in this line of research to extend the "Bellman heritage" to general-sum games (e.g., Nash-Q by [25] and CE-Q by [21] ). We do not cover these for two reasons: The description is more involved, and the results have been less satisfactory; more on the latter below.
 Regret minimization approaches Our third and final example of prior work in MAL is no-regret learning. It is an interesting example for two reasons. First, it has some unique properties that distinguish it from the work above. Second, both the AI and game theory communities appear to have converged on it independently. The basic idea goes back to early work on how to evaluate the success of learning rules [5, 23] , and has since been extended and rediscovered numerous times over the years under the names of universal consistency, no-regret learning, and the Bayes envelope (see [16] for an overview of this history). We will describe the algorithm proposed in [24] as a representative of this body of work. We start by defining the regret, r t i (a j , s i ) of agent i for playing the sequence of actions s i instead of playing action a j , given that the opponents played the sequence s −i . r t i (a j , s i |s −i ) = t k=1 R a j , s k −i − R s k i , s k −i The agent then selects each of its actions with probability proportional to max(r t i (a j , s i ), 0) at each time step t + 1. Recently, these ideas have also been adopted by researchers in the computer science community (e.g., [17, 27, 55] ). Note that the application of approaches based on regret minimization has been restricted to the case of repeated games. The difficulties of extending this concept to stochastic games are discussed in [39] .
 Some typical results One sees at least three kinds of results in the literature regarding the learning algorithms presented above, and others like them. These are: 1. Convergence of the strategy profile to an (e.g., Nash) equilibrium of the stage game in self play (that is, when all agents adopt the learning procedure under consideration). 2. Successful learning of an opponent's strategy (or opponents' strategies). 3. Obtaining payoffs that exceed a specified threshold. Each of these types comes in many flavors; here are some examples. The first type is perhaps the most common in the literature, in both game theory and AI. For example, while fictitious play does not in general converge to a Nash equilibrium of the stage game, the distribution of its play can be shown to converge to an equilibrium in zero-sum games [46] , 2 × 2 games with generic payoffs [41] , or games that can be solved by iterated elimination of strictly dominated strategies [42] . Similarly in AI, in [38] minimax-Q learning is proven to converge in the limit to the correct Q-values for any zero-sum game, guaranteeing convergence to a Nash equilibrium in self-play. This result makes the standard assumptions of infinite exploration and the conditions on learning rates used in proofs of convergence for single-agent Q-learning. Claus and Boutilier [14] conjecture that both single-agent Q-learners and the belief-based joint action learners they proposed converge to an equilibrium in common payoff games under the conditions of self-play and decreasing exploration, but do not offer a formal proof. Friend-or-Foe Q and Nash-Q were both shown to converge to a Nash equilibrium in a set of games that are a slight generalization on the set of zero-sum and common payoff games. Rational learning exemplifies results of the second type. The convergence shown is to correct beliefs about the opponent's repeated game strategy; thus it follows that, since each agent adopts a best response to their beliefs about the other agent, in the limit the agents will converge to a Nash equilibrium of the repeated game. This is an impressive result, but it is limited by two factors; the convergence depends on a very strong assumption of absolute continuity, and the beliefs converged to are only correct with respect to the aspects of history that are observable given the strategies of the agents. This is an involved topic, and the reader is referred to the literature for more details. The literature on no-regret learning provides an example of the third type of result, and has perhaps been the most explicit about criteria for evaluating learning rules. For example, in [19] two criteria are suggested. The first is that the learning rule be 'safe', which is defined as the requirement that the learning rule guarantee at least the minimax payoff of the game. (The minimax payoff is the maximum expected value a player can guarantee against any possible opponent.) The second criterion is that the rule should be 'consistent'. In order to be 'consistent', the learning rule must guarantee that it does at least as well as the best response to the empirical distribution of play when playing against an opponent whose play is governed by independent draws from a fixed distribution. They then define 'universal consistency' as the requirement that a learning rule do at least as well as the best response to the empirical distribution regardless of the actual strategy the opponent is employing (this implies both safety and consistency) and show that a modification of the fictitious play algorithm achieves this requirement. In [20] they strengthen their requirement by requiring that the learning rule also adapt to simple patterns in the play of its opponent. The requirement of 'universal consistency' is in fact equivalent to requiring that an algorithm exhibit no-regret, generally defined as follows, against all opponents. ∀ > 0, lim t→inf 1 t max a j ∈A i r t i (a j , s i |s −i ) < In both game theory and artificial intelligence, a large number of algorithms have been show to satisfy universal consistency or no-regret requirements. In addition, recent work [6] has tried to combine these criteria resulting in GIGA-WoLF, a no-regret algorithm that provably achieves convergence to a Nash equilibrium in self-play for games with two players and two actions per player. Meanwhile, the regret matching algorithm [24] described earlier guarantees that the empirical distributions of play converge to the set of correlated equilibria of the game. Other recent work by [2] has addressed concerns with only requiring guarantees about the behavior in the limit. Their algorithm is guaranteed to achieve -no-regret payoff guarantees with small polynomial bounds on time and uses only the agent's ability to observe what payoff it receives for each action.
 Some observations and questions We have so far described the work without comment; here we take a step back and ask some questions about this representative work. Our first comment concerns the settings in which the results are presented. While the learning procedures apply broadly, the results for the most part focus on self play (that is, when all agents adopt the learning procedure under consideration). They also tend to focus on games with only two agents. Why does most of the work have this particular focus? Is it technical convenience, or is learning among more than two agents, or among agents using different learning procedures, less relevant for some reason? Our second comment pertains to the nature of the results. With the exception of the work on no-regret learning, the results we described investigate convergence to equilibrium play of the stage game (albeit with various twists). Is this the pertinent yardstick? If the process (say of self play between two agents) does not converge to equilibrium play, should we be disturbed? More generally, and again with the exception of no-regret learning, the work focuses on the play to which the agents converge, not on the payoffs they obtain. Which is the right focus? No-regret learning is distinguished by its starting with criteria for successful learning, rather than a learning procedure. The question one might ask is whether the particular criteria are adequate. In particular, the requirement of consistency ignores the basic lesson regarding learning-vs.-teaching discussed in Section 3. By measuring the performance only against stationary opponents, we do not allow for the possibility of teaching opponents. Thus, for example, in an infinitely repeated Prisoners' Dilemma game, no-regret dictates the strategy of always defecting, precluding the possibility of cooperation (for example, by the mutually reinforcing Tit-For-Tat strategies). Our goal here is not to critique the existing work, but rather to shine a spotlight on the assumptions that have been made, and ask some questions that get at the basic issues addressed, questions which we feel have not been discussed as clearly and as explicitly as they deserve. In the next section we propose an organized way of thinking about these questions.
 Five distinct agendas in multi-agent learning After examining the MAL literature-the work surveyed here and much else-we have reached the conclusion that there are several distinct agendas at play, which are often left implicit and conflated. We believe that a prerequisite for success in the field is to be very explicit about the problem being addressed. We ourselves can identify five distinct possible goals of MAL research. There may well be others, but these are the ones we can identify. They each have a clear motivation and a success criterion that will allow researchers to evaluate new contributions, even if people's judgments may diverge regarding their relative importance or success to date. They can be caricatured as follows: 1. Computational 2. Descriptive 3. Normative 4. Prescriptive, cooperative 5. Prescriptive, non-cooperative We can now consider each of the five in turn. The first agenda is computational in nature. It views learning algorithms as an iterative way to compute properties of the game, such as solution concepts. As an example, fictitious play was originally proposed as a way of computing a sample Nash equilibrium for zero-sum games [10] , and replicator dynamics has been proposed for computing a sample Nash equilibrium in symmetric games. Other adaptive procedures have been proposed more recently for computing other solution concepts (for example, computing equilibria in local-effect games [35] ). These tend not to be the most efficient computation methods, but they do sometimes constitute quick-and-dirty methods that can easily be understood and implemented. The second agenda is descriptive-it asks how natural agents learn in the context of other learners. The goal here is to investigate formal models of learning that agree with people's behavior (typically, in laboratory experiments), or possibly with the behaviors of other agents (for example, animals or organizations). This same agenda could also be taken to apply to large-population models, if those are indeed interpreted as representing populations. This problem is clearly an important one, and when taken seriously calls for strong justification of the learning dynamics being studied. One approach is to apply the experimental methodology of the social sciences. There are several good examples of this approach in economics and game theory, for example [15] and [11] . There could be other supports for studying a given learning process. For example, to the extent that one accepts the Bayesian model as at least an idealized model of human decision making, one could justify Kalai and Lehrer's model of rational learning. 7 However, it seems to us that sometimes there is a rush to investigate the convergence properties, motivated by the wish to anchor the central notion of game theory in some process, at the expense of motivating that process rigorously. 8 The centrality of equilibria in game theory underlies the third agenda we identify in MAL, which for lack of a better term we called normative, and which focuses on determining which sets of learning rules are in equilibrium with each other. More precisely, we ask which repeated-game strategies are in equilibrium; it just so happens that in repeated games, most strategies embody a learning rule of some sort. For example, we can ask whether fictitious play and Q-learning, appropriately initialized, are in equilibrium with each other in a repeated Prisoner's Dilemma game. Although one might expect that game theory purists might flock to this approach, there are very few examples of it. In fact, the only example we know originates in AI rather than game theory [9] , and it is explicitly rejected by at least some game theorists [18] . We consider it a legitimate normative theory. Its practicality depends on the complexity of the stage game being played and the length of play; in this connection see our discussion of the problematic role of equilibria in Section 3. The last two agendas are prescriptive; they ask how agents should learn. The first of these involves distributed control in dynamic systems. There is sometimes a need or desire to decentralize the control of a system operating in a dynamic environment, and in this case the local controllers must adapt to each other's choices. This direction, which is most naturally modeled as a repeated or stochastic common-payoff (or 'team') game, has attracted much attention in AI in recent years. Proposed approaches can be evaluated based on the value achieved by the joint policy and the resources required, whether in terms of computation, communication, or time required to learn the policy. In this case there is rarely a role for equilibrium analysis; the agents have no freedom to deviate from the prescribed algorithm. Examples of this work include [12, 14, 22] to name a small sample. Researchers interested in this agenda have access to a large body of existing work both within AI and other fields such as control theory and distributed computing. In our final agenda, termed 'prescriptive, non-cooperative', we ask how an agent should act to obtain high reward in the repeated (and more generally, stochastic) game. It thus retains the design stance of AI, asking how to design an optimal (or at least effective) agent for a given environment. It just so happens that this environment is characterized by the types of agents inhabiting it, agents who may do some learning of their own. The objective of this agenda is to identify effective strategies for environments of interest. An effective strategy is one that achieves a high reward in its environment, where one of the main characteristics of this environment is the selected class of possible opponents. This class of opponents should itself be motivated as being reasonable and containing opponents of interest. Convergence to an equilibrium is not a goal in and of itself. There are various possible instantiations of the term 'high reward'. One example is the no-regret line of work which we discussed. It clearly defines what it means for a reward to be high enough (namely, to exhibit no regret); we also discussed the limitations of this criterion. A more recent example, this one from AI, is [7] . This work puts forward two criteria for any learning algorithm in a multi-agent setting: (1) The learning should always converge to a stationary policy, and (2) if the opponent converges to a stationary policy, the algorithm must converge to a best response. There are possible critiques of these precise criteria. They can be too weak since in many cases of interest the opponents will not converge on a stationary strategy. And they can be too strong since attaining a precise best response, without a constraint on the opponent's strategy, is not feasible. But this work, to our knowledge, marks the first time a formal criterion was put forward in AI. A third example of the last agenda is our own work in recent years. In [45] we define a criterion parameterized by a class of 'target opponents'; with this parameter we make three requirements of any learning algorithm: (1) (Targeted optimality) The algorithm must achieve an -optimal payoff against any 'target opponent', (2) (Safety) The algorithm must achieve at least the payoff of the security level strategy minus against any other opponent, and (3) (Autocompatibility) The algorithm must perform well in self-play (the precise technical condition is omitted here). We then demonstrate an algorithm that provably meets these criteria when the target set is the set of stationary opponents in general-sum two-player repeated games. More recent work has extended these results to handle opponents whose play is conditional on the recent history of the game [44] and settings with more than two players [51] .
 Summary In this article we have made the following points: 1. Learning in MAS is conceptually, not only technically, challenging. 2. One needs to be crystal clear about the problem being addressed and the associated evaluation criteria. 3. For the field to advance one cannot simply define arbitrary learning strategies, and analyze whether the resulting dynamics converge in certain cases to a Nash equilibrium or some other solution concept of the stage game. This in and of itself is not well motivated. 4. We have identified five coherent agendas. 5. Not all work in the field falls into one of these buckets. This means that either we need more buckets, or some work needs to be revisited or reconstructed so as to be well grounded. There is one last point we would like to make, which didn't have a natural home in the previous sections, but which in our view is important. It regards evaluation methodology. We have focused throughout the article on formal criteria, and indeed believe these to be essential. However, as is well known in computer science, many algorithms that meet formal criteria fail in practice, and vice versa. And so we advocate complementing the formal evaluation with an experimental one. We ourselves have always included a comprehensive bake-off between our proposed algorithms and the other leading contenders across a broad range of games. The algorithms we coded ourselves; the games were drawn from GAMUT, an existing testbed (see [43] and http://gamut.stanford.edu). GAMUT is available to the community at large. It would be useful to have a learning-algorithm repository as well. To conclude, we re-emphasize the statement made at the beginning: This article is meant to be the beginning of a discussion in the field, not its end. Fig. 1 .Fig. 2 . 12 Fig. 1. Stackelberg stage game: The payoff for the row player is given first in each cell, with the payoff for the column player following.
 We acknowledge a simplification of history here. There is definitely MAL work in AI that predates the last few years, though the relative deluge is indeed recent. Similarly, we focus on AI since this is where most of the action is these days, but there are also other areas in computer science that feature MAL material; we mean to include that literature here as well.
 We do acknowledge some degree of humor in the example. The detailed rules in http://www.rpschamps.com/rules.html make for additional
 We note that the term is used somewhat differently in the game theory literature.
 Although this is beyond the scope of this article, we note that the question of whether one can justify the Bayesian approach in an interactive setting goes beyond the familiar contravening experimental data; even the axiomatic justification of the expected-utility approach does not extend naturally to the multi-agent case. 8 It has been noted that game theory is somewhat unusual in having the notion of an equilibrium without associated dynamics that give rise to the equilibrium [1] .


Ethical guidelines for a superintelligence

The assumption that intelligence is a potentially infinite quantity 1 with a well-defined, one-dimensional value. Bostrom writes differential equations for intelligence, and characterizes their solutions. Certainly, if you asked Bostrom about this, he would say that this is a simplifying assumption made for the sake of making the analysis concrete. The problem is, that if you look at the argument carefully, it depends rather strongly on this idealization, and if you loosen the idealization, important parts of the argument become significantly weaker, such as Bostrom's expectation that the progress from human intelligence to superhuman intelligence will occur quickly. Of course, there are quantities associated with intelligence that do correspond to this description: The speed of processing, the size of the brain, the size of memory of various kinds. But we do not know the relation of these to intelligence in a qualitative sense. We do not know the relation in brain size to intelligence across animals, because we have no useful measure or even definition of intelligence across animals. And these quantities certainly do not seem to be particularly related to differences in intelligence between people. Bostrom, quoting Eliezer Yudkowsky, points out that the difference between Einstein and the village idiot is tiny as compared to the difference between man and mouse; which is true and important. But that in itself does not justify his conclusion that in the development of AI's it will take much longer to get from mouse to man than from average man to Einstein. For one thing, we know less about those cognitive processes that made Einstein exceptional, than about the cognitive processes that are common to all people, because they are much rarer. Bostrom claims that once you have a machine with the intelligence of a man, you can get a superintelligence just by making the thing faster and bigger. However, all that running faster does is to save you time. If you have two machines A and B and B runs ten times as fast as A, then A can do anything that B can do if you're willing to wait ten times as long. The assumption that a large gain in intelligence would necessarily entail a correspondingly large increase in power. Bostrom points out that what he calls a comparatively small increase in brain size and complexity resulted in mankind's spectacular gain in physical power. But he ignores the fact that the much larger increase in brain size and complexity that preceded the appearance in man had no such effect. He says that the relation of a supercomputer to man will be like the relation of a man to a mouse, rather than like the relation of Einstein to the rest of us; but what if it is like the relation of an elephant to a mouse? The assumption that large intelligence entails virtual omnipotence. In Bostrom's scenarios there seems to be essentially no limit to what the superintelligence would be able to do, just by virtue of its superintelligence. It will, in a very short time, develop technological prowess, social abilities, abilities to psychologically manipulate people and so on, incomparably more advanced than what existed before. It can easily resist and outsmart the united efforts of eight billion people who might object to being enslaved or exterminated. This belief manifests itself most clearly in Bostrom's prophecies of the messianic benefits we will gain if superintelligence works out well. He writes that if a superintelligence were developed, "[r]isks from nature -such as asteroid impacts, supervolcanoes, and natural pandemics -would be virtually eliminated, since super intelligence could deploy countermeasures against most such hazards, or at least demote them to the non-existential category (for instance, via space colonization)". Likewise, the superintelligence, having established an autocracy (a "singleton" in Bostrom's terminology) with itself as boss, would eliminate "risk of wars, technology races, undesirable forms of competition and evolution, and tragedies of the commons." On a lighter note, Bostrom advocates that philosophers may as well stop thinking about philosophical problems (they should think instead about how to instill ethical principles in AIs) because pretty soon, superintelligent AIs will be able to solve all the problems of philosophy. This prediction seems to me a hair less unlikely than the apocalyptic scenario, but only a hair. The unwarranted belief that, though achieving intelligence is more or less easy, giving a computer an ethical point of view is really hard. Bostrom writes about the problem of instilling ethics in computers in a language reminiscent of 1960's era arguments against machine intelligence; how are you going to get something as complicated as intelligence, when all you can do is manipulate registers? The definition [of moral terms] must bottom out in the AI's programming language and ultimately in primitives such as machine operators and addresses pointing to the contents of individual memory registers. When one considers the problem from this perspective, one can begin to appreciate the difficulty of the programmer's task. In the following paragraph he goes on to argue from the complexity of computer vision that instilling ethics is almost hopelessly difficult, without, apparently, noticing that computer vision itself is a central AI problem, which he is assuming is going to be solved. He considers that the problems of instilling ethics into an AI system is "a research challenge worthy of some of the next generation's best mathematical talent". It seems to me, on the contrary, that developing an understanding of ethics as contemporary humans understand it is actually one of the easier problems facing AI. Moreover, it would be a necessary part, both of aspects of human cognition, such as narrative understanding, and of characteristics that Bostrom attributes to the superintelligent AI. For instance, Bostrom refers to the AI's "social manipulation superpowers". But if an AI is to be a master manipulator, it will need a good understanding of what people consider moral; if it comes across as completely amoral, it will be at a very great disadvantage in manipulating people. There is actually some truth to the idea, central to The Lord of the Rings and Harry Potter, that in dealing with people, failing to understand their moral standards is a strategic gap. If the AI can understand human morality, it is hard to see what is the technical difficulty in getting it to follow that morality. Let me suggest the following approach to giving the superintelligent AI an operationally useful definition of minimal standards of ethics that it should follow. You specify a collection of admirable people, now dead. (Dead, because otherwise Bostrom will predict that the AI will manipulate the preferences of the living people.) The AI, of course knows all about them because it has read all their biographies on the web. You then instruct the AI, "Don't do anything that these people would have mostly seriously disapproved of." This has the following advantages: • It parallels one of the ways in which people gain a moral sense. • It is comparatively solidly grounded, and therefore unlikely to have an counterintuitive fixed point. • It is easily explained to people. Of course, it is completely impossible until we have an AI with a very powerful understanding; but that is true of all Bostrom's solutions as well. To be clear: I am not proposing that this criterion should be used as the ethical component of every day decisions; and I am not in the least claiming that this idea is any kind of contribution to the philosophy of ethics. The proposal is that this criterion would work well enough as a minimal standard of ethics; if the AI adheres to it, it will not exterminate us, enslave us, etc. This may not seem adequate to Bostrom, because he is not content with human morality in its current state; he thinks it is important for the AI to use its superintelligence to find a more ultimate morality. That seems to me both unnecessary and very dangerous. It is unnecessary because, as long as the AI follows our morality, it will at least avoid getting horribly out of whack, ethically; it will not exterminate us or enslave us. It is dangerous because it is hard to be sure that it will not lead to consequences that we would reasonably object to. The superintelligence might rationally decide, like the King of Brobdingnag, that we humans are "the most pernicious race of little odious vermin that nature ever suffered to crawl upon the surface of the earth," and that it would do well to exterminate us and replace us with some much more worthy species. However wise this decision, and however strongly dictated by the ultimate true theory of morality, I think we are entitled to object to it, and to do our best to prevent it. I feel safer in the hands of a superintelligence who is guided by 2014 morality, or for that matter by 1700 morality, than in the hands of one that decides to consider the question for itself. Bostrom considers at length solving the problem of the out-of-control computer by suggesting to the computer that it might actually be living in a simulated universe, and if so, the true powers that be might punish it for making too much mischief. This, of course, is just the belief in a transcendent God who punishes sin, rephrased in language appealing to twenty-first century philosophers. It is open to the traditional objection; namely, even if one grants the existence of God/Simulator, the grounds, either empirical or theoretical, for believing that He punishes sin and rewards virtue are not as strong as one might wish. However, Bostrom considers that the argument might convince the AI, or at least instill enough doubt to stop him in its nefarious plans. Certainly a general artificial intelligence is potentially dangerous; and once we get anywhere close to it, we should use common sense to make sure that it doesn't get out of hand. The programs that have great physical power, such as those that control the power grid or the nuclear bombs, should be conventional programs whose behavior is very well understood. They should also be protected from sabotage by AI's; but they have to be protected from human sabotage already, and the issues of protection are not very different. One should not write a program that thinks it has a blank check to spend all the resources of the world for any purpose, let alone solving the Riemann hypothesis or making paperclips. Any machine should have an accessible "off" switch; and in the case of a computer or robot that might have any tendency toward self-preservation, it should have an off switch that it cannot block. However, in the case of computers and robots, this is very easily done, since we are building them. All you need is to place in the internals of the robot, inaccessible to it, a device that, when it receives a specified signal, cuts off the power -or, if you want something more dramatic, triggers a small grenade. This can be done in a way that the computer probably cannot find out the details of how the grenade is placed or triggered, and certainly cannot prevent it. Even so, one might reasonably argue that the dangers involved are so great that we should not risk building a computer with anything close to human intelligence. Something can always go wrong, or some foolish or malicious person might create a superintelligence with no moral sense and with control of its own off switch. I certainly have no objection to imposing restrictions, in the spirit of the Asilomar guidelines for recombinant DNA research, that would halt AI research far short of human intelligence. (Fortunately, it would not be necessary for such restrictions to have any impact on AI research and development any time in the foreseeable future.) It is certainly worth discussing what should be done in that direction. However, Bostrom's claim that we have to accept that quasi-omnipotent superintelligences are part of our future, and that our task is to find a way to make sure that they guide themselves to moral principles beyond the understanding of our puny intellects, does not seem to me a helpful contribution to that discussion. To be more precise, a quantity potentially bounded only the finite size of the universe and other such cosmological considerations.