The Complexity of Decentralized Control of Markov Decision Processes

1. Introduction. Markov decision processes (MDPs) have received considerable attention, and there exist well-known algorithms for finding optimal control strategies in the case where a process is centrally controlled and the controller (or agent) has access to complete state information (Puterman 1994) . Less well understood is the case in which a process is controlled by multiple cooperating distributed agents, each with possibly different information about the state. We are interested in studying how hard these decentralized control problems are relative to analogous centralized control problems, from the point of view of computational complexity. In particular, we consider two different models of decentralized control of MDPs. One is a generalization of a partially observable Markov decision process (POMDP), which we call a decentralized partially observable Markov decision process (DEC-POMDP). In a DEC-POMDP, the process is controlled by multiple distributed agents, each with possibly different information about the state. The other is a generalization of an MDP, called a decentralized Markov decision process (DEC-MDP). A DEC-MDP is a DEC-POMDP with the restriction that at each time step the agents' observations together uniquely determine the state. The MDP, POMDP, and DEC-MDP can all be viewed as special cases of the DEC-POMDP. The relationships among the models are shown in Figure 1 . A number of different problems can be viewed as decentralized control of a Markov process. For example, consider problems involving the control of multiple distributed robots, such as robotic soccer (Coradeschi et al. 2000) . In these domains, it is necessary to develop a strategy for each robot under the assumption that the robots will have limited ability to communicate when they execute their strategies. Another problem that fits naturally within this framework is the distributed control of a power grid (Schneider et al. 1999) . Finally, several types of networking problems can be viewed within this framework (Altman 2001) . It would be beneficial to have general-purpose algorithms for solving these decentralized control problems. An algorithm for similar problems was proposed by Ooi et al. (1996) . Under the assumption that all agents share state information every K time steps, the authors developed a dynamic programming algorithm to derive optimal policies. A downside of this approach is that the state space for the dynamic programming algorithm grows doubly exponentially with K. The only known tractable algorithms for these types of problems rely on even more assumptions. One such algorithm was developed by Hsu and Marcus (1982) and works under the assumption that the agents share state information every time step (although it can take one time step for the information to propagate). Approximation algorithms have also been developed for these problems, although they can at best give guarantees of local optimality. For instance, Peshkin et al. (2000) studied algorithms that perform gradient descent in a space of parameterized policies. Is there something inherent in these problems that forces us to add assumptions and/or use approximation algorithms? Papadimitriou and Tsitsiklis (1982) presented some results aimed at answering this question. The authors proved that a simple decentralized decisionmaking problem is NP-complete, even with just two decision makers. They later noted that this implies that decentralized control of MDPs must be NP-hard (Papadimitriou and Tsitsiklis 1986) . We strengthen this result by showing that both the DEC-POMDP and DEC-MDP problems are NEXP-hard, even when the horizon is limited to be less than the number of states (and they are NEXP-complete in the latter case). Although it is not known whether the classes P, NP, and PSPACE are distinct, it is known that P = NEXP, and thus the problems we consider are provably intractable. Furthermore, assuming EXP = NEXP, the problems take superexponential time to solve in the worst case. This result is in contrast to the best known bounds for MDPs (P-hard) and POMDPs (PSPACE-hard) (Papadimitriou and Tsitsiklis 1987, Mundhenk et al. 2000) . Thus, we have gained insight into the possibility of a fundamental difference between centralized and decentralized control of Markov decision processes. In §2, we give a brief review of the concepts we will need from complexity theory. In §3, we define the MDP and POMDP models. Section 4 contains the definitions of the DEC-MDP and DEC-POMDP models, and a proof that the short-horizon versions of these problems fall within the complexity class NEXP. In §5, we present our main complexity result-a proof that these decentralized problems are NEXP-hard. Finally, §6 contains our conclusions. 2. Computational complexity. In this section, we give a brief introduction to the theory of computational complexity. More detail can be found in Papadimitriou (1994) . A complexity class is a set of problems where a problem is an infinite set of problem instances, each of which has a "yes" or "no" answer. To discuss the complexity of optimization problems, we must have a way of converting them to "yes/no" problems. The typical way this is done is to set a threshold and ask whether or not the optimal solution yields a reward that is no less than this threshold. The problem of actually finding the optimal solution can, of course, be no easier than the threshold problem. The first complexity class we consider is P, the set of problems that can be solved in polynomial time (in the size of the problem instance) on a sequential computer. NP is the set of problems that can be solved nondeterministically in polynomial time. A nondeterministic machine automatically knows the correct path to take any time there is a choice as to how the computation should proceed. An example of a problem that can be solved nondeterministically in polynomial time is deciding whether a sentence of propositional logic is satisfiable. The machine can guess an assignment of truth values to variables and evaluate the resulting expression in polynomial time. Of course, nondeterministic machines do not really exist, and the most efficient known algorithms for simulating them take exponential time in the worst case. In fact, it is strongly believed by most complexity theorists that P = NP (but this has not been proven formally). Complexity can also be measured in terms of the amount of space a computation requires. One class, PSPACE, includes all problems that can be solved in polynomial space. Any problem that can be solved in polynomial time or nondeterministic polynomial time can be solved in polynomial space (i.e., P ⊆ NP ⊆ PSPACE)-that P ⊆ PSPACE can be seen informally by observing that only polynomially much space can be accessed in polynomially many time steps. Moving up the complexity hierarchy, we have exponential time (EXP) and nondeterministic exponential time (NEXP). By exponential time, we mean time bounded by 2 n k , where n is the input size and k > 0 is a constant. It is known that PSPACE ⊆ EXP ⊆ NEXP, and it is believed that EXP = NEXP (but again this has not been proven). It has been proven that the classes P and EXP are distinct, however. The notion of a reduction is important in complexity theory. We say that a problem A is reducible to a problem B if any instance x of A can be converted into an instance f x of B such that the answer to x is "yes" if and only if the answer to f x is "yes." A problem A is said to be hard for a complexity class C (or C-hard) if any problem in C is efficiently reducible to A. If the complexity class in question is P, efficient means that f x can be computed using at most logarithmic space, while for the classes above P, efficient means that f x can be computed using at most polynomial time. A problem A is said to be complete for a complexity class C (or C-complete) if (a) A is contained in C, and (b) A is hard for C. For instance, the satisfiability problem mentioned above is NP-complete and P-hard. However, unless P = NP, satisfiability is not P-complete. 3. Centralized models. In this paper, we consider discrete-time finite sequential decision processes under the undiscounted finite-horizon optimality criterion. We build into our problem definitions the (unusual) assumption that the horizon is less than the number of states. Note that this assumption actually strengthens the hardness results; the general problems must be at least as hard as their short-horizon counterparts. Unfortunately, the assumption is needed for each of the upper bounds given below. Finding tight upper bounds for problems with arbitrary horizons remains an open problem (Blondel and Tsitsiklis 1999, §5) . Below we describe the partially observable Markov decision process and its associated decision problem. The Markov decision process is viewed as a restricted version of this model. A partially observable Markov decision process (POMDP) is defined as follows. We are given a tuple, S A P R O T K , where • S is a finite set of states, with distinguished initial state s 0 . • A is a finite action set. • P is a transition probability table. P s a s is a rational representing the probability of transitioning from s to s on taking action a. Here s s ∈ S and a ∈ A. • R is a reward function. R s a s is a rational representing the reward obtained from taking action a from state s and transitioning to state s . Again, s s ∈ S and a ∈ A. • is a finite set of observations. • O is a table of observation probabilities. O s a s o is a rational representing the probability of observing o when taking action a in state s and transitioning to state s as a result. Here s s ∈ S, a ∈ A, and o ∈ . • T is a positive integer representing the horizon (and T < S ). • K is a rational representing the threshold value. A POMDP is fully observable if there exists a mapping J → S such that whenever O s a s o is nonzero, J o = s . A Markov decision process (MDP) is defined to be a POMDP that is fully observable. A policy is defined to be a mapping from sequences of observations ō = o 1 • • • o t over to actions in A. We wish to find a policy that maximizes the expected total return over the finite horizon. The definitions below are used to formalize this notion. We use the symbol to denote the empty observation sequence. For an observation sequence ō = o 1 • • • o t , ōo is taken to represent the sequence o 1 • • • o t o. Definition. The probability of transitioning from a state s to a state s following policy while the agent sees observation sequence ō, written P s ō s , can be defined recursively as follows: P s s = 1 P s ōo s = q∈S P s ō q P q ō s O q ō s o where is the empty sequence. Definition. The value V T s of following policy from state s for T steps is given by the following equation: V T s = ō q∈S s ∈S P s ō q P q ō s R q ō s where the observation sequences have length at most T − 1. The decision problem corresponding to a finite-horizon POMDP is as follows. Given a POMDP D = S A P R O T K , is there a policy for which V T s 0 equals or exceeds K? It was shown in Papadimitriou and Tsitsiklis (1987) that the decision problem for POMDPs is PSPACE-complete and that the decision problem for MDPs is P-complete. 4. Decentralized models. We now describe extensions to the aforementioned models that allow for decentralized control. In these models, at each step, each agent receives a local observation and subsequently chooses an action. The state transitions and rewards received depend on the vector of actions of all the agents. A decentralized partially observable Markov decision process (DEC-POMDP) is defined formally as follows (for ease of exposition, we describe the two-agent case). We are given S A 1 A 2 P R 1 2 O T K , where • S is a finite set of states, with distinguished initial state s 0 . • A 1 and A 2 are finite action sets. • P is a transition probability table. P s a 1 a 2 s is a rational representing the probability of transitioning from s to s on taking actions a 1 a 2 . Here s s ∈ S, a 1 ∈ A 1 , and a 2 ∈ A 2 . • R is a reward function. R s a 1 a 2 s is a rational representing the reward obtained from taking actions a 1 a 2 from state s and transitioning to state s . Again s s ∈ S, a 1 ∈ A 1 , and a 2 ∈ A 2 . • 1 and 2 are finite sets of observations. • O is a table of observation probabilities. O s a 1 a 2 s o 1 o 2 is a rational representing the probability of observing o 1 o 2 when taking actions a 1 a 2 in state s and transitioning to state s as a result. Here s s ∈ S, a 1 ∈ A 1 , a 2 ∈ A 2 , o 1 ∈ 1 , and o 2 ∈ 2 . • T is a positive integer representing the horizon (and T < S ). • K is a rational representing the threshold value. A DEC-POMDP generalizes a POMDP by allowing for control by multiple distributed agents that together may not fully observe the system state (so we have only partial observability). We also define a generalization of MDP problems by requiring joint observability. We say that a DEC-POMDP is jointly observable if there exists a mapping J 1 × 2 → S such that whenever O s a 1 a 2 s o 1 o 2 is nonzero, J o 1 o 2 = s . We define a decentralized Markov decision process (DEC-MDP) to be a DEC-POMDP that is jointly observable. We define a local policy for agent i, i , to be a mapping from local histories of observations ōi = o i1 • • • o it over i , to actions in A i . A joint policy, = 1 2 , is defined to be a pair of local policies, one for each agent. We wish to find a joint policy that maximizes the expected total return over the finite horizon. As in the centralized case, we need some definitions to make this notion more formal. Definition. The probability of transitioning from a state s to a state s following joint policy = 1 2 while agent 1 sees observation sequence ō1 and agent 2 sees ō2 of the same length, written P s ō1 ō2 s , can be defined recursively as follows: P s s = 1 P s ō1 o 1 ō2 o 2 s = q∈S P s ō1 ō2 q P q 1 ō1 2 ō2 s O q 1 ō1 2 ō2 s o 1 o 2 where is the empty sequence. Definition. The value V T s of following policy = 1 2 from state s for T steps is given by the following equation: V T s = ō1 ō2 q∈S s ∈S P s ō1 ō2 q P q 1 ō1 2 ō2 s R q 1 ō1 2 ō2 s where the observation sequences are of length at most T − 1, and both sequences in any pair are of the same length. The decision problem is stated as follows. Given a DEC-POMDP D = S A 1 A 2 P R 1 2 O T K , is there a joint policy for which V T s 0 equals or exceeds K? We let DEC-POMDP m and DEC-MDP m denote the decision problems for the m-agent DEC-POMDP and the m-agent DEC-MDP, respectively. We conclude this section by showing a straightforward upper bound on the worst-case time complexity of DEC-POMDP m for any m ≥ 2. Because any DEC-MDP is trivially a DEC-POMDP, this upper bound also applies to DEC-MDP m . Theorem 1. For all m ≥ 2, DEC-POMDP m ∈ NEXP. Proof. We must show that a nondeterministic machine can solve any instance of DEC-POMDP m using at most exponential time. First, a joint policy can be "guessed" and written down in exponential time. This is because a joint policy consists of m mappings from local histories to actions; and because T < S , all histories have length less than S . A DEC-POMDP together with a joint policy can be viewed as a POMDP together with a policy, where the observations in the POMDP correspond to the observation m-tuples in the DEC-POMDP (one from each agent), and the POMDP actions correspond to m-tuples of DEC-POMDP actions (again, one from each agent). In exponential time, each of the exponentially many possible sequences of observations can be converted into a belief state (i.e., a probability distribution over the state set giving the probability of being in each state after seeing the given observation sequence). We note that every POMDP (Kaelbling et al. 1998 ) is equivalent to a "belief-state MDP" whose state set is the set of reachable belief states of the POMDP. The transition probabilities and expected rewards for the corresponding exponential-sized belief-state MDP can be computed in exponential time. Using standard MDP solution techniques (Puterman 1994) , it is possible to determine whether the guessed policy yields expected reward at least K in this belief-state MDP in time that is at most polynomial in the size of the belief-state MDP, which is exponential in the size of the original DEC-POMDP problem. Therefore, there exists an accepting computation path if and only if there is a policy that can achieve reward K. 5. Decentralized control of MDPs is NEXP-hard. We now turn our attention to proving that the upper bound just shown in Theorem 1 is tight-specifically, we show that NEXP is also a lower bound for the worst-case time complexity of decentralized problems by showing that any problem in the class NEXP can be reduced in polynomial time to a DEC-MDP 2 problem. It then follows that both DEC-MDP m and DEC-POMDP m are NEXPcomplete for any m ≥ 2. The proof of this lower bound is quite involved and will occupy most of the remainder of this paper. Each subsection of this section contains a piece of the development, and at the end of the section the main theorem is asserted. We begin by introducing the known NEXP-complete problem TILING used in the proof. We then present an overview of the proof and its major constituents. Next we present the reduction from TILING formally, and finally we prove that the reduction is correct. 5.1. The TILING problem. We can show this lower bound by reducing any NEXPcomplete problem to DEC-MDP 2 using a polynomial-time algorithm. For our reduction, we use an NEXP-complete problem called TILING (Lewis 1978 , Papadimitriou 1994 501), which is described as follows. We are given a board size n (represented compactly in binary), a set of tile types L = tile-0 tile-k , and a set of binary horizontal and vertical compatibility relations H V ⊆ L × L. A tiling is a mapping f 0 n − 1 × 0 n − 1 → L. A tiling f is consistent if and only if (a) f 0 0 = tile-0, and (b) for all x y f x y f x + 1 y ∈ H , and f x y f x y + 1 ∈ V . The decision problem is to determine, given L, H , V , and n, whether a consistent tiling exists. An example of a tiling instance and a corresponding consistent tiling is shown in Figure 2 . H = V = L = n = 4 a consistent tiling 0 1 2 3 0 1 2 3 0 1 2 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 1 1 2 2 1 1 2 2 2 2 2 2 2 2 Figure 2. An example of a tiling instance. In the remainder of this section, we assume that we have fixed an arbitrarily chosen instance of the tiling problem, so that L, H , V , and n are fixed. We then construct an instance of DEC-MDP that is solvable if and only if the selected tiling instance is solvable. We note that the DEC-MDP instance must be constructible in time polynomial in the size of the tiling instance (which in particular is logarithmic in the value of n), which will require the DEC-MDP instance to be at most polynomially larger than the tiling instance. 5.2. Overview of the reduction. The basic idea of our reduction is to create a twoagent DEC-MDP that randomly selects two tiling locations bit by bit, informing one agent of the first location and the other agent of the second location. The agents' local policies are observation-history based, so the agents can base their future actions on the tiling locations given to them. After generating the locations, the agents are simultaneously "queried" (i.e., a state is reached in which their actions are interpreted as answers to a query) for a tile type to place at the location given. We design the DEC-MDP problem so that the only way the agents can achieve nonnegative expected reward is to base their answers to the query on a single, jointly understood tiling that meets the constraints of the tiling problem. This design is complicated because the DEC-MDP state set itself cannot remember which tiling locations were selected (this would cause exponential blowup in the size of the state set, but our reduction must expand the problem size at most polynomially-we note that the tiling grid itself is not part of the tiling problem size, only the compactly represented grid size n is in the problem specification); the state will contain only certain limited information about the relative locations of the two tile positions. The difficulty of the design is also increased by the fact that any information remembered about the specified tiling locations must be shared with at least one of the agents to satisfy the joint observability requirement. To deal with these two issues, we have designed the DEC-MDP to pass through the following phases (a formal description follows later): Select Phase: Select two bit indices and values, each identifying a bit position and the value at that position in the location given to one of the agents. These are the only bits that are remembered in the state set from the locations given to the agents in the next phase-the other location bits are generated and forgotten by the process. The bit values remembered are called value-1 and value-2, and the indices to which these values correspond are called index-1 and index-2. Bit index-1 of the address given to agent 1 will have the value value-1, likewise for index-2, value-2, and agent 2. Generate Phase: Generate two tile locations at random, revealing one to each agent. The bits selected in the above select phase are used, and the other location bits are generated at random and immediately "forgotten" by the DEC-MDP state set. Query Phase: Query each agent for a tile type to place in the location that was specified to that agent. These tile types are remembered in the state. Echo Phase: Require the agents to echo the tile locations they received in the generate phase bit by bit. To enforce the accuracy of these location echoes, the DEC-MDP is designed to yield a negative reward if the bit remembered from the original location generation is not correctly echoed (the DEC-MDP is designed to ensure that each agent cannot know which bit is being checked in its echoes). As the agents echo the bits, the process computes state information representing whether the locations are equal or adjacent horizontally or vertically, and whether the agents' locations are both 0 0 (again, we cannot just remember the location bits because it would force an exponential state set). The echo phase allows us to compute state information about adjacency/equality of the locations after the tile types have been chosen, so that the agents' tile choices cannot depend on this information. This is critical in making the reduction correct. Test Phase: Check whether the tile types provided in the query phase come from a single consistent tiling. In other words, check that if the agents were asked for the same location they gave the same tile types during query, if they were asked for adjacent locations they gave types that satisfy the relevant adjacency constraints, and if the agents were both queried for location 0 0 they both gave tile type tile-0. The process gives a zero reward only if the tile types selected during the query phase meet any applicable constraints as determined by the echoed location bits. Otherwise, a negative reward is obtained. Note that because we are designing a DEC-MDP, we are required to maintain joint observability: The observations given to the agents at each time step must be sufficient to reconstruct all aspects of the DEC-MDP state at that time step. In particular, the bit indices and values selected in the select phase must be known to the agents (jointly), as well as the information computed in the echo phase regarding the relative position of the two locations. We achieve this joint observability by making all aspects of the DEC-MDP state observable to both agents, except for the indices and values selected in the select phase and the tile types that are given by the agents (and stored by the process) during the query phase. Each agent can observe which bit index and value are being remembered from the other agent's location, and each agent can observe the stored tile type it gave (but not the tile type given by the other agent). Because each agent can see what bit is saved from the other agent's location, we say that one location bit of each agent's location is visible to the other agent. We call the five phases just described "select," "generate," "query," "echo," and "test" in the development below. A formal presentation of the DEC-MDP just sketched follows below, but first we outline the proof that this approach represents a correct reduction.
 Overview of the correctness proof. Here we give an overview of our argument that the reduction sketched above is correct in the sense that there exists a policy that achieves expected total reward zero at the start state if and only if there is a solution to the tiling problem we started with. It is straightforward to show that if there exists a consistent tiling there must exist a policy achieving zero reward. The agents need only agree on a consistent tiling ahead of time and base their actions on the agreed-on tiling (waiting during selection and generation, giving the tile type present at the generated location during query, faithfully echoing the generated location during echo, and then waiting during test, at each point being guaranteed a zero reward by the structure of the problem). Note that it does not matter how expensive it might be to find and represent a consistent tiling or to carry out the policy just described because we are merely arguing for the existence of such a policy. We now outline the proof of the harder direction, that if there is no consistent tiling then there is no policy achieving expected reward zero. Note that because all rewards are nonpositive, any chance of receiving any negative reward forces the expected total reward to be negative. Consider an arbitrary policy that yields expected reward zero. Our argument rests on the following claims, which will be proved as lemmas in §5.5: Claim 1. The policy must repeat the two locations correctly during the echo phase. Claim 2. When executing the policy, the agents' selected actions during the query phase determine a single tiling, as follows. We define a query situation to be dangerous to an agent if and only if the observable bit value of the other agent's location (in the observation history) agrees with the bit value at the same index in the agent's own location (so that as far as the agent in danger knows, the other agent is being queried about the same location). During dangerous queries, the tile type selected by the agent in danger must depend only on the location queried (and not on the index or value of the bit observed from the other agent, on any other observable information, or on which agent is selecting the tile type). The agents' selected actions for dangerous queries thus determine a single tiling. Claim 3. The single tiling from Claim 2 is a consistent tiling. Claim 3 directly implies that if there is no consistent tiling, then all policies have negative expected reward, as desired. 5.4. Formal presentation of the reduction. Now we give the two-agent DEC-MDP D = S A 1 A 2 P R 1 2 O T K that is constructed from the selected tiling instance L H V n . We assume throughout that n is a power of two. It is straightforward to modify the proof to deal with the more general case-one way to do so is summarized briefly in Appendix A. 5.4.1. The state set. We describe the state set S of D below by giving a sequence of finite-domain "state variables" and then taking the state set to be the set of all possible assignments of values to the state variables. The finite-state automaton. One of the state variables will be maintained by a finitestate automaton (FSA), described in Appendix A. This variable, called rel-pos because it maintains a record of the relative position of the two location addresses echoed by the agents in the echo phase, can take on values from the state set of the FSA. Appendix A describes state set Q and also defines two functions (FSANEXT and FSA) based on the underlying FSA. These functions allow us to refer to the critical FSA behaviors here in our reduction while deferring most other details of the FSA to Appendix A. The function FSANEXT updates the state variable rel-pos based on one more bit of echoed location from each agent. The function FSA updates the state variable rel-pos based on a sequence of echoed location bits from each agent-so FSA is defined as a repeated application of the function FSANEXT. Appendix A also describes distinguished subsets of the FSA state set Q called apart, equal, hor, and ver representing possible relative positions for pairs of locations (not adjacent or equal, equal, horizontally adjacent, and vertically adjacent, respectively). These subsets are used below in defining the transition and reward behavior of the DEC-MDP D. Appendix A also defines the initial state q 0 of the FSA. The state variables. We now list the state variables defining the state set for D. We list the variables in three groups: the first group is observable to both agents, the second group only to agent 1, and the third group only to agent 2. These restrictions on observability are described in §5. We write a state by enumerating its variable values, e.g., as follows: gen 3 yes q 0 4 0 1 tile-1 5 1 0 tile-3 ∈ S. Semicolons are used to group together variables that have the same observability properties. We can represent sets of states by writing sets of values in some of the components of the tuple rather than just values. The \* symbol is used to represent the set of all possible values for a component. We sometimes use a state variable as a function from states to domain values for that variable. For instance, if q matches gen \* \* \* \* \* \* \* \* \* \* \* , then we will say phase q = gen. The initial state s 0 is as follows: select 0 yes q 0 0 0 0 tile-0 0 0 0 tile-0 . 5.4.2. The action sets and table of transition probabilities. We must allow "wait" actions, "zero" and "one" actions for echoing location address bits, and tile-type actions from the set of tile types L for answering during the query phase. We therefore take the action sets A 1 = A 2 to be wait 0 1 ∪ L. We give the transition distribution P s a 1 a 2 s for certain action pairs a 1 a 2 for certain source states s. For any source-state/action-pair combination not covered by the description below, the action pair is taken to cause a probability 1.0 self-transition back to the source state. The combinations not covered are not reachable from the initial state under any joint policy. Also, we note that the FSA-controlled state component rel-pos does not change from its initial state q 0 until the echo phase. Select phase. This is the first step of the process. In this step, the process chooses, for each agent, which of that agent's bits it will be checking in the echo phase. The value of that bit is also determined in this step. Transition probabilities when phase = select are given as follows: P s a 1 a 2 s = 1 4 log n 2 in the following situations: s = s 0 = select 0 yes q 0 0 0 0 tile-0 0 0 0 tile-0 s = gen 0 yes q 0 i 2 v 2 0 tile-0 i 1 v 1 0 tile-0 i 1 i 2 ∈ 0 2 log n − 1 and v 1 v 2 ∈ 0 1 Generate phase. During these steps, the two tile positions are chosen bit by bit. Note that we have to check for whether we are at one of the bits selected during the select phase, so that the value of the bit is the same as the value chosen during selection. Transition probabilities when phase = generate are given as follows. The second case describes the deterministic transition from the generate phase to the query phase. P s a 1 a 2 s = 1 h in the following situations: s = gen k yes q 0 i 2 v 2 \* tile-0 i 1 v 1 \* tile-0 where 0 ≤ k ≤ 2 log n − 1 s = gen k + 1 yes q 0 i 2 v 2 b 1 tile-0 i 1 v 1 b 2 tile-0 where b 1 = v 1 if k = i 1 else b 1 is either 0 or 1 b 2 = v 2 if k = i 2 else b 2 is either 0 or 1, and h is the number of allowed settings of b 1 b 2 from the previous two lines. P s a 1 a 2 s = 1 in the following situations: s = gen 2 log n yes q 0 i 2 v 2 \* tile-0 i 1 v 1 \* tile-0 and s = query 0 yes q 0 i 2 v 2 0 tile-0 i 1 v 1 0 tile-0 Query phase. The query phase consists of just one step, during which each agent chooses a tile type. Transition probabilities when phase = query are given as follows: P s a 1 a 2 s = 1 in the following situations: s = query 0 yes q 0 i 2 v 2 0 tile-0 i 1 v 1 0 tile-0 t 1 = a 1 if a 1 ∈ L tile-0 otherwise t 2 = a 2 if a 2 ∈ L tile-0 otherwise and s = echo 0 yes q 0 i 2 v 2 0 t 1 i 1 v 1 0 t 2 Echo phase. During the echo phase the agents are asked to repeat back the addresses seen in the generate phase, and information about the relative position of the addresses is calculated by the FSA described in Appendix A and recorded in the state. The FSA is accessed here using the function FSANEXT described in Appendix A. Transition probabilities when phase = echo are given as follows: P s a 1 a 2 s = 1 in the following situations: P s a 1 a 2 s = 1 in the following situations: s = test 0 \* \* \* \* 0 \* \* \* 0 \* and s = test 0 yes q 0 0 0 0 tile-0 0 0 0 tile-0 5.4.3. The reward function. We now describe the reward function for D. The reward R s a 1 a 2 s given when transitioning from state s to state s taking action pair a 1 a 2 is −1 in any situation except those situations matching one of the following patterns. Roughly, we give zero reward for waiting during select and generate, for answering with a tile type during query, for echoing a bit consistent with any remembered information during echo, and for having given tile types satisfying the relevant constraints during test. The relevant constraints during test are determined by the rel-pos state component computed by the FSA during the echo phase. s = echo k o q i 2 v 2 0 t 1 i 1 v 1 0 t 2 b 1 = a 1 if a 1 ∈ 0 1 0 otherwise b 2 = a 2 if a 2 ∈ 0 1 0 otherwise s = p k o FSANEXT q b 1 b 2 i 2 v 2 0 t 1 i 1 v 1 0 t 2 where p k = echo k + 1 for 0 ≤ k < 2 log n − 1 test 0 for k = 2 log R s a 1 a 2 s = 0 if and only if one of the following holds: The first four component fields of each state description are fully visible to both agents. The last eight state component fields are split into two groups of four, each group visible to only one agent. We therefore take the agent 1 observations 1 to be partial assignments to the following state variables: phase, index, origin, rel-pos, index-2, value-2, pos-bit-1, and tile-sel-1. Similarly, the observations 2 are partial assignments to the following state variables: phase, index, origin, rel-pos, index-1, value-1, pos-bit-2, and tile-sel-2. The observation distribution O s a 1 a 2 s o 1 o 2 simply reveals the indicated portion of the just-reached state s to each agent deterministically. We say that an observation sequence is p-phase if the sequence matches the pattern \* p \* \* \* \* \* \* \* , where the first "\*" stands for any observation sequence. Here, p can be any of gen, query, echo, or test. We take the horizon T to be 4 log n + 4, because the process spends one step in each of the select, query, and test phases, 2 log n + 1 steps in the generate phase, and 2 log n steps in the echo phase. We take the threshold value K to be 0. This completes the construction of the DEC-MDP by polynomial-time reduction from the selected tiling instance. An example of a zero-reward trajectory of the process is shown in Figure 3 . We now turn to correctness.
 Formal correctness argument. Next we show that the reduction presented above is indeed correct. Our main claim is that there exists a policy that achieves expected total reward zero at the start state if and only if there is a solution to the tiling problem we started with. To make our notation easier to read, we define the following abbreviations. Definition. Given an observation sequence ō1 over 1 , we write loc 1 ō1 for the location value represented by the bits transmitted to agent 1 in the generate phase of the process. We note that during the select and generate phases this value may be only partially specified (because not all of the bits have been generated). More precisely, loc 1 ō1 = b k • • • b 0 , where the b i values are chosen by the first match of the following sequence in ō1 (with k as large as possible while allowing a match): gen 1 \* \* \* \* b 0 \* • • • gen k + 1 \* \* \* \* b k \* We define loc 2 ō2 similarly. In addition, we define bit i l to be b i , where b k b 0 is the binary coding of the (possibly only partially specified) location l-we take bit i l to be undefined if the bit i is not specified in location l. By abuse of notation we treat loc 1 ō1 (or loc 2 ō2 ) as a tiling location x y sometimes (but only when it is fully specified) and as a bit string at other times. The easier direction of correctness is stated in the following lemma, which is formally proven in Appendix B. Lemma 1. If there exists a consistent tiling, then there must exist a policy achieving zero expected total reward. phase = gen index = 0 index-2 = 1 value-2 = 1 index-1 = 2 value-1 = 1 phase = gen index = 1 pos-bit-1 = 1 index-2 = 1 value-2 = 1 pos-bit-2 = 0 index-1 = 2 value-1 = 1 phase = gen index = 2 pos-bit-1 = 0 index-2 = 1 value-2 = 1 pos-bit-2 = 1 index-1 = 2 value-1 = 1 phase = gen index = 3 pos-bit-1 = 1 index-2 = 1 value-2 = 1 pos-bit-2 = 1 index-1 = 2 value-1 = 1 phase = gen index = 4 pos-bit-1 = 0 index-2 = 1 value-2 = 1 pos-bit-2 = 0 index-1 = 2 value-1 = 1 phase = query index-2 = 1 value-2 = 1 index-1 = 2 value-1 = 1 phase = echo index = 0 origin = yes rel-pos = q 0 index-2 = 1 value-2 = 1 tile-sel-1 = tile-0 index-1 = 2 value-1 = 1 tile-sel-2 = tile-1 phase = echo index = 1 origin = no rel-pos = q 1 index-2 = 1 value-2 = 1 tile-sel-1 = tile-0 index-1 = 2 value-1 = 1 tile-sel-2 = tile-1 phase = select phase = echo index = 2 origin = no rel-pos = q 2 phase = echo index = 3 origin = no rel-pos = q 3 phase = test origin = no rel-pos = q 4 ∈ hor tile-sel-1 = tile-0 tile-sel-2 = tile-1 index-2 = 1 value-2 = 1 tile-sel-1 = tile-0 index-1 = 2 value-1 = 1 tile-sel-2 = tile-1 index-2 = 1 value-2 = 1 tile-sel-1 = tile-0 index-1 = 2 value-1 = 1 tile-sel-2 = tile-1 1 1 1 2 1 0 1 1 0 0 0 1
 Select The process chooses indices to be checked in the echo phase.
 Generate The process generates address bits.
 Query The agents choose tiles.
 Echo The agents echo address bits. The process checks one bit for each agent and keeps track of information about the addresses echoed.
 Test The process checks that the relevant constraints are satisfied. phase = test 0 1 Figure 3 . An example of a zero-reward trajectory of the process constructed from the tiling example given in Figure 2 . The total reward is zero because the agents echo the "checked" bits correctly and choose tiles that do not violate any constraints, given the two addresses that are echoed. (For clarity, some state components are not shown.) We now discuss the more difficult reverse direction of the correctness proof. In the following subsections, we prove Claims 1 to 3 from §5.3 to show that if there is a policy which achieves nonnegative expected reward for horizon T , then there is also a consistent tiling. Throughout the remainder of the proof, we focus on a fixed, arbitrary policy that achieves zero expected reward. Given this policy, we must show that there is a consistent tiling. 5.5.1. Proof of Claim 1. Before proving the first claim, we need to formalize the notion of "faithfulness during echo." Definition. A pair of observation sequences ō1 ō2 over 1 and 2 , respectively, is said to be reachable if P s 0 ō1 ō2 s is nonzero for some state s. An observation sequence ō1 over 1 is said to be reachable if there exists an observation sequence ō2 over 2 such that the pair of observation sequences ō1 ō2 is reachable. Likewise, ō2 over 2 is reachable if there is some ō1 over 1 such that, ō1 ō2 is reachable. Definition. The policy = 1 2 is faithful during echo if it satisfies both of the following conditions for all indices k in 0 2 log n − 1 , and all reachable observation sequence pairs ō1 ō2 : 1 Much of our proof revolves around showing that the reachability of a pair of observation sequences is not affected by making certain changes to the sequences. We focus without loss of generality on changes to the observations of agent 2, but similar results hold for agent 1. The changes of particular interest are changes to the (randomly selected) value of the index-1 state component-this is the component that remembers which bit of agent 1's queried location will be checked during echo. It is important to show that agent 1 cannot determine which bit is being checked before that bit has to be echoed. To show this, we define a way to vary the observation sequences seen by agent 2 (preserving reachability) such that without changing the observations seen by agent 1 we have changed which agent 1 address bit is being checked. We now present this approach formally. Definition. We say that an observation sequence ō1 over 1 is superficially consistent if the values of the index-2 component and the value-2 component do not change throughout the sequence, and the value of the tile-sel-1 component is tile-0 for generate-phase and query-phase observations and some fixed tile type in L for echo-phase and test-phase observations. Given a superficially consistent observation sequence ō1 , we can write index-2 ō1 and value-2 ō1 to denote the value of the indicated component throughout the sequence. In addition, we can write tile-sel-1 ō1 to denote the fixed tile type for echo-phase and testphase observations (we take tile-sel-1 ō1 to be tile-0 if the sequence contains no echo-or test-phase observations). Corresponding definitions hold for observation sequences over 2 , replacing "1" by "2" and "2" by "1" throughout. Note that any reachable observation sequence must be superficially consistent, but the converse is not necessarily true. The following technical definition is necessary so that we can discuss the relationships between observation sequences without assuming reachability. Definition. We say that two superficially consistent observation sequences ō1 over 1 and ō2 over 2 are compatible if bit index-1 ō2 loc 1 ō1 = value-1 ō2 or this bit of loc 1 ō1 is not defined, and bit index-2 ō1 loc 2 ō2 = value-2 ō1 or this bit of loc 2 ō2 is not defined. Definition. Given an index i in 0 2 log n − 1 , a reachable pair of observation sequences ō1 ō2 , and an observation sequence ō 2 over 2 , we say that ō 2 is an i-index variant of ō2 relative to ō1 when ō 2 is any sequence compatible with ō1 that varies from ō2 only as follows: 1. index-1 has been set to i throughout the sequence, 2. value-1 has been set to the same value v throughout the sequence, 3. pos-bit-2 can vary arbitrarily from ō2 , and 4. For any echo-or test-phase observations, tile-sel-2 has been set to the tile type selected by on the query-phase prefix of ō 2 , or to tile-0 if selects a non-tile-type action on that query. If the pos-bit-2 components of ō2 and ō 2 are identical, we say that ō 2 is a same-address index variant of ō2 . We note that, given a reachable pair of observation sequences ō1 ō2 , there exists an iindex variant of ō2 relative to ō1 , for any i in 0 2 log n − 1 . This remains true even if we allow only same-address index variants. The following technical lemma asserts that index variation as just defined preserves reachability under very general conditions. Its proof is deferred to Appendix C. Lemma 2. Suppose is faithful during echo for the first k bits of the echo phase for some k. Let ō1 ō2 be a reachable pair of observation sequences that end no later than the kth bit of echo (i.e., the last observation in the sequence has index no greater than k if it is an echo-phase observation) and let ō 2 be an i-index variant of ō2 relative to ō1 for some i. If the observation sequences are echo-phase or test-phase, then we require that the index variation be a same-address variation. We can then conclude that ō1 ō 2 is reachable. We are now ready to assert and prove Claim 1 from §5.3.
 Lemma 3 (Claim 1). is faithful during echo. Proof. We argue by induction that faithfully echoes all 2 log n address bits. As an inductive hypothesis, we assume that faithfully echoes the first k bits, where 0 ≤ k < 2 log n. Note that if k equals zero, this is a null assumption, providing an implicit base case to our induction. Now suppose for contradiction that lies during the k + 1st step of the echo phase. Then one of the agents' policies must incorrectly echo bit k + 1; we assume without loss of generality that this is so for agent 1, i.e., under some reachable observation sequence pair ō1 ō2 of length 2 log n + k + 2, the policy 1 dictates that the agent choose action 1 − bit k loc 1 ō1 . Lemma 2 implies that the observation sequence pair ō1 ō 2 is also reachable, where ō 2 is any same-address k + 1-index variant of ō2 relative to ō1 . Because all the agent 1 observations are the same for both ō1 ō2 and ō1 ō 2 , when the latter sequence occurs, agent 1 chooses the same action 1−bit k loc 1 ō1 as given above for the former sequence, and a reward of −1 is obtained (because in this case it is bit k + 1 that is checked). Therefore, the expected total reward is not zero, yielding a contradiction. 5.5.2. Proof of Claim 2. Now we move on to prove Claim 2 from §5.3. We show that can be used to define a particular mapping from tile locations to tile types based on "dangerous queries." In §5.3, we defined an agent 1 observation sequence to be "dangerous" if it reveals a bit of agent 2's queried location that agrees with the corresponding bit of agent 1's queried location (and vice versa for agent 2 observation sequences). We now present this definition more formally. Definition. A query-phase observation sequence ō1 over 1 is dangerous if it is reachable and bit index-2 ō1 loc 1 ō1 = value-2 ō1 Likewise, a query-phase sequence ō2 over 2 is dangerous if it is reachable and bit index-1 ō2 loc 2 ō2 = value-1 ō2 . Dangerous query-phase sequences are those for which the agent's observations are consistent with the possibility that the other agent has been queried on the same location. We note that for any desired query location l, and for either agent, there exist dangerous observation sequences ō such that loc k ō = l. Moreover, such sequences still exist when we also require that the value of index-k ō be any particular desired value (where k is the number of the nonobserving agent). Lemma 4. Two same-length query-phase observation sequences, ō1 over 1 and ō2 over 2 , are reachable together as a pair if and only if they are compatible and each is individually reachable. Proof. The "only if" direction of the theorem follows easily-the reachability part follows from the definition of reachability, and the compatibility of jointly reachable sequences follows by a simple induction on sequence length given the design of D. The "if" direction can be shown based on the following assertions. First, a generate-phase observation sequence (for either agent) is reachable if and only if it matches the following pattern: gen 0 yes q 0 i v \* tile-0 • • • gen k yes q 0 i v \* tile-0 for some k i, and v; this can be established by a simple induction on sequence length based on the design of D. A similar pattern applies to the query phase. Given two compatible reachable sequences of the same length, ō1 and ō2 , we know by the definition of reachability that there must be some sequence ō 2 such that ō1 ō 2 is reachable. But given the patterns just shown for reachable sequences, ō2 and ō 2 can differ only in their choice of i, v, and in the address given to agent 2 via the pos-bit-2 component. It follows that ō2 is an i-index variant of ō 2 relative to ō1 , for some i. Lemma 2 then implies that the pair ō1 ō2 is reachable as desired. Lemma 5 (Claim 2). There exists a mapping f from tiling locations to tile types such that f loc i ō = i ō on all dangerous queries ō over i for both agents (i ∈ 1 2 ). Proof. To prove the lemma, we prove that for any two dangerous query sequences ōi and ōj over i and j , respectively, for arbitrary i j ∈ 1 2 , if loc i ōi = loc j ōj = l, then i ōi = j ōj . This implies that for any such ōi we can take f l = i loc i ōi to construct f satisfying the lemma. Suppose not. Then there must be a counterexample for which i = j-because given a counterexample for which i = j, either ōi or ōj must form a counterexample with any dangerous query ōk over 1−i such that loc 1−i ōk = l. We can now consider a counterexample where i = j. Let ō1 and ō2 be dangerous (and thus reachable) sequences over 1 and 2 , respectively, such that loc 1 ō1 = loc 2 ō2 but 1 ō1 = 2 ō2 . Note that loc 1 ō1 = loc 2 ō2 together with the fact that ō1 and ō2 are dangerous implies that ō1 and ō2 are compatible and thus reachable together (using Lemma 4). The faithfulness of echo under (proven in Claim 1, Lemma 3) then ensures that the extension (there is a single extension because D is deterministic in the echo and test phases) of these observation sequences by following to the test phase involves a faithful echo. The correctness of the FSA construction in Appendix A then ensures that the rel-pos state component after this extension will have the value equal. The reward structure of D during the test phase then ensures that to avoid a negative reward the tile types given during query, 1 ō1 and 2 ō2 , must be the same, contradicting our choice of ō1 and ō2 above and thus entailing the lemma. 5.5.3. Proof of Claim 3 and our main hardness theorem. We now finish the proof of our main theorem by proving Claim 3 from §5.3. We start by showing the existence of a useful class of pairs of dangerous observation sequences that are reachable together. Lemma 6. Given any two locations l 1 and l 2 sharing a single bit in their binary representations, there are dangerous observation sequences ō1 over 1 and ō2 over 2 such that: loc 1 ō1 = l 1 loc 2 ō2 = l 2 and ō1 ō2 is reachable Proof. It is straightforward to show that there exist dangerous observation sequences ō1 over 1 and ō2 over 2 such that loc 1 ō1 = l 1 and loc 2 ō2 = l 2 as desired. In these sequences, both index-1 and index-2 are set throughout to the index of a single bit shared by l 1 and l 2 . Because this bit is in common, these sequences are compatible, so by Lemma 4 they are reachable together. Lemma 7 (Claim 3). The mapping f defined in Lemma 5 is a consistent tiling. Proof. We prove the contrapositive. If the mapping f is not a consistent tiling, then there must be some particular constraint violated by f . It is easy to show that any such constraint is tested during the test phase if loc 1 ō1 and loc 2 ō2 have the appropriate values. (The faithfulness during echo claim proven in Lemma 3 implies that the origin and rel-pos components on entry to the test phase will have the correct values for comparing the two locations). For example, if a horizontal constraint fails for f , then there must be locations i j and i + 1 j such that the tile types f i j f i + 1 j are not in H ; because these two locations share a bit (in fact, all the bits in j, at least), Lemma 6 implies that there are dangerous ō1 and ō2 with loc 1 ō1 = i j and loc 2 ō2 = i + 1 j that are reachable together. During the test phase, the tile-sel-1 and tile-sel-2 state components are easily shown to be f i j and f i + 1 j , and then the definition of the reward function for D ensures a reachable negative reward. The arguments for the other constraints are similar. Claim 3 immediately implies that there exists a consistent tiling whenever there exists a policy achieving zero expected total reward. This completes the proof of the other direction of our main complexity result. We have thus shown that there exists a policy that achieves expected reward zero if and only if there exists a consistent tiling, demonstrating that DEC-MDP 2 is NEXP-hard. Theorem 2. DEC-MDP 2 is NEXP-hard. Corollary 1. For all m ≥ 2, both DEC-POMDP m and DEC-MDP m are NEXPcomplete. 6. Discussion. Using the tools of worst-case complexity analysis, we analyzed two variants of decentralized control of Markov decision processes. Specifically, we proved that the finite-horizon m-agent DEC-POMDP problem is NEXP-hard for m ≥ 2 and the finitehorizon m-agent DEC-MDP problem is also NEXP-hard for m ≥ 2. When the horizon is limited to be less than the number of states, the problems are NEXP-complete. The results have some theoretical implications. First, unlike the MDP and POMDP problems, the problems we studied provably do not admit polynomial-time algorithms, because P = NEXP. Second, we have drawn a connection between work on Markov decision processes and the body of work in complexity theory that deals with the exponential jump in complexity due to decentralization (Peterson and Reif 1979, Babai et al. 1991) . There are also more direct implications for researchers trying to solve problems of this nature. Consider the growing body of work on algorithms for obtaining exact or approximate solutions for POMDPs (e.g., Jaakkola et al. 1995 , Cassandra et al. 1997 , Hansen 1998 , Meuleau et al. 1999 , Lusena et al. 1999 , Zhang 2001 . For the finite-horizon case, we now have stronger evidence that there is no way to efficiently convert a DEC-MDP or DEC-POMDP into an equivalent POMDP and solve it using established techniques. This knowledge can provide direction for research on the development of algorithms for these problems. Finally, consider the infinite-horizon versions of the aforementioned problems. It has recently been shown that the infinite-horizon POMDP problem is undecidable (Madani et al. 1999 ) under several different optimality criteria. Because a POMDP is a special case of a DEC-POMDP, the corresponding infinite-horizon DEC-POMDP problems are also sets of reject states. From these sets we construct distinguished sets apart, equal, hor, and ver of cross-product automaton states as follows: apart = reject 1 × reject 2 × reject 3 equal = accept 1 × reject 2 × reject 3 hor = reject 1 × accept 2 × reject 3 ver = reject 1 × reject 2 × accept 3 The rest of the automaton's states comprise the set Q . Let q 1 0 , q 2 0 , and q 3 0 denote the start states of the three component automata. We define the start state of the cross-product automaton to be the state q 0 = q 1 0 q 2 0 q 3 0 . We now define two functions based on this automaton that are needed in the main body of the proof. One function takes as input the state of the automaton and a bit pair and returns the next state of the automaton. The second function takes as input a pair of bit strings of the same length and returns the state that the automaton will be in starting from its initial state and reading symbols formed by the corresponding bits in the two strings in sequence. Definition. For q ∈ Q and a 1 a 2 ∈ 0 1 , FSANEXT q a 1 a 2 = q , where q ∈ Q is the resulting state if the automaton starts in state q and reads the input symbol a 1 a 2 . Definition. The function FSA is defined inductively as follows: FSA = q 0 FSA b 0 • • • b k+1 c 0 • • • c k+1 = FSANEXT FSA b 0 • • • b k c 0 • • • c k b k+1 c k+1 Note that the range of FSA for inputs of length 2 log n is apart ∪ equal ∪ hor ∪ ver. In the proof given in §5 we assumed that the TILING grid size n was an exact power of two. We note that the proof can be adapted by adding two components to the crossproduct FSA described here, where the two new components are both FSAs over the same alphabet. The first new component accepts a string only when both x 1 and y 1 (as described above) are less than n (so that the tiling location represented by x 1 y 1 is in the tiling grid). The second new component behaves similarly for x 2 y 2 . The DEC-MDP can then be constructed using the smallest power of two larger than n but modified so that whenever either new component of the FSA rejects the (faithfully) echoed bit sequences, then the process gives a zero reward, regardless of the tile types returned during query. Each new component can be viewed as an FSA over a {0,1} alphabet, because each focuses either on just the agent 1 echoes or on just the agent 2 echoes. We describe the FSA for checking that x 1 is less than n; constructing the two components is then straightforward. Suppose that k = log n is the number of bits in the binary representation of n and that the bits themselves are given from least to most significant as b 1 • • • b k . Suppose also that there are j different bits equal to 1 among b 1 • • • b k and that these bits are at indices i 1 i j . We can then write a regular expression for detecting that its input of k bits from least to most significant represents a number in binary that is strictly less than n: 0 + 1 i 1 −1 0 b i 1 +1 • • • b k + 0 + 1 i 2 −1 0 b i 2 +1 • • • b k + • • • + 0 + 1 i j −1 0 b i j +1 • • • b k It can be shown that this regular expression has an equivalent FSA of size O log n 2 . Appendix B: Proof of Lemma 1. We assume there exists at least one consistent tiling, and we select a particular such mapping f . We describe a policy = 1 2 that achieves zero expected reward at the initial state. 1 is a mapping from sequences of observations in part of the mapping 1 is specified below-any unspecified observation sequence maps to the action wait. We note that 1 and 2 are symmetric. The local policy 2 is defined identically to 1 except that loc 1 is replaced by loc 2 . We first characterize the set of all reachable states from s 0 under the policy . We then note that taking the action prescribed by from any of these states yields a reward of zero. Thus, V T s 0 = 0. It is straightforward to show by induction that P s 0 ō1 ō2 s is zero except where one of the following patterns applies: • s = s 0 = select 0 yes q 0 0 0 0 tile-0 0 0 0 tile-0 . • s = gen k yes q 0 i 2 v 2 \* tile-0 i 1 v 1 \* tile-0 , where 0 ≤ k ≤ 2 log n, k ≤ i 1 or v 1 = bit i 1 loc 1 ō1 , and k ≤ i 2 or v 2 = bit i 2 loc 2 ō2 . • s = query 0 yes q 0 i 2 v 2 \* tile-0 i 1 v 1 \* tile-0 , where v 1 = bit i 1 loc 1 ō1 , and v 2 = bit i 2 loc 2 ō2 . • s = echo k o q i 2 v 2 \* t 1 i 1 v 1 \* t 2 , where 0 ≤ k ≤ 2 log n − 1, v 1 = bit i 1 loc 1 ō1 , v 2 = bit i 2 loc 2 ō2 , t 1 = f loc 1 ō1 , t 2 = f loc 2 ō2 , o = yes if and only if b j = c j = 0 for 0 ≤ j ≤ k − 1, with b j and c j as in the next item, and q = FSA b 0 • • • b k−1 c 0 • • • c k−1 , with b 0 • • • b k−1 and c 0 • • • c k−1 the least significant bits of loc 1 ō1 and loc 2 ō2 , respectively. • s = test \* o r \* \* \* t 1 \* \* \* t 2 , where o = yes if and only if loc 1 ō1 = 0 0 and loc 2 ō2 = 0 0 , r = FSA loc 1 ō1 loc 2 ō2 , t 1 = f loc 1 ō1 , and t 2 = f loc 2 ō2 . It can then be shown that the reward for any action prescribed by the policy given any of these reachable state/observation sequence combinations is zero given that f is a consistent tiling. Appendix C: Proof of Lemma 2. We need some new notation to carry out this proof. Given a state s, a state component c and corresponding value v from the domain of c, we define the state "s with c set to v" (written s c = v ) to be the state s that agrees with s at all state components except possibly c and has value v for state component c. We also write ō1 j for the first j observations in the sequence ō1 , and likewise ō2 j and ō 2 j . 839 For any state s j reachable while observing ō1 j ō2 j , we define a state s j that we will show is reachable while observing ō1 j ō 2 j , as follows: s j = s j index-1 = index-1 ō 2 j value-1 = value-1 ō 2 j tile-sel-2 = tile-sel-2 ō 2 j We can now show by an induction on sequence length j that for any state s j such that P s 0 ō1 j ō2 j s j is nonzero, then P s 0 ō1 j ō 2 j s j is also nonzero. From this we can conclude that ō1 ō 2 is reachable, as desired. For the base case of this induction, we take j to be 1, so that the observation sequences involved all have length 1, ending in the generate phase with index equal to zero. Inspection of the definition of the transition probabilities P shows that changing index-1 and value-1 arbitrarily has no effect on reachability. For the inductive case, we suppose some state s j is reachable by ō1 j ō2 j , and that state s j is reachable by ō1 j ō 2 j . Let a 1 be ō1 j , a 2 be ō2 j , and a 2 be ō 2 j . We must show that for any state s j+1 such that P s j a 1 a 2 s j+1 is nonzero, P s j a 1 a 2 s j+1 is also nonzero, for s j+1 . This follows from the following observations: • When phase(s j ) is select or generate, neither agent 2's action a 2 nor the values of index-1(s j ) or value-1(s j ) have any affect on P s j a 1 a 2 s j+1 being nonzero, as long as either index-1 is not equal to j or pos-bit-1(s j+1 ) equals value-1(s j ). However, this last condition is ensured to hold of the index-1 and value-1 components of s j and s j by the compatibility of ō 2 with ō1 . • When phase(s j ) is query, the action a 2 must equal the tile-sel-2 state component of s j+1 by the definitions of s j+1 and "index variant," and changes to the index-1 and value-1 components have no effect on P s j a 1 a 2 s j+1 being nonzero during the query phase. • When phase(s j ) is echo, the actions a 2 and a 2 must be a faithful echo of the location address bit indicated by the index state component (because we have assumed as part of out inductive hypothesis that is faithful during echo for at least j bits), and this bit's value does not vary between ō2 and ō 2 because if the observation sequences reach the echo phase we have the assumption that these are same-address variants. Thus a 2 = a 2 during echo. Again, changes to the index-1 and value-1 components have no effect on P s j a 1 a 2 s j+1 being nonzero during the echo phase. Figure 1 . 1 Figure 1. The relationships among the models.
 4.4Observable to both agents: phase ∈ select gen query echo test Current phase of the process index ∈ 0 2 log n Index of next location bit to be generated/echoed origin ∈ yes no Eventually true if both tile locations are 0 0 rel-pos ∈ QRelative tile positions during echo-controlled by the FSA Observable only to agent 1: index-2 ∈ 0 2 log n − 1 Index of bit remembered for agent 2 value-2 ∈ 0 1Value of bit remembered for agent 2 pos-bit-1 ∈ 0 1Bit for transmitting tile position to agent 1 tile-sel-1 ∈ L Tile type selected by agent 1 in query Observable only to agent 2: index-1 ∈ 0 2 log n − 1 Index of bit remembered for agent 1 value-1 ∈ 0 1Value of bit remembered for agent 1 pos-bit-2 ∈ 0 1 Bit for transmitting tile position to agent 2 tile-sel-2 ∈ L Tile type selected by agent 2 in query
 n − 1 and o = yes if and only if o = yes and a 1 = a 2 = 0 Test phase. The test phase consists of just one step terminating the process in a zeroreward absorbing state.
 Select phase s = select \* \* \* \* \* \* \* \* \* \* \* and a 1 = a 2 = wait Generate phase s = gen \* \* \* \* \* \* \* \* \* \* \* and a 1 = a 2 = wait Query phase s = query \* \* \* \* \* \* \* \* \* \* \* and both a 1 ∈ L and a 2 ∈ L Echo phase s = echo k \* \* i 2 v 2 \* \* i 1 v 1 \* \* and a 1 a 2 ∈ 0 1 where a 1 = v 1 or k = i 1 and a 2 = v 2 or k = i 2 830 D. S. BERNSTEIN, R. GIVAN, N. IMMERMAN, AND S. ZILBERSTEIN Test Phase i s = test \* o equal \* \* \* t 1 \* \* \* t 1 and a 1 = a 2 = wait where o = no or t 1 = tile-0 Test Phase ii s = test \* \* hor \* \* \* t 1 \* \* \* t 2 and a 1 = a 2 = wait where t 1 t 2 ∈ H Test Phase iii s = test \* \* ver \* \* \* t 1 \* \* \* t 2 and a 1 = a 2 = wait where t 1 t 2 ∈ V Test Phase iv s = test \* \* apart \* \* \* \* \* \* \* \* and a 1 = a 2 = wait 5.4.4. Observations, threshold, and horizon.
 . 1 ō1 = bit k loc 1 ō1 when ō1 = \* \* \* \* \* \* \* \* • • • echo k \* \* \* \* \* \* . 2. 2 ō2 = bit k loc 2 ō2 when ō2 = \* \* \* \* \* \* \* \* • • • echo k \* \* \* \* \* \* . We say the policy lies during echo otherwise. If the two conditions listed above are satisfied for all indices k in 0 d −1 , where 0 ≤ d ≤ 2 log n, we say that the policy faithfully echoes the first d bits.
 1 ō1 = a 1 when one of the following holds: Select phase: ō1 = select \* \* \* \* \* \* \* and a 1 = wait Generate phase: ō1 = \* gen \* \* \* \* \* \* \* and a 1 = wait Query phase: ō1 = \* query \* \* \* \* \* \* \* and a 1 = f loc 1 ō1 Echo phase: ō1 = \* echo k \* \* \* \* \* \* and a 1 = bit k loc 1 ō1 Test phase: ō1 = \* test \* \* \* \* \* \* \* and a 1 = wait
 D. S. BERNSTEIN, R. GIVAN, N. IMMERMAN, AND S. ZILBERSTEIN
 to actions in A 1 , and 2 from sequences over 2 to actions in A 2 . Only the reachable

Daniel S. Bernstein
Robert Givan
Neil Immerman
Shlomo Zilberstein


Using AI ethically to tackle covid-19

Taking a principled approach is crucial to the successful use of AI in pandemic management, say Stephen Cave and colleagues I n a crisis such as the covid-19 pandemic, governments and health services must act quickly and decisively to stop the spread of the disease. Artificial intelligence (AI), which in this context largely means increasingly powerful data driven algorithms, can be an important part of that action-for example, by helping to track the progress of a virus or to prioritise scarce resources. 1 To save lives it might be tempting to deploy these technologies at speed and scale. Deployment of AI can affect a wide range of fundamental values, however, such as autonomy, privacy, and fairness. AI is much more likely to be beneficial, even in urgent situations, if those commissioning, designing, and deploying it take a systematically ethical approach from the start. Ethics is about considering the potential harms and benefits of an action in a principled way. For a widely deployed technology, this will lay a foundation of trustworthiness on which to build. Ethical deployment requires consulting widely and openly; thinking deeply and broadly about potential impacts; and being transparent about goals being pursued, trade-offs being made, and values guiding these decisions. In a pandemic, such processes should be accelerated, but not abandoned. Otherwise, two main dangers arise: firstly, the benefits of the technology could be outweighed by harmful side effects, and secondly, public trust could be lost. 2 The first danger is that the potential benefits increase the incentive to deploy AI systems rapidly and at scale, but also increase the importance of an ethical approach. The speed of development limits the time available to test and assess a new technology, while the scale of deployment increases any negative consequences. Without forethought, this can lead to problems, such as a onesize-fits-all approach that harms already disadvantaged groups. 3 Secondly, public trust in AI is crucial. For example, contact tracing apps rely on widespread adoption for their success. 4 Both technology companies and governments, however, struggle to convince the public that they will use AI and data responsibly. After controversy over, for example, the partnership between the AI firm DeepMind and the Royal Free London NHS Foundation Trust, privacy groups have warned against plans to allow increased access to NHS data. 5 Similarly, concerns have been raised in China over the Health QR code system's distribution of data and control to private companies. 6 Overpromising on the benefits of technology or relaxing ethical requirements, as has sometimes happened during this crisis, 5 both risk undermining long term trust in the reputation of the entire sector. Whether potential harms become obvious immediately or only much later, adopting a consistently ethical approach from the outset will put us in a much better position to reap the full benefits of AI, both now and in the future.
 Bringing together AI ethics and health ethics AI can broadly be defined as digital systems that can make complex decisions or recommendations on the basis of data inputs. This simple definition highlights three reasons why ethical challenges arise from such systems. Firstly, AI applications, particularly in healthcare, often require a lot of personal data, and so invoke all the concerns about responsible data management, such as privacy, consent, security, and ownership. 7 Secondly, AI systems are often used to automate decision making processes that were previously made by humans. This automation gives rise to ethical challenges, such as who is to be held accountable for these decisions, or how stakeholders can know which value judgments are guiding them. 8 For example, is the system optimising a commercial value, the interests of a government, or the health of the individual? These concerns can arise even when an AI system is only recommending a course of action because of automation bias-the propensity for people to suspend their own judgement and over-rely on automated systems. 9 Thirdly, the operations of AI systems are often unclear, owing to the complexity of the data or the algorithm (especially many powerful and popular algorithms used in machine learning). 10 This lack of clarity, as well as compounding problems of accountability, can make it hard to assess ethically relevant factors, such as unintended biases in the system or the robustness of results across different populations. 11 Ethical decision making is, of course, already an integral part of healthcare practice, where it is often structured according to the four pillars of biomedical ethics: beneficence, non-maleficence, autonomy, and justice. 12 When considering the use of AI in a public health setting, such as a pandemic, it might therefore be useful to consider how the distinctive challenges posed by AI pertain to these four well established principles. 8
 Beneficence It might seem obvious that the use of AI in managing a pandemic is beneficent: it is intended to save lives. A risk exists, however, that the vague promise that a new technology will "save lives" can be used as a blanket justification for interventions we might not otherwise consider appropriate, such as widespread deployment of facial recognition software. 13 Those developing or deploying
 KEY MESSAGES • AI based technologies promise benefits for tackling a pandemic like covid-19, but also raise ethical challenges for developers and decision makers • If an ethical approach is not taken, the risks increase of unintended harmful consequences and a loss of stakeholder trust • Ethical challenges from the use of AI systems arise because they often require large amounts of personal data; automate decisions previously made by humans; and can be highly complex and opaque • The four pillars of biomedical ethics-beneficence, non-maleficence, autonomy, and justice-are a helpful way of seeing how these challenges can arise in public health • Open and transparent communication with diverse stakeholder groups during development of AI systems is the best way of tackling these challenges such a system must be clear about whom their intervention will benefit and how. Only by making this explicit can one ensure that the intervention is proportionate to its benefit. 14 For example, if a data driven contact tracing app does not require large amounts of location data to be collected and stored indefinitely, it would not be proportionate to engage in large scale data gathering that we would normally find excessive. Even if some additional benefit could be had, one needs to consider whether this benefit is sufficient to justify creating such a database.
 Non-maleficence To avoid unintended harms from the use of AI in the management of a pandemic, it is important to carefully consider the potential consequences of proposed interventions. Some interventions-for example, imposing self-isolation-may cause mental health problems for those who are already vulnerable (eg, elderly people) or carry high economic costs for individuals. AI systems seek to optimise a particular objective function-that is, a mathematical function representing the goals it has been designed to achieve. Any potential harms not represented by this function will not be considered in the system's predictions. For example, some systems designed to inform the prioritisation of hospital resources are optimised to predict death from covid-19, 15 but not other possible harms for patients (eg, "long covid"). If these other harms do not correlate with the risk of fatality, deciding how to prioritise health resources purely based on this system might considerably aggregate harm (depending on the incidence and severity of the other harms). Additionally, as these systems will be widely deployed, they must reliably perform as expected across different populations and potentially changing conditions. Trying to rapidly develop AI systems while our understanding of the virus is still limited, and with less time than usual to ensure the quality and representativeness of the data used, risks creating systems based on simplifying assumptions and datasets that do not cover all real world cases. For instance, a recent systematic review of 145 prediction models for diagnosis of covid-19 (including 57 using AI for image analysis) found all to have a high risk of statistical bias. 11 Inaccurate diagnoses or inappropriate interventions arising from such models could cost more lives than they save.
 Autonomy The benefits of new technologies almost always depend on how they affect peoples' behaviour and decision making: from the precautions an individual chooses to take, to treatment decisions by healthcare professionals, and politicians' prioritisation of different policy responses. Respecting peoples' autonomy is therefore crucial. Evidence from across cultures and age groups shows that people feel the need to be in control and endorse the use of technology, or its influence on their behaviour is likely to be limited. 16 A particular challenge for AI systems is that they might affect patients, healthcare professionals, and other stakeholders in more subtle and individualised ways than, for example, a mask or vaccine, where the desired behaviours are obvious. Designers can help users to understand and trust AI systems so that they feel able to use them with autonomy. 17 For example, diagnostic support systems used by healthcare professionals in a pandemic should provide sufficient information about the assumptions behind, and uncertainty surrounding, a recommendation, so that it can be incorporated into their professional judgment.
 Justice Data driven AI systems can differentially affect different groups, as is well documented. 18 When data of sufficient quality for some groups are lacking, AI systems can become biased, often in ways which discriminate against already disadvantaged groups, such as racial and ethnic minorities. 19 For example, smartphone apps are increasingly heralded as tools for monitoring and diagnosis, such as the MIT-Harvard model for diagnosing covid-19 through the sound of coughs. 20 But access to smartphones is unevenly distributed between countries and demographics, with global smartphone penetration estimated in 2019 to be 41.5%. 21 This limits both whose data are used to develop such apps and who has access to the service. If care is not taken to detect and counteract any biases, using AI for pandemic management could worsen health inequalities. 22 Again, the speed and scale at which systems might be deployed in response to the pandemic exacerbate these risks, making foresight and vigilance all the more crucial. More broadly, when AI systems are proposed for a response to a pandemic, difficult trade-offs between values could be introduced. For example, leading UK public health officials argued for a centralised approach to data collection in the design of the NHS digital contact tracing app, arguing that machine learning could be applied to the resultant dataset to aid in disease prediction. Legal and security experts, however, argued for a decentralised approach, citing concerns about privacy and data security. 23 The UK chose to use a decentralised app. These are inherently value laden choices, about which reasonable people might disagree, and particular groups might have much greater reasons for concern (eg, owing to worry about surveillance or historic discrimination). Involving diverse communities in decisions, and being open about the values and trade-offs, will help to reduce these risks. 24 Ethics in practice: a participatory approach Politicians and public health officials are charged with final decisions about deployment of AI, and thus are responsible for ensuring these ethical challenges are met. But this requires that they draw on the expertise of designers, engineers, and healthcare experts, as well as the views of affected groups. No single checklist is available that these decision makers can mechanically follow to ensure that AI is used ethically and responsibly. Especially during a crisis, there will be some trade-off between the use of AI for good and the need to mitigate its harms. Public decision makers should not tackle these trade-offs alone but must communicate with diverse stakeholder groups to ensure decisions about the use of AI are fair. To ensure that this can be done rapidly and effectively, even during a fast moving crisis, it is essential that processes are put in place in advance, detailing who should be consulted and how to do so if a public health crisis arises. Decision makers can then be held accountable for following those processes and for making transparent the reasoning behind their decisions to deploy AI. Broad stakeholder engagement means consulting with both a wide range of experts and diverse groups from across society, to better understand potential trade-offs involved in deploying a system and acceptable ways to resolve them. Consulting with experts might, for example, include talking to the engineers building AI systems to develop a fuller understanding of their weaknesses, limitations, and risks; experts in domains such as human centred design or value sensitive design to understand how the envisaged benefits of a system might depend on human behaviours and how to support adherence; and ethicists to understand where the use of AI systems might introduce value judgments into decision making processes. 18 Consultation with diverse public groups can highlight blind spots, identify previously ignored harms or benefits to different groups, and help decision makers to understand how trade-offs are perceived by different communities. 24 AI has the potential to help us solve increasingly important global problems, but deploying powerful new technologies for the first time in times of crisis always comes with risks. The better placed we are to deal with ethical challenges in advance, the easier it will be to secure public trust and quickly roll out technology in support of the public good. Contributors and sources: All five authors conduct research in the ethics of technology, in particular AI. SC and JW contributed significantly to planning and drafting; RN and SOhE contributed to planning and drafting; RAC contributed to planning and editing. SC and JW are joint first authors and guarantors. doi: 10.1136/bmj.n364 | BMJ 2021;372:n364 | the bmj
 on 23 March 2022 by guest. Protected by copyright. http://www.bmj.com/ BMJ: first published as 10.1136/bmj.n364 on 15 March 2021. Downloaded from
 the bmj | BMJ 2021;372:n364 | doi: 10.1136/bmj.n364

Stephen Cave
Jess Whittlestone
Rune Nyrup
Sean O. hEigeartaigh
Rafael A. Calvo