Fast reinforcement learning with generalized policy updates

The combination of reinforcement learning with deep learning is a promising approach to tackle important sequential decisionmaking problems that are currently intractable. One obstacle to overcome is the amount of data needed by learning systems of this type. In this article, we propose to address this issue through a divide-and-conquer approach. We argue that complex decision problems can be naturally decomposed into multiple tasks that unfold in sequence or in parallel. By associating each task with a reward function, this problem decomposition can be seamlessly accommodated within the standard reinforcement-learning formalism. The specific way we do so is through a generalization of two fundamental operations in reinforcement learning: policy improvement and policy evaluation. The generalized version of these operations allow one to leverage the solution of some tasks to speed up the solution of others. If the reward function of a task can be well approximated as a linear combination of the reward functions of tasks previously solved, we can reduce a reinforcement-learning problem to a simpler linear regression. When this is not the case, the agent can still exploit the task solutions by using them to interact with and learn about the environment. Both strategies considerably reduce the amount of data needed to solve a reinforcement-learning problem. artificial intelligence | reinforcement learning | generalized policy improvement | generalized policy evaluation | successor features R einforcement learning (RL) provides a conceptual framework to address a fundamental problem in artificial intelligence: the development of situated agents that learn how to behave while interacting with the environment (1) . In RL, this problem is formulated as an agent-centric optimization in which the goal is to select actions to get as much reward as possible in the long run. Many problems of interest are covered by such an inclusive definition; not surprisingly, RL has been used to model robots (2) and animals (3) , to simulate real (4) and artificial (5) limbs, to play board (6) and card (7) games, and to drive simulated bicycles (8) and radio-controlled helicopters (9) , to cite just a few applications. Recently, the combination of RL with deep learning added several impressive achievements to the above list (10) (11) (12) (13) . It does not seem too far-fetched, nowadays, to expect that these techniques will be part of the solution of important open problems. One obstacle to overcome on the track to make this possibility a reality is the enormous amount of data needed for an RL agent to learn to perform a task. To give an idea, in order to learn how to play one game of Atari, a simple video game console from the 1980s, an agent typically consumes an amount of data corresponding to several weeks of uninterrupted playing; it has been shown that, in some cases, humans are able to reach the same performance level in around 15 min (14) . One reason for such a discrepancy in the use of data is that, unlike humans, RL agents usually learn to perform a task essentially from scratch. This suggests that the range of problems our agents can tackle can be significantly extended if they are endowed with the appropriate mechanisms to leverage prior knowledge. In this article we describe one such mechanism. The framework we discuss is based on the premise that an RL problem can usually be decomposed into a multitude of "tasks." The intuition is that, in complex problems, such tasks will naturally emerge as recurrent patterns in the agent-environment interaction. Tasks then lead to a useful form of prior knowledge: once the agent has solved a few, it should be able to reuse their solution to solve other tasks faster. Grasping an object, moving in a certain direction, and seeking some sensory stimulus are examples of naturally occurring behavioral building blocks that can be learned and reused. Ideally, information should be exchanged across tasks whenever useful, preferably through a mechanism that integrates seamlessly into the standard RL formalism. The way this is achieved here is to have each task be defined by a reward function. As most animal trainers would probably attest, rewards are a natural mechanism to define tasks because they can induce very distinct behaviors under otherwise identical conditions (15, 16) . They also provide a unifying language that allows each task to be cast as an RL problem, blurring the (somewhat arbitrary) distinction between the two (17) . Under this view, a conventional RL problem is conceptually indistinguishable from a task self-imposed by the agent by "imagining" recompenses associated with arbitrary events. The problem then becomes a stream of tasks, unfolding in sequence or in parallel, whose solutions inform each other in some way. What allows the solution of one task to speed up the solution of other tasks is a generalization of two fundamental operations underlying much of RL: policy evaluation and policy improvement. The generalized version of these procedures, jointly referred to as "generalized policy updates," extend their standard counterparts from point-based to set-based operations. This makes it possible to reuse the solution of tasks in two distinct ways. When the reward function of a task can be reasonably approximated as a linear combination of other tasks' reward functions, the RL problem can be reduced to a simpler linear regression that is solvable with only a fraction of the data. When the linearity constraint is not satisfied, the agent can still leverage the solution of tasks-in this case, by using them to interact with and learn about the environment. This can also considerably reduce the amount of data needed to solve the problem. Together, these two strategies give rise to a divide-and-conquer approach to RL that can potentially help scale our agents to problems that are currently intractable.
 RL We consider the RL framework outlined in the Introduction: an agent interacts with an environment by selecting actions to get as much reward as possible in the long run (1) . This interaction happens at discrete time steps, and, as usual, we assume it can be modeled as a Markov decision process (MDP) (18) . An MDP is a tuple M ≡ (S, A, p, r , γ) whose components are defined as follows. The sets S and A are the state space and action space, respectively (we will consider that A is finite to simplify the exposition, but most of the ideas extend to infinite action spaces). At every time step t, the agent finds itself in a state s ∈ S and selects an action a ∈ A. The agent then transitions to a next state s , where a new action is selected, and so on. The transitions between states can be stochastic: the dynamics of the MDP, p(•|s, a), give the next-state distribution upon taking action a in state s. In RL, we assume that the agent does not know p, and thus it must learn based on transitions sampled from the environment. A sample transition is a tuple (s, a, r , s ) where r is the reward given by the function r (s, a, s ), also unknown to the agent. As discussed, here we adopt the view that different reward functions give rise to distinct tasks. Given a task r , the agent's goal is to find a policy π : S → A, that is, a mapping from states to actions, that maximizes the value of every state-action pair, defined as Q π r (s, a) ≡ E π ∞ i=0 γ i r (St+i , At+i , St+i+1) | St = s, At = a , [1] where St and At are random variables indicating the state occupied and the action selected by the agent at time step t, E π [•] denotes expectation over the trajectories induced by π, and γ ∈ [0, 1) is the discount factor, which gives less weight to rewards received further into the future. The function Q π r (s, a) is usually referred to as the "action-value function" of policy π on task r ; sometimes, it will be convenient to also talk about the "state-value function" of π, defined as V π r (s) ≡ Q π r (s, π(s)). Given an MDP representing a task r , there exists at least one optimal policy π \* r that attains the maximum possible value at all states; the associated optimal value function V \* r is shared by all optimal policies (18) . Solving a task r can thus be seen as the search for an optimal policy π \* r or an approximation thereof. Since the number of possible policies grows exponentially with the size of S and A, a direct search in the space of policies is usually infeasible. One way to circumvent this difficulty is to resort to methods based on dynamic programming, which exploit the properties of MDPs to reduce the cost of searching for a policy (19) . Policy Updates. RL algorithms based on dynamic programming build on two fundamental operations (1). Definition 1. "Policy evaluation" is the computation of Q π r , the value function of policy π on task r . Definition 2. Given a policy π and a task r , "policy improvement" is the definition of a policy π such that Q π r (s, a) ≥ Q π r (s, a) for all (s, a) ∈ S × A. [2] We call one application of policy evaluation followed by one application of policy improvement a "policy update." Given an arbitrary initial policy π, successive policy updates give rise to a sequence of improving policies that will eventually reach an optimal policy π \* r (18) . Even when policy evaluation and policy improvement are not performed exactly, it is possible to derive guarantees on the performance of the resulting policy based on the approximation errors introduced in these steps (20, 21) . Fig. 1 illustrates the basic intuition behind policy updates. What makes policy evaluation tractable is a recursive relation between state-action values known as the Bellman equation: Q π r (s, a) = E S ∼p(•|s,a) r (s, a, S ) + γQ π r (S , π(S )) . [3] Expression (Exp.) 3 induces a system of linear equations whose solution is Q π r . This immediately suggests ways of performing policy evaluation when the MDP is known (18) . Importantly, the Bellman equation also facilitates the computation of Q π r without knowledge of the dynamics of the MDP. In this case, one estimates the expectation on the right-hand side of Exp. 3 based on samples from p(•|s, a), leading to the well-known method of temporal differences (22, 23) . It is also often the case that in problems of interest the state space S is too big to allow for a tabular representation of the value function, and hence Q π r is replaced by an approximation Qπ r . As for policy improvement, it is in fact simple to define a policy π that performs at least as well as, and generally better than, a given policy π. Once the value function of π on task r is known, one can compute an improved policy π as π (s) ∈ arg max a∈A Q π r (s, a). [4] In words, the action selected by policy π on state s is the one that maximizes the action-value function of policy π on that state. The fact that policy π satisfies Definition 2 is one of the fundamental results in dynamic programming and the driving force behind many algorithms used in practice (18) . The specific way policy updates are carried out gives rise to different dynamic programming algorithms. For example, value iteration and policy iteration can be seen as the extremes of a spectrum of algorithms defined by the extent of the policy evaluation step (19, 24) . RL algorithms based on dynamic programming can be understood as stochastic approximations of these methods or other instantiations of policy updates (25) .
 Generalized Policy Updates From the discussion above, one can see that an important branch of the field of RL depends fundamentally on the notions of policy evaluation and policy improvement. We now discuss generalizations of these operations. Definition 3. "Generalized policy evaluation" (GPE) is the computation of the value function of a policy π on a set of tasks R.
 A B Fig. 1 . (A) Sequence of policy updates as a trajectory that alternates between the policy and value spaces and eventually converges to an optimal solution (1). (B) Detailed view of the trajectory across the value space for a state space with two states only. The shadowed rectangles associated with each value function represent the region of the value space containing the value function that will result from one application of policy improvement followed by policy evaluation (cf. Exp. 2). Definition 4. Given a set of policies Π and a task r , "generalized policy improvement" (GPI) is the definition of a policy π such that Q π r (s, a) ≥ sup π∈Π Q π r (s, a) for all (s, a) ∈ S × A. [5] GPE and GPI are strict generalizations of their standard counterparts, which are recovered when R has a single task and Π has a single policy. However, it is when R and Π are not singletons that GPE and GPI reach their full potential. In this case, they become a mechanism to quickly construct a solution for a task, as we now explain. Suppose we are interested in one of the tasks r ∈ R, and we have a set of policies Π available. The origin of these policies is not important: they may have come up as solutions for specific tasks or have been defined in any other arbitrary way. If the policies π ∈ Π are submitted to GPE, we have their value functions on the task r ∈ R. We can then apply GPI over these value functions to obtain a policy π that represents an improvement over all policies in Π. Clearly, this reasoning applies without modification to any task in R. Therefore, by applying GPE and GPI to a set of policies Π and a set of tasks R, one can compute a policy for any task in R that will in general outperform every policy in Π. Fig. 2 shows a graphical depiction of GPE and GPI. Obviously, in order for GPE and GPI to be useful in practice, we must have efficient ways of performing these operations. Consider GPE, for example. If we were to individually evaluate the policies π ∈ Π over the set of tasks r ∈ R, it is unlikely that the scheme above would result in any gains in terms of computation or consumption of data. To see why this is so, suppose again that we are interested in a particular task r . Computing the value functions of policies π ∈ Π on task r would require |Π| policy evaluations with a naive form of GPE (here, | • | denotes the cardinality of a set). Although the resulting GPI policy π would compare favorably to all policies in Π, this guarantee would be vacuous if these policies are not competent at task r . Therefore, a better allocation of resources might be to use the policy evaluations for standard policy updates, which would generate a sequence of |Π| policies with increasing performance on task r (compare Figs. 1 and 2 ). This difficulty in using generalized pol-Fig. 2 . Depiction of generalized policy updates on a state space with two states only. With GPE each policy π ∈ Π is evaluated on all tasks r ∈ R. The state-value function of policy π on task r, V π r , delimits a region in the value space where the next value function resulting from policy improvement will be (cf. Fig. 1 ). The analogous space induced by GPI corresponds to the intersection of the regions associated with the individual value functions (represented as dark gray rectangles in the figure). The smaller the space of value functions associated with GPI, the stronger the guarantees regarding the performance of the resulting policy. icy updates in practice is further aggravated if we do not have a fast way to carry out GPI. Next, we discuss efficient instantiations of GPE and GPI. Fast GPE with Successor Features. Conceptually, we can think of GPE as a function associated with a policy π that takes a task r as input and outputs a value function Q π r (26) . Hence, a practical way of implementing GPE would be to define a suitable representation for tasks and then learn a mapping from r to value functions Q π r (27) . This is feasible when such a mapping can be reasonably approximated by the choice of function approximator and enough examples (r , Q π r ) are available to characterize the relation underlying these pairs. Here, we will focus on a form of GPE that is based on a similar premise but leads to a form of generalization over tasks that is correct by definition. Let φ : S × A × S → R d be an arbitrary function whose output we will see as "features." Then, for any w ∈ R d , we have a task defined as rw(s, a, s ) = φ(s, a, s ) w, [6] where denotes the transpose of a vector. Let R φ ≡ {rw = φ w | w ∈ R d } be the set of tasks induced by all possible instantiations of w ∈ R d . We now show how to carry out an efficient form of GPE over R φ . Following Barreto et al. (28) , we define the "successor features" (SFs) of policy π as ψ π (s, a) ≡ E π ∞ i=0 γ i φ(St+i , At+i , St+i+1) | St = s, At = a . The ith component of ψ π (s, a) gives the expected discounted sum of φi when following policy π starting from (s, a). Thus, ψ π can be seen as a d -dimensional value function in which the features φi (s, a, s ) play the role of reward functions (cf. Exp. 1). As a consequence, SFs satisfy a Bellman equation analogous to Exp. 3, which means that they can be computed using standard RL methods like temporal differences (22) . Given the SFs of a policy π, ψ π , we can quickly evaluate π on task rw ∈ R φ by computing ψ π (s, a) w = E π ∞ i=0 γ i φ(St+i , At+i , St+i+1) w|St = s, At = a = E π ∞ i=0 γ i rw(St+i , At+i , St+i+1) | St = s, At = a = Q π rw (s, a) ≡ Q π w (s, a). [7] That is, the computation of the value function of policy π on task rw is reduced to the inner product ψ π (s, a) w. Since this is true for any task rw, SFs provide a mechanism to implement a very efficient form of GPE over the set R φ (cf. Definition 3). The question then arises as to how inclusive the set R φ is. Since R φ is fully determined by φ, the answer to this question lies in the definition of these features. Mathematically speaking, R φ is the linear space spanned by the d features φi . This view suggests ways of defining φ that result in a R φ containing all possible tasks. A simple example can be given for when both the state space S and the action space A are finite. In this case, we can recover any possible reward function by making d = |S | 2 × |A| and having each φi be an indicator function associated with the occurrence of a specific transition (s, a, s ). This is essentially Dayan's (29) concept of "successor representation" extended from S to S × A × S. The notion of covering the space of tasks can be generalized to continuous S and A if we think of φi as basis functions over the function space S × A × S → R, for we know that, under some technical conditions, one can approximate any function in this space with arbitrary accuracy by having an appropriate basis. Although these limiting cases are reassuring, more generally we want to have a number of features d |S |. This is possible if we consider that we are only interested in a subset of all possible tasks-which should in fact be a reasonable assumption in most realistic scenarios. In this case, the features φ should capture the underlying structure of the set of tasks R in which we are interested. For example, many tasks we care about could be recovered by features φ i representing entities (from concrete objects to abstract concepts) or salient events (such as the interaction between entities). In Fast RL with GPE and GPI, we give concrete examples of what the features φ may look like and also discuss possible ways of computing them directly from data. For now, it is worth mentioning that, even when φ does not span the space R, one can bound the error between the two sides of Exp. 7 based on how well the tasks of interest can be approximated using φ (proposition 1 in ref. 30 ). Fast GPI with Value-Based Action Selection. Thanks to the concept of value function, standard policy improvement can be carried out efficiently through Exp. 4. This raises the question whether the same is possible with GPI. We now answer this question in the affirmative for the case in which GPI is applied over a finite set Π. Following Barreto et al. (28) , we propose to compute an improved policy π as π (s) ∈ argmax a∈A max π∈Π Q π r (s, a). [8] Barreto et al. have shown that Exp. 8 qualifies as a legitimate form of GPI as per Definition 4 (theorem 1 in ref. 28 ). Note that the action selected by a GPI policy π on a state s may not coincide with any of the actions selected by the policies π ∈ Π on that state. This highlights an important difference between Exp. 8 and methods that define a higher-level policy that alternates between the policies π ∈ Π (32). In SI Appendix, we show that the policy π resulting from Exp. 8 will in general outperform its analog defined over Π. As discussed, GPI reduces to standard policy improvement when Π contains a single policy. this also happens with Exp. 8 when one of the policies in Π strictly outperforms the others on task r . This means that, after one application of GPI, adding the resulting policy π to the set Π will reduce the next application of Exp. 8 to Exp. 4. Therefore, if GPI is used in a sequence of policy updates, it will collapse back to its standard counterpart after the first update. In contrast with policy improvement, which is inherently iterative, GPI is an one-off operation that yields a quick solution for a task when multiple policies have been evaluated on it. The guarantees regarding the performance of the GPI policy π can be extended to the scenario where Exp. 8 is used with approximate value functions. In this case, the lower bound in Exp. 5 decreases in proportion to the maximum approximation error in the value function approximations (theorem 1 in ref. 28) . We refer the reader to SI Appendix for an extended discussion on GPI. The Generalized Policy. Strictly speaking, in order to leverage the instantiations of GPE and GPI discussed in this article, we only need two elements: features φ : S × A × S → R d and a set of policies Π = {πi } n i=1 . Together, these components yield a set of SFs Ψ = {ψ π i } n i=1 , which provide a quick form of GPE over the set of tasks R φ induced by the features φ. Specifically, for any w ∈ R d , we can quickly evaluate policy πi on task rw(s, a, s ) = φ(s, a, s ) w by computing Q π i w (s, a) = ψ π i (s, a) w. Once we have the value functions of policies πi on task rw, {Q π i w } n i=1 , Exp. 8 can be used to obtain a GPI policy π that will in general outperform the policies πi on rw. Observe that the same scheme can be used if we have approximate SFs ψπ i ≈ ψ π i , which will be the case in most realistic scenarios. Since the resulting value functions will also be approximations, Qπ i w ≈ Q π i w , we are back to the approximate version of GPI discussed in Fast GPI with Value-Based Action Selection. Given a set of SFs Ψ, approximate or otherwise, any w ∈ R d gives rise to a policy through GPE (Exp. 7) and GPI (Exp. 8). We can thus think of generalized updates as implementing a policy πΨ(s; w) whose behavior is modulated by w. We will call πΨ a "generalized policy." Fig. 3 provides a schematic view of the computation of πΨ through GPE and GPI. Since each component of w weighs one of the features φi (s, a, s ), changing them can intuitively be seen as setting the agent's current "preferences." For example, the vector w = [0, 1, −2] indicates that the agent is indifferent to feature φ1 and wants to seek feature φ2 while avoiding feature φ3 with twice the impetus. Specific instantiations of πΨ can behave in ways that are very different from its constituent policies π ∈ Π. We can draw a parallel with nature if we think of features as concepts like water or food and note how much the desire for these items can affect an animal's behavior. Analogies aside, this sort Fig. 3 . representation of how the generalized policy πΨ is implemented through GPE and GPI. Given a state s ∈ S, the first step is to compute, for each a ∈ A, the values of the n SFs: {ψ π i (s, a)} n i=1 . Note that ψ π i can be arbitrary nonlinear functions of their inputs, possibly represented by complex approximators like deep neural networks (30, 31) . The SFs can be laid out as an n × |A| × d tensor in which the entry (i, a, j) is ψ π i j (s, a). Given this layout, GPE reduces to the multiplication of the SF tensor with a vector w ∈ R d , the result of which is an n × |A| matrix whose (i, a) entry is Q π i (s, a). GPI then consists in finding the maximum element of this matrix and returning the associated action a. This whole process can be seen as the implementation of a policy πΨ(s; w) whose behavior is parameterized by a vector of preferences w. of arithmetic over features provides a rich interface for the agent to interact with the environment at a higher level of abstraction in which decisions correspond to preferences encoded as a vector w. Next, we discuss how this can be leveraged to speed up the solution of an RL task.
 Fast RL with GPE and GPI We now describe how to build and use the adaptable policy πΨ implemented by GPE and GPI. To make the discussion more concrete, we consider a simple RL environment depicted in Fig. 4 . The environment consists of a 10 × 10 grid with four actions available: A = {up, down, left, right}. The agent occupies one of the grid cells, and there are also 10 objects spread across the grid. Each object belongs to one of two types. At each time step t, the agent receives an image showing its position and the position and type of each object. Based on this information, the agent selects an action a ∈ A, which moves it one cell along the desired direction. The agent can pick up an object by moving to the cell occupied by it; in this case, it gets a reward defined by the type of the object. A new object then pops up in the grid, with both its type and location sampled uniformly at random (more details are in SI Appendix). This simple environment can be seen as a prototypical member of the class of problems in which GPE and GPI could be useful. This becomes clear if we think of objects as instances of (potentially abstract) concepts, here symbolized by their types, and note that the navigation dynamics are a proxy for any sort of dynamics that mediate the interaction of the agent with the world. In addition, despite its small size, the number of possible configurations of the grid is actually quite large, of the order of 10 15 . This precludes an exact representation of value functions and illustrates the need for approximations that inevitably arises in many realistic scenarios. By changing the rewards associated with each object type, one can create different tasks. We will consider that the agent wants to build a set of SFs Ψ that give rise to a generalized policy πΨ(s; w) that can adapt to different tasks through the vector of preferences w. This can be either because the agent does not know in advance the task it will face or because it will face more than one task. Defining a Basis for Behavior. In order to build the SFs Ψ, the agent must define two things: features φ and a set of policies Π. Since φ should be associated with rewarding events, we define each feature φi as an indicator function signaling whether an object of type i has been picked up by the agent (i.e., φ ∈ R 2 ). To be precise, we have that φi (s, a, s ) = 1 if the transition from state s to state s is associated with the agent picking up an object of type i, and φi (s, a, s ) = 0 otherwise. These features induce a set R φ where task rw ∈ R φ is characterized by how desirable or undesirable each type of object is. Now that we have defined φ, we turn to the question of how to determine an appropriate set of policies Π. We will restrict the policies in Π to be solutions to tasks rw ∈ R φ . We start with what is perhaps the simplest choice in this case: a set Π12 = {π1, π2} whose two elements are solutions to the tasks w1 = [1, 0] and w2 = [0, 1] (henceforth, we will drop the transpose superscript to avoid clutter). Note that the goal in tasks w1 and w2 is to pick up objects of one type while ignoring objects of the other type. We are now ready to compute the SFs Ψ induced by our choices of φ and Π. In our experiments, we used an algorithm analogous to Q-learning to compute approximate SFs ψπ 1 and ψπ 2 (pseudocode in SI Appendix). We represented the SFs using multilayer perceptrons with two hidden layers (33) . The set of SFs Ψ yields a generalized policy π Ψ(s ; w) parameterized by w. We now evaluate π Ψ on the task whose goal is to pick up objects of the first type while avoiding objects of the second type. Using φ defined above, this task can be represented as rw 3 (s, a, s ) = φ(s, a, s ) w3, with w3 = [1, −1]. We thus evaluate the generalized policy instantiated as π Ψ(s ; w3). Results are shown in Fig. 5A . As a reference, we also show the learning curve of Q-learning (23) using the same architecture to directly approximate Q π w 3 . GPE and GPI allow one to compute an instantaneous solution for a new task, without any learning on the task itself, that is competitive with the policies found by Q-learning when using around 6 × 10 4 sample transitions. The performance of the policy π Ψ synthesized by GPE and GPI corresponds to more than 70% of the performance eventually achieved by Q-learning after processing 10 6 transitions. This is quite an impressive result when we note that π Ψ managed to avoid objects of the second type even though its constituents policies π1 and π2 were never trained to actively avoid objects. We used a total of 10 6 sample transitions to learn both SFs ψπ 1 and ψπ 2 , which is the same amount of data used by Qlearning to achieve its final performance. The advantage of doing the former is that, once we have the SFs, we can use GPE and GPI to instantaneously compute a solution for any task in R φ . However, how well do GPE and GPI actually perform on R φ ? To answer this question, we ran a second round of experiments to assess the generalization of π Ψ over the entire set R φ . Since this evaluation clearly depends on the set of policies used, we consider two other sets in addition to Π12 = {π1, π2}. The new sets are Π34 = {π3, π4} and Π5 = {π5}, where the policies πi are solutions to the tasks w3 = [1, −1], w4 = [−1, 1], and w5 = [1, 1] . We repeated the previous experiment with each policy set and evaluated the resulting policies π Ψ over 19 tasks w evenly spread over the nonnegative quadrants of the unit circle (tasks in the negative quadrant are uninteresting because all of the agent must do is to avoid hitting objects). Results are shown in Fig. 6A . As expected, the generalization ability of GPE and GPI depends on the set of policies used. Perhaps more surprising is how well the generalized policy π Ψ induced by some of these sets perform across the entire space of tasks R φ , sometimes matching the best performance of Q-learning when solving each task individually. These experiments show that a proper choice of base policies Π can lead to good generalization over the entire set of tasks R φ . In general, though, it is unclear how to define an appropriate Π. Fortunately, we can refer to our theoretical understanding of GPE and GPI to have some guidance. First, we know from the discussion above that the larger the number of policies in Π the stronger the guarantees regarding the performance of the resulting policy π Ψ (Exp. 5). In addition to that, Barreto et al. have shown that it is possible to guarantee a minimum performance level for π Ψ on task w based on mini w − wi , where • is some norm and wi are the tasks associated with the policies πi ∈ Π used for GPE and GPI (theorem 2 in ref. 28 ). Together, these two insights suggest that, as we increase the size of Π, the performance of the resulting policy π Ψ should improve across R φ , especially on tasks that are close to the tasks wi . To test this hypothesis empirically, we repeated the previous experiment, but now, instead of comparing disjoint policy sets, we compared a sequence of sets formed by adding one by one the policies π2, π5, π1, and π3, in this order, to the initial set {π4}. The results, in Fig. 6B , confirm the trend implied by the theory. Task Inference. So far, we have assumed that the agent knows the vector w that describes the task of interest. Although this can be the case in some scenarios, ideally we would be able to apply GPE and GPI even when w is not provided. In this section and in Preferences as Actions, we describe two possible ways for the agent to learn w. Given a task r , we are looking for a w ∈ R d that leads to good performance of the generalized policy πΨ(s; w). We could in principle approach this problem as an optimization over w ∈ R d whose objective is to maximize the value of πΨ(s; w) across (a subset of) the state space. It turns out that we can exploit the structure underlying SFs to efficiently determine w without ever looking at the value of πΨ. Suppose we have a set of m sample transitions from a given task, {(si , ai , r i , s i )} m i=1 . Then, based on Exp. 6, we can infer w by solving the following minimization: min w m i=1 |φ(si , ai , s i ) w − r i | p , [9] where p ≥ 1 (one may also want to consider the inclusion of a regularization term, see ref. 33) . Observe that, once we have a solution w for the problem above, we can plug it in Exp. 7 and use GPE and GPI as we did before-that is, we have just turned an RL task into an easier linear regression problem. To illustrate the potential of this approach, we revisited the task w3 = [1, −1] tackled above, but now, instead of assuming we knew w3, we solved the problem in Exp. 9 using p = 2. We collected sample transitions using a policy π that picks actions
 A B Fig. 6 . Results on the space of tasks R φ induced by a two-dimensional φ. The sets of policies Π used in A are disjoint; in B these sets overlap. The evaluation of an algorithm on a task is represented as a vector whose direction indicates the task w and whose magnitude gives the average sum of rewards over 10 runs with 250 trials each. Q-learning learned each task individually, from scratch; the dotted curves correspond to its performance after having processed 5 × 10 4 , 1 × 10 5 , 2 × 10 5 , 5 × 10 5 , and 1 × 10 6 sample transitions. Our method only learned the policies in the sets Π and then generalized across all others tasks through GPE and GPI. The SFs used for GPE and GPI consumed 5 × 10 5 sample transitions to be trained each. uniformly at random and used the gradient descent method to refine an approximation w ≈ w online (SI Appendix). Results are shown in Fig. 5 A and B. The performance of the generalized policy π Ψ(s ; w3) using the correct w3 was recovered after around 800 sample transitions only. To put this number into perspective, recall that Q-learning needs around 60,000 transitions, or 75 times as much data, to achieve the same performance level. The number of transitions needed to learn w can be reduced further if we use the closed-form solution for the least-squares problem. The problem described in Exp. 9 can be solved even if the rewards are the only information available about a given task. Other interesting possibilities arise when the observations provided by the environment can be used to infer w, such as, for example, visual or auditory cues indicating the current task. In this case, it might be possible for the agent to determine w even before having observed a single nonzero reward (30) . So far, we have used handcrafted features φ. It turns out that Exp. 9 can be generalized to also allow φ to be inferred from data. Given sample transitions from k tasks, {(sij , aij , r ij , s ij )} with p ≥ 1. Note that the features φ can be arbitrary nonlinear functions of their inputs. As discussed by Barreto et al. (30) , if we make c = k we can solve a simplified version of the problem above in which wj do not show up. More generally, the problem in Exp. 10 can be solved as a multitask regression (34) . In Fig. 5B , we show the performance of π Ψ(s ; w) when using learned features φ. These results were obtained as follows. First, we used the random policy π to collect data from tasks w1, w2, w4, and w5 and applied the stochastic gradient descent algorithm to solve Exp. 10 with p = 2. We then discarded the resulting approximations wi appearing in Exp. 10 and solved Exp. 9 exactly as before but now using φ instead of φ. As with SFs, in order to represent the features, we used multilayer perceptrons with two hidden layers (SI Appendix). The results in Fig. 5B also show the effect of varying the dimension of the learned features on the performance of πΨ(s; w) (other illustrations are in refs. 28 and 30) . Preferences as Actions. In this section, we consider the scenario where, given features φ and a set of policies Π, there is no vector w that leads to acceptable performance of πΨ(s; w). To illustrate this situation, we use our environment with a slight twist: now, on picking up an object of a certain type, the agent gets a positive reward only if the number of objects of this type is greater or equal to the number of objects of the other type. Otherwise, the agent gets a negative reward. One can represent this problem using the previously defined features φ by switching between two instantiations of w: [1, −1] when the first type of object is more abundant and [−1, 1] otherwise. This means that the appropriate value for w is now state-dependent. To handle this situation, we introduce a function mapping states to preferences, ω : S → W, with W ⊆ R d , and redefine the generalized policy as πΨ(s; ω(s)). The RL problem then becomes to find a ω that correctly modulates πΨ (32, 35) . We can look at the computation of πΨ(s; ω(s)) as a twostage process: first, the agent computes a vector of preferences w = ω(s); then, using GPE and GPI, it determines the action to be executed in state s as a = πΨ(s; w). Under this view, ω(s) becomes an indirect way of selecting an action in s, which allows us to think of it as a policy defined in the space W. We have thus replaced the original problem of finding a policy π : S → A with the problem of finding a policy ω : S → W. To illustrate the benefits of applying this transformation to the problem, we will use the task described above in which the agent must pick up the type of object that is more abundant. We will tackle this problem by reusing the SFs Ψ induced by the two policies in Π12. Given Ψ, the problem comes down to finding a mapping ω : S → W that leads to good performance of π Ψ(s ; ω(s)). We define W = {−1, 0, 1} 2 and use Q-learning to learn ω. We compare the performance of π Ψ(s ; ω(s)) with two baselines: GPE and GPI using a fixed w, π Ψ(s ; w), and the standard version of Q-learning that learns a policy π(s) over primitive actions. The results, shown in Fig. 7 , illustrate how π Ψ(s ; ω(s)) can significantly outperform the baselines. It is easy to understand why this is the case when we compare π Ψ(s ; ω(s)) with π Ψ(s ; w), for the experiment was designed to render the latter unsuitable. However, why is it that replacing the fourdimensional space A with the nine-dimensional space W leads to better performance? The reason is that ω(s) can be easier to than π(s). As discussed, the vectors w represent the agent's current preferences for types of objects. Since, in general, such preferences should only change when an object is picked up, after selecting a specific w, the agent can hold on to it until it bumps into an object. That is, it is possible to define a good policy ω that maps multiple consecutive states s to the same w. In contrast, it will normally not be possible to execute a single primitive action a ∈ A between the collection of two consecutive objects. This means that the same policy can have a simpler representation as a mapping S → W than as a mapping S → A, making ω easier to learn with certain types of function approximators like neural networks (33) . Note that other choices of φ may induce other forms of regularities in ω-smoothness being an obvious example. Interestingly, the fact that the agent may be able to adhere to the same w for many time steps allows for a temporally extended policy πΨ(s; ω(s)) that is only permitted to change preferences at certain points in time, reducing even further the amount of data needed to learn the task (31) . As a final remark, we note that the ability to chain a sequence of preference vectors w changes the nature of the problem of computing an appropriate set of features φ. When the agent has to adhere to a single w, the features φi must span the space of rewards defining the tasks of interest. When the agent is allowed Fig. 7 . Average sum of rewards on the task whose goal is to pick up objects of the type that is more abundant. GPE and GPI used Π 12 as the base policies and the corresponding SFs consumed 5 × 10 5 transitions to be trained each. The results reflect the best performance of each algorithm over multiple parameter configurations (SI Appendix). Shadowed regions are one standard error over 100 runs. to learn a policy ω over preferences w, this linearity assumption is no longer necessary. Similarly, the use of a fixed w requires that the set of available SFs leads to a competent policy for all tasks w we may care about. Here, this requirement is alleviated, since, by changing w, the agent may be able to compensate for eventual imperfections on the induced GPI policies (31) . Lifelong Learning. We have looked at the different subproblems involved in the use of GPE and GPI. For didactic purposes, we have discussed these problems in isolation, but it is instructive to consider how they can jointly emerge in a real application. One scenario where the problems above would naturally arise is "lifelong learning," in which the interaction of the agent with the environment unfolds over a long period (36) . Since we expect some patterns to repeat during such an interaction, it is natural to break it in tasks that share some common structure. In terms of the concepts discussed in this article, this common structure would be φ, which can be handcrafted based on prior knowledge about the problem or learned through Exp. 10 at the beginning of the agent's lifetime. Based on the features φ, the agent can compute and store the SFs ψπ associated with the solutions π of the tasks encountered along the way. When facing a new task r , the agent can use the vector of preferences w to quickly adapt the generalized policy π Ψ. We have discussed two ways to do so. In the first one, the agent computes an approximation w through Exp. 9, which immediately yields a solution π Ψ(s ; w). The second possibility is to learn a mapping ω(s) : S → W and use it to have a per-state modulation of the generalized policy, π Ψ(s ; ω(s)). Regardless of how the agent computes w, it can then choose to hold on to π Ψ or use it as a initial solution for the task to be refined by a standard RL algorithm. Either way, the agent can also compute the SFs of the resulting policy, which can in turn be added to the library of SFs Ψ, improving even further its ability to tackle future tasks (28, 30) .
 Conclusion We have generalized two fundamental operations in RL, policy improvement and policy evaluation, from single to multiple operands (tasks and policies, respectively). The resulting operations, GPE and GPI, can be used to speed up the solution of an RL problem. We showed possible ways to efficiently implement GPE and GPI and discussed how their combination leads to a generalized policy πΨ(s; w) whose behavior is modulated by a vector of preferences w. Two ways of learning w were considered. In the simplest case, w is the solution of a linear regression problem. This reduces an RL task to a much simpler problem that can be solved using only a fraction of the data. This strategy depends on two conditions: 1) the reward function of the tasks of interest are approximately linear in the features φ used for GPE and 2) the set of policies Π used for GPI lead to good performance of πΨ on the tasks of interest. When these assumptions do not hold, one can still resort to GPE and GPI by looking at the preferences w as actions. This strategy can also improve sample efficiency if the mapping from states to preferences is simpler to learn than the corresponding policy. Many extensions of the basic framework presented in this article are possible. As mentioned, Barreto et al. (31) have proposed a way of implementing a generalized policy πΨ that explicitly leverages temporal abstraction by treating preferences w as abstract actions that persist for multiple time steps. Borsa et al. (37) introduced a generalized form of SFs that has a representation z ∈ Z of a policy πz as one of its inputs: ψ(s, a, z) = ψ πz (s, a). In principle, this allows one to apply GPI over arbitrary, potentially infinite, sets Π (for example, by replacing the maximization in Exp. 8 with sup z∈Z ψ(s, a, z) w). Hunt et al. (38) extended SFs to entropy-regularized RL, and, in doing so, they proposed solutions for many challenges that come up when applying GPE and GPI in continuous action spaces. More recently, Hansen et al. (39) proposed an approach that allows the features φ to be learned from data in the absence of a reward signal. Other extensions are possible, including different instantiations of GPE and GPI. All of these extensions expand the framework built upon GPE and GPI. Together, they provide a conceptual toolbox that allows one to decompose an RL problem into tasks whose solutions inform each other. This results in a divide-and-conquer approach to RL, which, combined with deep learning, has the potential to scale up our agents to problems currently out of reach. Data Availability. The source code used to generate all of the data associated with this article has been deposited in GitHub (https://github.com/deepmind/deepmind-research/tree/ master/option keyboard/gpe gpi experiments). Fig. 4 . 4 Fig. 4. Depiction of the environment used in the experiments. The shape of the objects (square or triangle) represents their type; the agent is depicted as a circle. We also show the first 10 steps taken by 3 policies, π 1 , π 2 , and π 3 , that would perform optimally on tasks w 1 = [1, 0], w 2 = [0, 1], and w 3 = [1, −1] for any discount factor γ ≥ 0.5.
 Fig. 5 . 5 Fig. 5. Average sum of rewards on task w 3 = [1, −1]. GPE and GPI used Π 12 = {π 1 , π 2 } as the base policies and the corresponding SFs consumed 5 × 10 5 sample transitions to be trained each. B is a zoomed-in version of A showing the early performance of GPE and GPI under different setups. The results reflect the best performance of each algorithm over multiple parameter configurations (SI Appendix). Shadowed regions are one standard error over 100 runs.
 j = 1, 2, . . . , k , we can formulate the problem as the search for a function φ :S × A × S → R c and k vectors wi ∈ R c satisfying min sij , aij , s ij ) wj − r ij | p , [10]
 | www.pnas.org/cgi/doi/10.1073/pnas.1907370117 Barreto et al. Downloaded from https://www.pnas.org by 205.240.4.254 on March 23, 2022 from IP address 205.240.4.254.
 Barreto et al. PNAS | December 1, 2020 | vol. 117 | no. 48 | 30081 Downloaded from https://www.pnas.org by 205.240.4.254 on March 23, 2022 from IP address 205.240.4.254.
 | www.pnas.org/cgi/doi/10.1073/pnas.1907370117 Barreto et al. Downloaded from https://www.pnas.org by 205.240.4.254 on March 23, 2022 from IP address 205.240.4.254.
 PNAS | December 1, 2020 | vol. 117 | no. 48 | 30083 Downloaded from https://www.pnas.org by 205.240.4.254 on March 23, 2022 from IP address 205.240.4.254.
 | www.pnas.org/cgi/doi/10.1073/pnas.1907370117 Barreto et al. Downloaded from https://www.pnas.org by 205.240.4.254 on March 23, 2022 from IP address 205.240.4.254.
 PNAS | December 1, 2020 | vol. 117 | no. 48 | 30085 Downloaded from https://www.pnas.org by 205.240.4.254 on March 23, 2022 from IP address 205.240.4.254.
 | www.pnas.org/cgi/doi/10.1073/pnas.1907370117 Barreto et al. Downloaded from https://www.pnas.org by 205.240.4.254 on March 23, 2022 from IP address 205.240.4.254.
 PNAS | December 1, 2020 | vol. 117 | no. 48 | 30087 Downloaded from https://www.pnas.org by 205.240.4.254 on March 23, 2022 from IP address 205.240.4.254.

André Barreto
Shaobo Hou
Diana Borsa
David Silver
Doina Precup


Long-Term Growth as a Sequence of Exponential Modes

Introduction Many have argued that current U.S. stock prices are much higher than can be justified by reasonable profit forecasts based on empirical models derived from the recorded history of the U.S. economy (Shiller, 2000) . While it might be theoretically possible for the economy to suddenly switch to a fundamentally new mode with much higher corporate profits, our experience with the U.S. economy offers little evidence that a mode switch of this magnitude has happened anytime in the last century or so (Barro & Sala-i-Martin, 1995) , though we may be able to understand smaller changes in stock prices in terms of milder fluctuations in profit trends (Barsky & De Long, 1993) . The arguments against high current U.S. stock prices are thus largely empirical, in contrast with the theoretical nature of the (typically informal) arguments offered in support of investor optimism. These empirical arguments, however, are based primarily on the last century or so of economic experience. Could empirical forecasts based on a longer term history of economic growth offer more support for the rationality of high stock prices? To address this question, this paper will take an empirical approach to modeling very long term data on economic growth, with an eye toward inferring the chances for very large changes in economic growth modes. In order to empirically model very long term economic history, we need one or more very long term time series, and we need to choose one or more functional forms to use in modeling those time series. Most attempts to formally model long term human history have focused on population time series, as population estimates have been offered going back many thousands of years, and such time series have been considered to be the most reliable long term series available. Also, for most of human history population seems to have reasonably reflected most fundamental changes, including areas colonized, capital accumulated, and technologies adopted. This seems to be because new productive abilities have mainly resulted in larger populations under Malthusian conditions of near-subsistence per capita consumption. Nearly constant per capita consumption, however, implies that reasonable estimates of total consumption can be obtained from reasonable estimates of population. We thus have reasonable estimates of world product up until about 1500, when per capita product began to rise. Economic historians have also provided us with reasonable estimates for world product over the past century or so (Maddison, 1995) . So all that has stood in the way of a full long term world product time series was to fill in the gap from roughly 1500 to 1850. Recently economic historian Brad De Long has cleverly used a simple empirical Malthusian relation to fill this gap, and has thus created a world product time series ranging from one million B.C. to today (De Long, 1998) . His estimates for before 10,000 B.C., however, are based on forty year old speculations (Deevey, 1960) . This paper will therefore replace those old speculations with recent published estimates going back to two million B.C. (Hawks, Hunley, Lee, & Wolpoff, 2000) . This paper will then attempt to empirically model that time series with a few simple functional forms. After considering models of gradually accelerating growth, which have been the most popular in formal models of long term population growth, we will turn to attempting to formalize the more common informal descriptions of history as a sequence of growth modes. And since exponential growth is the most common formal description of short term growth, we will consider modeling long term world product history as a sequence of exponential growth modes. The simplest sequence of exponentials is a sum of exponentials, and we will find that a sum of four exponential modes fits the data with less than a 2% average error, which is less than the average uncertainty in the estimates. One of these modes, however, appears to be there just to allow a more gradual transition between two other modes than a simple sum of exponentials will allow. So we will then try a constant elasticity of substitution (CES) form for combining exponentials, which can allow a transition to be smooth or sudden by varying a single parameter. We will find that a CES combination of three exponential modes also fits the data with less than a 2% average error. After describing these models, and discussing their interpretation, we will then use them to consider when we might see a transition to yet another mode of economic growth, and how fast growth might then be. It seems quite unreasonable to expect an infinite series of faster and faster growth modes, since that leads to an infinite world product in finite time. But it may be reasonable to wonder if a world that has so far seen at least three distinct growth modes might see yet one more growth mode.
 The Data Economic historians have reasonable estimates for world product over the last century or so, while other historians have reasonable estimates for world population over the last ten thousand years, and cruder estimates for the two million years before that. Until a few centuries ago, per-capita consumption was typically near subsistence levels, making it easy to infer world product from world population. Thus the obstacle to having a full long term world product time series has been to get reasonable estimates for the intermediate period, from roughly 1500 to 1850. Recently, economic historian Brad De Long found a simple linear relation between world population growth rates and per capita product, a relation which seems valid at the low values of per capita product observed from roughly 1850 to 1950 (De Long, 1998) . This linear relation has allowed him to created world product estimates for the missing intermediate period, and thus to create a world product time series ranging from one million B.C. to today. De Long developed several time series based on alternative assumptions, and this paper uses his preferred world product series. The only change made here is to substitute more recent estimates for population prior to 10,000 B.C. De Long continued the use by Kremer (Kremer, 1993) Deevey's estimates, however, were never published in a peer-reviewed journal, cite no other sources, and are now forty years old. The data set used here excises Deevey's forty year old estimates and instead substitutes two more recent population estimates. Hawks and coauthors (Hawks et al., 2000) estimate a "population bottleneck" of about ten thousand at about two million B.C., and that we "cannot reject exponential growth from" then until ten thousand B.C. Consistent with that conclusion, they also accept a Weiss (Weiss, 1984) The resulting world product time series has 54 data points, is graphed in Figure 1 , and is listed in the appendix. In Figure 1 , world product is described in units of an equivalent number of humans at a subsistence consumption level. 2 Time is described by the number of years before the somewhat arbitrary date of 2050. This data is also graphed later in Figure 2 , which shows world product growth rates versus levels, displaying each step between data points as a horizontal line. This graph looks noisier, because growth rates are noisier, but it has the virtue of avoiding the arbitrary choice of a reference date like 2050. 3
 Choosing a Functional Form What functional form should we use to model our world product time series? Simple exponential growth is probably the most popular general model for describing positive quantities that grow in time by many orders of magnitude, as both population and world product have. It is, for example, widely used to model the last half century of economic growth (Barro & Sala-i-Martin, 1995) and 1 These estimates are based on a "multi-regional" model of human evolution (Relethford, 1998 ). An alternative "out of Africa" model posits a similar population bottleneck at roughly 200 thousand B.C. (Rogers & Jorde, 1995; Ingman, Kaessmann, Paabo, & Gyllensten, 2000) . The multi-regional model is based more heavily on non-genetic fossil evidence. Genetic evidence may well estimate the number of our direct ancestors alive on a given date, but nongenetic fossils seem better estimates of the size of the economy or ecological niche occupied by our direct ancestors and their very close relatives on that date. Further footnotes will, however, mention how the alternative model changes this paper's estimates. 2 Specifically, "subsistence" is $80 1990 "International" dollars, which is the ratio of De Long's preferred world product and population estimates for one million B.C. This per capita product has been assigned to my substitute dates of two million B.C. and 750 thousand B.C., and retained De Long's estimates for all other dates. 3 In the analysis which follows, we will occasionally compare world product growth since two million B.C. to the earlier exponential growth of maximum brain size. Maximum animal mass and relative brain size have both grown roughly exponentially since the Cambrian explosion about 550 million years ago, when animals first appeared in large numbers in the fossil record. The mass of the currently largest animal doubled roughly every 70 million years, while maximum brain volume relative to body volume (raised to the 2/3 power) has doubled roughly every 50 million years (Russell, 1983) . (More details on the details of brain size history are given by Jerison (Jerison, 1991) .) Together these estimates suggest that maximum animal brain size has doubled roughly every 34 million years over the last 550 million years (for a total of about 16 doublings). For comparison, the exponential decay rate of Marine genera over this period was a half-life of about 23 million years (Valen, 1973; Newman & Sibani, 1999) . the last few centuries of growth in scientific literature (de Solla Price, 1963) . Simple variations on exponential growth, such as linear trends in growth rates, have been particularly popular in making short term population forecasts (Lee, 1990) , but seem clearly inadequate for longer term forecasts. Variations on logistic growth have also long been popular in estimating various population trends (Meyer & Ausubel, 1999) , but also seem clearly inadequate for modeling very long term population trends (Cohen, 1995) . Many researchers have had more empirical success explicitly modeling long term population histories with functions that describe accelerating change. The first such efforts used simple hyperbolics (von Foerster, Mora, & Amiot, 1960) , and soon became infamous warnings against taking one's model too literally, since they predicted an infinite human population in the early twenty first century. More recent related efforts (Kremer, 1993; Kapitza, 1996; Johansen & Sornette, 2000) have all included corrections which could describe a recent or upcoming slowdown and other complexities, mostly to allow growth to today be approaching fundamental limits, and hence a final end to accelerating growth. Some authors have also informally summarized world history as continually accelerating change, but the more common informal summary is of human history as a sequence of specific growth modes. History has, for example, been described as the slow expansion of hunter-gatherers, followed by faster growth following the domestication of plants and animals, followed by even faster growth with commerce, science, and industry (Cipolla, 1967) . Simple exponential growth is a very common model for describing particular historical periods. It is, for example, widely used to model the last half century of world economic growth (Barro & Sala-i-Martin, 1995) . Since historical periods tend to be described as exponential growth, formal models of transitions between periods tend to be models of transitions between exponential growth asymptotes. Many have modeled the industrial revolution this way (Marvin Goodfriend, 1995; Hansen & Prescott, 1998; Jones, 1999; Steinmann, Prskawetz, & Feichtinger, 1998; Galor & Weil, 2000; Lucas, 2000) . At least one earlier transition, to cities, has also been modeled this way (Marvin Goodfriend, 1995) . And possible current or upcoming transitions, such as due to computers or the internet, have also been modeled this way (Helpman, 1998) . Given all this, it seems natural to formally model the long term history of humanity as a sequence of exponential growth modes. Yet to my knowledge no one has done this. Some authors seem to have come close, by drawing exponential curves suggestively next to data on log-log graphs of population growth (Deevey, 1960; Kates, 1996; Livi-Bacci, 1997) . (Some of these authors, however, do not seem to have realized that it could be simple exponentials they were drawing, and instead of "exponential growth followed by a period of approximate stability" (Kates, 1996) .) One author did more explicitly consider and reject a sum of exponentials model for world population history, though without any explicit model fitting or test (Cohen, 1995) . (Another author considered a sum of two exponentials, one of which was decreasing (Lee, 1988) .) In this paper we thus take on the neglected task of more formally modeling long term human history as a sequence of exponential growth modes. In particular, we consider such a model of world product history. Once we have chosen to model history as a sequence of exponential modes, there still remains the question of how we should mathematically model transitions between exponential growth modes. We will first try a simple sum of exponentials, and then try to replace this sum with a CES (constant elasticity of substitution) form. Such a form allows each transition to be sudden or gradual, depending on a single free parameter per transition. We will end up with a model describing the three classic economic growth modes, which we might call hunting, farming, and industry.
 A Sum of Exponentials Model Apparently, no one has yet tried to formally model long-term growth as a sequence of exponentially growing modes. This paper thus attempts to do so for world product. One very simple model of a sequence of growth modes is a sum of exponentials, which for n modes we will write as Y (t) = Y n (t), where Y 0 (t) = 0, and for i from 1 to n, Y i (t) = Y i−1 (t) + M i (t), M i (t) = c i 2 t/τ i . The parameter τ i is the doubling time of mode i. If data consists of pairs (t j , Y j ), a min-square-error of logs method searches for model parameters of a function Y (t) that minimize j (log(Y (t j )) − log(Y j )) 2 . Applying this method to my data, the best fit sum of four exponential terms gives an eyeballpleasing fit, and a (root mean square) average percentage error of 1.8%. This is about the same magnitude as the smallest uncertainty or "indifference range" attributed to the population estimates that this data is based in part on, which is about 2% (McEvedy & Jones, 1978) . Figure 1 compares this model with the data, and indicates that the main errors seem to be due to unmodeled fluctuations with a period of roughly 500-1500 years. Figure 1 M i (d i ) = M i−1 (d i ). (The questionable doubling time factor for the hunting mode comes from comparing it to the previous growth rate of animal brains, as discussed in the footnotes. 6 ) Table 1 also gives a world product factor, defined as Y i (d i+1 )/Y i−1 (d i ), which says how much world product increased due to each term.
 CES-Combined Exponentials While most of the modes in the above model seem to have a familiar interpretation, the second to last mode, the one with a 58 year doubling time, seems more difficult to interpret. We might consider it to represent a commercial revolution, distinct from the later industrial revolution. But visibly, it seems to just be there to allow the model to describe a slower and smoother transition than a simple sum of exponentials will allow between the modes before and after it. To more directly model the possibility of a slow versus fast transitions between modes, we can combine terms not with a sum, but instead with a constant elasticity of substitution (CES) form (Barro & Sala-i-Martin, 1995) , as in Y i (t) = (Y i−1 (t) a i + M i (t) a i ) 1/a i . When the CES power is a i = 1, this is x + y, the previous case. When a i = 0, this is proportional to (Y i−1 M i ), which for exponential terms essentially makes an infinitely slow transition. When If we now fit such a CES combination of three exponential terms to our data, again using the min square error of logs method, we get an eyeball-pleasing fit, and an average percentage error of 1.7%. This model is described 7 in Table 2 , and compared to the data in Figures 1 and 2 . This model has the same number of free parameters as the four-term sum of exponentials model, and a modestly smaller total squared error of logs. a i is positive infinity, this is max(Y i−1 , M i ), Note that the industry mode will not begin to dominate until 2020, has a doubling time which is half of what we have observed over the last half century, and is already responsible for a larger world product factor than either of the previous two modes. The very low transition power allows a new mode to have a strong influence long before it officially dominates, and predicts a mid-transition growth rate of about half of the new mode growth rate.
 Interpreting the Modes By describing world product history as either a sum or CES-combination of exponentials, one seems to essentially be saying that among the thousands of large and important changes that the world economy has seen, only a handful are fundamentally big news. Other changes are either "small," at least in the big picture, or such that a change of broadly similar economic magnitude and timing was largely predetermined by the fundamental economic forces at play. Typically, the economy is dominated by one particular mode of economic growth, which produces a roughly constant growth rate. While there are often economic processes which grow exponentially at a rate much faster than that of the economy as a whole, such processes almost always slow down as they become limited by the size of the total economy. Very rarely, however, a faster process reforms the economy so fundamentally that overall economic growth rate accelerates to track this new process. The economy might then be thought of as composed of an old sector and a new sector, a new sector which continues to grow at its same speed even when it comes to dominate the economy. Following this line of interpretation, the simple sum of exponentials model seems to assume that during each transition, the "new" and "old" economies co-exist but do not influence each other. In contrast, the CES-combination form seems to describe interactions between modes. Modes might complement each other (0 < a < 1), making for a more gradual transition, or substitute for each other (a > 1), making for a sharper transition. The large transition centered around 5000 B.C. (but surely begin thousands of years earlier) is naturally interpreted as the transition to from "hunting" to "farming" (both broadly construed). Populations relying on food from the domestication of plants and animals seem to have grown exponentially for several millennia before this transition date, nicely fitting a simple interpretation of co-existing new and old economies (Diamond, 1997) . In the CES model, the power estimate for the farming transition suggests that farming mostly substituted for hunting, rather than complementing it. Because the data is poor, this suggestion can only be weak. But it does make sense, given that hunting and farming economic modes competed for the use of both land and labor. 8 The recent "industrial revolution" transition has a CES power describing a much more gradual transition, suggesting less substitution and more complementarity between the old and new economic growth modes. If we interpreted the old and new modes relatively literally as "farming" and "industry", this makes some sense, in that improved farming technology has freed workers for industry while improved industry technology has often transferred to farming technology. Such a literal interpretation, however, is argued against by the CES model estimate that we will not "officially" transition to the new mode until 2020, in the sense of having the new contribution to the world economy exceed the old part. For this to make sense, much of our non-farming economy needs to be interpreted as still dominated by the "old" growth mode. The form of our model suggests that the new growth mode might be found in small quantities well before it has a noticeable effect on total growth rates, and that what distinguishes this new economy inside the old one is that it is growing rapidly, at a rate closely related to what will become the new total economic growth rate. We might thus better understand what distinguishes the current "new" and "old" growth modes by looking for processes which doubled before the transition at a rate similar to the modeled new rates of 6 or 15 years. Some interesting candidates are found in measures of scientific progress. The number of scientific journals has doubled steadily about every 15 years since about 1750, even though the world product doubling time in 1750 was around a century. (From about 1650 to 1750 both journals and scientific societies doubled roughly every 30 years.) Narrower measures of progress grow slower, while broader measures grow faster. For example, the number of "important" discoveries has doubled every 20 years, while the number of U.S. engineers has doubled every 10 years (de Solla Price, 1963) . This rough coincidence in timing and doubling times suggests, though only weakly, that a new "scientific" evolution and diffusion of knowledge via a subject-specific network of articulate specialists is fundamental to the new industry growth mode. Perhaps the diffusion of seeds and artifacts via a simpler more spatial network of less specialized people is fundamental to the farming mode. If so, the new 15 year doubling time of the sum of exponentials model suggests that published papers are close to the fundamental unit of growth, while the 6 year doubling time of the CES model suggests that a very broad measure of "scientific"-like activity is closer to the fundamental unit of growth. How can we make sense of the basic idea of the world economy repeatedly switching to new growth modes? That is, if the economy is capable of faster growth modes, what prevented them from happening earlier? Among the many models mentioned before of transitions between historical economic growth modes, perhaps the simplest postulate minimum scale effects, so that a new growth mode will not occur in economies below a certain size, density, or technology level. A stochastic variation on this is to make the number of attempts or chances to discover and transition to a new growth mode be proportional to the current scale of the economy. This variation also predicts that a transition will occur at roughly the time when the economy reaches some predetermined scale. Such models could interpret the roughly stair-step shape of Figure 2 somewhat literally. The lip of each step would describe a new mode which would not be realized until world product, moving along the x-axis, reached the base of that step.
 Could It Happen Again? Current U.S. stock prices seem to be much higher than can be justified by the expectation that the U.S. economy will continue much as it has for the last century (Shiller, 2000) . If speculators are rational, it seems that they must be assigning a substantial probability to the possibility of a large change in the economy, such as a fall in the risk-premium investors demand, a fall in the real risk of corporate profits, a rise in the share of income taken by corporate profits, or a rise in real economic growth rates. It is this last possibility, of a large rise in economic growth rates, that seems to be offered some support by the empirical analysis of very long term economic growth. If one takes seriously the idea of long-term economic history as a sequence of exponential growth modes, one might naturally wonder if a world economy that seems to have so far seen at least three dramatic transitions to much faster growth modes will see yet another such transition. One pair of authors, Ausubel and Meyer, did informally "speculate on a fourth [Deevey-like] 'information' pulse starting now that would enable another order-of-magnitude rise" in population. But they took the idea no further (Ausubel & Meyer, 1994) . Are there ways an economy can grow fast that will not be possible until the world economy reaches a threshold in size, density, or the price of some key technology? While it is certainly possible that the economy is approaching fundamental limits to economic growth rates or levels, so that no faster modes are possible, we should also consider the other possibility. To use our models to estimate how soon the world economy might jump again to a faster growth mode, and how much faster that mode might be, we must make some assumptions about what it is that is similar across past and future modes and mode transitions. The apparent regularities in Figure 2 suggest one approach. They suggest that we treat multiples of world product and growth rates as similar across growth modes. More precisely, if we assume that at each transition new values of the doubling time factor and world product factor are drawn again from the same constant-in-time distributions for these parameters, then previous values of these parameters offer "sample" estimates of future parameter values. In the sum of four exponentials model, if the current mode were to last through as large a world product factor as one of the previous three growth modes, it would last until one of the sample dates of 1963, 2032, or 2066. If the next doubling time factor were the same as one of the last three values, the next doubling time would be either would be .05, 1, or 4 years. 9 The world product method expects that the world product factor, as previously defined, of the current mode will be similar to that of the previous modes. The doubling time method instead expects to see a similar value for (d i+1 − d i )/τ i , which is the number of doubling times during the period when a mode is dominant. Since the "current" mode does not even start to dominate until 2020, the doubling time method naturally gives later date estimates. The estimates of the new doubling time offer what seems to be a remarkably precise estimate of an amazingly fast growth rate. One is tempted to immediately reject such rapid growth as too preposterous to consider, until one remembers how preposterous it would have seemed to forecast the previous two transitions that did in fact occur. The robustness of these forecasts can be examined by considering what these types of models would have predicted using only the data up to 1900. From 1900 to 2000, world product increased by a factor of 37. The best fit four term sum of exponentials model forecasts that a factor of 35 increase, while the best fit three term CES model forecasts a factor of 22 increase. The main reason for the CES model failure here is estimating a transition date of about 2060, instead of the 2020 date suggested by the larger dataset. Other parameter estimates were in much closer agreement. 10 Another row corresponding to a brain growth mode would give sample dates of 2072 and 2120. Using the out of Africa population estimate would change the new doubling time estimates from 14.9?,9.3,15.8 to 1.49?,93,15.8 3 and Figure 4 show how a new transition might look, if the new mode had a 15 day doubling time and the same strong complementarity with our current mode that our current mode had with the previous mode, and if the new mode first started to have a noticeable effect on growth rates in 2041. (Transitions starting at any other date would look very similar.) Table 4 makes it clear that in this scenario, the new mode has an enormous effect on economic growth rates very soon after it has any noticeable effect on growth rates. If this growth mode did not hit limits to slow it down, then by 2047, i.e., within six years of becoming noticeable, the economy would then grow by a larger factor than it had from two million B.C. until 2040. This is thus not at all a reasonable model for describing the recent small but noticeable increase in U.S. economic growth rates that some now attribute to a "new industrial revolution." Thus if stock prices are now high in anticipation of a transition to a new growth mode, it must be in anticipation of a much smaller change in growth rates, of a change that has not yet detectably changed growth rates, or of a much more gradual transition between growth modes. In summary, if one takes seriously the model of economic growth as a series of exponential growth modes, and if relative change parameters of a new transition are likely to be similar to such parameters describing old transitions, then it seems hard to escape the conclusion that the world economy could see a very dramatic change within the next century, to a new economic growth mode with a doubling time of roughly two weeks or less. While it is hard to see in much detail how the world economy could possibly grow that fast, it is at least suggestive that computer hardware costs have in fact been consistently falling for many decades at a doubling time of one to two years (Moravec, 1998; Kurzweil, 1999) . Several observers have predicted a transition to a dramatically new economic mode where computers substitute for most human labor, perhaps not long after the raw computational ability of a desktop computer matches the computational ability of a human brain, around 2025 (Moravec, 1998; Kurzweil, 1999) . Simple formal models have even been developed suggesting that adding robots to very standard growth models can allow growth rates to naturally increase by a factor of ten, a hundred, or even more (Hanson, 1998) . Others have speculated that even faster growth rates are possible with mature nanotechnology. If the next mode had a "slow" doubling time of two years, and if it lasted through twenty doubling times, longer than any mode seen so far, it would still last only forty years. After that, it is not clear how many more even faster growth modes are possible before hitting fundamental limits. But it is hard to see how such fundamental limits would not be reached within a few decades at most.
 Conclusion Many people have tried to make sense of very long term time series estimating human population. Some have explicitly modeled this history as steadily accelerating growth, while others have more informally described it as a sequence of growth modes. And exponential growth is by far the most common formal model of shorter term growth. Meanwhile, Brad De Long has recently constructed long term world product estimates from recent world product estimates and older population estimates. This paper has thus attempted the neglected task of more formally describing long term world product growth as a sequence of exponential growth modes. We have found that a time series of world product over the last two million years can be comfortably described as either as a sum of four exponentials, or as a CES-combination of three exponentials. But the CES model seems preferable on theoretical grounds, which can be thought of as describing the three modes of "hunt-ing," "farming," and "industry," broadly construed. (An earlier period of exponential growth in brain sizes may be the relevant previous growth mode.) In the CES-combination model, there is a sharp transition between hunting and farming, and a smooth transition between farming and industry. These can be interpreted as due to the interaction between hunting and farming being more one of substitution, relative to a more complementary relation between farming and industry. The rough timing and doubling time coincidence between industry and measures of scientific progress weakly suggests that scientific-like creation and diffusion of knowledge might be a key to the current growth mode. Since there seem to be some rough regularities regarding how much the economy grows between transitions, and how much faster each new growth mode is, this paper has also considered what these regularities suggest about when we might see yet another transition to a much faster mode, if such faster modes are possible. The suggestion is fantastic, namely of a transition to a doubling time of two weeks or less sometime within roughly the next century. One might think this suggestion too fantastic to consider, were it not for the fact that similar predictions before previous transitions would have seemed similarly fantastic. We should also keep in mind that investors in U.S. stocks seem to be betting that a large change in the nature of the economy is likely soon. They may well be wrong, but this paper shows that the empirical case against such expectations is not as clear as one might expect from empirical analyzes looking at the last century or so. From a purely empirical point of view, very large changes are actually to be expected within the next century. of population estimates from McEvedy and Jones (McEvedy & Jones, 1978) going back to 10 thousand B.C., and of three population estimates from Deevey (Deevey, 1960) for 25 thousand B.C., 300 thousand B.C. and one million B.C.
 population estimate of about half a million between one million B.C. and 500 thousand B.C., which I've coded as at 750 thousand B.C. 1
 also shows the best fit hyperbolic model, 4Y (t) = c(T − t) −α ,which has a power of α = 1.46, goes to infinity at T = 2004, and has a much inferior average percentage error of 9.6%. (Using three terms instead of four in the sum of exponentials model also gives a visibly much worse fit.)For each term,
 with an infinitely fast transition. Other positive values of a i allow for other intermediate speeds of transition.

 Table 1 : 1 Table 1 lists a mode description, doubling time 5 , the factor by which the doubling Sum of Exponentials Growth Modes time accelerated between terms, and a "date began to dominate," defined as the d i that solves Growth Doubling Date Began Doubling World Product Mode Time (years) To Dominate Time Factor Factor Hunting 230,000 2,000,000 B.C. 149? 406 Farming 860 4700 B.C. 267 178 ?? 58 1730 A.D. 14.9 8 .9 Industry 15 1903 A.D. 3.8 > 78
 Table 2 : 2 CES-combined Exponentials Growth Modes Growth Doubling Date Began Doubling World Product Transition Mode Time (year) To Dominate Time Factor Factor CES Power Hunting 224,000 2,000,000 B.C. 153? 480 ? Farming 909 4860 B.C. 247 190 2.4 Industry 6.3 2020 A.D. 145 > 590 0.094
 Table 3 : 3 Table 3 describes sample forecasts based on the CES model (which the author prefers). It9 If the next mode were like the brain growth mode, it would start in 2162 and have a doubling time of 0.1 years. Using the CES Model to Forecast The New Mode includes sample forecasts of the next economic doubling time, and two kinds of estimates of the next transition date. 10 Growth Doubling Time New Doubling World Product New Date New Date Mode (DT) Factor Time (days) (WP) Factor from WP from DT Hunting 153? 14.9? 480 1996 2075 Farming 247 9.3 190 1976 2067 Industry 145 15.8 > 590
 Table 4 : 4 . New Transition Scenario, Instantaneous Annual Growth Rates For example, the up-to-1900 CES estimate of the new doubling time is 6.4 years, very close to the up-to-2000 estimate of 6.3 years. Timing estimates thus seem less reliable than growth rate estimates. 2039 2040 2041 2042 2043 2044 2045 2046 2047 6.1% 6.1% 6.6% 8.0% 14% 41% 147% 476% 1023%
 Table 4 , 4 Figure
 The best fit "log-periodic model," of the formY (t) = c(T − t) −α (1 + b cos(ω ln(T − t) + φ)) (Johansen & Sornette, 2000) , has a power of α = 1.48, goes to infinity at T = 2005, and has an average percentage error of 7.4%.5 Under the "out of Africa" theory, the hunting mode began ten times more recently, at 200 thousand B.C. Substituting this estimate for the two multi-regional estimates would reduce the hunting doubling time, increase the hunting doubling time factor, and reduce the farming doubling time factor, all by a factor of ten. All other entries would remain unchanged.
 For reference, an additional row of Table1describing the earlier brain size growth mode would have entries: 34,000,000, 550,000,000 B.C., ??, and "67,000." The last entry is the total increase of brain size, which isn't directly comparable to the other rows since it is not an increase in world product.
 An additional row describing the earlier brain size growth mode would have the same entries as before, and the "out of Africa" theory would change the same entries by a factor of ten.
 Maximum brain size does not translate very directly into world product, but the long roughly steady growth in maximum brain size has been mentioned in this paper because it may be the relevant growth mode prior to the early spread of the human population. Product growth in an economy of human-like creatures seems to have awaited the evolution of a land-based animal with hands, a large enough brain, and perhaps other unknown prerequisites.