Showing versus doing: Teaching by demonstration

Introduction Is there a difference between doing something and showing someone else how to do something? Consider cooking a chicken. To cook one for dinner, you would do it in the most efficient way possible while avoiding contaminating other foods. But, what if you wanted to teach a completely naïve observer how to prepare poultry? In that case, you might take pains to emphasize certain aspects of the process. For example, by ensuring the observer sees you wash your hands thoroughly after handling the uncooked chicken, you signal that it is undesirable (and perhaps even dangerous) for other ingredients to come in contact with raw meat. More broadly, how could an agent show another agent how to do a task, and, in doing so, teach about its underlying reward structure? To model showing, we draw on psychological research on learning and teaching concepts by example. People are good at this. For instance, when a teacher signals their pedagogical intentions, children more frequently imitate actions and learn abstract functional representations [6, 7] . Recent work has formalized concept teaching as a form of recursive social inference, where a teacher chooses an example that best conveys a concept to a learner, who assumes that the teacher is choosing in this manner [14] . The key insight from these models is that helpful teachers do not merely select probable examples of a concept, but rather choose examples that best disambiguate a concept from other candidate concepts. This approach allows for more effective, and more efficient, teaching and learning of concepts from examples. We can extend these ideas to explain showing behavior. Although recent work has examined userassisted teaching [8] , identified legible motor behavior in human-machine coordination [9] , and analyzed reward coordination in game theoretic terms [11] , previous work has yet to successfully model how people naturally teach reward functions by demonstration. Moreover, in Inverse Reinforcement Learning (IRL), in which an observer attempts to infer the reward function that an expert (human or artificial) is maximizing, it is typically assumed that experts are only doing the task and not intentionally showing how to do the task. This raises two related questions: First, how does a person showing how to do a task differ from them just doing it? And second, are standard IRL algorithms able to benefit from human attempts to show how to do a task? In this paper, we investigate these questions. To do so, we formulate a computational model of showing that applies Bayesian models of teaching by example to the reward function learning setting. We contrast this pedagogical model with a model of doing: standard optimal planning in Markov Decision Processes. The pedagogical model predicts several systematic differences from the standard planning model, and we test whether human participants reproduce these distinctive patterns. For instance, the pedagogical model chooses paths to a goal that best disambiguates which goal is being pursued (Experiment 1). Similarly, when teaching feature-based reward functions, the model will prioritize trajectories that better signal the reward value of state features or even perform trajectories that would be inefficient for an agent simply doing the task (Experiment 2). Finally, to determine whether showing is indeed better than doing, we train a standard IRL algorithm with our model trajectories and human trajectories.
 A Bayesian Model of Teaching by Demonstration Our model draws on two approaches: IRL [2] and Bayesian models of teaching by example [14] . The first of these, IRL and the related concept of inverse planning, have been used to model people's theory of mind, or the capacity to infer another agent's unobservable beliefs and/or desires through their observed behavior [5] . The second, Bayesian models of pedagogy, prescribe how a teacher should use examples to communicate a concept to an ideal learner. Our model of teaching by demonstration, called Pedagogical Inverse Reinforcement Learning, merges these two approaches together by treating a teacher's demonstration trajectories as communicative acts that signal the reward function that an observer should learn.
 Learning from an Expert's Actions
 Markov Decision Processes An agent that plans to maximize a reward function can be modeled as the solution to a Markov Decision Process (MDP). An MDP is defined by the tuple < S, A, T, R, γ >: a set of states in the world S; a set of actions for each state A(s); a transition function that maps states and actions to next states, T : S × A → S (in this work we assume all transitions are deterministic, but this can be generalized to probabilistic transitions); a reward function that maps states to scalar rewards, R : S → R; and a discount factor γ ∈ [0, 1]. Solutions to an MDP are stochastic policies that map states to distributions over actions, π : S → P (A(s)). Given a policy, we define the expected cumulative discounted reward, or value, V π (s), at each state associated with following that policy: V π (s) = E π ∞ k=0 γ k r t+k+1 | s t = s . (1) In particular, the optimal policy for an MDP yields the optimal value function, V \* , which is the value function that has the maximal value for every state (V \* (s) = max π V π (s), ∀s ∈ S). The optimal policy also defines an optimal state-action value function, Q \* (s, a) = E π [r t+1 + γV \* (s t+1 ) | s t = s, a t = a]. Q i = calculateActionValues(s, R i , T , γ) 4: π i = softmax(Q i , λ) 5: Π.add(π i ) 6: Calculate j = {j : s 1 ∈ s, length(j) ≤ l max , and ∃π ∈ Π s.t. (si,ai)∈j π(a i | s i ) > p min }. In the Reinforcement Learning setting, an agent takes actions in an MDP and receives rewards, which allow it to eventually learn the optimal policy [15] . We thus assume that an expert who knows the reward function and is doing a task selects an action a t in a state s t according to a Boltzmann policy, which is a standard soft-maximization of the action-values: P Doing (a t | s t , R) = exp{Q \* (s i , a i )/λ} a ∈A(si) exp{Q \* (s i , a )/λ} . (2) λ > 0 is an inverse temperature parameter (as λ → 0, the expert selects the optimal action with probability 1; as λ → ∞, the expert selects actions uniformly randomly). In the IRL setting, an observer sees a trajectory of an expert executing an optimal policy, j = {(s 1 , a 1 ), (s 2 , a 2 ), ..., (s k , a k )}, and infers the reward function R that the expert is maximizing. Given that an agent's policy is stationary and Markovian, the probability of the trajectory given a reward function is just the product of the individual action probabilities, P Doing (j | R) = t P Doing (a t | s t , R). From a Bayesian perspective [13] , the observer is computing a posterior probability over possible reward functions R: P Observing (R | j) = P Doing (j | R)P (R) R P Doing (j | R )P (R ) . (3) Here, we always assume that P (R) is uniform.
 Bayesian Pedagogy IRL typically assumes that the demonstrator is executing the stochastic optimal policy for a reward function. But is this the best way to teach a reward function? Bayesian models of pedagogy and communicative intent have shown that choosing an example to teach a concept differs from simply sampling from that concept [14, 10] . These models all treat the teacher's choice of a datum, d, as maximizing the probability a learner will infer a target concept, h: P Teacher (d | h) = P Learner (h | d) α d P Learner (h | d ) α . (4) α is the teacher's softmax parameter. As α → 0, the teacher chooses uniformly randomly; as α → ∞, the teacher chooses d that maximally causes the learner to infer a target concept h; when α = 1, the teacher is "probability matching". The teaching distribution describes how examples can be effectively chosen to teach a concept. For instance, consider teaching the concept of "even numbers". The sets {2, 2, 2} and {2, 18, 202} are both examples of even numbers. Indeed, given finite options with replacement, they both have the same probability of being randomly chosen as sets of examples. But {2, 18, 202} is clearly better for helpful teaching since a naïve learner shown {2, 2, 2} would probably infer that "even numbers" means "the number 2". This illustrates an important aspect of successful teaching by example: that examples should not only be consistent with the concept being taught, but should also maximally disambiguate the concept being taught from other possible concepts.
 Pedagogical Inverse Reinforcement Learning To define a model of teaching by demonstration, we treat the teacher's trajectories in a reinforcementlearning problem as a "communicative act" for the learner's benefit. Thus, an effective teacher will modify its demonstrations when showing and not simply doing a task. As in Equation 4 , we can define a teacher that selects trajectories that best convey the reward function: P Showing (j | R) = P Observing (R | j) α j P Observing (R | j ) α . (5) In other words, showing depends on a demonstrator's inferences about an observer's inferences about doing. This model provides quantitative and qualitative predictions for how agents will show and teach how to do a task given they know its true reward function. Since humans are the paradigm teachers and a potential source of expert knowledge for artificial agents, we tested how well our model describes human teaching. In Experiment 1, we had people teach simple goal-based reward functions in a discrete MDP. Even though in these cases entering a goal is already highly diagnostic, different paths of different lengths are better for showing, which is reflected in human behavior. In Experiment 2, people taught more complex feature-based reward functions by demonstration. In both studies, people's behavior matched the qualitative predictions of our models. 3 Experiment 1: Teaching Goal-based Reward Functions Consider a grid with three possible terminal goals as shown in Figure 1 . If an agent's goal is &, it could take a number of routes. For instance, it could move all the way right and then move upwards towards the & (right-then-up) or first move upwards and then towards the right (up-then-right). But, what if the agent is not just doing the task, but also attempting to show it to an observer trying to learn the goal location? When the goal is &, our pedagogical model predicts that up-then-right is the more probable trajectory because it is more disambiguating. Up-then-right better indicates that the intended goal is & than right-then-up because right-then-up has more actions consistent with the goal being #. We have included an analytic proof of why this is the case for a simpler setting in the supplementary materials. Additionally, our pedagogical model makes the prediction that when trajectory length costs are negligible, agents will engage in repetitive, inefficient behaviors that gesture towards one goal location over others. This "looping" behavior results when an agent can return to a state with an action that has high signaling value by taking actions that have a low signaling "cost" (i.e. they do not signal something other than the true goal). Figure 1d shows an example of such a looping trajectory. In Experiment 1, we tested whether people's showing behavior reflected the pedagogical model when reward functions are goal-based. If so, this would indicate that people choose the disambiguating path to a goal when showing.
 Experimental Design Sixty Amazon Mechanical Turk participants performed the task in Figure 1 . One was excluded due to missing data. All participants completed a learning block in which they had to find the reward location without being told. Afterwards, they were either placed in a Do condition or a Show condition. Participants in Do were told they would win a bonus based on the number of rewards (correct goals) they reached and were shown the text, "The reward is at location X", where X was one of the three symbols %, #, or &. Those in Show were told they would win a bonus based on how well a randomly matched partner who was shown their responses (and did not know the location of the reward) did on the task. On each round of Show, participants were shown text saying "Show your partner that the reward is at location X". All participants were given the same sequence of trials in which the reward locations were <%, &, #, &, %, #, %, #, &>.
 Results As predicted, Show participants tended to choose paths that disambiguated their goal as compared to Do participants. We coded the number of responses on & and % trials that were "showing" trajectories based on how they entered the goal (i.e. out of 3 for each goal). On & trials, entering from the left, and on % trials, entering from above were coded as "showing". We ran a 2x2 ANOVA with Show vs Do as a between-subjects factor and goal (% vs &) as a repeated measure. There was a main effect of condition (F (1, 57) = 16.17, p < .001; Show: M = 1.82, S.E. 0.17; Do: M = 1.05, S.E. 0.17 The model does not predict any difference between conditions for the # (lower right) goal. However, a visual analysis suggested that more participants took a "swerving" path to reach #. This observation was confirmed by looking at trials where # was the goal and comparing the number of swerving trials, which was defined as making more than one change in direction (Show: M = 0.83, Do: M = 0.26; two-sided t-test: t(44.2) = 2.18, p = 0.03). Although not predicted by the model, participants may swerve to better signal their intention to move 'directly' towards the goal.
 Discussion Reaching a goal is sufficient to indicate its location, but participants still chose paths that better disambiguated their intended goal. Overall, these results indicate that people are sensitive to the distinction between doing and showing, consistent with our computational framework.
 Experiment 2: Teaching Feature-based Reward Functions Experiment 1 showed that people choose disambiguating plans even when entering the goal makes this seemingly unnecessary. However, one might expect richer showing behavior when teaching more complex reward functions. Thus, for Experiment 2, we developed a paradigm in which showing how to do a task, as opposed to merely doing a task, makes a difference for how well the underlying reward function is learned. In particular, we focused on teaching feature-based reward functions that allow an agent to generalize what it has learned in one situation to a new situation. People often use feature-based representations for generalization [3] , and feature-based reward functions have been used extensively in reinforcement learning (e.g. [1] ). We used a colored-tile grid task shown in Figure 2 to study teaching feature-based reward functions. White tiles are always "safe" (reward of 0), while yellow tiles are always terminal states that reward 10 points. The remaining 3 tile types-orange, purple, and cyan-are each either "safe" or "dangerous" (reward of −2). The rewards associated with the three tile types are independent, and nothing about the tiles themselves signal that they are safe or dangerous. A standard planning algorithm will reach the terminal state in the most efficient and optimal manner. Our pedagogical model, however, predicts that an agent who is showing the task will engage in specific behaviors that best disambiguate the true reward function. For instance, the pedagogical model is more likely to take a roundabout path that leads through all the safe tile types, choose to remain on a safe colored tile rather than go on the white tiles, or even loop repeatedly between multiple safe tile-types. All of these types of behaviors send strong signals to the learner about which tiles are safe as well as which tiles are dangerous.
 Experimental Design Sixty participants did a feature-based reward teaching task; two were excluded due to missing data. In the first phase, all participants were given a learning-applying task. In the learning rounds, they interacted with the grid shown in Figure 2 while receiving feedback on which tiles won or lost points. Safe tiles were worth 0 points, dangerous tiles were worth -2 points, and the terminal goal tile was worth 5 points. They also won an additional 5 points for each round completed for a total of 10 points. Each point was worth 2 cents of bonus. After each learning round, an applying round occurred in which they applied what they just learned about the tiles without receiving feedback in a new grid configuration. They all played 8 pairs of learning and applying rounds corresponding to the 8 possible assignments of "safe" and "dangerous" to the 3 tile types, and order was randomized between participants. As in Experiment 1, participants were then split into Do or Show conditions with no feedback. Do participants were told which colors were safe and won points for performing the task. Show participants still won points and were told which types were safe. They were also told that their behavior would be shown to another person who would apply what they learned from watching the participant's behavior to a separate grid. The points won would be added to the demonstrator's bonus.
 Results Responses matched model predictions. Do participants simply took efficient routes, whereas Show participants took paths that signaled tile reward values. In particular, Show participants took paths that led through multiple safe tile types, remained on safe colored tiles when safe non-colored tiles were available, and looped at the boundaries of differently colored safe tiles.
 Model-based Analysis To determine how well the two models predicted human behaviors globally, we fit separate models for each reward function and condition combination. We found parameters that had the highest median likelihood out of the set of participant trajectories in a given reward function-condition combination. Since some participants used extremely large trajectories (e.g. >25 steps) and we wanted to include an analysis of all the data, we calculated best-fitting state-action policies. For the standard-planner, it is straightforward to calculate a Boltzmann policy for a reward function given λ. For the pedagogical model, we first need to specify an initial model of doing and distribution over a finite set of trajectories. We determine this initial set of trajectories and their probabilities using three parameters: λ, the softmax parameter for a hypothetical "doing" agent that the model assumes the learner believes it is observing; l max , the maximum trajectory length; and p min , the minimum probability for a trajectory under the hypothetical doing agent. The pedagogical model then uses an α parameter that determines the degree to which the teacher is maximizing. State-action probabilities are calculated from a distribution over trajectories using the equation P (a | s, R) = j P (a | s, j)P (j | R), where P (a | s, j) = |{(s,a):s=st,a=at∀(st,at)∈j}| |{(s,a):s=st∀(st,at)∈j}| . We fit parameter values that produced the maximum median likelihood for each model for each reward function and condition combination. These parameters are reported in the supplementary materials. The normalized median fit for each of these models is plotted in Figure 3 . As shown in the figure, the standard planning model better captures behavior in the Do condition, while the pedagogical model better captures behavior in the Show condition. Importantly, even when the standard planning model could have a high λ and behave more randomly, the pedagogical model better fits the Show condition. This indicates that showing is not simply random behavior.
 Behavioral Analyses We additionally analyzed specific behavioral differences between the Do and Show conditions predicted by the models. When showing a task, people visit a greater variety of safe tiles, visit tile types that the learner has uncertainty about (i.e. the colored tiles), and more frequently revisit states or "loop" in a manner that leads to better signaling. We found that all three of these behaviors were more likely to occur in the Show condition than in the Do condition. To measure the variety of tiles visited, we calculated the entropy of the frequency distribution over colored-tile visits by round by participant. Average entropy was higher for Show (Show: M = 0.50, SE = 0.03; Do: M = 0.39, SE = 0.03; two-sided t-test: t(54.9) = −3.27, p < 0.01). When analyzing time spent on colored as opposed to un-colored tiles, we calculated the proportion of visits to colored tiles after the first colored tile had been visited. Again, this measure was higher for Show (Show: M = 0.87, SE = 0.01; Do: M = 0.82, SE = 0.01; two-sided t-test: t(55.6) = −3.14, p < .01). Finally, we calculated the number of times states were revisited in the two conditions-an indicator of "looping"-and found that participants revisited states more in Show compared to Do (Show: M = 1.38, SE = 0.22; Do: M = 0.10, SE = 0.03; two-sided t-test: t(28.3) = −2.82, p < .01). There was no difference between conditions in the total rewards won (two-sided t-test: t(46.2) = .026, p = 0.80).
 Teaching Maximum-Likelihood IRL One reason to investigate showing is its potential for training artificial agents. Our pedagogical model makes assumptions about the learner, but it may be that pedagogical trajectories are better even for training off-the-shelf IRL algorithms. For instance, Maximum Likelihood IRL (MLIRL) is a state-of-the-art IRL algorithm for inferring feature-based reward functions [4, 12] . Importantly, unlike the discrete reward function space our showing model assumes, MLIRL estimates the maximum likelihood reward function over a space of continuous feature weights using gradient ascent. To test this, we input human and model trajectories into MLIRL. We constrained non-goal feature weights to be non-positive. Overall, the algorithm was able to learn the true reward function better from showing than doing trajectories produced by either the models or participants (Figure 2 ).
 Discussion When learning a feature-based reward function from demonstration, it matters if the demonstrator is showing or doing. In this experiment, we showed that our model of pedagogical reasoning over trajectories captures how people show how to do a task. When showing as opposed to simply doing, demonstrators are more likely to visit a variety of states to show that they are safe, stay on otherwise ambiguously safe tiles, and also engage in "looping" behavior to signal information about the tiles. Moreover, this type of teaching is even better at training standard IRL algorithms like MLIRL.
 General Discussion We have presented a model of showing as Bayesian teaching. Our model makes accurate quantitative and qualitative predictions about human showing behavior, as demonstrated in two experiments. Experiment 1 showed that people modify their behavior to signal information about goals, while Experiment 2 investigated how people teach feature-based reward functions. Finally, we showed that even standard IRL algorithms benefit from showing as opposed to merely doing. This provides a basis for future study into intentional teaching by demonstration. Future research must explore showing in settings with even richer state features and whether more savvy observers can leverage a showing agent's pedagogical intent for even better learning. 7 : 7 Construct hypothetical doing probability distribution P Doing (j | R) as an N x M array. 8: P Observing (R | j) = PDoing(j|R)P (R) R PDoing(j|R )P (R ) 9: P Showing (j | R) = PObserving(R|j) α j PObserving(R|j ) α 10: return P Showing (j | R) 2.1.2 Inverse Reinforcement Learning (IRL)
 Figure 1 : 1 Figure 1: Experiment 1: Model predictions and participant trajectories for 3 trials when the goal is (a) &, (b) %, and (c) #. Model trajectories are the two with the highest probability (λ = 2, α = 1.0, p min = 10 −6 , l max = 4). Yellow numbers are counts of trajectories with the labeled tile as the penultimate state. (d) An example of looping behavior predicted by the model when % is the goal.
 ) as well as a main effect of goal (F (1, 57) = 4.77, p < .05; %-goal: M = 1.73, S.E. = 0.18; &-goal: M = 1.15, S.E. = 0.16). There was no interaction (F (1, 57) = 0.98, p = 0.32).
 Figure 2 : 2 Figure 2: Experiment 2 results. (a) Column labels are reward function codes. They refer to which tiles were safe (o) and which were dangerous (x) with the ordering . Row 1: Underlying reward functions that participants either did or showed; Row 2: Do participant trajectories with visible tile colors; Row 3: Show participant trajectories; Row 4: Mean reward function learned from Do trajectories by Maximum-Likelihood Inverse Reinforcement Learning (MLIRL) [4, 12]; Row 5: Mean reward function learned from Show trajectories by MLIRL. (b) Mean distance between learned and true reward function weights for human-trained and model-trained MLIRL. For the models, MLIRL results for the top two ranked demonstration trajectories are shown.
 Figure 3 : 3 Figure 3: Experiment 2 normalized median model fits.
 Algorithm 1 Pedagogical Trajectory AlgorithmRequire: starting states s, reward functions {R 1 , R 2 , ..., R N }, transition function T , maximum showing trajectory depth l max , minimum hypothetical doing probability p min , teacher maximization parameter α, discount factor γ. 1: Π ← ∅ 2: for i = 1 to N do 3:

M. K. Ho
M. L. Littman
J. MacGlashan
F. Cushman
J. L. Austerweil


The case against economic values in the orbitofrontal cortex (or anywhere else in the brain)

The past 20 years have seen a great deal of interest in understanding how our brains implement economic choices (Camerer et al., 2005; Glimcher & Fehr, 2013; Loewenstein et al., 2008; Padoa-Schioppa, 2011; Rangel et al., 2008; Rushworth et al., 2011) . Much research in this field of neuroeconomics rests on the assumption that choices between options rely on an explicit valuation process (Kable & Glimcher, 2009; Levy & Glimcher, 2012; Montague & Berns, 2002; O'Doherty, 2014; Padoa-Schioppa, 2011) . That is, that the brain first assigns value to each option and then compares those values to determine choice. The concept of valuation-which stems from economic theory-is so ingrained that it may seem inevitable. How else would one literally compare apples to oranges? Indeed, the idea that value exists on a single cardinal scale, also called a "common currency," has been extended to encompass not only goods, but also effort costs and time delays; everything an agent needs to bundle into evaluation of a mode of action that is aimed at procuring a specific outcome. In parallel, the computational framework of reinforcement learning, which has been a cornerstone of neuroeconomics, also makes a scalar value signal central to its implementation (Niv, 2009; Sutton & Barto, 2018) . Specifically, in reinforcement learning, the value of an option-a state or an action-is the expected sum of future rewards contingent upon that choice. As a sum of future rewards that may be of different types (and include costs as negative rewards), reinforcement-learning value is naturally calculated in some unitless common currency. However, not all reinforcement learning algorithms rely on or even calculate values (Sutton & Barto, 2018) , and reinforcementlearning values, as sums of future rewards, are not synonymous with economic values of specific goods. Likewise, many empirically supported process models of choice get by with no valuation (Gigerenzer & Gaissmaier, 2011; Miller et al., 2019; Vlaev et al., 2011) . The fact that the brain can compute values to compare apples and oranges does not mean that it routinely does so, or that valuation is the primary process underlying choice. In this opinion paper, we argue that in many choice scenarios the brain may not be computing values at all, despite appearing to do so. We will demonstrate that the rationale for a value signal is weak, as are both behavioral and neural evidence supporting value computation. Finally, we propose an alternative-direct learning of action policies-and suggest there is scant direct evidence for value learning that cannot be explained by this and other alternative theories. Our hypothesis is important as it suggests a different interpretation of previous data and requires that studies attempting to resolve mechanisms of value computation in the brain first establish that in the specific situation studied, valuation is indeed occurring. In particular, much work has associated the orbitofrontal cortex (OFC) in representing the expected economic value of different goods or options (Bartra et al., 2013; Levy & Glimcher, 2012 )-a role that must be critically reconsidered if we agree that the brain may not necessarily be representing such values in many of the experiments so far used to test this hypothesis.
 Why Argue Against Value? Intuitively, our thesis is that while we may know that we prefer an orange to an apple (for one thing, oranges don't brown when exposed to oxygen), we may make this judgment without consulting an internal scalar (cardinal) or even a universal ordinal value signal. Consequently, we may be hard-pressed to express the precise value that we put on an orange. This is not merely an issue of conscious access to our internal value of oranges 1 , but may be due to the fact that deciding on preferences can utilize many alternative mechanisms that don't require or rely on calculation of such a value. Valuation is hard (Payne et al., 1992) . It is also often unnecessary: when choosing between, say, an orange and a car, it is immediately clear that one is better than the other without calculating the precise value of either. If you are extremely thirsty, you might choose the orange, whereas if you need to go somewhere a car is the only relevant option. Arguably, many real-life choices are between options that are sufficiently different in the needs that they fulfill as to be more similar to this extreme example than they are to the choice between apples and oranges (Juechems & Summerfield, 2019) . Valuation may only be necessary when choosing between two very similarly valued items. However, if the items are sufficiently similar in how they satisfy our needs, the brain may decide to choose randomly, according to some valuefree heuristic, or according to past choices-and move on (Chater, 2018) . Alternatively, the brain can try to calculate the exact values of the options to the necessary precision that arbitrates between them. Indeed, choosing between similar options often takes longer than choosing between very different options, even when making rather trivial choices that do not warrant the time and effort invested (Pirrone et al., 2018; Teodorescu et al., 2016 ). An example is the inordinate amount of time some of us may spend deciding what brand of tuna fish to buy, despite the fact that our time (as per our salary) is worth much more than the money we will save by correctly evaluating which brand provides the best "value for money." This scenario also illustrates how inept we are at comparing the value of different kinds-here, time and money-which should have been straightforward if the modus operandi of the brain was to evaluate everything in terms of some common currency (see below). So perhaps the brain can, when needed, calculate values. However, we argue that this is not the main means by which the brain makes decisions, and perhaps not the natural mode of decision making.
 What Is Value? Before moving on, we would like to delineate precisely how we are defining value for the sake of our argument. The term "value" has multiple uses (the problems this multiplicity raises are carefully laid out by O'Doherty, 2014). We consider "value" a hypothesized scalar variable that reflects the worth of a specific item or outcome. 2 Because it is scalar, it is necessarily abstract. It can refer to any good, and makes use of a common currency code that is comparable across goods of different types (e.g., food, water, recreation time). Value, by this definition, is cardinal, not ordinal, meaning it can be defined for an option per se, rather than solely relative to other options. That is, it reifies the idea of "utils"-quantifiable units of value, often used in a jocular manner in Economics classes. The idea of a common currency is key as it implies that all relative calculations have already happened-the value of an option is in some denomination that objectively defines it as compared to other such values. One might argue that this is too narrow a definition, but we believe this is what neuroeconomists have in mind when talking about representation of common currency values in the brain. For example, we know of no neuroeconomic model that imagines a neural implementation of an ordinal value scale. This having been said, even an ordinal scale should not change based on the comparison set, and should satisfy transitivity-which many of the neural signals attributed to value do not, as we detail below. Our definition of value, for the purpose of this paper, is different from other types of value that are discussed in reinforcement learning. Specifically, here we are discussing value as the reward worth of a single item/event, not the expected sum of all future rewards (R in reinforcement learning models, not V). One might argue that in order to compute such an expected sum V, the subjective worth of all individual rewards must be translated to a common currency that can be added. In this sense, reinforcement learning models do presuppose a common currency reward value. However, they don't necessarily commit to economic properties of such values, such as transitivity and consistency (see also Juechems & Summerfield, 2019 ), and we discuss below a class of reinforcement learning algorithms that can make do with evaluations that are only relative to the current options, and not applicable in other situations. One important feature of our definition of value is that in our view, although value is inferred from choice, it is not strictly identical to choice, nor necessarily implied by choice. For instance, if we observe a consistent preference for A over B, A is assumed to have a higher value than B. But choice of A over B isn't sufficient to infer the existence of a self-consistent value function: a decisionmaker may adhere to a heuristic policy that results in stable preference for Slovic, 2006) . Moreover, choice can be altered without manipulating the reward value of an item (Schonberg & Katz, 2020) , which suggests that value, as inferred from choice, is not untarnished by processes that are not economic in nature. Thus, value can be inferred from behavior given certain assumptions, but preferences do not always lead to a value function. [A > B], [B > C], and [C > A] (Lichtenstein & This document is copyrighted by the American Psychological Association or one of its allied publishers. This article is intended solely for the personal use of the individual user and is not to be disseminated broadly.
 The Common-Currency Hypothesis The notion that economic decision-makers make use of internal value functions to compare options is often dated to Daniel Bernoulli's proposal in 1738 of a logarithmic utility curve to explain preferences in the St. Petersburg Paradox (Martin, 2011) . Here, a decision-maker is offered a chance to play a game in which they win $2 (n−1) with n being the first time, in a series of coin tosses, that a coin falls on "heads." The paradox is that although the expected monetary value of this game (that is, the sum of expected wins multiplied by their probability P ∞ n=1 2 ðn−1Þ • 0.5 n = P ∞ n=1 0.5) is infinity, people are not willing to pay even $10 to play this game (Hayden & Platt, 2009) . The explanation that Bernoulli proposed proved foundational within microeconomic theory. He argued that decision-makers don't base calculations on the nominal, objective cash value of the potential gain, but rather on its subjective value. If the subjective value grows more slowly than the objective value (the idea of "diminishing marginal utility"), an optimizing decision-maker will appear riskaverse, for instance, when evaluating the St. Petersburg gamble. Utility, or subjective value, has been central to many if not most microeconomic models. These include the axiomatic approaches of Pareto, Von Neumann and Morgenstern, and Samuelson. It is also central to behavioral theories such as prospect theory and decision field theory (Busemeyer & Townsend, 1993; Kahneman & Tversky, 1979) . Ironically, however, explaining choices in the St. Petersburg paradox using such a subjective value function requires utility for money that diminishes so rapidly that it does not generalize to choices in other contexts. Instead, heuristic accounts-accounts that don't rely on ideas about value maximization-provide better quantitative matches for St. Petersburg choices (Hayden & Platt, 2009) . Despite its central importance in economics, economists are typically agnostic about whether the concept of value is just a convenient description, or whether it is instantiated in the brain. As Friedman argued, decision-makers behave as if we compute and compare values, but we cannot conclude that we actually do so (Friedman, 1953) . In that work, he famously compared economic agents to a trained billiards player who makes excellent shots as if having a sophisticated grasp of Euclidean geometry, although in fact such a theoretical understanding is not necessary for good billiards skill. Economists typically stop at the point of saying that economic models are "as if" models, with some arguing that the question of the underlying reality of economic variables is outside the domain of economics (Gul & Pesendorfer, 2008; Harrison, 2008) . In contrast, neuroeconomics has generally taken as a default assumption that these "as if" theories are reified in the brain (Kable & Glimcher, 2009; Levy & Glimcher, 2012; Montague & Berns, 2002; O'Doherty, 2014; Padoa-Schioppa, 2011; Rich & Wallis, 2016) . Modern tools such as neuroimaging and single unit physiology provide the opportunity to assess the implementation of decision making directly. Indeed, neuroscientists have had little trouble identifying correlates of value in several brain areas (Knutson et al., 2001; Levy & Glimcher, 2012; Plassmann et al., 2007; Wallis, 2007) , leading to the suggestion that the OFC is a nexus for representing the economic value of goods in the brain (see, e.g., Rushworth et al., 2011; Wallis, 2007) , alongside other brain areas that are important for computing and representing value such as the ventromedial prefrontal cortex (Bartra et al., 2013) and ventral striatum (Haber & Knutson, 2010) . However, it is not clear that the signals identified in these studies actually represent value as proposed-on a common-currency scale, for comparing options and making choices. Moreover, so called "value signals" only correlate with value, so may be signaling other quantities, such as attention, action plans, vigor, or preference, which also correlate with value (Maunsell, 2004; O'Doherty, 2014; Wallis & Rich, 2011) . Hence it is important to carefully consider alternative interpretations for these findings, as we do further below, after discussing some practical constraints and philosophical conundrums.
 Alternatives to Valuation in Decision Making While common-currency value provides a convenient and general mechanism for making choices, there are many possible alternatives (Gigerenzer & Gaissmaier, 2011; Kahneman et al., 1982; Lichtenstein & Slovic, 2006; Vlaev et al., 2011) , some of which are quite general and robust. Consider for example the "priority heuristic" (Brandstätter et al., 2006) . This de minimis heuristic approach proposes that decision-makers first identify a dimension along which options vary and then compare options along that dimension. If that results in a choice, they stop; otherwise, they move on to the next dimension. This heuristic can explain many phenomena, including Allais preferences (Allais, 1953) , the reflection effect (Fishburn & Kochenberger, 1979) , the certainty effect (Kahneman & Tversky, 1979) , the fourfold pattern in risky choice that motivates prospect theory (Tversky & Fox, 1995) , and several intransitivities-all without ever requiring computing of value (Brandstätter et al., 2006) . And the priority heuristic is one of a large number of heuristics that do a remarkable job at describing behavior (Gigerenzer & Gaissmaier, 2011) . These heuristics are generally motivated by psychological observations, and thus are consistent with known data from the psychology-if not the neuroscience-of choice. In particular, they reflect the assumption that calculating value is difficult, that humans typically use shortcuts whenever possible, and that heuristics are a good shortcut (Gigerenzer & Gaissmaier, 2011; Lieder & Griffiths, 2020; Payne et al., 1992) . Moreover, these heuristics are not limited to humans, but apply to other species, including monkeys (Heilbronner & Hayden, 2016; Marsh, 2002; Santos & Rosati, 2015; Shafir et al., 2002) . Importantly for our argument, the success of heuristic approaches demonstrates that value calculations are not a priori essential for neuroeconomic theories of choice. In particular, heuristic theories can solve many problems for which value is proposed to be needed (Kahneman et al., 1982; Lichtenstein & Slovic, 2006; Piantadosi & Hayden, 2015; Stevens, 2016; Tversky, 1969; Vlaev et al., 2011) . Heuristics readily allow for comparison of multi-dimensional goods and bundles, and for choice across dissimilar goods. While heuristics are not perfect, and lead to many choice anomalies, choice is, empirically, full of anomalies. Moreover, several non-heuristic process models also eschew value computation steps, including decision by sampling, query theory, and fuzzy trace theory; these all account for a wide range of choice behavior as well (Reyna, 2008; Stewart & Simpson, 2008; Stewart et al., 2006; Weber et al., 2007) .
 Practical Issues in the Neuroscience of Value Representation Despite the above, it is often considered axiomatic that an internal, neural, value scale must exist in the brain. The job of neuroscientists is then to find this signal-to pinpoint brain activity This document is copyrighted by the American Psychological Association or one of its allied publishers. This article is intended solely for the personal use of the individual user and is not to be disseminated broadly. that correlates with value. In this section, we challenge this viewpoint by considering some basic issues that come up in the neuroscience of value. The first problem is that it is impossible to precisely measure value. We can measure preferences between options and use the data to infer option values, but elicited preferences are noisy measurements that also reflect factors other than value. For example, depending on how preferences are elicited, they may also reflect the tendency to press the same button repeatedly rather than change actions, the tendency to switch between options due to a prior belief about depleting resources, or the amount of attention or looking time for each of the options (Armel et al., 2008; Schonberg & Katz, 2020; Shimojo et al., 2003; Sugrue et al., 2005) . Finally, value may shift from trial to trial, even depending on recent outcomes, meaning that methods that average across trials produce misleading value estimates (Sugrue et al., 2005) . One could, in theory, incorporate these factors into the inferred value of an option. For example, there may be inherent value in choosing the same thing twice in a row. However, it becomes unclear what the definition of the "option" is that is being evaluated: if the value of an apple after eating (or choosing, or even just viewing) an orange is different from the value of an apple without that proximal experience, can we determine the value of goods at all? And if we can change the value (read: preference) for an option just by directing more attention to it (Salomon et al., 2018; Schonberg et al., 2014) , or inducing choice of it (Izuma et al., 2010; Sharot et al., 2012; Voigt et al., 2017) , is preference really measuring the subjective economic worth of a good? The risk is circularity-if any choice behavior can be explained by supposing value that depends on local history, then the concept of value adds no additional explanatory power beyond that of recent events and choices. Different ways to measure value raise a second challenge of inconsistent measures. Indeed, if we had a single neural value function we called on, values elicited by different measures would match. Unfortunately, they do not (Lichtenstein & Slovic, 2006) . To explain this, in their seminal work, Sarah Lichtenstein and Paul Slovic proposed that preferences do not arise from any internal value function, but instead are constructed at the time of elicitation (see also Ariely et al., 2003; Payne et al., 1992; Tversky & Shafir, 1992) . That is, in the view of these and like-minded scholars, value doesn't sit in the brain waiting to be used; rather, preference is a complex and active process that takes place at the time the decision is made. Critically, in this theory we compute choices in a largely ad-hoc manner based on the available options, without an intermediary common-currency valuation stage. As such, there is no guarantee of consistency or reliability; any consistency or reliability observed may be explained as a result of strong attractor states in the way the system determines the choice, and deviations from consistency are evidence for the specific nature of the algorithm and its idiosyncrasies. This hypothesis, while denouncing value as a latent construct in the brain, nevertheless invites neuroscientific research to understand the active processes involved in choice, and how these relate to (relative) evaluation, if not to economic values. This having been said, experimenters can roughly estimate value, even if not measure it precisely. For example, in a given experiment they can examine choices and determine with confidence that a monkey (behaves as if it) places more value on a gamble as compared to a safe option with a matched expected value. One could then use this fact to try to identify a neural correlate of value. However, for this neural endeavor to be valid, it is important to identify all confounding variables and regress them out. O'Doherty's (2014) review delineates the practical difficulty of doing so, using the overall rubric of "visceral, autonomic, and skeletomotor" activity. In practice, confounding variables include both stimulus and outcome identity, information about the state or structure of the world, the surpriseness, informativeness, and informational value of stimuli, details of the action associated with selecting of consuming the reward, including its likelihood and vigor, and the attention and arousal engendered by the stimulus (e.g., Blanchard et al., 2015; Botvinik-Nezer et al., 2020; Niv et al., 2007; O'Doherty, 2007; Roesch & Olson, 2004; Roesch et al., 2006; Wilson et al., 2014; . Indeed, in studies that separately assess encoding of outcome identity versus outcome value, activity in brain areas that are often considered to be emblematic of economic value (in particular, the OFC) turns out to correlate with outcome identity instead (Klein-Flügge et al., 2013) . A final insurmountable problem is that it may be impossible, even in theory, to obtain a brain measure of value that is independent of behavior. Suppose for example, that we identify a particular class of neurons whose firing rates are perfectly correlated with value down to our ability to measure it through preference. Supporting this idea, we observe that any procedure that modifies value (as inferred from behavior) changes the firing rate of these neurons in a manner consistent with our predictions. We may tentatively hypothesize that these neurons are (or are among) the value neurons of the brain. However, as shown by Schonberg and colleagues, preference can be changed irrespective of changing the economic worth of goods (Schonberg & Katz, 2020) . Therefore, to differentiate value neurons from preference neurons, we would need to show that these neurons do not strictly follow expressed preference when the value function diverges from it. This is, of course, not possible if the value function never measurably diverges from preference. And if we assume that values can diverge from preference, it is not clear how to define values to start with. We call this "the neuroeconomic relativity problem" because, like Einstein's relativity problem, it reflects the fact that there is no external reference frame to which one can calibrate value inferences.
 Reconsidering the Motivation for Common Currency It may be worth asking, then, what does having a single common currency buy you? Why would the brain invest in such an organization? One advantage of a common-currency scale is that it simplifies comparing options that differ along multiple dimensions. For example, when hunting for an apartment, the options may differ along dimensions of price, area, neighborhood, and amenities. The logic is that these dimensions must be first combined into a single scalar per apartment so that the scalars can be compared. However, this is not the only way to solve the problem, and importantly, may not be the way humans make their decisions. For example, the apartment shopper may choose a single dimension and pick the winner along that dimension, as discussed above, or may compare separately on each dimension and choose the apartment that wins on most counts (Tversky, 1972) . Laboratory studies where humans can choose what attributes to view indeed suggest that people don't uncover all the attributes of one option (to calculate its value) and then continue to the next option, but rather prefer to view information for all options attribute by attribute (Fellows, 2006; Hunt et al., 2014) . Indeed, as mentioned, much empirical work indicates that human decision-makers broadly favor heuristic approaches that This document is copyrighted by the American Psychological Association or one of its allied publishers. This article is intended solely for the personal use of the individual user and is not to be disseminated broadly. THE CASE AGAINST ECONOMIC VALUES eschew a value stage in a large number of contexts (Brandstätter et al., 2006; Gigerenzer & Gaissmaier, 2011; Kahneman et al., 1982; Lieder & Griffiths, 2020) . Notably, systems that make use of heuristics may generate internal variables that are conceptually distinct from value but that correlate with value, thus leading to an interpretational confound. Consider, for example, a relatively well-understood implementation of choice in a (non-brain) distributed system: the selection of hive sites in bees (Apis mellifera, Seeley, 2010; Seeley & Buhrman, 1999; Seeley et al., 2006) . Bee swarms select a hive site by sending out scouts to investigate potential sites. Each site differs along roughly 20 dimensions of varying but measurable importance (size, safety from predators, exposure to sunlight, wind, etc.). Each scout bee that encounters the hive site performs an extremely poor estimation of the value of the site-it typically will only sample three-to-four of the dimensions and even then, estimate them poorly. The bee then returns and if the estimated site quality is sufficiently good, indicates an assessment of its quality to the other bees in the swarm. The quality it signals will be correlated with the overall value of the hive site, but only weakly, and, critically, will only integrate a subset of relevant value components. The overall comparison between options is indirect-the options race to attract adherents; the majority of the decision is made by a positive feedback process. This example is particularly relevant because scholars of perceptual decision making may recognize this process as strongly related to ideas about how the brain decides between options by racing to bounds, and there is evidence that deliberative processes in the brain follow the same principles as choice processes in bee swarms (Eisenreich et al., 2017; Franks et al., 2003; Mitchell, 2009; Pais et al., 2013; Passino et al., 2008; Pirrone et al., 2018) .
 Evaluating the Neural Evidence for the Common-Currency Hypothesis Evidence for the common-currency hypothesis comes from the observation that firing rates of single neurons or hemodynamic responses of voxels correlate with values of offers and outcomes (Kennerley et al., 2009 (Kennerley et al., , 2011 Levy & Glimcher, 2012; Padoa-Schioppa, 2011; Rangel et al., 2008) . These responses depend on multiple elements of offers (e.g., the expected reward, as well as the associated response costs), and are modulated by factors that affect subjective value, such as context or level of satiety (see Conen & Padoa-Schioppa, 2019; O'Neill & Schultz, 2010; Rudebeck et al., 2017 for only a few examples). Often, variations in these firing rates predict variations in choice (Conen & Padoa-Schioppa, 2015; Strait et al., 2014; Sugrue et al., 2004) . Such patterns have been taken as evidence that the neural signal encodes value, in particular implicating the OFC (Bartra et al., 2013; Levy & Glimcher, 2012; O'Reilly, 2020) . However, such patterns of neural responding do not definitively demonstrate a common-currency value signal. For example, in the original finding of value-encoding neurons in the OFC (Tremblay & Schultz, 1999) , neurons responded more strongly to a high-valued option than to a low-valued one. However, the principle of common currency only held within a given choice pair-responses for Option B were low when this option was paired with a better Option A, but were high when this same option was paired with a worse Option C. Other studies found value-correlated responses to be more stable across contexts (Padoa-Schioppa & Assad, 2008) , but most findings support an encoding that depends on the alternatives (e.g., Kobayashi et al., 2010; Padoa-Schioppa, 2009; Zimmermann et al., 2018) . One might interpret this finding as reflecting simple range adaptation of neurons that have a finite firing rate; however, a more parsimonious explanation is that the firing pattern is consistent with a relative preference code rather than an abstract value code. This interpretation places the putative neural representation of value one stage later than we would expect from a common-currency codeimmediately after comparison (because a relative valuation is itself a comparison) rather than as an input to comparison, raising the question of where the input came from. An interpretation of this code as something other than economic value would obviate this worry. Importantly, with a code that depends on alternatives, there is no real sense in which we can read out the "true subjective value" of an option. We can only know if an option's value is higher than another value-similar to the information provided by preferences and choice behavior. Another corollary of such a relative code is that it is unclear to what extent we can read out a meaningful value signal when only one option is available. Indeed, presenting a subject with one option at a time should, theoretically, provide the best ability to read out option values from neural activity in areas associated with value, such as OFC, ventromedial prefrontal cortex, and dorsal and subgenual anterior cingulate cortices. Doing so reveals that while neural activity in these regions does correlate with the value of the first option, neural responses to the second (alternative) option (presented on its own) correlate with the value difference, that is, they reflect the result of value comparison rather than valuation per se Hunt et al., 2018; Strait et al., 2014) . Moreover, even responses to the first offer do not simply encode its value, but also contain information about the likelihood that option will be chosen (presumably relative to the expected value of the second offer, Azab & Hayden, 2017) . These results suggest that even when evaluation is experimentally segregated from comparison, pure value encodings may not exist. In fact, the idea that prefrontal neurons are selective or responsive to a single experimenter-defined variable has been increasingly falling out of favor (Fusi et al., 2016; Raposo et al., 2014; Rigotti et al., 2013) , with "mixed selectivity" appearing to be a core operating principle of prefrontal cortex, including in ostensible value regions (Blanchard et al., 2018; Hayden & Platt, 2010; Kimmel et al., 2020; . A third issue is that although the activity of some neurons in the OFC is correlated with value (but not necessarily linearly related to value), activity of other neurons in this same area is anticorrelated with value, with the majority of neurons showing no relationship at all with value. This raises the possibility that the neurons are encoding something other than value, for instance, a distributed representation of the identity of each of the three options, A, B, and C above, or the identity of the stimulus representing the offer. With the small number of options evaluated in most experiments, such a distributed code of outcome or stimulus identity (but not value) can easily result in some neurons randomly firing most strongly for the higher-valued of the three options and least for the worse option, while in other neurons the relationship would be in the opposite direction. This would also explain why many neurons show a nonmonotonic relationship between value and their firing. Recently, using ensemble recordings in the OFC that allow analysis on the level of a single trial, Wallis and colleagues have attempted a more direct test of the hypothesis that OFC encodes value (Rich & Wallis, 2016) . In their task, monkeys were offered two options (denoted by images corresponding to the options) and asked to choose between them. On some trials, only one option was offered. Classifiers were trained to classify options corresponding to each of four offer-value levels (0.05, 0.10, 0.18, and 0.30 ml juice, or, in separate blocks, four levels of second-order reinforcer), aggregating over the two images corresponding to each value level. Using single-unit activity and local-field-potential recordings in the OFC, the authors could classify the offered value above chance, suggesting that OFC neurons encoded information about value. However, here too, offer values may have been encoded as different (outcome stimulus) identities, not different (cardinal) values in a common currency. One stringent test of the value-coding hypothesis would be to show that a combination of two offers of value level 0.05 ml resulted in similar activation pattern to a single offer of value level 0.10 ml. Ensemble recordings that give ample data in a single trial allow testing these hypotheses with novel combinations that have not been trained to predict the same identity of reward through multiple presentations, therefore testing the assumptions of the additivity of common currency directly. This critical test has not been conducted, to our knowledge. As mentioned, many factors that are conceptually distinct from value can influence choice-and are therefore closely correlated with value (Maunsell, 2004) . Reward value drives attention, promotes both short-and long-term learning, primes behavioral adjustments, updates internal models, activates circuitry that detect both positive and negative surprises, and elicits mental computations of cost-benefit tradeoffs as well as comparisons with what could have been chosen. All of these are different from scalar, commoncurrency value, but are known to drive neural activity in the regions usually associated with economic value. As such, they confound that interpretation. This problem is a long-standing and notorious one in neuroeconomics (Maunsell, 2004; O'Doherty, 2014; Roesch & Olson, 2003) . To overcome some of these potential confounds, a strong tradition in neuroeconomics research is to control for extraneous factors such as salience and response cost (although we note that many factors-for example, covert attention-cannot be controlled for). As a practical issue, it is difficult to control for alternative interpretations without making the task so convoluted that it becomes unnatural for the animal, in the ethological sense (O'Doherty, 2014) . As a result, it is not clear if findings of value computation (if we believe them to be so) in these tasks would naturally translate to how the brain computes choice in naturalistic situations. But the deeper issue remains that value as inferred in most experiments is definitionally a summary of aggregated choice behavior, and therefore, any variable that influences choice behavior will necessarily be correlated with value. It is for this reason that we suggest that a stronger demonstration of scalar value coding in the brain should show mathematical properties such as the additivity of value, or separation from preferences as these are changed without changing value, as suggested above. If the response of neurons to the novel sum of two stimuli each promising 0.1 ml of juice is similar to the response of those same neurons to a different stimulus associated with 0.2 ml of juice, and especially if this response did not change when preference for an option was induced by means such as the mere exposure effect (Schonberg & Katz, 2020) , it would be harder to argue that this is due to a shared motor plan, or attentional capture. We note that research to date has shown that the activity of brain areas associated with value, in particular the OFC, does change when preferences are modified through methods that should, in principle, not change economic value (Botvinik-Nezer et al., 2020) .
 An Alternative View: Direct Learning of Policies Although reinforcement-learning models have contributed to the assumption that the brain computes values, many reinforcementlearning algorithms do not learn or estimate values for different actions. Instead, they directly learn action policies. In fact, in the reinforcement-learning literature, the goal of an agent is to obtain as much reward as possible by executing optimal actions-calculating values is only one means to achieve that end. In explaining reinforcement-learning methods, Dayan and Abbott's (2001) textbook begins with the "direct actor"-an actor that learns actions without computing their values. The Actor-Critic model, a prominent algorithm for reinforcement learning that has been linked to the brain (Barto, 1995; Joel et al., 2002; Maia, 2010; O'Doherty et al., 2004; Takahashi et al., 2008) , does exactly that. In this algorithm, a Critic module learns values of states in terms of expected future rewards (here, the state includes all available actions, averaging over choices and explicitly not computing the value of each possible choice), and uses these to compute reward prediction errors. These prediction errors are used to learn an action policy in the Actor module: the probability of actions that are followed by positive prediction errors is increased, and the probability of actions that are followed by negative prediction errors is decreased. Under some reasonable conditions, this model learns correct reward-maximizing policies (Sutton & Barto, 2018) . However, the quantities learned by the Actor-tendencies to perform one action over the other-cannot be read out as action values. In particular, due to the way the algorithm learns, ties are broken between equally good actions such that eventually agents learn deterministic policies. To be clear, if four actions were to lead to 1, 2, 3, and 3 drops of juice, respectively, the model may learn to always choose the third option, or to always choose the fourth (both optimal policies). At the end of learning, action weights in the Actor may be 0, 0, 1, 0 respectively, losing all information about value, or relative value. Even before convergence to a deterministic policy, weights may be 0, 0.1, 0.95, 0.2 respectively (or any other combinationweights here do not have to sum up to 1, and these are just illustrative numbers). Importantly, if the basal ganglia indeed implement an Actor-Critic learning algorithm in the brain, there is no sense in which we can glean action values from the Actor or the Critic. The Actor-Critic model is only one of a class of reinforcementlearning algorithms that learn policies directly, that is, without calculating option values (e.g., Sutton et al., 2000; Williams, 1992) . In their general form, these algorithms maintain an action policy, use experience to evaluate a gradient direction for this policy (that is, what change in policy would increase the overall obtained reward), and change the policy in that direction. Another notable model, developed to explain behavioral patterns in choices, is the Experience-Weighted Attraction model of Camerer and Ho (1999) , which interpolates between value calculation and direct policy learning. Recent findings suggest that both behavior and learning signals measured in humans during decision making are more in line with a policy-learning algorithm than with value estimation. For instance, This document is copyrighted by the American Psychological Association or one of its allied publishers. This article is intended solely for the personal use of the individual user and is not to be disseminated broadly. THE CASE AGAINST ECONOMIC VALUES Li and Daw (2011) had participants choose between two options that gave reward with different probabilities. After each choice, both the outcome of the chosen option and the counterfactual outcome of the unchosen option were displayed. Behavior, as well as neural signals corresponding to prediction errors in the basal ganglia, suggested that subjects were updating both options in opposite directions, learning relative choice propensities (a policy) rather than tracking the expected value of each option. In another task in which humans chose between pairs of options and were able to view both the outcome of their choice and the counterfactual outcome of the forgone option, Palminteri et al. (2015) had subjects learn which of two probabilistically rewarding options is better, and which of two probabilistically punishing options was better. At test, subjects were asked to choose between pairs from both the rewarding and the punishing contexts. Surprisingly, when choosing between the less rewarding option and the less punishing option, subjects tended to choose the less punishing option. This is consistent with policy learning, as that option had been the favored option in the punishment context, whereas the less rewarding option had been the disfavored one in the reward context. However, the value of a sometimes-rewarding option is clearly higher than that of a sometimes-punishing option, hence value learning cannot explain this fundamentally suboptimal choice pattern. Direct learning of policies is consistent with basic tenets of decision making, such as the fact that choices are stochastic even when no exploration is necessary or warranted (e.g., choosing between two gambles that are fully described; Khaw, Li, et al., 2017) . Because policies are relative quantities (preference for one option implies unfavorability of another, as the probabilities of all choices have to sum to one), they also explain common violations of the independence axiom such as the effects of third-option "decoys" on choice (Soltani et al., 2012) , and temporal and spatial contextual influences on choice (Khaw, Glimcher, et al., 2017) , although these phenomena can also be explained by valuation models that involve range normalization.
 Conclusion The field of neuroeconomics started with the putative identification of pure value signals (Platt & Glimcher, 1999) . The meaning of these signals was disputed early on (Maunsell, 2004; Roesch et al., 2006) , but deep questions were set aside as researchers continued to identify brain areas with parts of the computational processes of choice, and in particular, identified the OFC with the seat of economic value. However, while those debates have subsided, the problems they raised have not been resolved. Progress on these issues will require additional data, but we stress that not every experiment that involves choices also implies valuation, and one must be careful in interpreting data not only because of potential confounds, but also because we must be wary of treating a hypothesis-that the brain computes value-as an axiom. We also suggest the need for additional philosophical work to define value in a way that is-at least in principle-dissociable from other factors that promote choices (Juechems & Summerfield, 2019) . In other words, we argue for a return to the productive debates of the early days of the field, 20 years ago (for neuroeconomics), and earlier (for its psychological underpinnings). Bolstered by an additional 20 years of new data, such debates would surely benefit the field of neuroeconomics moving forward. They would also potentially help reconcile conflicting views on what the OFC might or might not be doing in decision making (Stalnaker et al., 2015) . We note here that, in neuroeconomics, value-based choices are generally thought to be part of explicit, aware, goal-directed, or model-based decision making that relies on frontal-cortex areas, in particular, the OFC.2 We consider subjective worth as revealed by preference, following neuroeconomic theory, but note the limiting focus on only the "wanting" not the "liking" side of value (Berridge, 1996) .
 This document is copyrighted by the American Psychological Association or one of its allied publishers.This article is intended solely for the personal use of the individual user and is not to be disseminated broadly.

It has been common in the neuroeconomics literature to assume (based on past research) that the brain explicitly computes some notion of value in order to make choices. This paper argues that this is wrong: it is plausible that the brain does not in fact explicitly calculate values, and instead directly learns a policy that produces actions.

If you previously were optimistic about inverse reinforcement learning and similar techniques because you thought they could infer the same notion of value that the brain computes, this seems like an important counterargument. However, it should be noted that the authors are _not_ arguing that the brain is not optimizing some specific notion of value: just that it is not _explicitly computing_ such a notion of value. (Similarly, policies learned by RL optimize the reward function, even though they need not explicitly calculate the reward of every state in order to choose an action.) So you could still hope that the brain is optimizing some notion of value that isn’t explicitly computed, and then use inverse RL to recover that notion of value.