Learning from Physical Human Corrections, One Feature at a Time

Figure 1 : Participant pushes on the robot to teach it to go closer to the table. In the process of giving this correction, the human changes both the robot's distance from table and -inadvertently -the orientation of a cup which the robot is grasping (blue arrows). Typically, the robot would learn about both cup and table features from this one correction (top right). We propose that robots interacting with humans should learn about only one feature at a time (bottom right). correct the robot as it is moving. For example, the robot is moving a fragile cup from a cabinet to the table, and a nearby human notices that the robot is carrying the cup too high above the table: if the cup were to drop from that height, it would likely break! To correct the robot's behavior, the human intuitively pushes the robot's end-effector towards the table to signal their motion preference. Ideally, the human's correction will only affect the cup's distance from the table; in practice, however, human actions are noisy and imperfect [7, 19, 20, 24] , especially when kinesthetically maneuvering robotic manipulators while trying to carefully orchestrate their multiple degrees of freedom [1] . As a consequence, when the person pushes down on the end-effector, they accidentally change not only the robot's distance from the table, but also the orientation of the cup (see Fig. 1 ). This single human interaction has therefore adjusted two task features: the cup's distance from the table and the cup's orientation. From the robot's perspective, it is not immediately clear what the person actually intends: do they (a) want the robot to carry the cup closer to the table, or do they (b) additionally want the robot to carry the cup at a new orientation? State-of-the-art algorithms default to the latter interpretation. Prior work has built on Inverse Reinforcement Learning (IRL) [13, [16] [17] [18] 24] to formalize learning from physical human corrections as an estimation problem: the robot estimates the objective function that it should optimize during the task by treating human corrections as evidence about the objective function's parameters 1 . Under the ideal objective function parameters, the corrected behavior has to have a lower cost than the robot's current behavior [3, 11] . Therefore, when the person's correction changes multiple features -however slightly -a rich hypothesis space will lead to the robot updating its understanding about the importance of all of these features (top right in Fig. 1 ). This traditional approach works well with perfect or near-perfect corrections; however, with real people come aspects of corrections that are not always intended. These unintended corrections lead to unintended learning. In other words, the robot attempts to learn from and alter its behavior based on all the inputs, even those that are superfluous. Returning to our previous example, if all features were updated the robot would learn (correctly) that the cup should be lower, and (incorrectly) that the cup should be carried at a different orientation. In general, because of the inherent physical difficulty in simultaneously correcting many degrees of freedom of a robotic arm, learning about all features at once may systematically cause the robot to infer more from the human's corrections than desired. Our insight is that we can alleviate unintended robot learning by focusing the learning on only one feature at a time. For tasks where the human is attempting to change the importance of just one feature, this insight helps the robot reject inadvertent adjustments on the other features (bottom right in Fig. 1 ). But even for tasks in which the human wants to correct several features, learning one feature at a time enables people to break down the task and teach sequentially. Indeed, sequential teaching may come more naturally to people collaborating with robots [21, 22] , and reduces the burden on users to coordinate all aspects of the task simultaneously during each individual correction. Based on our insight, we make the following contributions: Online Feature Identification. As the robot is executing its task, the human collaborator can intervene and provide physical corrections. We formulate the problem of identifying which one feature the person is trying to correct at each time step, derive a solution, and justify a simple approximation for online performance. We hypothesize that this approach will result in a better learning process, with a more accurate objective function being inferred by the robot at each time step, and a better final outcome. User Study Testing One-at-a-Time Learning. After validating our algorithm in 2-D simulations with an approximately optimal human, we put our hypothesis to the test in a user study on a 7-DoF robotic manipulator. These experiments compare one-ata-time and all-at-once learning within a factorial design, across tasks that need just one feature to be corrected, and tasks that need multiple features to be corrected. We find that one-at-a-time learning is especially helpful in the second case, where the person's teaching task is more complex. People also prefer it, finding that the robot is better at understanding their corrections and requires less reteaching. Overall, our work provides a practical improvement for learning objective functions online from physical human-robot interaction.
 ONE-AT-A-TIME OBJECTIVE LEARNING FROM PHYSICAL HUMAN INTERACTION 2.1 Why Learn from Physical Corrections? When a human and robot are collaborating in close proximity, physical interaction -in which the human touches, pushes, pulls, or otherwise guides the robot -is almost inevitable. The way in which a robot responds to such physical human-robot interaction (pHRI) depends on how the robot interprets those corrections. Traditionally, the human's interactions are treated in one of three ways [9] : as disturbances to be rejected [5, 12, 23] , as collisions to be detected and avoided [4] , or as operator signals to be followed by switching into a compliant mode [8, 10, 14] . In all cases, the robot does not learn from the human's actions; once the human stops interacting, the robot resumes its original behavior. In contrast, we argue that interactions are intentional, and therefore informative -the human interacts with the robot because it is doing something wrong, and the human's correction indicates how the robot should behave. Furthermore, since the way in which the robot chose its behavior was by optimizing an objective function, interaction suggests that this objective function was incorrect. Thus, rather than stubbornly continuing to optimize the same wrong objective, the robot should instead leverage the human's feedback in order to update its understanding of the objective function.
 Learning Problem Statement Assume the robot starts in some configuration q 0 at time t = 0. Let Ξ be the space of trajectories beginning at q 0 and ending at a feasible goal configuration, where each ξ ∈ Ξ is a sequence of configurations. Next, let Φ : Ξ → R F be a vector-valued function mapping trajectories to feature values, with Φ i (ξ ) signifying the value of the i-th feature. Similar to prior IRL work [13, 16, 18, 24] , the robot's objective function (here a cost function) is parametrized by θ ∈ R F , which weights the importance of these features along the entire trajectory: C(ξ ) = θ • Φ(ξ ) (1) The robot starts off with an initial objective function θ 0 at time t = 0, and optimizes this objective function to produce its initial trajectory: ξ 0 = arg min Ξ θ 0 • Φ(ξ ) (2) After identifying ξ 0 , the robot starts to execute this initial trajectory. The person interacting with the robot has some desired objective function that they want the robot to optimize, denoted as θ \* . The robot does not have access to these parameters -they are internal to the person (and here assumed to be constant). However, at every time step t, the person might intervene to move the robot away from its current configuration by some ∆q t . The robot should then treat the human's correction ∆q t as an observation about θ \* , and update its objective from θ t to θ t +1 , such that this new objective function is closer to θ \* .
 All-at-Once Learning Following [3] , we interpret the change in configuration ∆q t as an indication of the corrected trajectory, ξ t c , that the human would prefer for the robot to execute: ξ t c = ξ t + M −1 (0, .., ∆q t , ...0) T (3) Here ξ t is the robot's current trajectory -optimal under θ t -and M is a matrix that smoothly propagates the local correction ∆q t along the rest of the trajectory [6] . Next, based on [11] and [18] , we make the core assumption that the corrected trajectory ξ t c is better than the current trajectory ξ t with respect to the ground truth θ \* . Recalling that our objective Session We-1A: Machine Learning for HRI HRI'18, March 5-8, 2018, Chicago, IL, USA function is a cost function, this implies: θ \* • Φ(ξ t c ) < θ \* • Φ(ξ t ) (4) To now find a θ t +1 closer to θ \* , we select a weight vector that is both (a) near the current θ t and (b) maximally makes (4) hold: θ t +1 = arg min θ ∈Θ θ • (Φ(ξ t c ) − Φ(ξ t )) + 1 2α ||θ − θ t || 2 (5) Note that α > 0. This optimization problem is a quadratic in θ , so we will take the gradient of ( 5 ) and set it equal to 0: ∇ θ = Φ(ξ t c ) − Φ(ξ t ) + 1 α (θ − θ t ) = 0 (6) Rearranging ( 6 ), we finally obtain: θ t +1 = θ t − α(Φ(ξ t c ) − Φ(ξ t )) (7) Interestingly, (7) is the same update rule from co-active learning [11] and online maximum margin planning [18] , shown by [3] to be an approximate solution to the partially observable Markov decision process that treats θ \* as the hidden state and optimizes the cost parametrized by θ \* . This update rule has an intuitive interpretation: if a feature has a higher value in corrected trajectory than in the current trajectory, (7) decreases corresponding weight -making it lower-cost -and thus encourages the optimizer to generate subsequent trajectories where that feature also has a higher value. Under this method, the robot updates the weights on all features that the person changed with their correction during the current time step.
 One-at-a-Time Learning A natural solution for restricting the number of learned features might be to switch the regularization term in (5) to the L 1 norm [13, 15] , which encourages sparsity of the weight update. However, there is no guarantee that this will result in changing just one weight; it may still update all the features that the human corrected, including those that were accidentally changed. In this work, to capture one-at-a-time learning, we now make a different assumption about ξ t c , the intended corrected trajectory. While the actual corrected trajectory, ξ t c , might change multiple features, we assume that the human's intended corrected trajectory, ξ t c , changes only a single feature. We simplify the intended corrected trajectory into an intended change in features, ∆Φ t c , and impose the constraint that ∆Φ t c can only have one non-zero entry: this entry represents the feature which the person wants to update. Note that our one-at-a-time strategy does not mean that only one feature ever changes throughout the task. Instead, at every time step t there can be a different intended feature change, and so the person can sequentially change the weights to match their desired objective over multiple corrections. Without loss of generality, assume that the human is attempting to change the i th entry in θ t , the robot's current feature weights. If the human interacts to only update the weight on the i t h feature, then their correction of the robot's current trajectory, ξ t , should change the feature count in the direction J (θ i ) = ∂Φ(ξ t ) ∂θ t i . In other words, given that the person is an optimal corrector and that their interaction was meant to change just the weight on the i t h feature, then we would expect them to correct the trajectory such that they produce a feature difference exactly in the direction J (θ i ). Realistically, however, human corrections are noisy -even for expert users [2] -and will not necessarily induce the optimal feature difference during every correction. Despite these imperfections, we assume that the result of their correction will still noisily optimize the distance (dot product) in the optimal direction. This provides us with an observation model, from which we can find the likelihood of observing a specific feature difference given the one feature which the human is attempting to update: P(∆Φ|i) ∝ e J (θ i )•∆Φ (8) Accordingly, for the observed feature difference ∆Φ = Φ(ξ t c )−Φ(ξ t ), the feature which the human is most likely trying to change is: i \* = arg max i P(Φ(ξ t c ) − Φ(ξ t ) i) = arg max i J (θ i ) • (Φ(ξ t c ) − Φ(ξ t )) (9) Using ( 9 ), we can estimate which feature the person wanted to update during their physical correction. Next, by leveraging i \* and the observed feature difference, we can reconstruct ∆Φ t c , the human's intended feature difference. Recall that -if the human wanted to only update feature i \* -their intended feature difference would ideally be in the direction J (θ i \* ) = ∂Φ(ξ t ) ∂θ t i \* , and so we can choose ∆Φ t c ∝ J (θ i \* ). In practice, however, we will simplify this derivative by projecting the actual feature difference induced by the human's interaction onto the i \* t h axis, ∆Φ t c = (0, .., Φ i \* (ξ t c ) − Φ i \* (ξ t ), ..0) T . Thus, once we have identified which feature the person most wants to change during their current interaction, i \* , we argue that the intended feature correction should only change this one feature 2 . Evaluating J (θ i ) requires numerical differentiation, i.e., finding an optimal trajectory at least F + 1 times at each time step (where F is the number of features). To make this process run in real-time, we approximate J (θ i ) as proportional to (0, .., 1, ..0) T . In other words, we assume that when the i t h weight changes, it predominantly causes a change in the i t h feature along the corresponding optimal trajectory. Substituting this simplification back into (2.4), we have reduced our method for finding the feature which the human intends to change into a simple, yet intuitive, heuristic: only the feature that changed the most as a result of the human's correction should be updated. We note, however, that this heuristic has its roots in the more principled approach that was detailed above. Our update rule now becomes θ t +1 = θ t − α ∆Φ t c (10) Overall, isolating a single feature at every time step is meant to prevent unintended learning. If the person is trying to correct multiple features, they can still do so: the robot will pick up on what seems like the most dominant feature in the correction, adjust that, and then give the person a chance to correct whatever remains during the next time step. Due to the noisy nature of human corrections, we hypothesize that this one-at-a-time update strategy will lead to shorter trajectories through the learned weight spacewhich reach the ideal weight more directly -when compared to a strategy that tries to update everything at once. In what follows, we first show some simulation analysis with optimal and noisy humans, and then test our hypothesis in a user study.
 SIMULATIONS In order to better validate and compare the all-at-once and oneat-a-time learning methods described in Section 2, we conducted human-robot interaction simulations. These simulations show that updating one feature per interaction can help prevent unintended learning, particularly when the human interacts sub-optimally. Setting. We will consider a vertical planar environment, where the y-axis corresponds to height above a table and the x-axis is parallel to that table. The simulated robot is attempting to move from a fixed start position, s, to a fixed goal position, д. The robot is modeled as a single point, and the robot's configuration is its current (x, y) position. A simulated human is standing beside the table near the start position, and physically interacts with the robot to correct its behavior when necessary. The robot does not know the true feature weights of the human's objective function, θ \* , but the robot does know that there are three different features which the human might care about: the length of the robot's trajectory (length), the robot's height above the table (table), and the robot's distance from the human (human). Here the table feature corresponds to the height along the y-axis, since the table is a surface at y = 0, and the human feature corresponds to the distance along the x-axis, since the human is standing at x = 0. The weight of the length feature is fixed, and the robot learns the relative weights associated with table and human features over the course of the task. The human's true reward parameter is θ \* = [0.5, 0], where 0.5 is the true weight associated with table and 0 is the true weight associated with human. Initially, the robot believes that θ 0 = [0, 0], and so the robot is unaware that it should move closer to the table. Simulated Human. We consider two different simulated humans: (a) an optimal human, who corrects the robot to exactly follow their desired trajectory and (b) a noisy human, who imperfectly corrects the robot's trajectory. At the start of the task, the optimal human identifies a desired trajectory: ξ \* H = arg min Ξ θ \* • Φ(ξ ). During the task, the human does not change ξ \* H , and interacts with the robot to make it follow this desired trajectory. At each time step t the human provides a correction ∆q t that changes the robot's current configuration to the desired configuration, ξ \* H (t), but the human only provides this correction if the robot's distance from ξ \* H (t) is greater than some acceptable margin of error. In contrast, the noisy human takes actions sampled from a Gaussian distribution: these actions are centered at the optimal human action with a bias in the x-direction. This bias introduces a systematic error, where the noisy human accidentally pulls the robot closer to their body when attempting to significantly correct the vertical table feature. As a result of this noise and bias, the noisy human may unintentionally correct the human feature. Analysis. We performed two different simulations: one with the optimal human (see Fig. 2 ), and one with the noisy human (see Fig. 3 ). When the human optimally corrects the robot's table feature in Fig. 2 , they never unintentionally affect the weight of the human feature, and so all-at-once and one-at-a-time learning both yield the exact same results for the optimal human. By contrast, the noisy human unintentionally corrects the human features at the start of the task (when trying to correct the table features), and, as such, we observed different behavior for all-at-once and one-at-a-time learning in Fig. 3 . Although the robot follows a similar mean trajectory for both learning methods, and eventually converges to the correct feature weights in each case, we observe that all-at-once had a longer learning process and more persistent human interaction. In particular, the length of the mean path in feature space from θ 0 to θ T was 0.57 for all-at-once vs. 0.49 for one-at-a-time; the length of the mean path specifically for the human feature weight was 0.23 for all-at-once vs. 0.001 for one-at-a-time. Recall that the robot was constrained to reach its goal position in 10 steps; we found that, in the all-at-once case, the human interacted with the robot during an average of 5.24 steps, and, in the one-at-a-time case, the human interacted with the robot during an average of 3.56 steps. These simulations showcase that, when the human interacts sub-optimally, their corrections can lead to unintended learning on the robot's part, which the human must then exert additional effort to undo. For the simulation we have described, updating only one feature per time step helps to mitigate accidental learning, demonstrating the potential benefits of our proposed one-at-a-time learning method.
 EXPERIMENTS We conducted an IRB-approved user study to investigate the benefits of one-at-a-time learning. During each experimental task, the robot began with a number of incorrect weights in its objective, and the participants intervened to physically correct the robot.
 Independent Variables We use a 2 by 2 factorial design. We manipulated the learning strategy with two levels, all-at-once and one-at-a-time, as well as the number of feature weights that need correction, one feature weight and all the feature weights. In the all-at-once learning strategy, the robot updated all the feature weights from a given interaction with the gradient update from Equation ( 7 ) and then replanned a new trajectory with the updated weights. In the one-at-a-time condition, the robot chose the feature that changed the most using Equation (2.4), updated according to Equation (10) , and then replanned a new trajectory withe the updated θ .
 Dependent Measures
 Objective. To analyze the objective performance of the two learning strategies, we split the objective measures into four categories: Final Learned Reward: These measure how closely the learned reward matched the optimal reward by the end of the trajectory. We measured the dot product between the optimal and final reward vector, denoted DotFinal = θ \* • θ T . We also analyzed the regret of the final learned reward, which is the weighted feature difference between the ideal trajectory and the learned trajectory RegretFinal = θ \* • Φ(ξ θ \* ) − θ \* • Φ(ξ θ T ) and the individual feature differences between the ideal reward and the trajectory induced by the final learned reward TableDiffFinal = |Φ T b (ξ θ \* ) − Φ T b (ξ θ T )| CupDiffFinal = |Φ C (ξ θ \* ) − Φ C (ξ θ T )| Learning Process: Measures about the learning process, i.e. θ = {θ 0 , θ 1 , . . . , θ T }, included the average dot product between the true reward and the estimated reward over time: DotAvg = 1 T T i=0 θ \* •θ i . We also measured the length of the θ path through weight space for both cup, θC , and table, θT b weights. Finally, we computed the number of times the cup and table weights were updated away from the optimal θ \* (denoted by CupAway and TableAway). Executed Trajectory: For the actual executed trajectory, ξ act , we measured the regret Regret = θ \* • Φ(ξ θ \* ) − θ \* • Φ(ξ act ) and the individual table and cup feature differences between the ideal and actual trajectory TableDiff = |Φ T b (ξ θ \* ) − Φ T b (ξ act )| CupDiff = |Φ C (ξ θ \* ) − Φ C (ξ act )| Interaction: Interaction measures on the forces applied by the human, {u 0 H , u 1 H , . . . , u T H }, included the total interaction force, Iact-Force = T t =0 ||u t H || 1 and total interaction time. 4.2.2 Subjective. For each condition, we administered a 7-point Likert scale survey about the participant's interaction experience (see Table 1 for questions). We separated our survey questions into four scales: success in teaching the robot about the task, correctness of update, needing to undo corrections because the robot learned something wrong, and ease of undoing. The final learned weight vector with one-at-a-time is closer to the ideal weight vector for the task where two feature weights are incorrect (left). Looking at the individual feature differences from ideal: while the final cup weight is closer to ideal for one-at-a-time for both tasks (center), the ideal table weight is actually significantly further away from the ideal for the one-at-a-time strategy during the one-feature task (right). However, for the two feature task, the one-at-a-time method outperforms the all-at-once for final learned cup and table weights.
 Hypotheses H1. Updating one feature at a time significantly increases the final learned reward, enables a better learning process, results in lower regret for the executed trajectory, and leads to less interaction effort and time compared to all-at-once update. H2. Participants will perceive the robot as more successful at accomplishing the task, correctly updating its knowledge of the task, less likely to learn about extraneous aspects of the task, and be easier to correct if it did learn something wrong in the one-at-a-time condition.
 Tasks We designed two experimental household manipulation tasks for the robot to perform in a shared workspace (see Fig. 4 for setup). For each experimental task, the robot carried a cup from a start to end pose with an initially incorrect objective. One of the tasks focused on participants having to correct a single aspect of the incorrect objective, while the other needed them to correct all parts of the objective. Participants were instructed to physically intervene to correct the robot's behavior during the task. Similar to state-of-theart methods, all the features in the robot's objective were chosen to be intuitive to a human to ensure that participants could understand how to correct the robot. In Task 1, the robot's objective had only one feature weight incorrect. The robot's default trajectory took a cup from the participant and put it down on the table, but carried the cup too far above the table (top of Fig. 4 ). In Task 2, all the feature weights started out incorrect in the robot's objective. The robot also took a cup from the participant and put it down on the table, but this time it initially grasped the cup at the wrong angle and was also carrying the cup too high above the table (bottom of Fig. 4 ).
 Participants We used a within-subjects design and counterbalanced the order of the conditions during experiments. In total, we recruited 12 participants (7 female, 4 male, 1 non-binary trans-masculine, aged 18-30) from the campus community, 11 of which had technical backgrounds and 1 of which did not. None of the participants had experience interacting with the robot used in our experiments.
 Procedure Before beginning the experiment, participants performed a familiarization task to become comfortable teaching the robot with physical corrections. The robot's original trajectory moved a cup from a shelf to a table, but the robot did not initially care about tilting the cup mid-task. The robot's objective contained only one aspect of the task (cup orientation) and participants had to correct only this one aspect. Afterwards, for each experimental task, the participants were shown the robot's default trajectory as well as what their desired trajectory looks like. They were also told what aspects of the task the robot is always aware of (cup orientation and distance of end-effector to table) as well as which learning strategy they were interacting with. Participants were told the difference between the two learning strategies in order to minimize in-task learning effects. Note, however, that we did not tell participants to teach the robot in any specific style (like one aspect as a time), only about how the robot reasons about their corrections.
 Analysis 4.7.1 Objective. Final Learned Reward. We ran a factorial repeated-measures ANOVA with learning strategy and number of features as factors, and user ID as a random effect, for each of the measures capturing the quality of the final learning outcome. Fig. 5 summarizes our findings about the final learned weights for each learning strategy. For the final dot product with the true reward, we found a significant main effect of the learning strategy (F (1, 81) = 29.86, p < .0001), but also an interaction effect with the number of features (F (1, 81) = 13.07, p < .01). The post-hoc analysis with Tukey HSD revealed that one-at-a-time led to a higher dot product on the two feature task (p < .0001), but there was no significant difference on the one-feature task (where one-at-a-time led to slightly higher dot product). We next looked at the final regret, i.e. the difference between the cost of the learned trajectory and that of the ideal trajectory. For this metric we found an interaction effect, suggesting that one-ata-time led to lower regret for the two-feature task but not for the one-feature task. Looking separately at the feature values for table and cup, we found that one-at-a-time led to a significantly lower difference for the cup feature across the board (F (1, 81) = 11.30, + Cup All-at-Once One-at-a-Time (b) In contrast to (a), when two feature weights are wrong, the one-ata-time strategy outperforms the all-at-once strategy when it came to a higher dot product over the duration of the trajectory.
 Figure 6: The one-at-a-time strategy shows significantly more consistent alignment between the estimated weight vector, θ t , and the ideal weight vector, θ \* , than the all-at-once for the two feature task. This indicates that when multiple aspects of the objective need changing, the one-at-a-time method enables more accurate learning. p < .01, no interaction effect), but that one-at-a-time only improved the difference for the table on the two feature task (p < .0001) -it actually significantly increased the difference on the one feature task (p < .001). Overall, we see that one-at-a-time learns something significantly better across the board for the two-feature task. When it comes to the one feature task, the results are mixed: it led to a significantly better result for the cup orientation, but significantly worse for the table distance feature. Learning Process. For the average dot product between the estimated and true reward over time, our analysis revealed almost identical outcomes to before, when we were looking at the final reward only (see Fig. 6 ). We also found that one-at-a-time resulted in significantly fewer updates in the wrong direction for the cup weight across the board (F (1, 81) = 44.91, p < .0001) and for the table weight (F (1, 81) = 22.02, p < .0001), with no interaction effect. Fig. 7 highlights these findings and their connection to the subjective metrics. Looking at the length of the path through the space of weights, we found a main effect of learning strategy (F (1, 81) = 26.82, p < .0001), but also an interaction effect (F (1, 81) = 6.55, p = .01). The posthoc analysis with Tukey HSD revealed that for the the one-feature task, one-at-a-time resulted in significantly shorter path traversed through weight space (p < .0001). The path was shorter with the two-feature task as well, but the difference was not significant. The effect was mainly due to the one-at-a-time method resulting in a shorter path for the cup weight on the one-feature task, as revealed by the posthoc analysis (p < .0001). Overall, we see that the quality of the learning process was significantly higher for the one-at-a-time strategy across both tasks. When one aspect and all aspects of the objective were wrong, oneat-a-time led to fewer wrong weight updates and resulted in the learned reward across time being closer to the true reward. The Executed Trajectory. We found no significant main effect of the learning strategy on the regret of the executed trajectory: the two strategies lead to relatively similar actual trajectories with respect to regret. Both regret as well as the feature differences from ideal for cup and table showed significant interaction effects. Interaction Metrics. We found no significant effects on interaction time or force. Summary of Objective Metric Analysis. Taken together, these results indicate that a one-at-a-time learning strategy leads to a better learning process across the board. On the more complex two-feature task, this strategy also leads to unquestionably better learning outcomes. For the one-feature task, learning one feature at a time enables users to better avoid the wrong perturbation of the correct weight (on the cup feature), but is not as good as the all-at-once method at enabling users to properly correct the wrong weight (on the table feature). Thus, H1 was partially supported: although updating one feature weight at a time does not improve task performance when there is only one aspect of the objective wrong, reasoning about one feature weight at a time leads to significantly better learning and task performance when all aspects of the objective are wrong. 4.7.2 Subjective. We ran a repeated measures ANOVA on the results of our participant survey. After testing the reliability of our 4 scales, we found that the correct update and undoing scale were significantly reliable, so we grouped these into a combined score (see Chronbach's α in Table 1 ). We analyzed success and undoing ease separately as they were not reliable. For the correct update scale, we found a significant effect of learning strategy (F (1, 33) = 5.09, p = 0.031), showing that participants perceived the one-at-a-time strategy as better at updating the robot's objective according to their corrections. Additionally, the undoing scale showed a significant effect of learning strategy (F (1, 33) = 10.35, p < 0.01), with the one-at-at-time strategy being less likely to learn the wrong thing and cause the participants to have to undo a correction. For ease of undoing, when analyzing Q9 and Q10 individually we found no significant effect of strategy. The one-at-a-time strategy results in significantly less weight updates that are away from the optimum weight across all tasks (left top, left bottom). These findings are consistent with the subjective likert data from the undoing scale, where participants perceived the one-at-a-time method as less likely to learn the wrong thing and need an additional undoing action. Summary of Subjective Metric Analysis. The subjective data echoes some of the objective data results. Participants perceived that one-at-a-time better understood their corrections and required less undoing due to unintended learning, partially supporting H2.
 DISCUSSION In this paper, we compared the performance of one-at-a-time and allat-once learning for two tasks: one that required correcting a single feature, and another that required correcting multiple features of a robot's objective. For the multiple feature task, learning about one feature at a time was objectively superior: it led to a better final learning outcome (Fig. 5 ), took a shorter path to the optimum, and had fewer incorrect inferences and undoings along the way (Fig. 6 ). However, the results were not as clear for the single feature task: the one-at-a-time method lessened unintended learning on the weights that were initially correct, but it hindered learning for the incorrect weights. However, participants subjectively preferred the one-at-a-time strategy overall: they thought it was better at learning the correct aspects of the task and required less undoing. We hypothesize that the superior objective performance of the one-at-a-time strategy in the second task is due to the increased complexity of the task. It appears that one-at-a-time learning is more useful as the teaching task becomes more complex and requires fixing more aspects of the robot's objective. However, with simple teaching tasks that only require one aspect of the objective to change, it is not yet clear whether one-at-a-time is a significantly better learning strategy.
 Limitations and Future Work It is both a limitation and a strength that we chose the simplest possible feature selection method for the one-at-a-time task. On the one hand, this is an intuitive and computationally inexpensive method to examine as a first exploration into teaching robot objectives online via physical interaction. At the same time, our simple learning strategy was not consistently superior in the simple task. This opens the door for analyzing more sophisticated methods that perform Bayesian inference on the intended feature, or low-pass filtering to prevent high frequency changes in which features gets Table 1 : Likert scale questions were grouped into four categories: success in accomplishing the task, correctness of update (reliable), needing to undo corrections because of unintended learning (reliable), and ease of undoing.
 Likert Questions Cronbach's α .93 Q7: Sometimes my corrections were just meant to fix the effect of previous corrections I gave. Q8: I had to re-teach the robot about an aspect of the task that it started off knowing well. undo ease Q9: When the robot learned something wrong, it was difficult for me to undo that. .66 Q10: It was easy to re-correct the robot whenever it misunderstood a previous correction of mine. updated to improve overall learning and usability. Additionally, while our method worked well with intuitive features like "distance to table", additional work is needed to investigate how well each method works when the features are non-intuitive to the human. Perhaps our largest limitation in this work is our demographics: our study participants were primarily individuals with a technical background (with one exception). Future work must consider a more diverse user population to ensure external validity. Not only do we need algorithms that can learn from humans, but the methods must also reason about the difficulties humans experience when trying to kinesthetically teach a complex robotic system. To simplify the teaching process, we propose that robots should learn one aspect of the objective at a time from physical corrections. While our user studies indicate the benefits of this method, it is only a first step towards seamless human-robot interaction. Optimal human: Teaching the robot (b) Optimal human: Learning weights
 Figure 2 : 2 Figure 2: Simulation with optimal human. (a) Human corrects the robot during the first few time steps, and the robot follows the human's desired trajectory afterwards. (b) The robot's estimated feature weights converge to the human's true feature weights.
 Figure 3 : 3 Figure 3: Simulation with noisy human. (a) The human noisily corrects the robot's trajectory, where the ellipses show the robot's states with 95% confidence over 100 simulations. (b)With all-at-once, the robot initially learns that the human feature is important, and the person must undo that unintended learning. One-at-at-time learning reduces the unintended effects of the human's noisy corrections; this causes the robot to converge towards the human's desired trajectory more rapidly.
 (a) Task 1 : 1 Correct one feature, the distance to table (b) Task 2: Correct two features, the cup orientation and distance to table
 Figure 4 : 4 Figure 4: Depictions of the robot trajectories for each of the two experimental tasks. The black path represents the original trajectory and the blue path represents the human's desired trajectory.
 Figure 5 : 5 Figure5: The final learned weight vector with one-at-a-time is closer to the ideal weight vector for the task where two feature weights are incorrect (left). Looking at the individual feature differences from ideal: while the final cup weight is closer to ideal for one-at-a-time for both tasks (center), the ideal table weight is actually significantly further away from the ideal for the one-at-a-time strategy during the one-feature task (right). However, for the two feature task, the one-at-a-time method outperforms the all-at-once for final learned cup and table weights.
 One-at-a-Time (a) In the task with only one wrong feature weight, there is no significant difference between the two methods in average dot product over time.
 Figure 7 : 7 Figure7: The one-at-a-time strategy results in significantly less weight updates that are away from the optimum weight across all tasks (left top, left bottom). These findings are consistent with the subjective likert data from the undoing scale, where participants perceived the one-at-a-time method as less likely to learn the wrong thing and need an additional undoing action.
 succ Q1: I successfully taught the robot how to do the task. correct update Q2: The robot correctly updated its understanding about aspects of the task that I did want to change. .84 Q3: The robot wrongly updated its understanding about aspects of the task I did NOT want to change. Q4: The robot understood which aspects of the task I wanted to change, and how to change them. Q5: The robot misinterpreted my corrections.undoing Q6: I had to try to undo corrections that I gave to the robot, because it learned the wrong thing.
 Similar to prior IRL work, we will assume that the correct features for the task have been identified a priori, and are known to both the human and the robot.
 To ensure that all features are equally sensitive, we normalized each feature by the maximal attainable feature difference by computing optimal trajectories offline with a range of θ values.

Andrea Bajcsy
Dylan P. Losey
Marcia K. O'Malley
Anca D. Dragan


Explainable Robotic Systems

As artificial agents, and especially socially interactive robots, enter human society, the demands for these agents to be transparent and explainable grow rapidly. When systems are able, for example, to explain how they made classifications or arrived at a decision, users are better able to judge the systems' accuracy and have more calibrated trust in them (e.g., [8] [10] ). More and more AI systems process vast amounts of information and make classifications or recommendations that humans use for financial, employment, medical, military, and political decisions. More precariously yet, autonomous social robots, by definition, make decisions reaching beyond direct commands and perform actions with social and moral significance for humans. The demand for these social robots to become transparent and explainable is particularly urgent. However, to make robots explainable, we need to understand how people interpret the behavior of such systems and what expectations they have of them. In this workshop, we will address the topics of transparency and explainability, for robots in particular, from both the cognitive science perspective and the computer science and robotics perspective. Cognitive science elucidates how people interpret robot behavior; computer science and robotics elucidate how the computational mechanisms underlying robot behavior can be structured and communicated so as to be human-interpretable. The implementation and use of explainable robotic systems may prevent the potentially frightening confusion over why a robot is behaving the way it is. Moreover, explainable robot systems may allow people to better calibrate their expectations of the robot's capabilities and be less prone to treating robots as almost-humans.
 BACKGROUND When people interact with a robotic system, they construct mental models to understand and predict its actions. However, people's mental models of robots stem from their interactions with living beings. Thus, people easily run the risk of establishing incorrect or inadequate models of robotic systems, which may result in self-deception or even harm [12] . Moreover, a long-term study [9] showed that initially established (incorrect) mental models of an intelligent information system remained robust over time, even when details of the system's implementation were given and initial beliefs were challenged with contradictory evidence. Incorrect mental models of AIS can have significant consequences for trust in such systems and, as a result, for acceptance of and collaboration with these systems [10] . Several studies indicate that people distrust a robotic system when they are unable to understand its actions. When a robot fails to communicate its intentions, people not only perceive the robot as creepy or unsettling [11] , they also perceive such robots as erratic and untrustworthy even when they follow a clear decision-making process [5] . Indeed, when a robot is not transparent about its intentions (i.e., not providing any explanations for its behavior), people may even question its correct task performance and blame the robotic agent for its alleged errors [4] . In addition to such cases of distrust, incorrect mental models of AIS can also lead to the opposite situation. People sometimes over-trust artificial agents, such as when they comply with a faulty robot's unusual requests [7] or follow the lead of a potentially inept robot [6] . Thus, there should be little doubt about the value of making robotic systems perform easily interpretable actions and explain their own decisions and behaviors. This goal for explainable robots demands innovative work in computer science, robotics, and cognitive science, and our proposed workshop provides strong representation from all these fields. Additionally, we invite research that examines if and how such structures lead to better acceptance of robotic systems, to more accurate mental models of such systems, and to calibrated trust in these systems.
 WORKSHOP ACTIVITIES The aim of this full-day workshop is to provide a forum to share and learn about recent research on requirements for artificial agents' explanations as well as the implementation of transparent, predictable and explainable robotic systems. Extended time for discussions will highlight and document promising approaches and encourage further work. We welcome prospective participants to submit extended abstracts (max. 2 pages) that contribute to the discussion of people's interpretation of robot actions as well as the implementation of transparent, predictable, and explainable behaviors in robotic systems. After the conference, and with permission of the authors, we will provide online access to the workshop proceedings on a dedicated workshop website: explainableroboticsystems.wordpress.com. We encourage researchers to attend the workshop even without a paper submission, as a major portion of the workshop involves community engagement and discussion to foster uptake of concepts regarding explainable robotic systems within the field of HRI. The morning session will cover invited talks from Rachid Alami, Joanna Bryson, Bradley Hayes, and Alessandra Sciutti, as well as short presentations by authors of accepted papers, followed by general discussion. The afternoon will be devoted to break-out sessions to discuss the next steps in research and development of explainable and transparent robotic systems. Groups will be composed of representatives of different disciplines in order to work on integrating the multiple necessary perspectives in this endeavor. To boost the discussions, we will ask presenters to prepare questions or raise pressing issues that provide starting points for the discussion groups. is a postdoctoral research associate at the Department of Cognitive, Linguistic, and Psychological Sciences, Brown University, USA. She has a Bachelor of Business Administration in Communication Management (2005), a Master of Science in Media Communication (2011), and a PhD in Communication Science and Human-Robot Interaction (2015). Her research research interest focuses on peopleâĂŹs social, emotional and cognitive responses to robots including the societal and ethical consequences of those responses. Bertram Malle is Professor of Psychology in the Department of Cognitive, Linguistic, and Psychological Sciences at Brown University and Co-Director of the Humanity-Centered Robotics Initiative at Brown. He was trained in psychology, philosophy, and linguistics at the University of Graz, Austria, and received his Ph.D. in Psychology from Stanford University in 1995. His research focuses on social cognition, moral psychology, and human-robot interaction. Anca Dragan is an Assistant Professor in Electrical Engineering and Computer Sciences at UC Berkeley. She received her PhD from the Robotics Institute at Carnegie Mellon University in 2015 and now runs a lab on algorithmic human-robot interaction. She has expertise in transparent and explainable robotics and AI, with applications in personal robots and autonomous cars. Tom Ziemke is Professor of Cognitive Science at the University of Skövde and Professor of Cognitive Systems at Linköping University, Sweden. He received his Ph.D. in Computer Science from Sheffield University, UK, in 2000. His main research interests are embodied cognition and social interaction, and in recent years in particular people's interaction with different types of autonomous systems, ranging from social robots to automated vehicles.

Maartje M.A. de Graaf
Bertram F. Malle
Anca Dragan
Tom Ziemke


An AI Race for Strategic Advantage: Rhetoric and Risks

Introduction Global leadership in different areas of fundamental and applied research in artificial intelligence has emerged as a strategic priority, both for major companies and nation states. The last two years have seen heightened competition between industry research groups for talented researchers and startups (Metz 2017 , CB Insights 2017), and the release of strategic plans aimed at establishing research leadership in AI by the United States (NSTC, 2016), China (Fa, 2017) , and other research-leading nations (Hall and Prescenti 2017 , NEDO 2017 , CIFAR 2017 . As the applications of AI research become more lucrative, and the impacts of AI on economic, scientific and military applications more transformative, this competition can be expected to intensify. A narrative has begun to emerge, in both policy and public contexts (Simonite 2017, Allen and Husain 2017, Stewart 2017, Allen and Kania 2017, Allen and Chan 2017), framing future trajectories relating to the development of AI in terms typically associated with a race for technological superiority. The China State Council's 2017 "A Next Generation Artificial Intelligence Development Plan" includes in its aims to seize the major strategic opportunity for the development of AI, to build China's first-mover advantage in the development of AI. The United States National Science and Technology Council's Research and Development Strategic Plan highlights that the US no longer leads the world in numbers of deep learning-related publications. Strong public statements have been made by political and technology leaders: notable examples include Russia's President Vladimir Putin stating that "whoever becomes the leader in this sphere will become the ruler of the world" (RT 2017) and Open-AI cofounder Elon Musk tweeting that "competition for AI superiority at national level [is the] most likely cause of WW3" (Musk 2017 ). These developments reflect a perception that global leadership in AI could confer scientific, infrastructural and economic advantages to frontrunners. Such comparative advantage could also be self-perpetuating, if it helps to recruit the best research talent, or leads to greater wealth and spending power, or to the accumulation of better computing and data resources, or if the application of AI to the process of scientific research itself leads to further breakthroughs. Similarly, although many breakthroughs are likely to be local in application, the perception that some fundamental advances can be applied to a broad range of research challenges further fuels the idea that a winner could take it all. Such notions of 'winner takes all' or the importance of technological leadership, foster the framing of AI development as a race. We will refer to this framing as that of the 'race for technological superiority' in AI. This kind of race is the primary concern of this paper. But at the same time, it is important to note that there are also concerns about a specifically military AI arms race. Although this is not the primary concern of this paper, there are some noteworthy interrelations with the broader race for technological superiority: most significantly, if AI fulfils its promise as a widely-applicable and transformational technology, then general superiority in AI will to a significant degree imply also military superiority. Many aspects of AI progress are likely to have dual-use relevance to both civilian and military applications (for example, advances enabling greater autonomy and a wider range of capabilities in autonomous vehicles) not least given the deep connection between AI research and the defence community (Geist 2016) . And if systems approach general intel-ligence or superintelligence, they are likely to confer a very significant strategic advantage that could encompass, but go well beyond, conventional weaponry, with radical implications for security and geopolitics (Allen and Chan 2017). The central concern of this paper is what might happen if the rhetoric noted above around first mover advantage or the pursuit of AI superiority becomes a widespread or even dominant framing of AI development. Our concerns arise both from use of the language of a race for technological superiority (independently of whether such a race is happening) and from such a race becoming a reality.
 The Dangers of an AI Race for Technological Advantage: In Rhetoric and in Reality What is so bad about framing the development of AI in terms of a race for technological advantage? After all, it is widely agreed that AI brings enormous potential benefits across many sectors. One recent report estimated that it could add £232 billion by 2030 to the UK economy alone, with healthcare one of the sectors most enhanced, potentially bringing faster, better service to consumers (PwC 2017). There is a widespread belief that competition and other market forces are central to such innovation. So in as much as a race to develop AI technology means these kinds of benefit come sooner, we have reason to view it positively. But at the same time, the development of such a potentially powerful new technology will need to be steered if it is to be as beneficial as possible while minimising the risks it might pose (Crawford and Calo 2016). In the words of the Future of Life Institute's open letter on 'Research Priorities for Robust and Beneficial Artificial Intelligence', signed by over 8,000 people including many leading figures in AI, work should focus "not only on making AI more capable, but also on maximizing the societal benefit of AI." (Future of Life Institute, 2017a). The danger of an AI race is that it makes exactly this thoughtful steering towards broadly beneficial outcomes more difficult.
 Three Sets of Risks We want to distinguish between three sets of risks: i) The dangers of an AI 'race for technological advantage' framing, regardless of whether the race is seriously pursued; ii) The dangers of an AI 'race for technological advantage' framing and an actual AI race for technological advantage, regardless of whether the race is won; iii) The dangers of an AI race for technological advantage being won.
 (i) Risks Posed by a Race Rhetoric Alone It is possible that the trend towards 'race for technological advantage' terminology in AI, including suggestions such as Vladimir Putin's that the winner of such a race "will become the ruler of the world," could pose risks even if the race is not pursued in earnest, let alone won. We perceive two main risks here: (i.a) The kind of thoughtful consideration of how to achieve broadly beneficial AI, as mentioned above, will require subtle, inclusive, multi-stakeholder deliberation over a prolonged peri-od. The rhetoric of the race for technological advantage, with its implicit or explicit threat that dire consequences will follow if some other group wins that race, is not likely to be conducive to such deliberation. Indeed, rhetoric around technological superiority (such as the 'arms race' rhetoric used in the Cold War in the US), played into what has been called a politics of fear, or a politics of insecurity --that is, a political climate that discourages debate in favour of unquestioning support for a prescribed agenda (Griffith 1987 ). In The Politics of Insecurity, Jef Huysmans argues that use of the language of security (by which he means militarised language, which would include 'arms race' and related rhetoric) "is a particular technique of framing policy questions in logics of survival with a capacity to mobilize politics of fear in which social relations are structured on the basis of distrust" (Huysmans 2006 ). (i.b) Second, if the rhetoric of a competitive, 'winner takes all' AI race is used in the absence of an actual race, it could contribute to sparking such a race. (ii) Risks Posed by a Race Emerging If the rhetoric of a race for technological advantage became an actual race to develop sophisticated AI, the risks increase further: (ii.a) First, there is the risk that racing to achieve powerful AI would not be conducive to taking the proper safety precautions that such technology will require (Armstrong, Bostrom, and Shulman 2016). We mentioned above the need for broad consultation about the role AI should play in the life of a community. This might help address important considerations such as avoiding biased systems, or maximising fairness. But in addition to these goals, serious attention must also be given to ensuring humans do not lose control of systems. Such considerations become particularly important if AI approaches general intelligence or superintelligence (Bostrom 2014 ), but also long before, particularly when AI systems are performing critical functions. The risk is that as the perceived benefit to winning the race increases, so correspondingly does the incentive to cut corners on these safety considerations. (ii.b) Second, a 'race for technological advantage' could increase the risk of competition in AI causing real conflict (overt or covert). Huysmans argues that militarised language such as this has "a specific capacity for fabricating and sustaining antagonistic relations between groups. In the case of the race for technological advantage, it encourages us to see competitors as threats or even enemies. The belief that a country intends in earnest to win an AI race, and that this would result in technological dominance, could, for example, prompt other countries to use aggression to prevent this (akin to the cyberattacks made against Iranian nuclear facilities attributed to the US and Israel) (Nakashima 2012), or motivate the targeting of key personnel (precedents --though specific to their historical context --might include Operation Paperclip, during which over 1,600 German scientists and engineers who had worked on military technology were taken to the US (Jacobsen 2014), or the apparently ongoing operations to encourage the defection of nuclear scientists between nations) (Golden 2017) . Such scenarios would also in-crease the risk that a general race for technological superiority became increasingly a military AI arms race.
 (iii) Risks Posed by Race Victory The third category of risks of an AI race for technological superiority are those that would arise if a race were won. We will not explore these in detail here --and the forms they take will anyway depend on the precise nature of the technology in question. But as an example, these risks include the concentration of power in the hands of whatever group possesses this transformative technology. If we survey the current international landscape, and consider the number of countries demonstrably willing to use force against others, as well as the speed with which political direction within a country can change, and the persistence of non-state actors such as terrorist groups, we might conclude that the number of groups we would not trust to responsibly manage an overwhelming technological advantage exceeds the number we would.
 Choices for the Research Community Given the dangers noted above of talking about AI development as a competitive race, one could argue that it would be better for the community of researchers considering AI and its impacts to avoid this terminology altogether. In this way, the language of the race for dominance would be seen as (in Nick Bostrom's terms) an information hazard --perhaps either an idea hazard (a general idea whose dissemination could increase risk) or an attention hazard (if we consider that the dangerous idea already exists, but is as yet largely unnoticed) (Bostrom 2011 ). But if we believe that the idea is already being disseminated by influential actors, such as states, including major powers, then the information hazard argument is weakened. It might still apply to particularly influential researchers --those whose thoughts on the future of AI can become headline news around the world. But even in their case, and particularly in the case of lesser-known figures, there is a strong countervailing argument: that if the potentially dangerous idea of an AI race is already gaining currency, then researchers could make a positive impact by publicly drawing attention to these dangers, as well as by pursuing research dedicated to mitigating the risks of such a framing. Of course, many leading researchers are already speaking out against an AI arms race in the sense of a race to develop autonomous weapons --see for example the Future of Life Institute's open letter on this, signed by over three thousand AI researchers and over fifteen thousand others (Future of Life Institute 2015b). We believe this community could also usefully direct its attention to speaking out against an AI race in this other sense of a competitive rush to develop powerful general-purpose AI as fast as possible. Both media and governments are currently giving considerable attention to AI, yet are still exploring ways of framing it and its impacts. We believe that there is therefore an opportunity now for researchers to influence this framing 1 . One of the principles on AI agreed at the 2017 Asilomar conference offers a precedent on which to build --it states: Race Avoidance: Teams developing AI systems should actively cooperate to avoid corner-cutting on safety standards (Future of Life Institute 2017).
 Alternatives to a Race Approach If we are not to pursue (and talk about pursuing) AI development as a race, then how should we pursue (and talk about pursuing) such development? This is too big a topic to consider thoroughly here. But we note these promising directions for alternative narratives around progress in, and benefits of, AI:
 AI Development as a Shared Priority for Global Good As advances in AI find application to an ever-wider range of scientific and societal challenges, there is a burgeoning discussion around harnessing the benefits of AI for global benefit. This reflects a widespread view among AI scientists and a growing number of policymakers that AI presents tremendous opportunities for making progress on global societal ills, and aiding in tackling some of the biggest challenges we face in the coming century --among them climate change and clean energy production, biodiversity loss, healthcare, global poverty and education. Emphasising these benefits could counteract a race approach in a number of ways: First, if there is global scientific consensus that some of the key aims of AI should be to benefit humanity in these ways, then it becomes less important in which companies or countries key breakthroughs occur. Second, it makes clear that cooperation on the development of AI stands to result in faster progress on these pressing challenges. Lastly, if the aims of the field are to benefit humanity worldwide, then the global community represent stakeholders in the process of AI development; this narrative therefore promotes inclusive and collaborative development and deployment of AI.
 Cooperation on AI as it is Applied to Increasingly Safety-Critical Settings Globally The next decade will see AI applied in an increasingly integral way to safety-critical systems; healthcare, transport, infrastructure to name a few. In order to realise these benefits as quickly and safely as possible, sharing of research, datasets, and best practices will be critical. For example, to ensure the safety of autonomous cars, pooling expertise and datasets on vehicle performances across as wide as possible a range of environments and conditions (including accidents and near-accidents) would provide substantial benefits for all involved. This is particularly so given that the research, data, and testing needed to refine and ensure the safety of such systems before deployment may be considerably more costly and time-consuming than the research needed to develop the initial technological capability. Promoting recognition that deep cooperation of this nature is needed to deliver the benefits of AI robustly may be a powerful tool in dispelling a 'technological race' narrative; and a 'cooperation for safe AI' framing is likely to become increasingly important as more powerful and broadly capable AI systems are developed and deployed.
 Responsible Development of AI and Public Perception AI is the focus for a growing range of public concerns as well as optimism (Ipsos MORI 2017, Fast and Horvitz 2017). Many stakeholders, including in industry, recognise the importance of public trust in the safety and benefits offered by AI if it is to be deployed successfully (Banavar 2017) . It is the view of the authors that a narrative focused on global cooperation and safe, responsible development of AI is likely to inspire greater public confidence than a narrative focused more on technological dominance or leadership. Other powerful new technologies, such as genetically modified organisms and nuclear power, have in the past proven controversial, with significant communities arguing for a cautious, safety-first approach, to which the rhetoric of the race is antithetical.
 Recent Narratives There have been encouraging developments promoting the above narratives in recent years. 'AI for global benefit' is perhaps best exemplified by the 2017's ITU summit on AI for Global Good (Butler 2017), although it also features prominently in narratives being put forward by the IEEE's Ethically Aligned Design process (IEEE 2016), the Partnership on AI, and programmes and materials put forward by Microsoft, DeepMind and other leading companies. Collaboration on AI in safetycritical settings is also a thematic pillar for the Partnership on AI 2 . Even more ambitious cooperative projects have been proposed by others, for example the call for a 'CERN for AI' from Professor Gary Marcus, through which participants "share their results with the world, rather than restricting them to a single country or corporation" (Marcus 2017). Finally, the overall narrative of cooperation was clearly expressed in a statement issued by a Beneficial AI conference in Tokyo 3 : The challenges of ensuring that [AI] is beneficial are challenges for us all. We urge that these challenges be addressed in a spirit of cooperation, not competition. (Beneficial AI Tokyo, 2017) .
 Conclusion Although artificial intelligence has been discussed for decades, only recently has it received serious and sustained attention from governments, industry and the media. Among the various ways in which this technology is framed, we have highlighted one that we consider to be potentially dangerous: that of the race for technological advantage. Partly, we believe that a general race to develop AI would be dangerous because it would also encompass --given the dual use of this technology --a literal, military arms race. But even if this were not the case, we believe there would be risks -e.g. from corner-cutting in safety and consultation. Although some might argue that the research community should altogether avoid using AI 'race' terminology for fear of giving it currency, we believe that the idea is already current enough to justify interventions that draw attention to the dangers. Much work remains to be done in understanding the dynamics of a possible race, and in developing alternative For example, few countries have published formal strategies or legislation on AI, but a number have commissioned reviews that have sought expert opinion, eg, the UK Government-commissioned report on AI(Hall and Prescenti, 2017) and the White House report (NSTC, 2016).
 See https://www.partnershiponai.org/thematic-pillars/ 3 In which both authors were involved. framings for AI development --but there are encouraging examples on which to build.