Confidence-aware motion prediction for real-time collision avoidance 1

Introduction Motion planning serves a key role in robotics, enabling robots to automatically compute trajectories that achieve the specified objectives while avoiding unwanted collisions. In many situations of practical interest, such as autonomous driving and unmanned aerial vehicle (UAV) navigation, it is important that motion planning account not just for the current state of the environment, but also for its predicted future state. Often, certain objects in the environment may move in active, complex patterns that cannot be readily predicted using straightforward physics models; we shall refer to such complex moving objects as agents. Examples of agents considered in this paper include pedestrians and human-driven vehicles. Predicting the future state of these agents is generally a difficult problem. Some of the key challenges include unclear and varying intents of other agents, mismatches between dynamics models and reality, incomplete sensor information, and interaction effects. One popular approach to addressing the challenge of a priori unknown agent intent is to use rule-based or data-driven algorithms to predict individual trajectories for each agent, as in  Schmerling et al. (2017) . Alternatively,  Ziebart et al. (2009) ,  Bandyopadhyay et al. (2013) , and  Kochenderfer et al. (2010)  explicitly predict an agent's full state distribution over time; this representation may be better suited to capturing uncertainty in an agent's dynamics and the environment itself.  Wang et al. (2019)  and  Fisac et al. (2018b)  pose the prediction problem game-theoretically to model coupled human-robot interaction effects explicitly. Unfortunately, a significant problem still remains: if an agent suddenly moves in a way that is not predicted, or not assigned sufficient probability, the robot may not react appropriately. For example, in Figure  1  a pedestrian is walking around an obstacle that the robot, a quadcopter, cannot detect. To the robot, such behavior may be assigned very low probability, which could lead the robot to plan a dangerous trajectory. In this particular example, this inaccuracy caused the quadcopter to collide with the pedestrian (Figure  1 , left). To prepare for this eventuality, we introduce the idea of confidence-aware prediction. We argue that, in addition to predicting the future state of an agent, it is also crucial for a robot to assess the quality of the mechanism by which it is generating those predictions. That is, a robot should reason about how confident it is in its predictions of other agents before attempting to plan future motion. For computational efficiency, the quadcopter uses a simplified model of pedestrian dynamics and decision-making. Thus equipped, it generates a time-varying probability distribution over the future state of the pedestrian, and plans trajectories to a pre-specified goal that maintain a low probability of collision. Figure  1 (right) illustrates how this approach works in practice. The quadcopter maintains a Bayesian belief over its prediction confidence. As soon as the pedestrian moves in a way that was assigned low probability by the predictive model, the quadcopter adjusts its belief about the accuracy of that model. Consequently, it is less certain about what the pedestrian will do in the future. This leads the quadcopter's onboard motion planner, which attempts to find efficient trajectories with low probability of collision, to generate more cautious, and perhaps less efficient, motion plans. In order to improve the robustness of generated motion plans, we employ the recent FaSTrack framework from  Herbert et al. (2017)  for fast and safe motion planning and tracking. FaSTrack quantifies the maximum possible tracking error between a high-order dynamical model of the physical robot and the (potentially lower-order) dynamical model used by its motion planner. Solving an offline Hamilton-Jacobi reachability problem yields a guaranteed tracking error bound and the corresponding safety controller. These may be used by an out-of-the-box real-time motion planning algorithm to facilitate motion plans with strong runtime collision-avoidance guarantees. The remainder of this paper is organized as follows. Section 2 places this work in the context of existing literature in human motion modeling and prediction, as well as robust motion planning. Section 3 frames the prediction and planning problems more formally, and introduces a running example used throughout the paper. Section 4 presents our main contribution: confidence-aware predictions. Section 5 showcases confidence-aware predictions in operation in several examples. Section 6 describes the application of the robust motion planning framework from FaSTrack to this setting, in which predictions are probabilistic. Section 7 explores a connection between our approach and reachability theory. Section 8 presents experimental results from a hardware demonstration. Finally, Section 9 concludes with a discussion of some of the limitations of our work and how they might be addressed in specific applications, as well as suggestions for future research.   1 . When planning around humans, accurate predictions of human motion (visualized here in pink and blue, representing high and low probability, respectively) are an essential prerequisite for safety. Unfortunately, these approaches may fail to explain all observed motion at runtime (e.g., human avoids unmodeled spill on the ground), leading to inaccurate predictions, and potentially, collisions (left). Our method addresses this by updating its predictive model confidence in real time (right), leading to more conservative motion planning in circumstances when predictions are known to be suspect.
 Prior work 2
 Human modeling and prediction One common approach for predicting human actions is to collect data from real-world scenarios and train a machine learning model via supervised learning. Such techniques use the human's current state, and potentially her prior state and action history, to predict future actions directly.  Amor et al. (2014) ,  Ding et al. (2011) ,  Koppula and Saxena (2013) ,  Lasota and Shah (2015) , and  Hawkins et al. (2013)  demonstrated the effectiveness of this approach for inference and planning around human arm motion. In addition,  Hawkins et al. (2013)  focused on multi-step tasks such as assembly, and  Schmerling et al. (2017)  and  Driggs-Campbell et al. (2018)  addressed the prediction problem for human drivers. Rather than predicting actions directly, an alternative is for the robot to model the human as a rational agent seeking to maximize an unknown objective function. The human's actions up to a particular time may be viewed as Bayesian evidence from which the robot may infer the parameters of that objective. Assuming that the human seeks to maximize this objective in the future, the robot can predict her future movements (e.g.,  Bai et al., 2015; Baker et al., 2007; Ng and Russell, 2000; Ziebart et al., 2009) . In this paper, we build on this work by introducing a principled online technique for estimating confidence in such a learned model of human motion.
 Safe robot motion planning Once armed with a predictive model of the human motion, the robot may leverage motion planning methods that plan around uncertain moving obstacles and generate real-time dynamically feasible and safe trajectories. To avoid moving obstacles in real time, robots typically employ reactive and/or path-based methods. Reactive methods directly map sensor readings into control, with no memory involved (e.g.,  Belkhouche, 2009) . Path-based methods such as rapidly-exploring random trees from  Karaman and Frazzoli (2011)  and A* from  Hart et al. (1968)  find simple kinematic paths through space and, if necessary, time. These path-based methods of planning are advantageous in terms of efficiency, yet, while they have in some cases been combined with probabilistically moving obstacles as in  Aoude et al. (2013)  and  Ziebart et al. (2009) , they do not consider the endogenous dynamics of the robot or exogenous disturbances such as wind. As a result, the robot may deviate from the planned path and potentially collide with obstacles. It is common for these plans to try to avoid obstacles by a heuristic margin of error.  Herbert et al. (2017) and Fridovich-Keil et al. (2018)  proposed FaSTrack, a recent algorithm that provides a guaranteed tracking error margin and corresponding errorfeedback controller for dynamic systems tracking a generic planner in the presence of bounded external disturbance. Our work builds upon FaSTrack to create an algorithm that can safely and dynamically navigate around uncertain moving obstacles in real time.
 Problem setup We consider a single mobile robot operating in a shared space with a single human agent (e.g., a pedestrian or human-driven car). For simplicity, we presume that the robot has full knowledge of its own state and that of the human, although both would require online estimation in practice. As we present each formal component of this problem, we will provide a concrete illustration using a running example in which a quadcopter is navigating around a pedestrian.
 Dynamical system models and safety We will model the motion of both the human and the robot as the evolution of two dynamical systems. Let the state of the human be x H 2 R n H , where n H is the dimension of the human state space. We similarly define the robot's state, for planning purposes, as x R 2 R n R . In general, these states could represent the positions and velocities of a mobile robot and a human in a shared environment, the kinematic configurations of a human and a robotic manipulator in a common workspace, or the positions, orientations, and velocities of human-driven and autonomous vehicles in an intersection. We express the evolution of these states over time as a family of ordinary differential equations: _ x H = f H (x H , u H ), _ x R = f R (x R , u R ) ð1Þ where u H 2 R m H and u R 2 R m R are the control actions of the human and robot, respectively. Running example: We introduce a running example for illustration purposes throughout the paper. In this example we consider a small quadcopter that needs to fly to goal location g R 2 R 3 in a room where a pedestrian is walking. For the purposes of planning, the quadcopter's 3D state is given by its position in space x R = ½p x , p y , p z , with velocity controls assumed decoupled in each spatial direction, up to v R = 0:25 m/s. The human can only move by walking and therefore her state is given by planar coordinates x H = ½h x , h y evolving as _ x H = ½v H cos u H , v H sin u H . Intuitively, we model the human as moving with a fixed speed and controlling their heading angle. At any given time, the human is assumed to either move at a leisurely walking speed (v H '1 m/s) or remain still (v H '0). Ultimately, the robot needs to plan and execute an efficient trajectory to a pre-specified goal state (g R ), without colliding with the human. We define the keep-out set K & R n H × R n R as the set of joint robot-human states to be avoided (for example, because they imply physical collisions). To avoid reaching this set, the robot must reason about the human's future motion when constructing its own motion plan. Running example: In our quadcopter-avoidingpedestrian example, K consists of joint robot-human states in which the quadcopter is flying within a square of side length l = 0:3 m centered around the human's location, while at any altitude, as well as any joint states in which the robot is outside the environment bounds defined as a box with a square base of side L = 3:66 m and height H = 2 m, regardless of the human's state.
 Robust robot control Provided an objective and a dynamics model, the robot must generate a motion plan that avoids the keep-out set K. Unfortunately, this safety requirement is difficult to meet during operation for two main reasons. 1. Model mismatch. The dynamical system model f R will never be a perfect representation of the real robot. This mismatch could lead to unintended collision. 2. Disturbances. Even with a perfect dynamics model, there may be unobserved, external ''disturbance'' inputs such as wind or friction. Without accounting for these disturbances, the system is not guaranteed to avoid K, even if the planned trajectory is pointwise collision-free. To account for modeling error and external disturbances, we could in principle design a higher-fidelity dynamical model directly in a robust motion planning framework. Unfortunately, however, real-time trajectory optimization in high dimensions can be computationally burdensome, particularly when we also require some notion of robustness to external disturbance. Ideally, we would like to enjoy the computational benefits of planning with a lower-fidelity model while enforcing the safety constraints induced by the higher-fidelity model. To characterize this model mismatch, we consider a higher fidelity and typically higher-order dynamical representation of the robot, with state representation s R 2 R n S . This dynamical model will also explicitly account for external disturbances as unknown bounded inputs, distinct from control inputs. In order to map between this higher-fidelity ''tracking'' state s R and the lower-fidelity ''planning'' state x R , we shall assume a known projection operator p : R n S ! R n R . Fortunately, we can plan in the lower-dimensional state space at runtime, and guarantee robust collision avoidance via an offline reachability analysis that quantifies the effects of model mismatch and external disturbance. This framework, called FaSTrack and first proposed by  Herbert et al. (2017) , is described in further detail in Section 6. Running example: We model our quadcopter with the following flight dynamics (in the near-hover regime, at zero yaw with respect to a global coordinate frame): _ p x _ p y _ p z 2 4 3 5 = v x v y v z 2 4 3 5 , _ v x _ v y _ v z 2 4 3 5 = a g tan u u Àa g tan u f u T À a g 2 4 3 5 ð2Þ where ½p x , p y , p z is the quadcopter's position in space and ½v x , v y , v z is its velocity expressed in the fixed global frame. We model its control inputs as thrust acceleration u T and attitude angles (roll u f and pitch u u ), and denote the acceleration due to gravity as a g . The quadcopter's motion planner generates nominal kinematic trajectories in the lowerdimensional ½p x , p y , p z position state space. Therefore, we have a linear projection map p(s R ) = ½I 3 , 0 3 s R , that is, x R retains the position variables in s R and discards the velocities.
 Predictive human model In order to predict the human's future motion, the robot uses its internal model of human dynamics, f H . Under this modeling assumption, the human's future trajectory depends upon the choice of control input over time, u H ( Á ). Extensive work in econometrics and cognitive science, such as that of  Von Neumann and Morgenstern (1945) ,  Luce (1959), and Baker et al. (2007) , has shown that human behavior (that is, u H ) can be well modeled by utility-driven optimization. Thus, the robot models the human as optimizing a reward function, r H (x H , u H ; u), that depends on the human's state and action, as well as a set of parameters u. This reward function could be a linear combination of features as in many inverse optimal control implementations (where the goal or feature weighting u must be learned, either online or offline), or more generally learned through function approximators such as deep neural networks, where u are the trained weights as in  Finn et al. (2016) . We assume that the robot has a suitable human reward function r H , either learned offline from prior human demonstrations or otherwise encoded by the system designers. Thus, endowed with r H , the robot can model the human's choice of control action as a probability distribution over actions conditioned on state. Under maximumentropy assumptions  (Ziebart et al., 2008)  inspired by noisy-rationality decision-making models  (Baker et al., 2007) , the robot models the human as more likely to choose (discrete) actions u H with high expected utility, in this case the state-action value (or Q-value): P(u H jx H ; b, u) = e bQ H (x H , u H ;u) P ũ e bQ H (x H , ũ;u) ð3Þ We use a temporally and spatially discretized version of human dynamics, fH . These discrete-time dynamics may be found by integrating f H over a fixed time step of Dt with fixed control u H over the interval. Section 5 provides further details on this discretization. Running example: The quadcopter's model of the human assumes the human intends to reach some target location g H 2 R 2 in a straight line. The human's reward function is given by the distance traveled over time step Dt, i.e., r H (x H , u H ; g H ) = À v H Dt, and human trajectories are constrained to terminate at g H . The state-action value, parameterized by u = g H , captures the optimal cost of reaching g H from x H when initially applying u H for a duration Dt: Q H (x H , u H ; g H ) = À v H DtÀ k x H + v H Dt½cos u H , sin u H > À g H k 2 : Often, the coefficient b is termed the rationality coefficient, because it quantifies the degree to which the robot expects the human's choice of control to align with its model of utility. For example, taking b # 0 yields a model of a human who appears ''irrational,'' choosing actions uniformly at random and completely ignoring the modeled utility. At the other extreme, taking b " ' corresponds to a ''perfectly rational'' human, whose actions exactly optimize the modeled reward function. As we will see in Section 4, b can also be viewed as a measure of the robot's confidence in the predictive accuracy of Q H . Note that Q H (x H , u H ; u) only depends on the human state and action and not on the robot's. Thus far, we have intentionally neglected discussion of human-robot interaction effects. These effects are notoriously difficult to model, and the community has devoted a significant effort to building and validating a variety of models (e.g.,  Sadigh et al., 2016; Trautman and Krause, 2010) . In that spirit, we could have chosen to model human actions u H as dependent upon robot state x R in (3), and likewise defined Q H to depend upon x R . This extended formulation is sufficiently general as to encompass all possible (Markov) interaction models. However, in this work we explicitly do not model these interactions; indeed, one of the most important virtues of our approach is its robustness to precisely these sorts of modeling errors.
 Probabilistically safe motion planning Ideally, the robot's motion planner should generate trajectories that reach a desired goal state efficiently, while maintaining safety. More specifically, in this context ''safety'' indicates that the physical system will never enter the keepout set K during operation, despite human motion and external disturbances. That is, we would like to guarantee that (p(s R ), x H ) 6 2 K for all time. To make this type of strong, deterministic, a priori safety guarantee requires the robot to avoid the set of all human states x H which could possibly be occupied at a particular time, i.e., the human's forward reachable set. If the robot can find trajectories that are safe for any possible human trajectory then there is no need to predict the human's next action. Unfortunately, the forward reachable set of the human often encompasses such a large volume of the workspace that it is impossible for the robot to find a guaranteed safe trajectory to the goal state. This motivates refining our notion of prediction: rather than reasoning about all the places where the human could be, the robot can instead reason about how likely the human is to be at each location. This probabilistic reasoning provides a guide for planning robot trajectories with a quantitative degree of safety assurance. Our probabilistic model of human control input (3) coupled with dynamics model f H allows us to compute a probability distribution over human states for every future time. By relaxing our conception of safety to consider only collisions that might occur with sufficient probability P th , we dramatically reduce the effective volume of this set of future states to avoid. In practice, P th should be chosen carefully by a system designer in order to trade off overall collision probability with conservativeness in motion planning. The proposed approach in this paper follows two central steps to provide a quantifiable, high-confidence collision avoidance guarantee for the robot's motion around the human. In Section 4 we present our proposed Bayesian framework for reasoning about the uncertainty inherent in a model's prediction of human behavior. Based on this inference, we demonstrate how to generate a real-time probabilistic prediction of the human's motion over time. Next, in Section 6, we extend a state-of-the-art, provably safe, real-time robotic motion planner to incorporate our time-varying probabilistic human prediction.
 Confidence-aware human motion prediction Any approach to human motion prediction short of computing a full forward reachable set must, explicitly or implicitly, reflect a model of human decision-making. In this work, we make that model explicit by assuming that the human chooses control actions in a Markovian fashion according to the probability distribution (3). Other work in the literature, such as that of  Schmerling et al. (2017) , aims to learn a generative probabilistic model for human trajectories; implicitly, this training procedure distills a model of human decision-making. Whether explicit or implicit, these models are by nature imperfect and liable to make inaccurate predictions eventually. One benefit of using an explicit model of human decision-making, such as (3), is that we may reason directly and succinctly about its performance online. In particular, the entropy of the human control distribution in (3) is a decreasing function of the parameter b. High values of b place more probability mass on highutility control actions u H , whereas low values of b spread the probability mass more evenly between different control inputs, regardless of their modeled utility Q H . Therefore, b naturally quantifies how well the human's motion is expected to agree with the notion of optimality encoded in Q H . The commonly used term ''rationality coefficient,'' however, seems to imply that discrepancies between the two indicate a failure on the human's part to make the ''correct'' decisions, as encoded by the modeled utility. Instead, we argue that these inevitable disagreements are primarily a result of the model's inability to fully capture the human's behavior. Thus, instead of conceiving of b as a rationality measure, we believe that b can be given a more pragmatic interpretation related to the accuracy with which the robot's model of the human is able to explain the human's motion. Consistently, in this paper, we refer to b as model confidence. An important related observation following from this interpretation of b is that the predictive accuracy of a human model is likely to change over time. For example, the human may change their mind unexpectedly, or react suddenly to some aspect of the environment that the robot is unaware of. Therefore, we shall model b as an unobserved, time-varying parameter. Estimating it in real time provides us with a direct, quantitative summary of the degree to which the utility model Q H explains the human's current motion. To do this, we maintain a Bayesian belief about the possible values of b. Initially, we begin with a uniform prior over b and over time this distribution evolves given measurements of the human's state and actions.
 Real-time inference of model confidence We reason about the model confidence b as a hidden state in a hidden Markov model (HMM) framework. The robot starts with a prior belief b 0 À over the initial value of b. In this work, we use a uniform prior, although that is not strictly necessary. At each discrete time step k 2 f0, 1, 2, . . .g, it will have some belief about model confidence b k À (b). 3 After observing a human action u k H , the robot will update its belief to b k + by applying Bayes' rule. The hidden state may evolve between subsequent time steps, accounting for the important fact that the predictive accuracy of the human model may change over time as unmodeled factors in the human's behavior become more or less relevant. As, by definition, we do not have access to a model of these factors, we use a naive ''e-static'' transition model: at each time k, b may, with some probability e, be re-sampled from the initial distribution b 0 À , and otherwise retains its previous value. We define the belief over the next value of b (denoted by b 0 ) as an expectation of the conditional probability P(b 0 jb), i.e., b k À (b 0 ) :¼ E b;b kÀ1 + ½P(b 0 jb). Concretely, this expectation may be computed as b k À (b 0 ) = (1 À e)b kÀ1 + (b 0 ) + eb 0 À (b 0 ) ð4Þ By measuring the evolution of the human's state x H over time, we assume that, at every time step k, the robot is able to observe the human's control input u k H . This observed control may be used as evidence to update the robot's belief b k À about b over time via a Bayesian update: b k + (b) = P(u k H jx k H ; b, u)b k À (b) P b P(u k H jx k H ; b, u)b k À ( b) ð5Þ with b k + (b) :¼ P(bjx 0:k H , u 0:k H ) for k 2 f0, 1, . . .g, and P(u k H jx k H ; b, u) given by (3). It is critical to be able to perform this update rapidly to facilitate real-time operation; this would be difficult in the original continuous hypothesis space b 2 ½0, '), or even in a large discrete set. Fortunately, our software examples in Section 5 and hardware demonstration in Section 8 suggest that maintaining a Bayesian belief over a relatively small set of N b = 5 discrete values of b distributed on a log scale achieves significant improvement relative to using a fixed value. The ''e-static'' transition model leads to the desirable consequence that old observations of the human's actions have a smaller influence on the current model confidence distribution than recent observations. In fact, if no new observations are made, successively applying time updates asymptotically contracts the belief towards the initial distribution, that is, b k À ( Á ) ! b 0 À ( Á ). The choice of parameter e effectively controls the rate of this contraction, with higher e leading to more rapid contraction.
 Human motion prediction Equipped with a belief over b at time step k, we are now able to propagate the human's state distribution forward to any future time via the well-known Kolmogorov forward equations, recursively. In particular, suppose that we know the probability that the human is in each state x k H at some future time step k. We know that (according to our utility model) the probability of the human choosing control u k H in state x k H is given by (3). Accounting for the otherwise deterministic dynamics model fH , we obtain the following expression for the human's state distribution at the following time step k + 1: P(x k + 1 H ; b, u) = X x k H , u k H P(x k + 1 H jx k H , u k H ; b, u) Á P(u k H jx k H ; b, u)P(x k H ; b, u) ð6Þ for a particular choice of b. Marginalizing over b according to our belief at the current step time k, we obtain the overall occupancy probability distribution at each future time step k: P(x k H ; u) = E b;b k P(x k H ; b, u) ð7Þ Note that (  6 ) is expressed more generally than is strictly required. Indeed, because the only randomness in dynamics model fH originates from the human's choice of control input u H , we have P( x k + 1 H jx k H , u k H ; b, u) = 1fx k + 1 H = fH (x k H , u k H )g.
 Model confidence with auxiliary parameter identification Thus far, we have tacitly assumed that the only unknown parameter in the human utility model (  3 ) is the model confidence, b. However, often one or more of the auxiliary parameters u are also unknown. These auxiliary parameters could encode one or more human goal states or intents, or other characteristics of the human's utility, such as her preference for avoiding parts of the environment. Further, much like model confidence, they may change over time. In principle, it is possible to maintain a Bayesian belief over b and u jointly. The Bayesian update for the hidden state (b, u) is then given by ) the prior at time step k. This approach can be practical for parameters taking finitely many values from a small, discrete set, e.g., possible distinct modes for a human driver (distracted, cautious, aggressive). However, for certain scenarios or approaches it may not be practical to maintain a full Bayesian belief on the parameters u. In such cases, it is reasonable to replace the belief over u with a point estimate u, such as the maximum likelihood estimator or the mean, and substitute that estimate into (6). Depending on the complexity of the resulting maximum likelihood estimation problem, it may or may not be computationally feasible to update the parameter estimate u at each time step. Fortunately, even when it is computationally expensive to estimate u, we can leverage our model confidence as an indicator of when reestimating these parameters may be most useful. That is, when model confidence degrades that may indicate poor estimates of u. b k + (b, u) = P(u k H jx k H ; b, u)b k À (b, u) P b, ũ P(u k H jx k H ; b, ũ)b k À ( b, ũ) ð8Þ
 Prediction examples We illustrate these inference steps with two sets of examples: our running pedestrian example and a simple model of a car.
 Pedestrian model (running example) So far, we have presented a running example of a quadcopter avoiding a human. We use a deliberately simple, purely kinematic model of continuous-time human motion: _ x H = _ h x _ h y ! = v H cos u H v H sin u H ! ð9Þ However, as discussed in Section 3.3, the proposed prediction method operates in discrete time (and space). The discrete dynamics corresponding to (9) are given by x k + 1 H À x k H [x H (t + Dt) À x H (t) = v H Dt cos u H (t) v H Dt sin u H (t) ! ð10Þ for a time discretization of Dt.
 Dubins car model To emphasize the generality of our method, we present similar results for a different application domain: autonomous driving. We will model a human-driven vehicle as a dynamical system whose state x H evolves as _ x H = _ h x _ h y _ h f 2 4 3 5 = v H cos h f v H sin h f u H 2 4 3 5 ð11Þ Observe that, while (11) appears very similar to (9), in this Dubins car example the angle of motion is a state, not a control input. We discretize these dynamics by integrating (11) from t to t + Dt, assuming a constant control input u H : x k + 1 H À x k H [x H (t + Dt) À x H (t) = v H u H (t) sin (h f (t) + u H (t)Dt) À sin (h f (t)) À Á À v H u H (t) cos (h f (t) + u H (t)Dt) À cos (h f (t)) À Á u H Dt 2 6 4 3 7 5 For a specific goal position g = ½g x , g y , the Q-value corresponding to state-action pair (x H , u H ) and reward function r H (x H , u H ) = À v H Dt (until the goal is reached) may be found by solving a shortest path problem offline.
 Accurate model First, we consider a scenario in which the robot has full knowledge of the human's goal, and the human moves along the shortest path from a start location to this known goal state. Thus, human motion is well-explained by Q H . The first row of Figure  2  illustrates the probability distributions our method predicts for the pedestrian's future state at different times. Initially, the predictions generated by our Bayesian confidence-inference approach (right) appear similar to those generated by the low model confidence predictor (left). However, our method rapidly discovers that Q H is an accurate description of the pedestrian's motion and generates predictions that match the high model confidence predictor (center). The data used in this example was collected by tracking the motion of a real person walking in a motion capture arena. See Section 8 for further details. Likewise, the first row of Figure  3  shows similar results for a human-driven Dubins car model (in simulation) at an intersection. Here, traffic laws provide a strong prior on the human's potential goal states. As shown, our method of Bayesian model confidence inference quickly infers the correct goal and learns that the human driver is acting in accordance with its model Q H . The resulting predictions are substantially similar to the high-b predictor. The data used in this example was simulated by controlling a Dubins car model along a pre-specified trajectory.
 Unmodeled obstacle Often, robots do not have fully specified models of the environment. Here, we showcase the resilience of our approach to unmodeled obstacles that the human must avoid. In this scenario, the human has the same start and goal as in the accurate model case, except that there is an obstacle along the way. The robot is unaware of this obstacle, however, which means that in its vicinity the human's motion is not well-explained by Q H , and b(b) ought to place more probability mass on higher values of b. The second rows of Figure  2  and Figure  3  illustrate this type of situation for the pedestrian and Dubins car, respectively. In Figure  2 , the pedestrian walks to an a priori known goal location and avoids an unmodeled spill on the ground. Analogously, in Figure  3  the car swerves to avoid a large pothole. By inferring model confidence online, our approach generates higher-variance predictions of future state, but only in the vicinity of these unmodeled obstacles. At other times throughout the episode when Q H is more accurate, our approach produces predictions more in line with the high model confidence predictor.
 Unmodeled goal In most realistic human-robot encounters, even if the robot does have an accurate environment map and observes all obstacles, it is unlikely for it to be aware of all human goals. We test our approach's resilience to unknown human goals by constructing a scenario in which the human moves between both known and unknown goals. The third row of Figure  2  illustrates this situation for the pedestrian example. Here, the pedestrian first moves to one known goal position, then to another, and finally back to the start which was not a modeled goal location. The first two legs of this trajectory are consistent with the robot's model of goal-oriented motion, though accurate prediction does require the predictor to infer which goal the pedestrian is walking toward. However, when the pedestrian returns to the start, her motion appears inconsistent with Q H , skewing the robot's belief over b toward zero. Similarly, in the third row of Figure  3  we consider a situation in which a car makes an unexpected turn onto an unmapped access road. As soon as the driver initiates the turn, our predictor rapidly learns to distrust its internal model Q H and shift its belief over b upward.
 Safe probabilistic planning and tracking Given probabilistic predictions of the human's future motion, the robot must plan efficient trajectories that avoid collision with high probability. In order to reason robustly about this probability of future collision, we must account for potential tracking errors incurred by the real system as it follows planned trajectories. To this end, we build on the recent FaSTrack framework of  Herbert et al. (2017) , which provides control-theoretic robust safety certificates in the presence of deterministic obstacles, and extend it to achieve approximate probabilistic collision-avoidance.
 Background: fast planning, safe tracking Recall that x R is the robot's state for the purposes of motion planning, and that s R encodes a higher-fidelity, potentially higher-dimensional notion of state (with associated dynamics). The recently proposed FaSTrack framework from  Herbert et al. (2017)  uses Hamilton-Jacobi reachability analysis to quantify the worst-case tracking performance of the s R -system as it follows trajectories generated by the x R -system. For further reading on reachability analysis refer to  Evans and Souganidis (1984) ,  Mitchell et al. (2005) , and  Bansal et al. (2017) . A byproduct of this FaSTrack analysis is an error feedback controller that the s R system can use to achieve this worst-case tracking error. The tracking error bound may be given to one of many off-the-shelf real-time motion planning algorithms operating in x R -space in order to guarantee real-time collision avoidance by the s Rsystem. Formally, FaSTrack precomputes an optimal tracking controller, as well as a corresponding compact set E in the robot's planning state space, such that (p(s R (t))À x R, ref (t)) 2 E for any reference trajectory proposed by the lower-fidelity planner. This bound E is a trajectory tracking certificate that can be passed to an online planning algorithm for real-time safety verification: the dynamical robot is guaranteed to always be somewhere within the bound relative to the current planned reference point x R, ref (t). This tracking error bound may sometimes be expressed analytically; otherwise, it may be computed numerically offline using level set methods (e.g.,  Mitchell, 2009) . Equipped with E, the planner can generate safe plans online by ensuring that the entire tracking error bound around the nominal state remains collision-free throughout the trajectory. Efficiently checking these E-augmented trajectories for collisions with known obstacles is critical for real-time performance. Note that the planner only needs to know E (which is computed offline) and otherwise requires no explicit understanding of the high-fidelity model. Running example: As dynamics (2) are decoupled in the three spatial directions, the bound E computed by FaSTrack is an axis-aligned box of dimensions E x × E y × E z . For further details refer to Fridovich-Keil et al. (  2018 ).
 Robust tracking, probabilistic safety Unfortunately, planning algorithms for collision checking against deterministic obstacles cannot be readily applied to our problem. Instead, a trajectory's collision check should return the probability that it might lead to a collision. Based on this probability, the planning algorithm can discriminate between trajectories that are sufficiently safe and those that are not. As discussed in Section 3.4, a safe online motion planner invoked at time t should continually check the probability that, at any future time t, (p(s R (t)), x H (t)) 2 K. The tracking error bound guarantee from FaSTrack allows us to conduct worst-case analysis on collisions given a human state x H . Concretely, if no point in the Minkowski sum fx R + Eg is in the collision set with x H , we can guarantee that the robot is not in collision with the human. The probability of a collision event for any point x R (t) along a candidate trajectory is then P coll (x R (t)) :¼ P((x R , x H ) 2 K) ð12Þ Assuming worst-case tracking error bound E, this quantity can be upper-bounded by the total probability that x H (t) will be in collision with any of the possible robot states xR 2 fx R (t) + Eg. For each robot planning state x R 2 R n R we define the set of human states in potential collision with the robot: H E (x R ) :¼ fx H 2 R n H : 9x R 2 fx R + Eg, (x R , xH ) 2 Kg ð13Þ Running example: Given K and E, H E (x R ) is the set of human positions within the rectangle of dimensions (l + E x ) × (l + E y ) centered on ½p x , p y . A human anywhere in this rectangle could be in collision with the quadcopter. The following result follows directly from the definition of the tracking error bound and a union bound. Proposition 1. The probability of a robot with worst-case tracking error E colliding with the human at any trajectory point x R (t) is bounded above by the probability mass of x H (t) contained within H E (x R (t)). We consider discrete-time motion plans. The probability of collision along any such trajectory from current time step k to final step k + K is upper-bounded by P k:k + K coll ł P k:k + K coll :¼ 1 À Y k + K k = k P(x k H 6 2 H E (x k R )jx k H 6 2 H E (x s R ), k ł s\k) ð14Þ Evaluating the right-hand side of (  14 ) exactly requires reasoning about the joint distribution of human states over all time steps and its conditional relationship on whether collision has yet occurred. This is equivalent to maintaining a probability distribution over the exponentially large space of trajectories x k:k + K H that the human might follow. As The International Journal of Robotics Research 39(2-3) motion planning occurs in real time, we shall resort to a heuristic approximation of (  14 ). One approach to approximating (  14 ) is to assume that the event x k 1 H 6 2 H E (x k 1 R ) is independent of x k 2 H 6 2 H E (x k 2 R ), for all k 1 6 ¼ k 2 . This independence assumption is equivalent to removing the conditioning in (  14 ). Unfortunately, this approximation is excessively pessimistic; if there is no collision at time step k, then collision is also unlikely at time step k + 1 because both human and robot trajectories are continuous. In fact, for sufficiently small time discretization Dt and nonzero collision probabilities at each time step, the total collision probability resulting from an independence assumption would approach 1 exponentially fast in the number of time steps K. We shall refine this approximation by finding a tight lower bound on the right-hand side of (  14 ). Because collision events are correlated in time, we first consider replacing each conditional probability P( x k H 6 2 H E (x k R )jx s H 6 2 H E (x s R ), k ł s\k) by 1 for all k 2 fk + 1, . . . , k + Kg. This effectively lower bounds P k:k + K coll by the worst-case probability of collision at the current time step k: P k:k + K coll ø 1 À P(x k H 6 2 H E (x k R )) = P(x k H 2 H E (x k R )) ð15Þ This bound is extremely loose in general, because it completely ignores the possibility of future collision. However, note that probabilities in the product in (  14 ) may be conditioned in any particular order (not necessarily chronological). This commutativity allows us to generate K À k + 1 lower bounds of the form P k:k + K coll ø P(x k H 2 H E (x k R )) for k 2 fk, . . . , k + Kg. Taking the tightest of all of these bounds, we can obtain an informative, yet quickly computable, approximator for the sought probability: P k:k + K coll ø max k2fk:k + Kg P(x k H 2 H E (x k R ))'P k:k + K coll ð16Þ To summarize, the left inequality in (  16 ) lower-bounds P k:k + K coll with the greatest marginal collision probability at any point in the trajectory. On the right-hand side of (  16 ), we take this greatest marginal collision probability as an approximator of the actual probability of collision over the entire trajectory. In effect, we shall approximate P k:k + K coll with a tight lower bound of an upper bound. While this type of approximation may err on the side of optimism, we note that both the robot's ability to replan over time and the fact that the left-hand side of (  16 ) is an upper bound on total trajectory collision probability mitigate this potentially underestimated risk.
 Safe online planning under uncertain human predictions This approximation of collision probability allows the robot to discriminate between valid and invalid candidate trajectories during motion planning. Using the prediction methodology proposed in Section 4, we may quickly generate, at every time t, the marginal probabilities in (  16 ) at each future time k 2 fk, . . . , k + Kg, based on past observations at times 0, . . . , k. The planner then computes the instantaneous probability of collision P(x k H 2 H E (x k R )) by integrating P(x t H jx 0:k H ) over H E (x k R ) , and rejects the candidate point x k R if this probability exceeds P th . Note that for graph-based planners that consider candidate trajectories by generating a graph of time-stamped states, rejecting a candidate edge from this graph is equivalent to rejecting all further trajectories that would contain that edge. This early rejection rule is consistent with the proposed approximation (16) of P k:k + K coll while preventing unnecessary exploration of candidate trajectories that would ultimately be deemed unsafe. Throughout operation, the robot follows each planned trajectory using the error feedback controller provided by FaSTrack, which ensures that the robot's high-fidelity state representation s R and the lower-fidelity state used for planning, x R , differ by no more than the tracking error bound E. This planning and tracking procedure continues until the robot reaches its desired goal state. Running example: Our quadcopter is now required to navigate to a target position shown in Figure  4  without colliding with the human. Our proposed algorithm successfully avoids collisions at all times, replanning to leave greater separation from the human whenever her motion departs from the model. In contrast, robot planning with fixed model confidence is either overly conservative at the expense of time and performance or overly aggressive at the expense of safety.
 Connections to reachability analysis In this section, we present an alternative, complementary analysis of the overall safety properties of the proposed approach to prediction and motion planning. This discussion is grounded in the language of reachability theory and worst-case analysis of human motion.
 Forward reachable set Throughout this section, we frequently refer to the human's time-indexed forward reachable set. We define this set formally in the following. Definition 1. (Forward reachable set) For a dynamical system _ x = f (x, u) with state trajectories given by the function j(x(0), t, u( Á ))ex(t), the forward reachable set FRS(x, t) of a state x after time t has elapsed is FRS(x, t) :¼ fx 0 : 9u( Á ), x 0 = j(x, t, u( Á ))g That is, a state x 0 is in the forward reachable set of x after time t if it is reachable via some applied control signal u( Á ). Remark 1. (Recovery of FRS) For P th = 0 and any finite b, the set of states assigned probability greater than P th is identical to the forward reachable set, up to discretization errors. This is visualized for low, high, and Bayesian model confidence in Figure  5 .
 A sufficient condition for the safety of individual trajectories In Section 6.2, we construct an approximation to the probability of collision along a trajectory, which we use during motion planning to avoid potentially dangerous states. To make this guarantee of collision avoidance for a motion plan even stronger, it would suffice to ensure that the robot never comes too close to the human's forward reachable set. More precisely, a planned trajectory is safe if fx R (t) + Eg \ FRS(x H , t) = ;, for every state x R (t) along a motion plan generated when the human was at state x H . The proof of this statement follows directly from the properties of the tracking error bound E described in Section 6. While this condition may seem appealing, it is in fact highly restrictive. The requirement of avoiding the full forward reachable set is not always possible in confined spaces; indeed, this was our original motivation for wanting to predict human motion (see Section 3.4). However, despite this shortcoming, the logic behind this sufficient condition for safety provides insight into the effectiveness of our framework.
 Recovering the forward reachable set Though it will not constitute a formal safety guarantee, we analyze the empirical safety properties of our approach by examining how our predicted state distributions over time relate to forward reachable sets. During operation, our belief over model confidence b evolves to match the degree to which the utility model Q H explains recent human motion. The ''time constant'' governing the speed of this evolution may be tuned by the system designer to be arbitrarily fast by choosing the parameter e to be small, as discussed in Section 4.1. Thus, we may safely assume that b(b) places high probability mass on small values of b as soon as the robot observes human motion that is not well explained by Q H . Figure  6  shows the sets of states with ''high enough'' (.P th ) predicted probability mass overlaid on the human's forward reachable set at time t, which is a circle of radius v H t centered on x H for the dynamics in our running example. When b is high (10), we observe that virtually all of the probability mass is concentrated in a small number of states in the direction of motion predicted by our utility model. When b is low (0:05) we observe that the set of states assigned probability above our collision threshold P th occupies a much larger fraction of the reachable set. A typical belief b(b) recorded at a moment when the human was roughly moving according to Q H yields an intermediate set of states. Figure  7  illustrates the evolution of these sets of states over time, for the unmodeled obstacle example of Section 5.4 in which a pedestrian avoids a spill. Each row corresponds to the predicted state distribution at a particular point in time. Within a row, each column shows the reachable set and the set of states assigned occupancy Interestingly, as the Bayesian model confidence decreases, which occurs when the pedestrian turns to avoid the spill at t'6 s, the predicted state distribution assigns high probability to a relatively large set of states, but unlike the low-b predictor that set of states is oriented toward the known goal. Of course, had b(b) placed even more probability mass on lower values of b then the Bayesian confidence predictor would converge to the low confidence one. In addition, we observe that, within each row as the prediction horizon increases, the area contained within the forward reachable set increases and the fraction of that area contained within the predicted sets decreases. This phenomenon is a direct consequence of our choice of threshold P th . Had we chosen a smaller threshold value, a larger fraction of the forward reachable set would have been occupied by the lower-b predictors. This observation may be viewed prescriptively. Recalling the sufficient condition for safety of planned trajectories from Section 7.2, if the robot replans every T replan seconds, we may interpret the fraction of FRS( Á , t + T replan ) assigned occupancy probability greater than P th by the low-confidence predictor as a rough indicator of the safety of an individual motion plan, robust to worst-case human movement. As this fraction tends toward unity, the robot is more and more likely to be safe. However, for any P th .0, this fraction approaches zero for T replan " '. This immediately suggests that, if we wish to replan every T replan seconds, we can achieve a particular level of safety as measured by this fraction by choosing an appropriate threshold P th . In summary, confidence-aware predictions rapidly place high-probability mass on low values of b whenever human motion is not well-explained by utility model Q H . Whenever this happens, the resulting predictions encompass a larger fraction of the forward reachable set, and in the limit that P th # 0 we recover the forward reachable set exactly. The larger this fraction, the more closely our approach satisfies the sufficient condition for safety presented in Section 7.2.
 Hardware demonstration We implemented confidence-aware human motion prediction (Section 4) and integrated it into a real-time, safe probabilistic motion planner (Section 6), all within the Robot Operating System (ROS) software framework of  Quigley et al. (2009) . To demonstrate the efficacy of our methods, we tested our work for the quadcopter-avoiding-pedestrian example used for illustration throughout this paper. Human trajectories were recorded as (x, y) positions on the ground plane at roughly 235 Hz by an OptiTrack infrared motion capture system, and we used a Crazyflie 2.0 micro-quadcopter, also tracked by the OptiTrack system. 4 Figure  4  illustrates the unmodeled obstacle case from Section 5.4, in which the pedestrian turns to avoid a spill on the ground. Using a low model confidence results in motion plans that suddenly and excessively deviate from the ideal straight-line path when the pedestrian turns to avoid the spill. By contrast, the high-confidence predictor consistently predicts that the pedestrian will walk in a straight line to the goal even when they turn; this almost leads to collision, as shown in detail in Figure  8 . Our proposed approach for Bayesian model confidence initially assigns high confidence and predicts that the pedestrian will walk straight to the goal, but when they turn to avoid the spill, the predictions become less confident. This causes the quadcopter to make a minor course correction, shown in further detail in Figure  9 .
 Conclusion When robots operate in complex environments in concert with other agents, safety often depends upon the robot's ability to predict the agents' future actions. While this prediction problem may be tractable in some cases, it can be extremely difficult for agents such as people who act with intent. In this paper, we introduce the idea of confidenceaware prediction as a natural coping mechanism for predicting the future actions of intent-driven agents. Our approach uses each measurement of the human's state to reason about the accuracy of its internal model of human decision-making. This reasoning about model confidence is expressed compactly as a Bayesian filter over the possible values of a single parameter, b, which controls the entropy of the robot's model of the human's choice of action. In effect, whenever the human's motion is not wellexplained by this model, the robot predicts that the human could occupy a larger volume of the state space.  We couple this notion of confidence-aware prediction with a reachability-based robust motion planning algorithm, FaSTrack, which quantifies the robot's ability to track a planned reference trajectory. Using this maximum tracking error allows us to bound an approximation of the probability of collision along planned trajectories. In addition, we present a deeper connection between confidence-aware prediction and forward reachable sets, which provides an alternative explanation of the safety of our approach. We demonstrate the proposed methodology on a ROS-based quadcopter testbed in a motion capture arena.
 Limitations There are several important limitations of this work, which we summarize and discuss in the following. 9.1.1. State discretization. As presented, our approach to prediction requires a discrete representation of the human's state space. This can be tractable for the relatively simple dynamical models of human motion we consider in this work. Fortunately, one of the strongest attributes of confidenceaware prediction is that it affords a certain degree of robustness to modeling errors by design. Still, our approach is effectively limited to low-order dynamical models. 9.1.2. FaSTrack complexity. FaSTrack provides a strong safety guarantee vis-a `-vis the maximum tracking error that could ever exist between a higher-fidelity dynamical model of the robot and a lower-order model used for motion planning. Unfortunately, the computational complexity of finding this maximum tracking error and the corresponding safety controller scales exponentially with the dimension of The subsequent motion plans may not be safe; here, poor prediction quality leads to a collision. the high-fidelity model. In some cases, these dynamics are decomposable and analytic solutions exist (e.g.,  Chen et al., 2018; Fridovich-Keil et al., 2018) , and in other cases conservative approximations may be effective (e.g.,  Chen et al., 2016; Royo et al., 2018) . 9.1.3. Boltzmann distributional assumption. We model the human's choice of control input at each time step as an independent, random draw from a Boltzmann distribution (3). This distributional assumption is motivated from the literature in cognitive science and econometrics and is increasingly common in robotics, yet it may not be accurate in all cases. Maintaining an up-to-date model confidence belief b(b) can certainly mitigate this inaccuracy, but only at the cost of making excessively conservative predictions. 9.1.4. Safety certification. Our analysis in Section 7 makes connections to forward reachability in an effort to understand the safety properties of our system. As shown, whenever our confidence-aware prediction method detects poor model performance it quickly yields predictions that approximate the human's forward reachable set. Although this approximation is not perfect, and hence we cannot provide a strong safety certificate, the connection to reachability is in some sense prescriptive. That is, it can be used to guide the choice of collision probability threshold P th and replanning frequency. However, even if we could provide a strong guarantee of collision avoidance for a particular motion plan, that would not, in general, guarantee that future motion plans would be recursively safe. This recursive property is much more general and, unsurprisingly, more difficult to satisfy.
 Future directions Future work will aim to address each of these shortcomings. We are also interested in extending our methodology for the multi-robot, multi-human setting; our preliminary results are reported by  Bajcsy et al. (2018) . In addition, we believe that our model confidence inference approach could be integrated with other commonly used probabilistic prediction methods besides the Boltzmann utility model. Finally, we are excited to test our work in hardware in other application spaces, such as manipulation and driving. Fig. Fig.1. When planning around humans, accurate predictions of human motion (visualized here in pink and blue, representing high and low probability, respectively) are an essential prerequisite for safety. Unfortunately, these approaches may fail to explain all observed motion at runtime (e.g., human avoids unmodeled spill on the ground), leading to inaccurate predictions, and potentially, collisions (left). Our method addresses this by updating its predictive model confidence in real time (right), leading to more conservative motion planning in circumstances when predictions are known to be suspect.
 with b k + (b, u) :¼ P(b, ujx 0:k H , u 0:k H ) the running posterior and b k À (b, u) :¼ P(b, ujx 0:kÀ1 H , u 0:kÀ1 H
 Fig. 2 . 2 Fig. 2. Snapshots of pedestrian trajectory and probabilistic model predictions. Top row: Pedestrian moves from the bottom right to a goal marked as a red circle. Middle row: Pedestrian changes course to avoid a spill on the floor. Bottom row: Pedestrian moves to one known goal, then to another, then to a third which the robot has not modeled. The first two columns show predictions for low and high model confidence; the third column shows the predictions using our Bayesian model confidence. For all pedestrian videos, see https://youtu.be/lh_E9rW-MJo.
 Fig. 3 . 3 Fig. 3. Snapshots of Dubins car and probabilistic predictions. Top row: Car moves straight ahead toward one of two known goals (red arrows), staying in its lane. Middle row: Car suddenly swerves to the left to avoid a pothole. Bottom row: Car turns to the right, away from the only known goal. The left and center columns show results for low and high confidence predictors, respectively, and the right column shows our approach using Bayesian inferred model confidence. For all Dubins car videos, see https://youtu.be/sAJKNnP42fQ.
 Fig. 4 . 4 Fig. 4. Scenario from the middle row of Figure 2 visualized with robot's trajectory. When b is low and the robot is not confident, it makes large deviations from its path to accommodate the human. When b is high, the robot refuses to change course and comes dangerously close to the human. With inferred model confidence, the robot balances safety and efficiency with a slight deviation around the human.
 Fig. 5 . 5 Fig. 5. The human (black dot) is moving west towards a goal. Visualized are the predicted state distributions for 1 second into the future when using low, high, and Bayesian model confidence. Higher-saturation indicates higher likelihood of occupancy. The dashed circle represents the pedestrian's 1 second forward reachable set.
 Fig. 6 . 6 Fig.6. Visualization of the states with probability greater than or equal to the collision threshold, P th = 0:01. The human's forward reachable set includes the set of states assigned probability greater than P th . We show these ''high probability'' predicted states for predictors with fixed low and high b, as well as our Bayesian-inferred b.
 Fig. 7 . 7 Fig.7. The human (black dot) is walking towards the known goal (red dot) but has to avoid an unmodeled coffee spill on the ground. Here we show the snapshots of the predictions at various future times (columns) as the human walks around in real time (rows). The visualized states have probability greater than or equal to P th = 0:01. Each panel displays the human prediction under low confidence (in yellow), high confidence (in dark purple), and Bayesian confidence (colored as per the most likely b value), as well as the forward reachable set. The human's actual trajectory is shown in red.
 Fig. 8 . 8 Fig. 8. Predicting with fixed-b (in this case, b = 20) can yield highly inaccurate predictions (and worse, confidently inaccurate ones).The subsequent motion plans may not be safe; here, poor prediction quality leads to a collision.
 Fig. 9 . 9 Fig. 9. Inferring b leads to predicted state distributions whose entropy increases whenever the utility model Q H fails to explain observed human motion. The resulting predictions are more robust to modeling errors, resulting in safer motion plans. Here, the quadcopter successfully avoids the pedestrian even when they turn unexpectedly.
			 The International Journal of Robotics Research 39(2-3)

David Fridovich-Keil
Andrea Bajcsy
Jaime F Fisac
Sylvia L Herbert
Steven Wang
Anca D Dragan
Claire J Tomlin


Doing more with less: meta-reasoning and meta-learning in humans and machines[☆](#aep-article-footnote-id6)
===========================================================================================================

Artificial intelligence systems use an increasing amount of computation and data to solve very specific problems. By contrast, human minds solve a wide range of problems using a fixed amount of computation and limited experience. We identify two abilities that we see as crucial to this kind of general intelligence: meta-reasoning (deciding how to allocate computational resources) and meta-learning (modeling the learning environment to make better use of limited data). We summarize the relevant AI literature and relate the resulting ideas to recent work in psychology.

Section snippets
----------------

Meta-reasoning
--------------

The term ‘meta-reasoning’ contains within it the solution to the problem of how to efficiently deploy computational resources: meta-reasoning is reasoning about reasoning, which means making intelligent decisions about how to think [5]. In research on artificial intelligence, meta-reasoning plays a central role in the definition and design of rational agents that can operate on performance-limited hardware and interact with their environment in real-time [2,6, 7, 8,4]. Considering this problem

Meta-learning
-------------

While meta-reasoning focuses on the efficient use of cognitive resources, a related challenge is the efficient use of data. One of the most striking aspects of human learning is the ability to learn new words or concepts from limited numbers of examples [33]. This ability contrasts strongly with traditional machine learning methods, which typically require many labeled examples in order to even begin to classify stimuli accurately. Recent work in machine learning has aimed to narrow this gap,

Towards general intelligence
----------------------------

Meta-reasoning and meta-learning individually capture key components of human intelligence, but it may be by combining them that we come closest to identifying what is necessary to achieve the kind of general intelligence that currently eludes AI systems. While meta-learning has begun to explore problems that involve learning to perform many different tasks in parallel, those tasks remain relatively similar to one another. By contrast, what characterizes human environments is the need to

Conflict of interest statement
------------------------------

Nothing declared.

References and recommended reading
----------------------------------

Papers of particular interest, published within the period of review, have been highlighted as:* • of special interest
* •• of outstanding interest

References (50)
---------------

* S.J. Russell *et al.*### [Principles of metareasoning](/science/article/pii/000437029190015C)

### Artif Intell

(1991)
* S.J. Russell### [Rationality and intelligence](/science/article/pii/S000437029700026X)

### Artif Intell

(1997)
* E.J. Horvitz *et al.*### [Computational tradeoffs under bounded resources](/science/article/pii/S0004370201000510)

### Artif Intell

(2001)
* G. Gigerenzer *et al.*### [Heuristic decision making](/science/article/pii/S1364661315002041)

### Annu Rev Psychol

(2011)
* D. Amodei *et al.*### AI and Compute

(2018)
* S.J. Russell *et al.*### Do the Right Thing: Studies in Limited Rationality

(1991)
* T.L. Griffiths *et al.*### Rational use of cognitive resources: levels of analysis between the computational and the algorithmic

### Top Cognit Sci

(2015)
* S.J. Gershman *et al.*### Computational rationality: a converging paradigm for intelligence in brains, minds, and machines

### Science

(2015)
* M.T. Cox *et al.*### Metareasoning: Thinking about Thinking

(2011)
* S.J. Russell *et al.*### Provably bounded-optimal agents

### J Artif Intell Res

(1994)

- N. Hay *et al.*### Selecting computations: theory and applications
- F. Lieder *et al.*### Algorithm selection by rational metareasoning as a model of human strategy selection

### Advances in Neural Information Processing Systems

(2014)
- S. Milli *et al.*### When does bounded-optimal metareasoning favor few cognitive systems?
- F. Callaway *et al.*### Learning to select computations

### Uncertainty in Artificial Intelligence: Proceedings of the Thirty-Fourth Conference

(2018)
- C.H. Lin *et al.*### Metareasoning for Planning Under Uncertainty

### Proceedings of the 24th International Conference on Artificial Intelligence

(2015)
- F. Lieder *et al.*### An automatic method for discovering rational heuristics for risky choice

### Proceedings of the 39th Annual Meeting of the Cognitive Science Society (CogSci)

(2017)
- E. Vul *et al.*### One and done? Optimal decisions from very few samples

### Cognit Sci

(2014)
- S.J. Russell### Rationality and intelligence: a brief update
- L. Kocsis *et al.*### Bandit based Monte-Carlo planning

### European Conference on Machine Learning

(2006)
- F. Lieder *et al.*### Burn-in, bias, and the rationality of anchoring

### Advances in Neural Information Processing Systems

(2012)
- J. Dunlosky *et al.*### Metacognition

(2008)
- M. Oaksford *et al.*### A rational analysis of the selection task as optimal data selection

### Psychol Rev

(1994)
- T.M. Gureckis *et al.*### Self-directed learning: a cognitive and computational perspective

### Perspect Psychol Sci

(2012)
- J.W. Payne *et al.*### The Adaptive Decision Maker

(1993)
- Y.-L. Boureau *et al.*### Deciding how to decide: self-control and meta-decision making

### Trends Cognit Sci

(2015)
View more referencesCited by (74)
-------------

* ### [Experience-driven recalibration of learning from surprising events](/science/article/pii/S0010027722003328)

2023, CognitionCitation Excerpt :This conclusion is congruent with findings that statistical experience can shape learning and decision processes in a variety of domains (Constantino & Daw, 2015; McGuire & Kable, 2012; Ossmy et al., 2013; Schweighofer et al., 2006). Our findings support a class of models in which patterns of belief updating are guided by higher-order beliefs about environmental structure (Razmi & Nassar, 2022; Yu et al., 2021) and highlight the need to extend such models to incorporate experiential structure-learning or metalearning processes (Griffiths et al., 2019; Wang, 2021). Findings from the present work will provide useful constraints for further model development.

Show abstractDifferent environments favor different patterns of adaptive learning. A surprising event that in one context would accelerate belief updating might, in another context, be downweighted as a meaningless outlier. Here, we investigated whether people would spontaneously regulate the influence of surprise on learning in response to event-by-event experiential feedback. Across two experiments, we examined whether participants performing a perceptual judgment task under spatial uncertainty (*n* = 29, *n* = 63) adapted their patterns of predictive gaze according to the informativeness or uninformativeness of surprising events in their current environment. Uninstructed predictive eye movements exhibited a form of *metalearning* in which surprise came to modulate event-by-event learning rates in opposite directions across contexts. Participants later appropriately readjusted their patterns of adaptive learning when the statistics of the environment underwent an unsignaled reversal. Although significant adjustments occurred in both directions, performance was consistently superior in environments in which surprising events reflected meaningful change, potentially reflecting a bias towards interpreting surprise as informative and/or difficulty ignoring salient outliers. Our results provide evidence for spontaneous, context-appropriate recalibration of the role of surprise in adaptive learning.
* ### [Learning From Major Accidents: A Meta-Learning Perspective](/science/article/pii/S092575352200323X)

2023, Safety ScienceShow abstractLearning from the past is essential to improve safety and reliability in the chemical industry. In the context of Industry 4.0 and Industry 5.0, where Artificial Intelligence and IoT are expanding throughout every industrial sector, it is essential to determine if an artificial learner may exploit historical accident data to support a more efficient and sustainable learning framework. One important limitation of Machine Learning algorithms is their difficulty in generalizing over multiple tasks. In this context, the present study aims to investigate the issue of meta-learning and transfer learning, evaluating whether the knowledge extracted from a generic accident database could be used to predict the consequence of new, technology-specific accidents. To this end, a classification algorithm is trained on a large and generic accident database to learn the relationship between accident features and consequence severity from a diverse pool of examples. Later, the acquired knowledge is transferred to another domain to predict the number of fatalities and injuries in new accidents. The methodology is evaluated on a test case, where two classification algorithms are trained on a generic accident database (i.e., the Major Hazard Incident Data Service) and evaluated on a technology-specific, lower-quality database. The results suggest that automated algorithms can learn from historical data and transfer knowledge to predict the severity of different types of accidents. The findings indicate that the knowledge gained from previous tasks might be used to address new tasks. Therefore, the proposed approach reduces the need for new data and the cost of the analyses.
* ### [A meta-learning approach to improving radiation response prediction in cancers](/science/article/pii/S001048252200871X)

2022, Computers in Biology and MedicineCitation Excerpt :Many statistical methods have been proposed to solve this dilemma, “the curse of dimensionality”, including taking prior knowledge into account and borrowing information from other similar diseases [25–27]. Meta-learning, which has emerged in recent years, is a framework that relies on previously learned knowledge from similar tasks and quickly adapt to a new task with a small amount of training data [28,29]. Unlike traditional machine learning methods, the learning algorithm of which are usually predefined and handcrafted, meta-learning performs the optimization activity autonomously and explores the diversity of multiple learning algorithms [30].

Show abstractPredicting the efficacy of radiotherapy in individual patients has drawn widespread attention, but the limited sample size remains a bottleneck for utilizing high-dimensional multi-omics data to guide personalized radiotherapy. We hypothesize the recently developed meta-learning framework could address this limitation.

By combining gene expression, DNA methylation, and clinical data of 806 patients who had received radiotherapy from The Cancer Genome Atlas (TCGA), we applied the Model-Agnostic Meta-Learning (MAML) framework to tasks consisting of pan-cancer data, to obtain the best initial parameters of a neural network for a specific cancer with smaller number of samples. The performance of meta-learning framework was compared with four traditional machine learning methods based on two training schemes, and tested on Cancer Cell Line Encyclopedia (CCLE) and Chinese Glioma Genome Atlas (CGGA) datasets. Moreover, biological significance of the models was investigated by survival analysis and feature interpretation.

The mean AUC (Area under the ROC Curve) [95% confidence interval] of our models across nine cancer types was 0.702 [0.691–0.713], which improved by 0.166 on average over other the four machine learning methods on two training schemes. Our models performed significantly better (*p* < 0.05) in seven cancer types and performed comparable to the other predictors in the rest of two cancer types. The more pan-cancer samples were used to transfer meta-knowledge, the greater the performance improved (*p* < 0.05). The predicted response scores that our models generated were negatively correlated with cell radiosensitivity index in four cancer types (*p* < 0.05), while not statistically significant in the other three cancer types. Moreover, the predicted response scores were shown to be prognostic factors in seven cancer types and eight potential radiosensitivity-related genes were identified.

For the first time, we established the meta-learning approach to improving individual radiation response prediction by transferring common knowledge from pan-cancer data with MAML framework. The results demonstrated the superiority, generalizability, and biological significance of our approach.
* ### [Thermodynamic fluctuation theorems govern human sensorimotor learning](https://doi.org/10.1038/s41598-023-27736-8)

2023, Scientific Reports
* ### [Shaped-Charge Learning Architecture for the Human–Machine Teams](https://doi.org/10.3390/e25060924)

2023, Entropy
* ### [A computational process-tracing method for measuring people’s planning strategies and how they change over time](https://doi.org/10.3758/s13428-022-01789-5)

2023, Behavior Research Methods

[View all citing articles on Scopus](http://www.scopus.com/scopus/inward/citedby.url?partnerID=10&rel=3.0.0&eid=2-s2.0-85062041121&md5=eaa134c3594586f3b3e7bdb1f7bdc1)Recommended articles (6)
------------------------

* Research article### [MicroRNAs change the games in central nervous system pharmacology](/science/article/pii/S0006295219302473)

Biochemical Pharmacology, Volume 168, 2019, pp. 162-172Show abstractMicroRNAs (miRNAs) represent a class of important post-transcriptional regulators of gene expression, enabling cells to follow their intrinsic developmental program. By directly binding to their targets, miRNAs can both promote transcriptional patterns in crucial steps of cell growth, and act as powerful buffering system that titrate protein content in case of aberrant gene expression. The literature of the last decade showed that the presence of tissue-enriched miRNAs in body fluids could be reminiscent of disease state. This is particularly relevant in neurodegenerative disorders, in which peripheral biomarkers could be helpful means to detect disease onset. However, dysregulation of miRNAs is not merely a consequence of disease, but directly contributes to pathological outcomes. On this basis, increasing interest is growing in the development of pharmacological agents targeting specific miRNAs. Actually, this apparently futuristic approach is already part of the current therapies. In fact, several drugs approved for CNS disorders, such as L-Dopa or valproic acid, were also demonstrated to restore some miRNAs. Moreover, ongoing clinical trials demonstrated that miRNA-based drugs are effective against tumors, suggesting that miRNAs also represent a promising class of therapeutic molecules. However, several issues still need to be addressed, particularly in case of CNS diseases, in which stability and delivery are crucial aspects of the therapy. In this commentary, we highlighted potential advantages and limitations of miRNAs as next generation targets in CNS pharmacology, focusing on multiple sclerosis, a chronic demyelinating disease lacking specific therapeutic targets and bona-fide biomarkers.
* Research article### [Distributed semantic representations for modeling human judgment](/science/article/pii/S2352154618301712)

Current Opinion in Behavioral Sciences, Volume 29, 2019, pp. 31-36Show abstractPeople make judgments about thousands of different objects and concepts on a day-to-day basis; however, capturing the knowledge that subserves these judgments has been difficult. Recent advances in computational linguistics are filling this gap, as the statistics of language use yield rich, distributed semantic representations for natural objects and concepts. These representations have been shown to predict semantic and linguistic judgments, such as judgments of meaning and relatedness, and more recently, high-level judgments, including probability judgment and forecasting, stereotyping and various types of social judgment, consumer choice, and perceptions of risk. Distributed semantic representations are now a key component of computational models that represent knowledge, make evaluations and attributions, and give responses, in a human-like manner.
* Research article### [Linking neural responses to behavior with information-preserving population vectors](/science/article/pii/S2352154618302092)

Current Opinion in Behavioral Sciences, Volume 29, 2019, pp. 37-44Show abstractAll systems for processing signals, both artificial and within animals, must obey fundamental statistical laws for how information can be processed. We discuss here recent results using information theory that provide a blueprint for building circuits where signals can be read-out without information loss. Many properties that are necessary to build information-preserving circuits are actually observed in real neurons, at least approximately. One such property is the use of logistic nonlinearity for relating inputs to neural response probability. Such nonlinearities are common in neural and intracellular networks. With this nonlinearity type, there is a linear combination of neural responses that is guaranteed to preserve Shannon information contained in the response of a neural population, no matter how many neurons it contains. This read-out measure is related to a classic quantity known as the population vector that has been quite successful in relating neural responses to animal behavior in a wide variety of cases. Nevertheless, the population vector did not withstand the scrutiny of detailed information-theoretical analyses that showed that it discards substantial amounts of information contained in the responses of a neural population. We discuss recent theoretical results showing how to modify the population vector expression to make it ‘information-preserving’, and what is necessary in terms of neural circuit organization to allow for lossless information transfer. Implementing these strategies within artificial systems is likely to increase their efficiency, especially for brain–machine interfaces.
* Research article### [Planning at decision time and in the background during spatial navigation](/science/article/pii/S2352154618301682)

Current Opinion in Behavioral Sciences, Volume 29, 2019, pp. 69-76Show abstractPlanning is the model-based approach to solving control problems. The hallmark of planning is the endogenous generation of dynamical representations of future states, like goal locations, or state sequences, like trajectories to the goal location, using an internal model of the task. We review recent evidence of model-based planning processes and the representation of future goal states in the brain of rodents and humans engaged in spatial navigation tasks. We highlight two distinct but complementary usages of planning as identified in artificial intelligence: ‘at decision time’, to support goal-directed choices and sequential memory encoding, and ‘in the background’, to learn behavioral policies and to optimize internal models. We discuss how two kinds of internally generated sequences in the hippocampus – theta and SWR sequences – might participate in the neuronal implementation of these two planning modes, thus supporting a flexible model-based system for adaptive cognition and action.
* Research article### [Analogues of mental simulation and imagination in deep learning](/science/article/pii/S2352154618301670)

Current Opinion in Behavioral Sciences, Volume 29, 2019, pp. 8-16Show abstractMental simulation — the capacity to imagine what will or what could be — is a salient feature of human cognition, playing a key role in a wide range of cognitive abilities. In artificial intelligence, the last few years have seen the development of methods which are analogous to mental models and mental simulation. This paper outlines recent methods in deep learning for constructing such models from data and learning to use them via reinforcement learning, and compares such approaches to human mental simulation. Model-based methods in deep learning can serve as powerful tools for building and scaling cognitive models. However, a number of challenges remain in matching the capacity of human mental simulation for efficiency, compositionality, generalization, and creativity.
* Research article### [The value of abstraction](/science/article/pii/S2352154619300026)

Current Opinion in Behavioral Sciences, Volume 29, 2019, pp. 111-116
[☆](#baep-article-footnote-id6)This research was supported by contract number FA9550-18-1-0077 from the Air Force Office of Scientific Research and by contract number FA8650-18-2-7832 from the Defence Advanced Research Projects Agency.

[View full text](/science/article/pii/S2352154618302122)

Thomas L Griffiths
Frederick Callaway
Michael B Chang
Erin Grant
Paul M Krueger
Falk Lieder


China’s New AI Governance Initiatives Shouldn’t Be Ignored

Over the past six months, the Chinese government has rolled out a series of policy documents and public pronouncements that are finally putting meat on the bone of the country’s governance regime for artificial intelligence (AI). Given China’s track record of leveraging AI for mass surveillance, it’s tempting to view these initiatives as little more than a fig leaf to cover widespread abuses of human rights. But that response risks ignoring regulatory changes with major implications for global AI development and national security. Anyone who wants to compete against, cooperate with, or simply understand China’s AI ecosystem must examine these moves closely.

These recent initiatives show the emergence of three different approaches to AI governance, each championed by a different branch of the Chinese bureaucracy, and each at a different level of maturity. Their backers also pack very different bureaucratic punches. It’s worth examining the three approaches and their backers, along with how they will both complement and compete with each other, to better understand where China’s AI governance is heading.

| **Three Approaches to Chinese AI Governance** |
| **Organization** | **Focus of Approach** | **Relevant Documents** |
| Cyberspace Administration of China | Rules for online algorithms, with a focus on public opinion | - [Internet Information Service Algorithmic Recommendation Management Provisions](https://digichina.stanford.edu/work/translation-internet-information-service-algorithmic-recommendation-management-provisions-opinon-seeking-draft/)
 - [Guiding Opinions on Strengthening Overall Governance of Internet Information Service Algorithms](https://digichina.stanford.edu/work/translation-guiding-opinions-on-strengthening-overall-governance-of-internet-information-service-algorithms/) |
| China Academy of Information and Communications Technology | Tools for testing and certification of “trustworthy AI” systems | - [Trustworthy AI white paper](https://cset.georgetown.edu/publication/white-paper-on-trustworthy-artificial-intelligence/)
 - [Trustworthy Facial Recognition Applications and Protections Plan](https://www.sohu.com/a/501708742_100207327) |
| Ministry of Science and Technology | Establishing AI ethics principles and creating tech ethics review boards within companies and research institutions | - [Guiding Opinions on Strengthening Ethical Governance of Science and Technology](http://www.most.gov.cn/tztg/202107/t20210728_176136.html)
 - [Ethical Norms for New Generation Artificial Intelligence](https://cset.georgetown.edu/publication/ethical-norms-for-new-generation-artificial-intelligence-released/) |

The strongest and most immediately influential moves in AI governance have been made by the Cyberspace Administration of China (CAC), a relatively new but [very powerful](https://qz.com/2039292/how-did-chinas-top-internet-regulator-become-so-powerful/) regulator that writes the rules governing certain applications of AI. The CAC’s approach is the most mature, the most rule-based, and the most concerned with AI’s role in disseminating information.

The CAC made headlines in August 2021 when it released a [draft set of thirty rules](https://digichina.stanford.edu/work/translation-internet-information-service-algorithmic-recommendation-management-provisions-opinon-seeking-draft/) for regulating internet recommendation algorithms, the software powering everything from TikTok to news apps and search engines. Some of those rules are China-specific, such as the one stipulating that recommendation algorithms “vigorously disseminate positive energy.” But other provisions [break ground](https://digichina.stanford.edu/work/experts-examine-chinas-pioneering-draft-algorithm-regulations/) in ongoing international debates, such as the requirement that algorithm providers be able to “[give an explanation](https://digichina.stanford.edu/work/translation-internet-information-service-algorithmic-recommendation-management-provisions-opinon-seeking-draft/)” and “remedy” situations in which algorithms have infringed on user rights and interests. If put into practice, these types of provisions could spur Chinese companies to experiment with new kinds of disclosure and methods for algorithmic interpretability, an [emerging but very immature](https://cset.georgetown.edu/publication/key-concepts-in-ai-safety-interpretability-in-machine-learning/) area of machine learning research.

Soon after releasing its recommendation algorithm rules, the CAC came out with a much more ambitious effort: a [three-year road map](http://www.cac.gov.cn/2021-09/29/c_1634507915623047.htm) for governing all internet algorithms. Following through on that road map will require input from many of the nine regulators that co-signed the project, including the Ministry of Industry and Information Technology (MIIT).

The second approach to AI governance has emerged out of the China Academy of Information and Communications Technology (CAICT), an [influential think tank under the MIIT](https://www.newamerica.org/cybersecurity-initiative/digichina/blog/profile-china-academy-information-and-communications-technology-caict/). Active in policy formulation and many aspects of technology testing and certification, the CAICT has distinguished its method through a focus on creating the tools for measuring and testing AI systems. This work remains in its infancy, both from technical and regulatory perspectives. But if successful, it could lay the foundations for China’s larger AI governance regime, ensuring that deployed systems are robust, reliable, and controllable.

In July 2021, the CAICT teamed up with a research lab at the Chinese e-commerce giant JD to release the country’s first [white paper](https://cset.georgetown.edu/publication/white-paper-on-trustworthy-artificial-intelligence/) on “trustworthy AI.” Already popular in European and U.S. discussions, trustworthy AI refers to many of the more technical aspects of AI governance, such as testing systems for robustness, bias, and explainability. The way the CAICT defines trustworthy AI in its core principles [looks very similar](https://macropolo.org/beijing-approach-trustworthy-ai/?rp=m) to the definitions that have come out of U.S. and European institutions, but the paper was notable for how quickly those principles are being converted into concrete action.

The CAICT [is working with](http://aiiaorg.cn/index.php?m=content&c=index&a=show&catid=34&id=115) China’s AI Industry Alliance, a government-sponsored industry body, to test and certify different kinds of AI systems. In November 2021, it issued its first batch of [trustworthy AI certifications](https://www.sohu.com/a/501708742_100207327) for facial recognition systems. Depending on the technical rigor of implementation, these types of certifications could help accelerate progress on algorithmic interpretability—or they could simply turn into a form of bureaucratic rent seeking. On policy impact, the CAICT is often viewed as representing the views of the powerful MIIT, but the MIIT’s leadership has yet to issue its own policy documents on trustworthy AI. Whether it does will be a strong indicator of the bureaucratic momentum behind this approach.

Finally, the Ministry of Science and Technology (MOST) has taken the lightest of the three approaches to AI governance. Its highest-profile publications have focused on laying down ethical guidelines, relying on companies and researchers to supervise themselves in applying those principles to their work.

In July 2021, MOST [published guidelines](http://www.most.gov.cn/tztg/202107/t20210728_176136.html) that called for universities, labs, and companies to set up internal review committees to oversee and adjudicate technology ethics issues. Two months later, the main AI expert committee operating under MOST released its own [set of ethical norms for AI](https://cset.georgetown.edu/publication/ethical-norms-for-new-generation-artificial-intelligence-released/), with a special focus on weaving ethics into the entire life cycle of development. Since then, MOST has been encouraging leading tech companies to establish their own ethics review committees and audit their own products.

MOST’s approach is similar to those of international organizations such as the [United Nations Educational, Scientific and Cultural Organization](https://www.computerweekly.com/news/252510287/Unesco-member-states-adopt-AI-ethics-recommendation) and the [Organisation for Economic Co-operation and Development](https://oecd.ai/en/ai-principles), which have released AI principles and encouraged countries and companies to adopt them. But in the Chinese context, that tactic feels quite out of step with the country’s increasingly hands-on approach to technology governance, a disconnect that could undermine the impact of MOST’s efforts.

One unanswered question is how these three approaches will fit together. Chinese ministries and administrative bodies are [notoriously competitive](https://sgp.fas.org/crs/row/R41007.pdf) with one another, constantly jostling to get their pet initiatives in front of the country’s central leadership in hopes that they become the chosen policies of the party-state. In this contest, the CAC’s approach appears to have the clear upper hand: It is the most mature, the most in tune with the regulatory zeitgeist, and it comes from the organization with the most bureaucratic heft. But its approach can’t succeed entirely on its own. The CAC requires that companies be able to explain how their recommendation algorithms function, and the tools or certifications for what constitutes explainable AI are likely to come from the CAICT. In addition, given the sprawling and rapidly evolving nature of the technology, many practical aspects of trustworthy AI will first surface in the MOST-inspired ethics committees of individual companies.

The three-year road map for algorithmic governance offers a glimpse of some bureaucratic collaboration. Though the CAC is clearly the lead author, the document includes new references to algorithms being trustworthy and to companies setting up ethics review committees, additions likely made at the behest of the other two ministries. There may also be substantial shifts in bureaucratic power as AI governance expands to cover many industrial and social applications of AI. The CAC is traditionally an internet-focused regulator, and future regulations for autonomous vehicles or medical AI may create an opening for a ministry like the MIIT to seize the regulatory reins.

The potential impact of these regulatory currents extends far beyond China. If the CAC follows through on certain requirements for algorithmic transparency and explainability, China will be running some of the world’s largest regulatory experiments on topics that European regulators have [long debated](https://www.law.ox.ac.uk/business-law-blog/blog/2018/05/rethinking-explainable-machines-next-chapter-gdprs-right-explanation). Whether Chinese companies are able to [meet these new demands](https://digichina.stanford.edu/work/experts-examine-chinas-pioneering-draft-algorithm-regulations/) could inform analogous debates in Europe over the right to explanation.

On the security side, as AI systems are woven deeper into the fabrics of militaries around the world, governments want to ensure those systems are robust, reliable, and controllable for the sake of international stability. The CAICT’s current experiments in certifying AI systems are likely not game-ready for those kinds of high-stakes deployment decisions. But developing an early understanding of how Chinese institutions and technologists approach these questions could prove valuable for governments who may soon find themselves negotiating over aspects of autonomous weapons and arms controls.

With 2022 marking [a major year](https://thediplomat.com/2020/12/china-looks-ahead-to-20th-party-congress-in-2022/) in the Chinese political calendar, the people and bureaucracies building out Chinese AI governance are likely to continue jostling for position and influence. The results of that jostling warrant close attention from AI experts and China watchers. If China’s attempts to rein in algorithms prove successful, they could imbue these approaches with a kind of technological and regulatory soft power that shapes AI governance regimes around the globe.