Key Concepts in AI Safety: An Overview

MARCH 2021
Key Concepts in AI Safety: An Overview
CSET Issue Brief
AUTHORS Tim G. J. Rudner Helen Toner

 Center for Security and Emerging Technology | 1 This paper is the first installment in a series on “AI safety,” an area of machine learning research that aims to identify causes of unintended behavior in machine learning systems and develop tools to ensure these systems work safely and reliably. Below, we introduce three categories of AI safety issues: problems of robustness, assurance, and specification. Other papers in this series elaborate on these and further key concepts. Introduction The past decade has seen the emergence of modern artificial intelligence and a variety of AI-powered technological innovations. This rapid transformation has predominantly been driven by machine learning, a subfield of AI in which computers learn patterns and form associations based on data. Machine learning has achieved success in application areas including image classification and generation, speech and text generation, and decision making in complex environments such as autonomous driving, video games, and strategy board games. However, unlike the mathematical and computational tools commonly used in engineering, modern machine learning methods do not come with safety guarantees. While advances in fields such as control theory have made it possible to build complex physical systems, like those found in various types of aircraft and automobiles, that are validated and guaranteed to have an extremely low chance of failure, we do not yet have ways to produce similar guarantees for modern machine learning systems. As a result, many machine learning systems cannot be deployed without risking the system encountering a previously unknown scenario that causes it to fail. The risk of system failures causing significant harm increases as machine learning becomes more widely used, especially in areas where safety and security are critical. To mitigate this risk, research into “safe” machine learning seeks to identify potential causes of unintended behavior in machine learning systems and develop tools to reduce the likelihood of such behavior occurring. This area
 Center for Security and Emerging Technology | 2 of research is referred to as “AI safety”1 and focuses on technical solutions to ensure that AI systems operate safely and reliably. Many other challenges related to the safe deployment of AI systems—such as how to integrate them into existing networks, how to train operators to work effectively with them, and so on—are worthy of substantial attention, but are not covered here. Problems in AI safety can be grouped into three categories: robustness, assurance, and specification. Robustness guarantees that a system continues to operate within safe limits even in unfamiliar settings; assurance seeks to establish that it can be analyzed and understood easily by human operators; and specification is concerned with ensuring that its behavior aligns with the system designer’s intentions.2 Modern Machine Learning Machine learning methods are designed to learn patterns and associations from data.3 Typically, a machine learning method consists of a statistical model of the relationship between inputs and outputs (for example, the relationship between an audio recording and a text transcription of it) and a learning algorithm specifying how the model should change as it receives more information about this input–output relationship. The process of updating the model as more data is made available is called “training,” and recent advances in fundamental research and engineering have enabled efficient training of highly complex models from large amounts of data. Once trained successfully, a machine learning system can be used to make predictions (such as whether or not an image depicts an object or a human), to perform actions (such as autonomous navigation), or to generate synthetic data (such as images, videos, speech, and text). Many modern machine learning systems use deep neural networks—statistical models that can represent a wide range of complex associations and patterns and that work particularly well with large amounts of data. Examples of useful application areas for deep neural networks include image classification and sequential decision-making in autonomous systems, as well as text, speech, and image generation.
 Center for Security and Emerging Technology | 3 Machine learning systems derive associations and patterns from data rather than from a prespecified set of rules. As a result, these systems are only as good as the data they were trained on. While modern machine learning systems usually work remarkably well in settings similar to those encountered during training, they often fail in settings that are meaningfully different. For example, a deep neural network trained to classify images of cats and dogs in black and white is likely to succeed at classifying similar images of cats and dogs in color. However, it will not be able to correctly classify a fish if it has never encountered an image of one during training. While machine learning systems do not use explicit rules to represent associations and patterns, they do use rules to update the model during training. These rules, also called “learning algorithms,” encode how the human designer of a machine learning system wants it to “learn” from data. For example, if the goal is to correctly classify images of cats and dogs, the learning algorithm should include a set of steps that update the model to gradually become better at classifying cats and dogs. This goal can be encoded in a learning algorithm in many ways, and it is the task of the human designer of such a system to do so. Robustness In order to be reliable, a machine learning system must operate safely under a wide range of conditions. Building into the system the ability to quantify whether or not it is confident about a prediction may reduce the chance of failure in situations it is not well-prepared to handle. The system, upon recognizing it is in a setting it was not trained for, could then revert to a safe fallback option or alert a human operator. Challenging inputs for machine learning systems can come in many shapes and guises, including situations a system may never have encountered before (as in the fish classification example above). Operating safely in such scenarios means that a system must, first, recognize that it has not been trained for such a situation and, second, have a way to act safely—for example, by notifying a human operator to intervene. An active area of research around this problem seeks to train machine learning models to estimate
 Center for Security and Emerging Technology | 4 confidence levels in their predictions. These estimates, called predictive uncertainty estimates, would allow the system to alert a human operator if it encounters inputs meaningfully different from those it was trained on. Consider, for example, a machine learning system tasked to identify buildings in satellite imagery. If trained on satellite imagery of a certain region, the system learns to identify buildings that look similar to those in the training data. If, when deployed, it encounters an image of a building that looks meaningfully unlike anything it has seen during training, a robust system may or may not classify the image as showing a building, but would invariably alert a human operator about its uncertainty, prompting manual human review. Assurance To ensure the safety of a machine learning system, human operators must understand why the system behaves the way it does, and whether its behavior will adhere to the system designer’s expectations. A robust set of assurance techniques already exist for previous generations of computer systems. However, they are poorly suited to modern machine learning systems such as deep neural networks. Interpretability (also sometimes called explainability) in AI refers to the study of how to understand the decisions of machine learning systems, and how to design systems whose decisions are easily understood, or interpretable. This way, human operators can ensure a system works as intended and, in the case of unexpected behavior, receive an explanation for said behavior. It is worth noting that researchers and engineers working with and developing modern machine learning systems do understand the underlying mathematical operations inside so-called “black-box” models and how they lead from inputs to outputs. But this type of understanding is difficult to convert into typical human explanations for decisions or predictions—say, “I liked the house because of its large kitchen,” or “I knew that dog was a Dalmatian because it had spots.” Interpretability, then, seeks to understand
 Center for Security and Emerging Technology | 5 how trained machine learning systems “reason”—that is, how certain types of inputs or input characteristics inform a trained system’s predictions. Some of the best tools we have for this so far include generating visualizations of the mathematical operations inside a machine learning system or indicating which input characteristics are most responsible for a model’s outputs. In high-stakes settings where humans interact with machine learning systems in real time, interpretability will be crucial in giving human operators the confidence to act on predictions obtained from such systems. Specification “Specification” of machine learning systems refers to defining a system’s goal in a way that ensures its behavior aligns with the human operator’s intentions. Machine learning systems follow a pre-specified algorithm to learn from data, enabling them to achieve a specific goal. Both the learning algorithm and the goal are usually provided by a human system designer. Examples of possible goals include minimizing a prediction error or maximizing a reward. During training, a machine learning system will try to reach the given goal, regardless of how well it reflects the designer’s intent. Therefore, designers must take special care to specify an objective that will lead to the desired behavior. If the goal set by the system designer is a poor proxy for the intended behavior, the system will learn the wrong behavior and be considered “misspecified.” This outcome is likely in settings where the specified goal cannot fully capture the complexities of the desired behavior. Poor specification of a machine learning system’s goal can lead to safety hazards if a misspecified system is deployed in a high-stakes environment and does not operate as intended. Misspecification has already arisen as a problem in YouTube’s video recommendation algorithms. This algorithm was designed to optimize for engagement—the length of time a user spends watching videos—to maximize ad revenue. However, an unintended side effect manifested: To maximize viewing time, in
 Center for Security and Emerging Technology | 6 some cases, the recommendation algorithm gradually steered users toward extremist content—including videos from white supremacist and other political and religious extremist channels—because it predicted these recommendations would cause the user to stay engaged longer. The extent of this phenomenon remains disputed, and YouTube has changed its algorithms since this issue first gained considerable attention. Yet the underlying idea—that optimizing for engagement could have unintended effects—demonstrates the hazards of goal misspecification.4 Conclusion Safety considerations must precede the deployment of modern machine learning systems in high-stakes settings. Robustness, assurance, and specification are key areas of AI safety that can guide the development of reliably safe machine learning systems. While all three are the subjects of active and ongoing research, it remains uncertain when we will be able to consider machine learning systems reliably safe.
Center for Security and Emerging Technology | 7 Further Reading
This paper is the first in a series of three primers describing issues
in AI safety. Along with the remaining installments in this series,
interested readers could consult the following works for more on
AI, machine learning, and AI safety.
On AI safety:
•Pedro Ortega et al., “Building Safe Artificial Intelligence:
Specification, Robustness, and Assurance,” Medium,
September 27, 2018,
https://medium.com/@deepmindsafetyresearch/building-
safe-artificial-intelligence-52f5f75058f1.
•Dario Amodei et al., “Concrete Problems in AI Safety,” arXiv
[cs.AI] (July 25, 2016), preprint,
https://arxiv.org/abs/1606.06565.
On machine learning in general:
•Ben Buchanan and Taylor Miller, “Machine Learning for
Policymakers,” (Belfer Center for Science and International
Affairs, June 26, 2017),
https://www.belfercenter.org/publication/machine-learning-
policymakers.
•Greg Allen, “Understanding AI Technology,” (Department of
Defense Joint AI Center, April 2020),
https://www.ai.mil/docs/Understanding%20AI%20Technolo
gy.pdf.
•Ian Goodfellow, Yoshua Bengio, and Aaron Courville, Deep
Learning (Cambridge: MIT Press, 2016),
https://www.deeplearningbook.org.
Center for Security and Emerging Technology | 8 Authors
Tim G. J. Rudner is a non-resident AI/ML fellow with CSET and a
PhD candidate in computer science at the University of Oxford.
Helen Toner is director of strategy at CSET.
Acknowledgements
For feedback and assistance, we would like to thank Igor Mikolic-
Torreira, Shelton Fitch, Alexandra Vreeman, Lynne Weil, Larry Lewis, Jack Clark, Dewey Murdick, and Michael Page.
© 2021 by the Center for Security and Emerging Technology. This
work is licensed under a Creative Commons Attribution -Non
Commercial 4.0 International License.
To view a copy of this license, visit
https://creativecommons.org/licenses/by-nc/4.0/.
Document Identifier: doi: 10.0.201.137/ 20190040

Center for Security and Emerging Technology | 9 Endnotes
1 Dario Amodei, Chris Olah, Jacob Steinhardt, Paul Christiano, John Schulman,
 Dan Mané, “Concrete Problems in AI Safety,” arXiv [cs.AI] (July 25, 2016),
preprint, https://arxiv.org/abs/1606.06565 .
2 For additional details about this taxonomy, see Pedro Ortega, Vishal Maini, et
al., “Building Safe Artificial Intelligence: Specification, Robustness, and
Assurance,” Medium (blog), September 27, 2018, https://medium.com/
@deepmindsafetyresearch/building- safe-artifi cial-intelligence-52f5f7 5058f1.
3 For more on how machine learning systems work, see Ben Buchanan and
Taylor Miller, “Machine Learning for Policymakers” (Belfer Center for Science and
International Affairs, June 26, 2017), https://www.belfercenter.org/publication/
machine-learn ing-polic ymakers; Greg Allen, “Understanding AI
Technology,” (Department of Defense Joint AI Center, April 2020), https://
www.ai.mil/docs/Understanding%20AI%20Technology.pdf; and Ian
Goodfellow, Yoshua Bengio, and Aaron Courville, Deep Learning
(Cambridge: MIT Press, 2 016), https://www.deeplearningbook.org.
4 Chico Q. Camargo, “YouTube’s Algorithms Might Radicalise People – but the
Real Problem Is We’ve No Idea How They Work,” The Conversation, January,
21, 2020, https://theconversation.com/youtubes-algorithms-might-
radicalisepeople-but-the-real-problem-is-weve-no-idea-how-they-
work-129955.


MARCH 2021
Key Concepts in AI Safety: Interpretability in Machine Learning
CSET Issue Brief
AUTHORS Tim. G. J. Rudner Helen Toner

 Center for Security and Emerging Technology | 1 This paper is the third installment in a series on “AI safety,” an area of machine learning research that aims to identify causes of unintended behavior in machine learning systems and develop tools to ensure these systems work safely and reliably. The first paper in the series, “Key Concepts in AI Safety: An Overview,” described three categories of AI safety issues: problems of robustness, assurance, and specification. This paper introduces interpretability as a means to enable assurance in modern machine learning systems. Introduction Interpretability, also often referred to as explainability, in artificial intelligence (AI) refers to the study of how to understand the decisions of machine learning systems, and how to design systems whose decisions are easily understood, or interpretable. This way, human operators can ensure a system works as intended and receive an explanation for unexpected behaviors. Modern machine learning systems are becoming prevalent in automated decision making, spanning a variety of applications in both the private and public spheres. As this trend continues, machine learning systems are being deployed with increasingly limited human supervision, including in areas where their decisions may have significant impacts on people’s lives. Such areas include automated credit scoring, medical diagnoses, hiring, and autonomous driving, among many others.1 At the same time, machine learning systems are also becoming more complex, making it difficult to analyze and understand how they reach conclusions. This increase in complexity—and the lack of interpretability that comes with it—poses a fundamental challenge for using machine learning systems in high-stakes settings. Furthermore, many of our laws and institutions are premised on the right to request an explanation for a decision, especially if the decision leads to negative consequences.2 From a job candidate suing for discrimination in a hiring process, to a bank customer inquiring about the reason for receiving a low credit limit, to a
Center for Security and Emerging Technology | 2 soldier explaining their actions before a court-martial, we assume that there is a process for assessing how a decision was made and whether it was in line with standards we have set. This assumption may not hold true if the decisionmaker in question is a machine learning system which is unable to provide such an explanation. In order for modern machine learning systems to safely integrate into existing institutions in high-stakes settings, they must be interpretable by human operators. Why Are Modern Machine Learning Systems Not Interpretable? Many modern machine learning systems use statistical models called deep neural networks which are able to represent a wide range of complex associations and patterns. To understand why decisions by deep neural networks are hard to interpret, consider two types of systems that are interpretable. One example is an earlier generation of AI systems which use human-specified rules instead of relying on data. As a simplified example, the autopilot system on an airplane uses a set of if-this-then-that rules to keep the plane on course—if the nose drops, lift it; if the plane banks left, roll a little right, and so on. While the rules in real systems are more complicated, they nonetheless allow humans to look back on system behavior and recognize which this triggered which that. A second example is a linear model, a simple kind of machine learning model. Like all machine learning systems, a linear model uses number values called parameters to represent the relationship between inputs and outputs. For example, one could create a model to predict someone’s salary from their age and years of schooling, two explanatory variables. In a linear model, the main parameters would be one number value to be multiplied by the explanatory variable “age,” and one number to be multiplied by the other explanatory variable “years of schooling.” Determining what value those two parameters should take is the learning part of machine learning. For a linear model, good parameter values can be found by simple calculations that may take a computer less than a
Center for Security and Emerging Technology | 3 second to perform. More importantly, because each parameter in a linear model is directly associated with one explanatory variable, understanding how the model works is simple. If, say, the parameter that gets multiplied by “age” is much higher than the parameter for “years of schooling,” then the model is predicting age is a more important determinant of salary. Deep neural networks are different. By design, they have far more parameters than linear models, and each parameter is connected in complex ways with inputs and other parameters, rather than directly linking explanatory variables to the outcome that the model seeks to predict. This complexity is a double-edged sword. On one hand, models can represent highly complex associations and patterns, allowing them to solve problems previously considered out of reach for computers, including image recognition, autonomous driving, and playing the game of Go. On the other hand, unlike older or simpler computer systems, the internal functioning of each model is very difficult to understand. At this point it is worth noting why the term “black box,” often used in this context, is not quite right to describe why deep neural networks are hard to understand. Machine learning researchers understand perfectly well how the mathematical operations underlying these systems work, and it is easy to look at the parameter values that make up the model. The problem lies in understanding how these millions (or even billions) of number values connect to the concepts we care about, such as why a machine learning model may erroneously classify a cat as a dog. In other words, interpreting deep neural networks requires both the ability to understand which high-level features in the data—such as a certain part of an image or a specific sequence of words—affect a model’s predictions and why a model associates certain high-level features with a corresponding prediction—that is, how deep neural networks “reason” about data.
Center for Security and Emerging Technology | 4 How to Make Modern Machine Learning S ystems More
Interpretable
Researchers are pursuing a range of different approaches to
improving the interpretability of modern machine learning systems. A fundamental challenge for this work is that clear, well -defined
concepts have yet to be developed around what it would mean for different types of systems to be interpretable. So far, interpretability research seeks to build tools making it somewhat more possible for a human operator to understand a system’s
outputs and inner workings.
Figure 1. Examples of images and their corresponding saliency maps indicating
which parts of the images contribute most to how a machine learning system
trained on a large set of images would classify them.
Source: Kare n Simonyan, Andrea Vedaldi, a nd Andrew Zisserman, “Dee p Inside
Convolutional Networks: Visualising Image Classification Models and Saliency
Maps,” arXiv [cs.CV] (April 19, 2014), preprint, https://arxiv.org/abs/1312.6034 .
Saliency maps are one popular set of tools for making modern
machine learning systems used for computer-vision applications
more interpretable. Broadly speaking, saliency maps visualize
which areas of an image led to a model’s classification of the same
image.3 For example, we might investigate why a deep learning
model has learned to identify images of cats and dogs from a large
dataset of cat and dog images labeled as such. If we wish to
understand why the model classified an image of a German
Shepherd as a dog, a saliency map may highlight the parts of the
image containing features present in dogs, but not in cats (for

Center for Security and Emerging Technology | 5 example, a large muzzle). In this way, a saliency map
communicates to a human operator which part of the image prompted the machine learning system to classify the image as it
did.
Another popular method for making deep neural networks more
interpretable is to visualize how different components of the model relate to high-level concepts that may affect the model’s predictions—concepts such as textures and objects in image
classifiers, grammar and tone in language models, or short -term vs.
long-term planning in sequential decision-making models.
4
Figure 2. A visualization showing examples of what different layers of an image
classification network “see.” The left -most column, depicting an early layer, is
picking up lines; middle layers are detecting textures and patterns of increasing
complexity; later layers, shown on the right, are looking for objects.
Source: Chris Olah, Alexander Mordvintsev, and Ludwig Schubert, “Feature
Visualization,” Distill, November 7, 2017, https://distill.pub/2017/feature-
visualization.
Unfortunately, existing methods to make modern machine learning
systems interpretable fall short; they typically only provide one
angle from which to view the system instead of taking a holistic
view. To fully understand how a machine learning system works,
we must understand how the data and learning algorithm affected
training, whether training could have resulted in a different model
under modified training conditions, and how all of these factors
ultimately affect the system’s predictions. At this time, our
understanding of these questions is still very limited and the
insights obtained from existing methods are fallible, require human
supervision, and only apply to a small subset of application areas.

Center for Security and Emerging Technology | 6 For example, the saliency maps in Figure 1 do shed some light on how the model in question works. One can see, for instance, that the model focuses on the dog to classify it as such, but identifies the sailboat in part by looking at the ocean. Saliency maps, however, do not help a human observer understand what might have led to different outcomes. Likewise, Figure 2 shows a method for understanding what the different parts of an image classifier detect. Unfortunately, looking at this type of visualization does not help a human operator evaluate whether the system is likely to be accurate, fair, or reliable. The lack of a common, well-defined vocabulary for interpretable machine learning systems further exacerbates these shortcomings.5 Key concepts, such as trustworthiness, reliability, transparency, or verifiability, are often used loosely or interchangeably rather than referring to standardized or generally accepted technical definitions of such terms, making it difficult to measure progress and to accurately and reliably communicate research results to the public. Outlook Interpretability will allow us to understand potential failure modes for machine learning systems, enable regulatory compliance and audits, and reduce the risk of models with algorithmic or data-induced bias being deployed. Rendering modern machine learning systems like deep neural networks interpretable will help us ensure that any such system deployed in safety-critical settings works as intended. Unfortunately, while an active and ongoing area of research, existing approaches to achieving interpretability do not yet provide satisfactory solutions. It remains unclear when—or even if—we will be able to deploy truly interpretable deep learning systems. In the meantime, our best option may be to stick with simpler, more inherently interpretable models whenever possible.6
Center for Security and Emerging Technology | 7 Authors
Tim G. J. Rudner is a non-resident AI/ML fellow with CSET and a
PhD candidate in computer science at the University of Oxford.
Helen Toner is director of strategy at CSET.
Acknowledgements
For feedback and assistance, we would like to thank Igor Mikolic-
Torreira, Shelton Fitch, Alexandra Vreeman, Lynne Weil, Larry
Lewis, Jack Clark, Dewey Murdick, and Michael Page.
© 2021 by the Center for Security and Emerging Technology. This
work is licensed under a Creative Commons Attribution-Non Commercial 4.0 International License.
To view a copy of this license, visit
https://creativecommons.org/licenses/by-nc/4.0/.
Document Identifier: doi: 10.0.201.137/20190042

Center for Security and Emerging Technology | 8 Endnotes
1 Danny Yadron and Dan Tynan, “Tesla Driver Dies in First Fatal Crash While
Using Autopilot Mode,” The Guardian, June 30, 2016, https://
www.theguardian.com/technology/2016/jun/30/tesla-autopilot-death-self-
driving-car-elon-musk; Forough Poursabzi-Sangdeh, Daniel G. Goldstein, Jake M.
Hofman, Jennifer Wortman Vaughan, Hanna Wallach, “Manipulating and
Measuring Model Interpretability,” arXiv [cs.AI] (November 8, 2019), arXiv,
preprint, https://arxiv.org/abs/1802.07 810; and Jennifer Valentino- DeVries,
“How the Police Use Facial Recognition, and Where It Falls Short,” The New
York Times, January 12 , 2020, https://www.nytimes.com/2 020/01/12/
technology/facial- recognition-police.html.
2 For a more in-d epth discussion of what an “explanation” is in a legal context,
 why explanations are necessary, and how AI explanations compare to human
explanations, see Finale Doshi-V elez, Mason Kortz, Ryan Budish, Chris Bavitz,
Sam Gershman, David O'Brien, Kate Scott, Stuart Schieber, James Waldo, David
Weinberger, Adrian Weller, Alexandra Wood, “Accountability of AI Under the
Law: The Role of Explanation,” arXiv [cs.AI] (December 20, 2019), preprint,
https://arxiv.org/abs/1 711.01134.
3 Simonyan, Vedaldi, and Zisserman, “Deep Inside Convolutional Networks”;
Julius Adebayo et al., “Sanity Checks for Saliency Maps,” arXiv [cs.CV] (October
28, 2018), arXiv, preprint, https://arxiv.org/abs/1810.03292; Ruth Fong and
Andrea Vedaldi, “Interpretable Explanations of Black Boxes by Meaningful
Perturbation,” arXiv [cs.CV] (January 10, 2018), arXiv, preprint,
https://arxiv.org/abs/1704.03296; Ruth Fong, Mandela Patrick, and Andrea
Vedaldi, “Understanding Deep Networks via Extremal Perturbations and
Smooth Masks,” arXiv [cs.CV] (October 18, 2019), preprint,
https://arxiv.org/abs/1910.08485.
4 Jesse Vig, “A Multiscale Visualization of Attention in the Transformer Model,”
arXiv [cs.HC] (June 12, 2019), arXiv, preprint, https://arxiv.org/abs/1906.057 14;
OpenAI, “Dota 2 with Large Scale Deep Reinforcement Learning,” arXiv [cs.LG]
(December 13, 2019), preprint, https://arxiv.org/abs/1912.06680 .
5 Zachary C. Lipton, “The Mythos of Model Interpretability,” arXiv [cs.LG] (March
6, 2017), arXiv, preprint, https://arxiv.org/abs/1606.0 3490.
6 Cynthia Rudin, “Stop Explaining Black Box Machine Learning Models for High
Stakes Decisions and Use Interpretable Models Instead,” Nature Machine
Intelligence 1 (May 2019): 206–215, https://www.https://www.nature.com/
articles/s42256-019- 0048-x.


MARCH 2021
Key Concepts in AI Safety: Robustness and Adversarial Examples
CSET Issue Brief
AUTHORS Tim G. J. Rudner Helen Toner

 Center for Security and Emerging Technology | 1 This paper is the second installment in a series on “AI safety,” an area of machine learning research that aims to identify causes of unintended behavior in machine learning systems and develop tools to ensure these systems work safely and reliably. The first paper in the series, “Key Concepts in AI Safety: An Overview,” described three categories of AI safety issues: problems of robustness, assurance, and specification. This paper introduces adversarial examples, a major challenge to robustness in modern machine learning systems. Introduction As machine learning becomes more widely used and applied to areas where safety and reliability are critical, the risk of system failures causing significant harm rises. To avoid such failures, machine learning systems will need to be much more reliable than they currently are, operating safely under a wide range of conditions.1 In this paper, we introduce adversarial examples—a particularly challenging type of input to machine learning systems—and describe an artificial intelligence (AI) safety approach for preventing system failures caused by such inputs. Machine learning systems are designed to learn patterns and associations from data. Typically, a machine learning method consists of a statistical model of the relationship between inputs and outputs, as well as a learning algorithm. The algorithm specifies how the model should change as it receives more information (in the form of data) about the input–output relationship it is meant to represent. This process of updating the model with more data is called “training.” Once a machine learning model has been trained, it can make predictions (such as whether an image depicts an object or a human), perform actions (such as autonomous navigation), or generate synthetic data (such as images, videos, speech, and text). An important trait in any machine learning system is its ability to work well, not only on the specific inputs it was shown in training, but also on other inputs. For example, many image classification
Center for Security and Emerging Technology | 2 models are trained using a dataset of millions of images called ImageNet; these models are only useful if they also work well on real-life images outside of the training dataset. Modern machine learning systems using deep neural networks—a prevalent type of statistical model—are much better in this regard than many other approaches. For example, a deep neural network trained to classify images of cats and dogs in black and white is likely to succeed at classifying similar images of cats and dogs in color. However, even the most sophisticated machine learning systems will fail when given inputs that are meaningfully different from the inputs they were trained on. A cat-and-dog classifier, for example, will not be able to classify a fish as such if it has never encountered an image of a fish during training. Furthermore, as the next section explores in detail, humans cannot always intuit which kinds of inputs will appear meaningfully different to the model. Adversarial Examples One of the most significant current challenges in AI safety is creating machine learning systems that are robust to adversarial examples. Adversarial examples are model inputs (for example, images) designed to trick machine learning systems into incorrect predictions. In the case of a machine learning system designed to distinguish between cats and dogs, an adversarial example could be an image of a cat modified to appear to the model as a dog. Since machine learning systems process data differently from humans, the cat image could be altered in ways imperceptible to humans but meaningfully different to the machine learning system. The modified image may still resemble a cat to humans, but to a machine learning system, it “looks like” a dog. Adversarial examples can be generated systematically, either by digitally altering the input to a system or by directly altering the appearance of objects in the physical world. Unlike other adversarial attacks, such as “data poisoning,” which seeks to attack the algorithm used to train a machine learning model, adversarial examples are designed to attack already trained models.
Center for Security and Emerging Technology | 3 Figures 1 and 2 show systematically generated adversarial examples. Specifically, the adversarial example in Figure 1c digitally modifies the original image by an imperceptibly small amount, whereas the adversarial example in Figure 2b is created by adding patches to the image designed to mimic irregularities found in the physical world (such as graffiti or stickers). Both adversarial examples are generated via so-called white-box attacks, which assume the attacker knows how the trained classification model works and can exploit this knowledge to create adversarial examples that trick the model into making incorrect predictions. Figure 1. An example of a “white-box” adversarial example from Goodfellow et al. (2015). The original image (a) is classified as “panda” with 57.7 percent probability. After being overlaid with a minimal amount of noise—the adversarial alteration (b) multiplied by a factor of 0.007—the resulting image (c) is classified as “gibbon” with 99.3 percent probability. The difference between (a) and (c) is imperceptible to the human eye. a) Original image b) Adversarial alteration c) Adversarial example Source: Ian J. Goodfellow, Jonathon Shlens, and Christian Szegedy, “Explaining and Harnessing Adversarial Examples,” International Conference on Learning Representations, Vol. 3, San Diego, CA, USA, May 7-9, 2015, https://arxiv.org/abs/1412.6572.

Center for Security and Emerging Technology | 4 Figure 2. An example of a white-box adversarial example designed to generate physical alterations for physical-world objects. The adversarial alteration (b), which is designed to mimic the appearance of graffiti (a), tricks an image classifier into not seeing a stop sign.
 a) Graffiti on stop sign b) Adversarial Example Source: Kevin Eykholt, Ivan Evtimov, Earlence Fernandes, Bo Li, Amir Rahmati, Chaowei Xiao, Atul Prakash, Tadayoshi Kohno, Dawn Song, "Robust Physical-World Attacks on Deep Learning Visual Classification,” 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA, June 18-23, 2018, pp. 1625-1634, https://arxiv.org/abs/1707.08945. Although modern machine learning systems usually generalize remarkably well to data similar to the data used for training, adversarial examples can be created from surprisingly simple modifications to model inputs. Changes such as blurring or cropping images, or altering the appearance of the physical-world objects shown in an image, can fool an otherwise reliable system. In Figure 3b, an adversarial example is constructed by reducing the resolution of the original image, thereby changing the model’s prediction from correct to incorrect. Unlike the adversarial examples in Figures 1 and 2, the adversarial example in Figure 3 was created via a black-box attack—that is, created without access to the trained classification model. It is not as subtle as the alteration in Figure 1 and not as targeted as the alteration in Figure 2. However, it demonstrates that modern machine learning systems can be fooled with little effort and no knowledge of the prediction model.

Center for Security and Emerging Technology | 5 Figure 3. An example of a black-box adversarial example. The original image (a) is classified as “washer” with 53 percent probability. The image is altered by reducing its resolution to create an adversarial example (b), which is classified as “safe” with 37 percent and as “loudspeaker” with 24 percent probability.
 a) Original image b ) Adversarial example Source: Alexey Kurakin, Ian Goodfellow, and Samy Bengio, “Adversarial Examples in the Physical World,” arXiv [cs.CV] (February 11, 2017), preprint, https://arxiv.org/abs/1607.02533. Robustness to Adversarial Examples Robust machine learning systems need to be able to identify data that is meaningfully different from training data and provide a defense against adversarial examples. There are a wide range of different research areas attempting to make progress in this direction. One such research direction aims to incorporate predictive uncertainty estimates into machine learning systems. This way, any prediction from the system would come with an estimate of certainty. If the machine learning system indicates uncertainty about the correctness of its prediction, a human operator can be alerted. To understand predictive uncertainty estimates and how they can make machine learning systems more robust to adversarial examples, consider the classification “probability scores” given in the descriptions of Figures 1 and 3. In reality, these scores, which express the probability of an input belonging to a certain class (e.g., the class “cat” or “dog”), are misleading. While they do express a

Center for Security and Emerging Technology | 6 probability, they do not actually express the model’s level of certainty about the correctness of the predictions. To fully understand this point, consider a machine learning system trained to distinguish between two classes: cats and dogs. Such a system will by design have two outputs: one for the class “cat” and one for the class “dog.” If the model is given an image of a dog, it will output values between zero and one for each class—for instance, 90 percent and 10 percent for the classes “dog” and “cat,” respectively, so that the values sum up to 100 percent. However, if given an image of a fish, the model will still make predictions for the two classes on which it was trained, unaware that it is being asked to identify an object it was not trained to recognize. In a best-case scenario, it would give outputs of 50 percent for each class, indicating that the input is equally likely to be either a cat or a dog. In a worst-case scenario, it would give a high probability score for one class, providing a false sense of certainty. But the way most machine learning systems are designed, they cannot give a low score to both the “cat” and “dog” labels. As such, these outputs should not be read as the machine learning system’s “confidence” in the correctness of its classification. Predictive uncertainty estimates can fill this gray spot. They complement the regular model outputs by expressing the model’s uncertainty about the correctness of its predictions. If a machine learning system has good predictive uncertainty estimates, then the probability scores in Figure 3 would be accompanied by a high uncertainty score, indicating that the model is highly uncertain about the correctness of the predictions. Such uncertainty estimates can help a human operator avoid wrong predictions in safety-critical settings and ensure the system’s reliability and safety, as demonstrated in Figure 4.
Center for Security and Emerging Technology | 7 Figure 4. An example of predictive uncertainty estimates for autonomous vehicles. The first column shows the image fed into the system, the second column shows the ground truth classification of objects in the image (buildings, sky, street, sidewalk, etc.), the third column shows the model’s classification, and the rightmost column shows the system’s uncertainty about its classification. As can be seen from the image on the bottom right, the system is uncertain about its classification of parts of the sidewalk and could alert the human operator to take over the steering wheel.
 Source: Alex Kendall and Yarin Gal, “What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision?” Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, Vol. 30, Long Beach, CA, USA, December 4-9, 2017, pp. 5574-5584, https://arxiv.org/abs/1703.04977. Unfortunately, obtaining reliable predictive uncertainty estimates for modern machine learning systems remains an unsolved problem. While several existing methods can generate uncertainty estimates, there are no mathematical guarantees that these uncertainty estimates are actually accurate. Furthermore, while empirical studies demonstrate that certain methods produce good predictive uncertainty estimates in some settings, those results cannot be generalized to any setting. Like other areas of robustness research, developing methods that yield reliably well-calibrated uncertainty estimates for modern machine learning systems is an active and ongoing area of research.

Center for Security and Emerging Technology | 8 Outlook While modern machine learning systems often perform well on narrowly defined tasks, they can fail when presented with tasks meaningfully different from those seen during training. Adversarial attacks exploit this vulnerability by presenting inputs to machine learning systems specifically designed to elicit poor predictions. Adversarially robust machine learning systems seek to fix this vulnerability through mechanisms allowing the system to recognize when an input is meaningfully different from data seen during training, making the system more reliable in practice. Unfortunately, while an active area of research, existing approaches to detecting and defending against adversarial attacks do not yet provide satisfactory solutions, and the timeline to develop and deploy truly robust modern machine learning systems remains uncertain. For now, anyone considering deploying modern machine learning systems in safety-critical settings must therefore grapple with the fact that in doing so, they are introducing safety risks that we do not yet know how to mitigate effectively.2
Center for Security and Emerging Technology | 9 Authors Tim G. J. Rudner is a non-resident AI/ML fellow with CSET and a PhD candidate in computer science at the University of Oxford. Helen Toner is director of strategy at CSET. Acknowledgements For feedback and assistance, we would like to thank Igor Mikolic-Torreira, Shelton Fitch, Alexandra Vreeman, Lynne Weil, Larry Lewis, Jack Clark, Dewey Murdick, and Michael Page. © 2021 by the Center for Security and Emerging Technology. This work is licensed under a Creative Commons Attribution-Non Commercial 4.0 International License. To view a copy of this license, visit https://creativecommons.org/licenses/by-nc/4.0/. CSET Product ID #: 20190041 Document Identifier: doi: 10.0.201.137/20190041

Center for Security and Emerging Technology | 10 Endnotes
1 Organizational factors will also play an important role in whether the
 deployment of a given AI system is safe. For more on the importance of robust
organizational practices, see Thomas G. Dietterich, “Robust Artificial
Intelligence and Robust Human Organizations,” arXiv [cs.AI] (November 27,
2018), preprint, https://arxiv.org/abs/1811.10 840.
2 Andrew Lohn, “Hacking AI” (Center for Security and Emerging Technology,
December 2020), https://cset.georgetown.edu/research/hacking-ai/.


May 2021
Machine Intelligence for Scientific Discovery and Engineering Invention
CSET Data Brief
AUTHORS Matthew Daniels Autumn Toney Melissa Flagg Charles Yang

 Center for Security and Emerging Technology | 1 Introduction Artificial intelligence is driving new discoveries and advancements across many fields of science and engineering. Modern applications of AI and machine learning have the potential to change the practice of research and development (R&D) in the United States over the next decade. This data brief is a first step toward understanding how modern AI and ML have begun accelerating growth across various science and engineering disciplines in recent years. The application of AI to science is not new. Pioneers at Stanford University in the 1960s used early AI systems to automate analyses of chemical structure and mass spectra, supported by the Advanced Research Projects Agency (ARPA), the predecessor to the Defense Advanced Research Project Agency (DARPA), and NASA.1 This work grew out of an interest in using AI systems for complex reasoning in scientific problems.2 Today, deep learning methods, a specific subset of ML, can predict the 3D structure of proteins to within the width of an atom—substantial progress on a 50-year challenge achieved with modern machine learning.3 This tool is important because it may contribute new capabilities for disease treatment and pandemic response. In the years ahead, modern ML may help advance the engineering design of viable, cost-effective fusion reactors, a large-scale source of clean energy that has also been out of reach for more than 70 years.4 AI applications can drive advancements in fields as disparate as gene editing and chip design. American institutions have begun to adapt to this future. MIT, for example, announced its largest structural change since the 1950s and invested $1 billion to found a new college focused on the intersections between computing, AI, and the university’s existing disciplines of science and engineering.5 Students will be encouraged to be “bilingual” between computing and other disciplines.6 As modern AI applications begin to transform science and engineering, a key question for high-level leaders shaping science
 Center for Security and Emerging Technology | 2 and technology strategies is what disciplines will be affected first, and will these early impacts be narrowly confined to a few disciplines (like materials science and biomedical fields) or broadly applicable? Accelerated emergence and growth of new fields is especially noteworthy: bioengineering, computer science, and materials science have all emerged as enabling disciplines that changed science and engineering broadly in the last century—and were followed by national and global changes. This data brief provides an initial look at how modern AI and ML have begun accelerating growth across a wide array of research disciplines in recent years, and focuses on direct applications of AI to making new scientific and engineering breakthroughs.7 We first summarize illustrative examples of AI driving innovation in medicine, fundamental sciences, and engineering research. We then analyze global scientific publications data to see if these recent achievements represent real trends in published research. Illustrative Examples of Machine Learning for Science and Engineering Within the long history of AI for science, deep learning has recently emerged as a promising new tool for scientists and engineers. Here, we provide illustrations of the great potential of modern AI and ML to impact R&D in new ways. These examples motivate a more systematic look in the next section using CSET data on global research clusters. In medicine, one potential application of AI is its ability to improve healthcare diagnostics, particularly the ability to accurately and rapidly assess radiology scans.8 Facebook and New York University researchers have used a deep learning approach to significantly reduce the time required for Magnetic Resonance Imaging (MRI) scans.9 Various U.S. agencies overseeing healthcare regulations have begun approving AI applications in the healthcare industry. The Food and Drug Administration has already given clearance to a private startup to begin using AI tools to assist with reading prostate scans10 and the Center for Medicare and Medicaid
 Center for Security and Emerging Technology | 3 Services has also approved an AI startup to use automated imagery analysis to quickly help identify strokes in patients.11 In the fundamental sciences, AI is helping scientists make sense of complex, high-dimensional data. Physicists at MIT demonstrated an AI system able to re-derive 100 fundamental physics equations from the canonical Feynman Lectures based on simulated data.12 In astrophysics and astronomy, researchers have used graph neural networks to propose a new analytical equation for dark matter phenomena that better matches observed data than previous human-derived equations.13 Others identified previously missed exoplanets in large-scale astronomy data with deep learning.14 Applications of ML are also helping scientists discover new materials that can advance cleaner energy sources. For example, researchers have recently used neural networks to better understand the high-dimensional spectroscopy of nanoscale ferroelectric materials15 and run high throughput screening of possible battery cathode materials.16 Scientists also used AI to optimize fast-charging protocols for Li-ion batteries while preserving battery health,17 which could have important implications for electric vehicle charging. In the life sciences and bioengineering, modern AI and ML have shown promise in protein engineering,18 improving the gene targeting capabilities of clustered regularly interspaced short palindromic repeats (CRISPR) technologies,19 and discovering new classes of antibiotics that retain effectiveness against drug-resistant bacteria.20 An international collaborative effort between universities in the United States and Europe showed how deep learning methods can significantly accelerate a variety of important scientific simulations, including those from geology, nuclear fusion, and environmental science.21 Researchers at Google also used a deep learning model to demonstrate high-resolution, real-time weather forecasts that outperform the current best Numerical Weather Prediction system available from the National Oceanic and Atmospheric Administration.22 In engineering and manufacturing, AI is also slowly becoming an important tool in developing new engineering design approaches and advancing U.S. manufacturing capabilities. Researchers at
 Center for Security and Emerging Technology | 4 Google have also used deep reinforcement learning to accelerate the design process for semiconductor computer chips.23 General Motors has partnered with Autodesk to develop more lightweight vehicles using AI generative modelling.24 Advanced manufacturing techniques like 3D printing have use-cases for AI to help improve the tuning of manufacturing parameters.25 Finally, convolutional neural networks show potential in improving manufacturing quality control and helping reduce defects in manufacturing lines.26 These are examples of how modern AI applications can drive innovation. The next section examines more systematically how widespread the application of AI is to other fields of science and engineering. We do this by looking at all the lines of globally published research that are merging AI with other fields. Findings To begin understanding the breadth of AI’s current impact, we analyze the growth of AI-related papers across different scientific disciplines. We do this by leveraging a large, ongoing effort at CSET to explore scientific research publications. CSET uses a merged dataset that accounts for roughly 90 percent of the world's scientific publications. We explore this dataset via an automated structuring of scholarly literature, which we refer to as research clusters.27 RCs are generated using direct citation links from publications, meaning that clusters are defined by their citation relationships as opposed to a social network of authors. Rahkovsky et al. provide full details on the clustering methodology using CSET’s merged corpus and the application of Dunham et al.’s AI labeling model, which we use to identify AI-related papers.28 We programmatically assign each RC a research discipline label from a list of 19 broad research areas using Microsoft Academic Graph’s (MAG) field of study:29 Art, Biology, Business, Chemistry, Computer Science, Economics, Engineering, Environmental Science, Geography, Geology, History, Materials Science, Mathematics, Medicine, Philosophy, Physics, Political Science, Psychology, and Sociology. This label corresponds to the majority of MAG papers in the RC; papers in a RC that do not come from the MAG dataset do not contribute to this label. Because RCs are
 Center for Security and Emerging Technology | 5 defined by their direct citation links, however, they are not restricted to containing publications from only one discipline—any RC can encompass cross-disciplinary research. Any clustering of multidisciplinary papers comes with caveats (see our GitHub archive for further details),30 but for the purposes of this paper, these RCs provide a useful representation of AI's spread into diverse areas of science. For our analysis, we are interested in RCs that are AI-related, growing rapidly, and specifically affecting disciplines beyond computer science.31 From the entire set of RCs covering all research disciplines (126,925 in total), there are 5,221 RCs with 10 percent or more AI-related publications. Of those AI-related RCs, 1,170 do not have a computer science research discipline label. From these AI-related RCs outside computer science, we identify 340 RCs, encompassing 308,208 scientific research publications, with sustained rapid growth (20 percent annual growth or more) over the last three years. Applying a doubling growth threshold (100 percent growth or more), we narrow this further to 50 RCs, which encompass 39,488 scientific research publications. Figure 1 provides the breakdown of RCs with rapid growth and doubling growth for each research discipline, excluding computer science. For both rapid growth and doubling growth RCs, engineering, mathematics, and medicine are the top three research disciplines with cross-disciplinary AI research. While engineering, mathematics, and medicine lead progress, AI is entwined with growth in a wide span of science and engineering, including chemistry, biology, materials science, environmental science, and a strong presence in psychology. Figure 1 also shows this wide distribution of disciplines for the 340 rapidly growing RCs and the 50 RCs with doubling growth—notably, even with a threshold for an extremely high degree of growth, we still see impact across a broad set of disciplines. This analysis supports the growing impression that AI is influencing rapidly emerging areas of science and engineering far beyond computer science. It also suggests that U.S. research will benefit from AI-relevant skills dispersed widely in other technical communities.
 Center for Security and Emerging Technology | 6 Figure 1. AI-related research clusters with rapid growth (≥20% annually) and doubling growth annually by discipline
 Source: CSET Map of Science. Takeaways Building U.S. competitiveness in AI is a complementary priority to maintaining the United States’ wider leadership in science and technology. This data brief provides an initial look at AI’s appearance in rapidly growing clusters of published research worldwide. Applications of AI are a substantial part of rapidly growing research clusters across a broad array of disciplines—not just in one or two areas. Mathe-matics, medicine, engineering, psychology, biology, materials science, and environmental science have especially large quantities of rapidly growing research areas with noteworthy interdisciplinary roles for modern AI. For U.S. leaders, this may become a source of both opportunity and surprise. Most of all, these advancements make it clear that U.S. R&D organizations will have growing opportunities to apply AI to scientific discovery and engineering invention in many areas. On a national scale, these applications may accelerate the emergence

 Center for Security and Emerging Technology | 7 and growth of new domains of science at the intersections of more established disciplines; disciplines far from computer science will change in potentially surprising ways as AI offers new tools for R&D.32 Large R&D organizations, both inside and outside of the U.S. government, have particular opportunities: ● Education investments that include more and better cross-training in AI for scientists and engineers. For U.S. government organizations, this could build on precedent like the Department of Energy’s Computational Science Graduate Fellowship, but oriented to focus on AI and computing broadly.33 Such education investments should give special attention to diversity and inclusion, which is important both for future AI research and for enabling cross-disciplinary research. These investments will empower the next generation workforce to apply AI to problems across the U.S. government, from health to spaceflight. ● Increasing openness of access to scientific publications and data, especially in formats accessible and useful to “machine readers.” For example, the National Science Foundation, Department of Energy, and the National Institutes of Health might require funded researchers to open source data and publications (analogous to Plan S in the European Union,34 and building on the 2013 policy memorandum by the Office of Science and Technology Policy Director, John Holdren, directing U.S. federal agencies with more than $100 million in R&D expenditures to develop plans to make results of federally funded research freely available to the public35). One example of early work in this direction has been the COVID-19 Open Research Dataset (CORD-19), a resource of more than 400,000 scholarly articles in machine-readable format about SARS-CoV-2 and related viruses for use by the global ML community.36 ● Providing AI and data toolkits for scientists and engineers in different disciplines. The U.S. government should especially fund AI building blocks that allow scientists to cheaply
 Center for Security and Emerging Technology | 8 leverage AI tools. For example, this might include supporting the creation of pre-trained AI models with baked-in scientific knowledge.37 ● Providing scientific/engineering challenges and datasets more widely to AI researchers. For example, in the early 2000s, DARPA Grand Challenges helped jump-start the development of today’s autonomous vehicles. Progress in other areas of AI, such as image recognition38 and the ability to successfully play strategy board games, has benefited from large, robust benchmarks. Scientific funding agencies can articulate analogous challenges based on problems that are relevant to the scientific community to encourage the development of AI scientists. ● Pursuing international collaboration with U.S. allies, particularly the United Kingdom and Japan. In February 2020, the U.S. Department of Defense funded a workshop hosted at the Alan Turing Institute in London, which convened researchers from the United States, the United Kingdom, and Japan—and supported a new planetary challenge for AI systems able to make Nobel-worthy scientific discoveries. U.S. allies already share interest in this topic, and can provide opportunities for wider collaboration. Modern AI is helping to reshape the global research landscape beyond computer science. The security, health, and economic vitality of nations depend in part on the inventions they have ready access to—and modern applications of AI can help accelerate the creation of new inventions in the years ahead.
 Center for Security and Emerging Technology | 9 Authors Matthew Daniels is a senior fellow at CSET, where Autumn Toney is a data research analyst and Melissa Flagg is also a senior fellow. Charles Yang is a graduate student at the University of California, Berkeley in the Department of Electrical Engineering and Computer Science. Acknowledgments Thanks to Igor Mikolic-Torreira, Helen Toner, Dewey Murdick, Shelton Fitch, Brendan Tracey, Ayodeji Coker, and Jared Dunnmon for useful insights and feedback. We are also grateful to colleagues at the Alan Turing Institute for hosting discussions on AI for science, and to Hiroaki Kitano for his early vision of a global grand challenge focused on AI scientists. The authors are solely responsible for the views expressed in this publication and for any errors. © 2021 by the Center for Security and Emerging Technology. This work is licensed under a Creative Commons Attribution-Non Commercial 4.0 International License. To view a copy of this license, visit https://creativecommons.org/licenses/by-nc/4.0/. Document Identifier: doi: 10.51593/20200099

 Center for Security and Emerging Technology | 10 Endnotes
 1 Joshua Lederberg and Edward A. Feigenbaum, “Mechanization of Inductive Inference in Organic Chemistry,” in Benjamin Kleinmuntz, ed., Formal Representations for Human Judgment (New York: Wiley, 1968). Available online: https://exhibits.stanford.edu/feigenbaum/catalog/bm350jn4518. 2 Bruce G. Buchanan and Edward A. Feigenbaum, "DENDRAL and Meta-DENDRAL: Their applications dimension," in Readings in artificial intelligence (Burlington: Morgan Kaufmann, 1981), 313-322. 3 Ewen Callaway, “‘It will change everything’: DeepMind’s AI makes gigantic leap in solving protein structures,” Nature, November 30, 2020, https://www.nature.com/articles/d41586-020-03348-4. 4 See E.A. Baltz, E. Trask, M. Binderbauer et al., “Achievement of Sustained Net Plasma Heating in a Fusion Experiment with the Optometrist Algorithm,” Scientific Reports 7: 6425 (2017), https://doi.org/10.1038/s41598-017-06645-7 and Julian Kates-Harbeck, Alexey Svyatkovskiy, and William Tang, “Predicting disruptive instabilities in controlled fusion plasmas through deep learning,” Nature 568: 526–531 (2019), https://doi.org/10.1038/s41586-019-1116-4. 5 MIT News Office, “MIT reshapes itself to shape the future,” Massachusetts Institute of Technology, October 15, 2018, https://news.mit.edu/2018/mit-reshapes-itself-stephen-schwarzman-college-of-computing-1015. 6 Kaveh Waddell, “MIT is investing $1 billion into AI research,” Axios, October 18, 2018, https://www.axios.com/mit-ai-research-1-billion-b20ade0a-ba12-4300-922a-c8033dd47fe9.html. 7 This goal has had a resurgence in the 2010s: in 2016 Hiroaki Kitano, creator of Robocup, proposed a grand challenge for AI systems capable of making Nobel-worthy scientific discoveries: Hiroaki Kitano, "Artificial Intelligence to Win the Nobel Prize and Beyond: Creating the Engine for Scientific Discovery," AI Magazine 37, no. 1 (2016): 39-49. In future work we will discuss broader ways that AI may accelerate S&E. 8 One particular concern when developing AI systems for healthcare is that such technology may present racial biases. The development of equitable, fair, and transparent healthcare technology is an important challenge that will need to be addressed in the areas where AI, science, and people intersect. In particular, the National Academy of Medicine released a report that details the challenge and opportunity of AI in healthcare that is a useful starting reference. See Michael Matheny, Sonoo Thadaney Israni, Mahnoor Ahmed, and Danielle Whicher, eds., “Artificial Intelligence in Health Care: The Hope, the Hype, the Promise, the Peril”
 Center for Security and Emerging Technology | 11 (National Academy of Medicine, 2019), https://nam.edu/wp-content/uploads/2019/12/AI-in-Health-Care-PREPUB-FINAL.pdf. 9 Michael Recht et al., “Using Deep Learning to Accelerate Knee MRI at 3 T: Results of an Interchangeability Study,” American Journal of Roentgenology 215, no. 6 (2020): 1421-1429, https://www.ajronline.org/doi/10.2214/AJR.20.23313. 10 Emi Gal, “Ezra Announces First FDA-cleared Prostate AI,” Ezra, October 20, 2020, https://ezra.com/ezra-announces-first-fda-cleared-prostate-ai/. Letter from the Food and Drug Administration, June 11, 2020, https://www.accessdata.fda.gov/cdrh\_docs/pdf19/K192969.pdf. 11 See Viz.ai, “Viz.ai Granted Medicare New Technology Add-on Payment,” Cision PR Newswire, September 3, 2020, https://www.prnewswire.com/news-releases/vizai-granted-medicare-new-technology-add-on-payment-301123603.html and ContaCT, Federal Register 85, no. 182 (September 18, 2020): 58625, https://www.govinfo.gov/content/pkg/FR-2020-09-18/pdf/2020-19637.pdf. 12 Silviu-Marian Udrescu and Max Tegmark, “AI Feynman: A physics-inspired method for symbolic regression,” Science Advances 6, no. 16 (2020): https://doi.org/10.1126/sciadv.aay2631. 13 Miles Cranmer et al., “Discovering Symbolic Models from Deep Learning with Inductive Biases,” arXiv [cs.LG], arXiv (November 18, 2020), https://arxiv.org/pdf/2006.11287.pdf. 14 Anne Dattilo et al., “Identifying Exoplanets with Deep Learning. II. Two New Super-Earths Uncovered by a Neural Network in K2 Data,” The Astronomical Journal 157, no. 5 (2019): https://iopscience.iop.org/article/10.3847/1538-3881/ab0e12. 15 Joshua Agar et al., “Revealing ferroelectric switching character using deep recurrent neural networks,” Nature Communications 10 (2019): 4809, https://doi.org/10.1038/s41467-019-12750-0. 16 Rajendra Joshi et al., “Machine Learning the Voltage of Electrode Materials in Metal-Ion Batteries,” ACS Applied Materials & Interfaces 13, no. 14 (2019), https://doi.org/10.1021/acsami.9b04933. 17 Peter M. Attia et al., “Closed-loop optimization of fast-charging protocols for batteries with machine learning,” Nature 578 (2020): 397-402, https://doi.org/10.1038/s41586-020-1994-5. 18Ali Madani et al., “ProGen: Language Modeling for Protein Generation,” bioRxiv, 2020, https://doi.org/10.1101/2020.03.07.982272.

 Center for Security and Emerging Technology | 12 19 Ayal B. Gussow et al., “Machine-learning approach expands the repertoire of anti-CRISPR protein families,” Nature Communications 11 (2020): 3784, https://doi.org/10.1038/s41467-020-17652-0. 20 Jonathan M. Stokes et al., “A Deep Learning Approach to Antibiotic Discovery,” Cell 180, no. 4 (2020): https://doi.org/10.1016/j.cell.2020.01.021. 21 Muhammad Kasim et al., “Building high accuracy emulators for scientific simulations with deep neural architecture search,” arXiv [stat.ML], arXiv (October 8, 2020), https://arxiv.org/pdf/2001.08055.pdf. 22 Casper Kaae Sønderby et al., “MetNet: A Neural Weather Model for Precipitation Forecasting,” arXiv [cs.LG], arXiv (March 30, 2020), https://arxiv.org/abs/2003.12140. 23 Azalia Mirhoseini et al., “Chip Placement with Deep Reinforcement Learning,” arXiv [cs.LG], arXiv (April 22, 2020), https://arxiv.org/pdf/2004.10746.pdf. 24 “Advanced Software Design Technology Leads GM into Next Generation of Vehicle Lightweighting,” General Motors, May 3, 2018, https://media.gm.com/media/us/en/gm/news.detail.html/content/Pages/news/us/en/2018/may/0503-lightweighting.html. 25 Daniel Oberhaus, “Massive, AI-Powered Robots Are 3D-Printing Entire Rockets,” WIRED, October 14, 2019, https://www.wired.com/story/massive-ai-powered-robots-are-3d-printing-entire-rockets/. 26 “Landing AI Unveils AI Visual Inspection Platform to Improve Quality and Reduce Costs for Manufacturers Worldwide,” Landing AI, October 21, 2020, https://landing.ai/landing-ai-unveils-ai-visual-inspection-platform-to-improve-quality-and-reduce-costs-for-manufacturers-worldwide/. 27 Research clusters (RCs) are groupings of scientific research articles linked by citation that provide a representation of related research. The articles used to identify clusters are included in CSET’s merged dataset of scholarly literature containing Dimensions (https://www.dimensions.ai/), Microsoft Academic Graph (https://www.microsoft.com/en-us/research/project/microsoft-academic-graph/), CNKI (http://online.eastview.com/projects/cnki/), and Web of Science (https://clarivate.com/webofsciencegroup/solutions/web-of-science/) data. More details can be found at: https://github.com/georgetown-cset/MI-for-Discovery-and-Innovation. 28 See Ilya Rahkovsky et al., “AI Research Funding Portfolios and Extreme Growth,” Frontiers in Research Metrics and Analytics (2021) 6 and James Dunham, Jennifer Melot, and Dewey Murdick, “Identifying the Development and
 Center for Security and Emerging Technology | 13 Application of Artificial Intelligence in Scientific Text,” arXiv [cs.DL], arXiv (May 28, 2020), https://arxiv.org/abs/2002.07143. 29 See “Topics Analytics,” Microsoft Academic, https://academic.microsoft.com/topics/. 30 The GitHub archive with related details is available at: https://github.com/georgetown-cset/MI-for-Discovery-and-Innovation. 31 More details can be found at https://github.com/georgetown-cset/MI-for-Discovery-and-Innovation. 32 Further, the diversity of recent major advances within CS beyond supervised learning, such as in generative modeling (e.g., OpenAI’s GPT-3) and deep reinforcement learning (e.g., DeepMind’s AlphaStar) suggest this trend of increased collaboration between applications of AI and science has more room to grow and will likely continue in the years ahead. 33 Office of Energy Efficiency and Renewable Energy, “Department of Energy Computational Science Graduate Fellowship,” U.S. Department of Energy, https://www.energy.gov/eere/education/events/department-energy-computational-science-graduate-fellowship. 34 “Plan S Principles,” Plan S, https://www.coalition-s.org/plan\_s\_principles/. 35 Michael Stebbins, “Expanding Public Access to the Results of Federally Funded Research,” Obama White House Blog, February 22, 2013, https://obamawhitehouse.archives.gov/blog/2013/02/22/expanding-public-access-results-federally-funded-research. 36 Center for Security and Emerging Technology, “COVID-19 Open Research Dataset (CORD-19)” (CSET, March 16, 2020), https://cset.georgetown.edu/research/covid-19-open-research-dataset-cord-19/. For statistics on CORD-19 current as of May 2021, see: https://www.kaggle.com/allen-institute-for-ai/CORD-19-research-challenge. 37 As an early example in this area, DARPA has funded work on physics-informed neural networks (PINNs), see Maziar Raissi, Paris Perdikaris, and George E. Karniadakis, "Physics-informed neural networks: A deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations," Journal of Computational Physics 378 (2019): 686-707, https://www.sciencedirect.com/science/article/pii/S0021999118307125?casa\_token=IgwKulORSTYAAAAA. 38 See http://www.image-net.org/.