Future Indices

OCTOBER 2020
Future Indices
AUTHORS
Michael Page
Catherine Aiken
Dewey Murdick How Crowd Forecasting
Can Inform the Big Picture
CSET Issue Brief
Center for Security and Emerging Technology | 2 Executive Summary
What will the world look like in 2025? How will changing geopolitical and
tech-security trends—such as U.S.-China relations, remote work, and public interest in automation—shape the world we occupy? These questions about tomorrow are on the minds of policymakers today. Presenting recent forecast
data collected through CSET’s Foretell project ( cset-foretell.com), this brief
shows how crowd forecasting can inform policy by providing data on future trends and linking those trends to future policy-relevant scenarios.
We illustrate Foretell’s methodology with a
concrete example: First, we describe three possible
scenarios , or ways in which the tech-
security landscape might develop over the next five years. Each scenario reflects different ways in which U.S.-China tensions and the fortunes of the artificial intelligence industry might develop. Then, we break each scenario down into near-term
predictors and
identify one or more metrics for each
predictor. We then ask the crowd to forecast the metrics. Lastly, we compare the crowd’s forecasts with projections based on historical data to identify
trend departures : the extent to
which the metrics are expected to depart from their historical trajectories.
Our preliminary findings suggest two outcomes —both involving increasing
U.S.-China tensions and Department of Defense AI R&D investments—are
most likely. Forthcoming data on commercial AI R&D investments,globalization, and industry-DoD tensions will inform which of these two
scenarios is more likely.
Foretell’s approach is a variation on a proposal by Philip E. Tetlock, co-
founder of the Good Judgment Project, which won the Intelligence Advanced
Research Projects Activity (IARPA)-funded Aggregative Contingency Estimation (ACE) forecasting tournament. We believe a scaled-up version of Foretell would contribute to a more evidence-based policymaking environment.

Center for Security and Emerging Technology | 3 Table of Contents
Executive Summary ............................................................................................... 2
Introduction ........................................................................................................... 4
What Will the Tech-Security Landscape Look Like in 2025? ........................... 5
Our Methodology ................................................................................................ 8
Step 1: Decompose scenarios into predictors ............................................... 8
Step 2: Identify metrics for the predictors ....................................................... 9
Step 3: Collect historical and forecast data ................................................. 10
Step 4: Estimate trend departure ................................................................... 10 Sensitivity to Technical Choices ..................................................................... 13
Looking Forward ............................................................................................ .....13
Acknowledgments ..............................................................................................15
Endnotes ............................................................................................................. 16

Center for Security and Emerging Technology | 4 Introduction Experts disagree about what the world will look like in five years. In terms of geopolitical competition, some experts predict an ascendant China,1 others predict the collapse of the Chinese Communist Party,2 and others predict a U.S.-China cold war.3 While expert predictions are indispensable, they are limited. It is difficult to build on them, locate where others might disagree, and identify what evidence supports or undermines them. Yet such predictions are critical tools to help frame policy debates. Therefore, greater insight on the relative likelihoods of qualitative expert predictions is of great value to policymakers. To address this gap, CSET launched Foretell, a pilot project that uses crowd forecasting and data analytics to inform tech-security policy.4 Our aim is to build on big picture scenarios, such as qualitative expert predictions, by making them more amenable to quantitative analysis. Building on previous research, notably IARPA’s ACE forecasting tournament, Foretell relies on the wisdom of the crowd—the collective opinion of a large group—to generate probabilistic forecasts on specific, near-term questions.5 But as noted by the founders of the winning ACE team, Philip E. Tetlock and Barbara Mellers, along with J. Peter Scoblic,“the specificity required to make questions rigorously resolvable precludes asking ‘big’ questions.”6 Tetlock, Mellers, and Scoblic were describing an example of the problem of measurement: the gap between what we can measure and what we want to measure. To generate quantitative insights into big-picture concepts such as “U.S.-China tensions” or “a strong tech sector,” we must identify observable metrics that approximate the concept. To address both the prediction and measurement problems, Tetlock proposed a series of tournaments to generate “clusters of short-term questions that, taken individually, are rigorously resolvable but that can collectively tip the scales of plausibility in high-stakes debates.”7 We are implementing a simplified version of this method on Foretell by selecting metrics that inform big picture scenarios and aggregating the extent to which, for each metric, the crowd forecasts depart from their historical trajectories (trend departure). Section I discusses an example application, linking three possible scenarios to a set of metrics for which we have forecast

Center for Security and Emerging Technology | 5 data. Section II overviews the methodology in more detail. Section III discusses where we go from here. What Will the Tech-Security Landscape Look Like in 2025? To launch Foretell, we developed three scenarios depicting what the tech-security landscape might look like in 2025.8 We focused on two important issues for tech-security policy, U.S.-China tensions and the fortunes of the U.S. artificial intelligence industry. Table 1 shows how by varying these issues, we could end up in very different worlds. Table 1. Three Possible 2025 Scenarios U.S.-China tensions increase U.S.-China tensions decrease AI industry booms Tense Economic-Security Balance Virtually Integrated AI industry declines Domestic & Securitized Omitted In the first scenario (Tense Economic-Security Balance), U.S.-China tensions have risen as the AI industry has grown, leading to recurring conflicts between economic and security goals and deep divisions between the Department of Defense and the AI industry. In the second scenario (Virtually Integrated), U.S.-China tensions have subsided as the AI industry has flourished, aided by the public’s heightened interest in automation and the development of remote, global workforces. In the third scenario (Domestic & Securitized), U.S.-China tensions have risen as the economy has deteriorated, leading to a less globalized economy and an AI industry more reliant on defense funding. We did not include a scenario in which tensions decrease and the AI industry declines because we determined, after consultation with experts, that this is an unlikely scenario. Each of these scenarios calls for a different policy approach. For example, a virtually integrated AI industry impacts the role of export controls, and a remote AI workforce impacts immigration policy. Because a variation on each scenario is possible, the best policy approaches are robust to all scenarios,

Center for Security and Emerging Technology | 6 but robustness comes at a cost. The more probable one of these scenarios becomes relative to the others, the more policy should target that scenario, and the less important it is to hedge on the others. The key question then is how likely the different scenarios are, and how we will know when relative likelihoods change. It’s difficult to assess the relative likelihoods of our 2025 scenarios for two reasons. First, they describe events three to seven years from now, and forecasting is most accurate over shorter time periods. And second, they are complex and not directly observable. Whether analyzing historical events or future events, quantitative methods require well-defined observables. To solve these problems, we broke each scenario down into predictors and metrics, and posed the metrics to the crowd as forecast questions. Section II describes this process in more detail (see Figures 1 and 2). We then used the crowd forecasts to identify trend departures, meaning areas in which the policy environment appears to be changing faster or slower than one would expect based on projections from historical data. Table 2 shows trend departures for our 2025 scenarios based on a sampling of the metrics and predictors identified for each scenario.

Center for Security and Emerging Technology | 7 Table 2: How Foretell aggregates crowd forecasts to inform big picture scenarios Metric [trend departure] Predictor [trend departure] Scenario Domestic & Securitized Virtually Integrated Tense Economic- Security Balance Decreasing U.S.-China trade9 0.9 Increasing U.S.-China tensions 0.6
✅ Increasing U.S.-China tensions are a predictor
❌ Decreasing U.S.-China tensions are a predictor
✅ Increasing U.S.-China tensions are a predictor Decreasing Chinese O visas10 1.1 Increasing unfavorable public view on China11 0.5 Increasing Chinese incursions of Japanese air space12 0 Increasing DOD AI R&D contracts13 0.4 Increasing DOD AI R&D investment 0.2
✅ Increasing DoD AI R&D investment is a predictor
❌ Decreasing DoD AI R&D investment is a predictor
✅ Increasing DoD AI R&D investment is a predictor Increasing DOD AI grants14 0 Increasing big tech revenue15 0.3 Increasing commercial AI R&D investment 0.2
❌ Decreasing commercial AI R&D investment is a predictor
✅ Increasing commercial AI R&D investment is a predictor
✅ Increasing commercial AI R&D investment is a predictor Increasing private tech fundraising16 0.3 Increasing machine learning job postings17 0 Increasing big tech H-1B visas18 -0.3 Increasing skilled-labor migration -0.3
✅ Decreasing skilled-labor migration is a predictor
❌ Increasing skilled-labor migration is a predictor Not a predictor Increasing remote software engineer jobs19 5.2 Increasing remote tech economy 5.2
❌ Stable or decreasing remote tech economy is a predictor
✅ Increasing remote tech economy is a predictor Not a predictor Source: Foretell. For the underlying data and model, see the Foretell GitHub repository, https://github.com/georgetown-cset/public-foretell.

Center for Security and Emerging Technology | 8 While these preliminary results are best understood in combination with more conventional analytical tools, the results are illustrative. They currently point to a close battle between the Domestic & Securitized scenario and Tense Economic-Security Balance scenarios, as reflected by the green check marks in Table 2. The crowd to-date expects U.S.-China tensions and DOD AI R&D investment to increase relative to historical trend projections, both of which are predictors of these scenarios and indicate movement away from the Virtually Integrated scenario. Meanwhile, a significant differentiator of the Domestic & Securitized and Virtually Integrated scenarios is commercial AI R&D investment, for which we presently see mild trend departure favoring Tense Economic-Security Balance. Forthcoming forecast questions on globalization and DoD-industry tensions will provide predictors that help identify which of these two scenarios is more likely. Our Methodology Below is the five-step process we are using to aggregate historical and forecast data to inform big picture scenarios, as illustrated in Table 2. Table 3 summarizes the steps. The data and model underlying Table 2 are available on GitHub.20 Table 3: Collecting and aggregating crowd forecast data Step 1: Decompose scenarios into predictors Step 2: Identify metrics for the predictors Step 3: Collect historical and forecast data Step 4: Estimate trend departure Step 5: Aggregate Break down a policy-relevant scenario into the trends that precede it Find one or more metrics that adequately capture each predictor For each metric, collect historical data points and ask the crowd to forecast future data points Compare crowd forecasts with projections from historical data; look for divergence Aggregate trend departure across metrics to inform the likelihood of policy-relevant outcomes Step 1: Decompose scenarios into predictors We begin with a big picture scenario: a description of the policy environment approximately three to seven years from now. Scenarios can be constructed in multiple ways. In the example in Section I, we used the 2x2 matrix

Center for Security and Emerging Technology | 9 technique. An alternative approach is to begin with qualitative expert predictions. For example, experts regularly make predictions about the future of U.S.-China relations,21 or whether we’re heading toward a high--tech dystopia.22 Such expert predictions can serve as starting points for the quantitative analysis described here. We then break the scenarios down into predictors: the near-term drivers of the scenarios of interest. Figure 1 shows the predictors for the Domestic & Securitized scenario from Section I. Figure 1. Breaking scenarios down into predictors Step 2: Identify metrics for predictors Because most predictors are not directly observable, we identify metrics that, alone or in combination with others, approximate the predictor. For example, quantity of AI publications is a common metric for quantity of AI research.23 For more complex predictors, multiple metrics can form an index that approximates the concept of interest. For example, as shown in Figure 2, the predictor of increasing U.S.-China tensions could be measured by trade levels, immigration flows, public opinion, and military actions.24 Figure 2. Identifying metrics for predictors

Center for Security and Emerging Technology | 10 Step 3: Collect historical and forecast data The next step is to gather data for the metrics. We first collect data on the metrics’ historical values. By projecting the historical values forward, we create a baseline for the trend departure measure discussed in Step 4. We then provide the historical values to the crowd and ask them to forecast the metrics’ future values. Foretell currently has about 1,000 registered forecasters, comprising primarily graduate students in relevant fields.25 Figure 3 provides an example of the historical data we collect and make available to the forecasters, supplemented with real-time forecast data. Figure 3: The data we collect and provide to forecasters
Step 4: Estimate trend departure At this point, we have two forecasts: one based entirely on historical data (historical projection) and the other from the crowd (crowd forecast). The difference between the two is the trend departure. Trend departure can be understood as a surprise factor, a signal of whether a metric’s value should cause an analyst to stop and reconsider their assumptions. Consider the U.S.-China trade metric in Figure 4. What’s noteworthy about 2019 is not its absolute value, $560 billion, or even that the 2019 value is 15 percent lower than the 2018 value. Rather, what’s noteworthy is that the 15 percent decrease in 2019 deviated so significantly from historical trends, coming in $109 billion below the historical projection.

Center for Security and Emerging Technology | 11 Figure 4. U.S.-China trade in 2019 was $109 billion below the historical projection
 The actual (solid blue) is U.S. Census Bureau data. The projection (solid red) is based on the AAA ETS (exponential smoothing) algorithm. The upper and lower bounds (dashed red) are that projection’s 95 percent confidence interval. The example in Figure 4 involves a historical data point, U.S.-China trade in 2019, but trend departure can be calculated in the same manner for forecasted data points. In the case of U.S.-China trade, as shown in Figure 5, the crowd forecasts a 2020 value of $491 billion and a 2021 value of $505 billion, $131 billion and $142 billion below the historical projections, respectively.

Center for Security and Emerging Technology | 12 Figure 5. The crowd forecasts significant trend departure in U.S.-China trade in 2021
 The actual (solid blue) is U.S. Census Bureau data, and the projection (dashed blue) is Foretell forecast data. The historical projection (solid red) is based on the AAA ETS (exponential smoothing) algorithm. The upper and lower bounds (dashed red) are that projection’s 95 percent confidence interval. Step 5: Aggregate Finally, we put all the pieces back together. To create a common scale across metrics with different levels of variation, we divide trend departure by the historical projection’s confidence interval. The confidence interval provides a measure of what range of values is expected in light of a metric’s historical values. If the historical trend is very consistent, the confidence interval will be small and therefore moderate departures surprising. That’s the case in Figure 4, which shows a $71 billion confidence interval. By contrast, if the historical values vary greatly, the confidence interval will be large and therefore moderate departures less surprising. That’s the case in Figure 5, which shows a larger $156 billion confidence interval for 2021 after taking into account the anomalous 2019 value. Therefore, although the absolute trend departure amount is greater in Figure 5 (2021 forecast) than in Figure 4 (2019 actual), after dividing by the confidence interval, the trend departure is greater in 2019 (1.5) than in 2021 (0.9). We can aggregate trend departure at the metric or predictor level. Table 1 shows aggregation at the metric level. For the U.S.-China Tensions predictor,

Center for Security and Emerging Technology | 13 for example, the crowd expects that each of the four metrics will increase relative to their historical trends, forecasting trend departures between 0 and 1.1. Taking the simple average of the four metrics yields a predictor-level trend departure of 0.6. Alternatively, we could give the metrics different weights. For example, if trade seems particularly important and Chinese incursions of Japanese airspace unimportant, we could give the former a weight of 50 percent, the latter a weight of 10 percent, and the others weights of 20 percent, which would yield a predictor-level trend departure of 0.8. Aggregating trend departure at the predictor level works in the same manner, meaning, in principle, we could quantify the extent to which the crowd thinks we are heading toward one scenario or another. Until we have more metrics and predictors to capture the scenario of interest, however, adding a second level of quantification magnifies sources of error—such as the selection and weighting of predictors and metrics—without adding offsetting insight. Sensitivity to Technical Choices This brief describes our methodology at a conceptual level, but to implement it, we made many technical choices, such as what algorithm to use to create the historical projection, how to calculate trend departure, and how to standardize trend departure values across metrics with different levels of historical variation. Our underlying data and the model used to generate our results is publicly available and we encourage others to improve upon our technical choices. Ultimately, however, many of these technical choices are incidental to the results. A virtue of our focus on trend departure is that we are interested only in big changes, and the big changes should not be sensitive to debatable technical choices. Looking Forward We believe using crowd forecasting to inform big picture scenarios can improve policymaking in two ways. First, it can foster productive disagreement by helping policy analysts identify where they disagree and what data would advance the debate. Policy disagreements might be more manageable once reduced to specific, measurable uncertainties, such as the effect an export control would have on U.S. semiconductor manufacturers.

Center for Security and Emerging Technology | 14 Second, it could serve as a warning system. In a complex, dynamic environment, it can be difficult to appreciate the significance of ongoing change. For example, two policymakers might disagree about whether we are heading toward the Domestic & Securitized scenario described in Section I, but agree about what to do if we are headed toward that scenario. By continually monitoring and forecasting dozens of metrics that inform the likelihood of the Domestic & Securitized outcome, we can effectively automate our ability to notice changes that should trigger a reconsideration of strategic policy. Many potential obstacles remain. Among others, this methodology assumes forecasters are acting in good faith. If instead, forecasters attempt to manipulate the results to achieve their desired policy goals, they would undermine the integrity of the system. We believe such risks are best addressed in concrete cases, and in the abstract, do not pose enough risk to offset the potential upside. Foretell is still at the proof-of-concept stage. Over the remainder of its pilot, we will study the crowd’s accuracy relative to subject-matter experts and see what insights we can extract from other crowd-generated data, including rationales and the full distribution of probabilities, rather than simply the point estimates. However, for this method to realize its ultimate potential, scale is necessary. With sufficient policymaker interest, we believe a scaled-up forecasting project—with more metrics, forecasters, and end-use applications—will flourish.
Center for Security and Emerging Technology | 15 Acknowledgments
We gratefully acknowledge input from James Dunham, Robert Cardillo,
Alexander Friedland, Robert Hartman, Carolyn Mehta, Igor Mikolic-Terreira,
Luke Muehl auser, Vanessa Pineda, Ben Roesch, Peter Scoblic, Adam Siegel,
Philip Tetlock, Helen Toner
, and Lynne Weil. We thank Cultivate Labs—and
Vanessa Pineda and Carolyn Mehta in particular—for creating and
maintaining the forecasting platform. The authors are solely responsible for
the views expressed in this piece and for any errors.
© 2020 by the Center for Security and Emerging Technology. This work is
licensed under a Creative Commons Attribution-Non Commercial 4.0
International License.
To view a copy of this license, visit
https://creativecommons.org/licenses/by-nc/4.0/.
Document Identifier: doi: 10.51593/20200093

Center for Security and Emerging Technology | 16 Endnotes
1Michael Morell, “Analysis: The national security implications of COVID -19,” C BS News,
May 8, 2020, https://www.cbsnews.com/news/coronavirus -national -security-
implications -analysis/ .
2 Minxin Pei, “China’s Coming Upheaval,” Foreign Affairs, May/June 2020,
https://www.foreignaffairs.com/articles/united-s tates/2020-0 4-03/chinas-c oming-
upheaval.
3 John Haltiwanger, “The US and China are on the brink of a new Cold War that could
devastat e the global economy,” Business Insider, May 14, 2020,
https://www.businessinsider.com.au/the -us-china -entering- new-cold-war-amid -
coronavirus -2020-5 .
4 To lear n mor e about Foretell, c heck out cset -foretell.com . Foretell’s approa ch is a variation
on a proposal by Phili p Tetlock, co -founder o f the G ood J udgment Project, which w on th e
IARPA -funded A ggregative C ontingent Estimation (A CE) forecasting to urnament.
5 ACE was a four -year tournament designed to identif y what methods ar e most e ffective a t
estimati ng the likeli hood of geopolitical events betw een one mont h and one year i n the
future. The G ood J udgment Project, a te am le d by University o f Pennsylvania p rofessors
Phillip Tetlo ck and Barbara Mellers, wo n the tournament b y recruiting thousands o f
volunteers, training them to forec ast, and aggregating their judgments. Tetlo ck and Dan
Gardner describe t heir approach i n Tetlock, Gardner, Superforecasting: T he Art and Science
of Prediction (Broadway B ooks, 2015).
6 Tetlo ck et al., “Bringi ng probabilit y judgments into poli cy debates via forecasting
tournaments,” S cience V ol 355 Issue 6 324, (February 5 , 2017), 481 -483,
https://science.sciencemag.org/content/355/6324/481. full. See a lso S coblic & Tetlock,
“A Better Crystal Ball: The Right Wa y to Thi nk About the Future, ” Foreig n Affairs,
November/December 2000, https://www.foreignaffairs.com/articles/united -
states/2020 -10-13/better -crystal -ball.
7 Philip E. Tetlock, “Full -Inference -Cycle Tournaments: The Quality of our Questions Matters
as Much as th e Accuracy of our Answers,” Prepared for IARPA, August 30, 2017 , available
through D ropbox a t https://t.co/dLO0CXac8A?amp=1 . Scoblic & T etlock, supra n ote 6 ,
makes a similar proposal, suggesting ho w qualitative scenario planning and probabilistic
forecasting c an be combined to c reate w arning s ystems fo r policymakers.
8 Aiken, Page, “Three Possible 2025 Worlds t hat Should Inform Policy Today,” Foretell, June
11, 2020, https://www.cset -foretell.com/blog/three\_worl ds.

Center for Security and Emerging Technology | 17 9 Forecast data is available on Foretell, https://www.cset-foretell.com/questions/76-what-will-total-u-s-trade-volume-with-china-imports-and-exports-of-goods-be-in-2021. Data on a related question covering an earlier time period is also available, https://www.cset-foretell.com/questions/22-how-much-total-trade-will-there-be-between-the-u-s-and-china-in-q4-2020. 10 Forecast data is available on Foretell associated with two questions, both conditional on different election outcomes. See https://www.cset-foretell.com/questions/78-conditional-on-trump-winning-the-election-what-percentage-of-o-visas-will-go-to-chinese-nationals-in-fy-2021-october-1-2020-to-september-30-2021; https://www.cset-foretell.com/questions/79-conditional-on-biden-s-election-what-percentage-of-o-visas-will-go-to-chinese-nationals-in-fy-2021-october-1-2020-to-september-30-2021. Data on a related question covering an earlier time period is also available, https://www.cset-foretell.com/questions/47-what-percent-of-o-1-visas-issued-between-june-1-and-september-30-2020-will-go-to-people-from-china. 11 Forecast data is available on Foretell, https://www.cset-foretell.com/questions/80-what-percentage-of-u-s-citizens-will-have-an-unfavorable-view-of-china-as-reflected-in-the-2021-pew-global-attitudes-survey. 12 Forecast data is available on Foretell, https://www.cset-foretell.com/questions/32-how-many-sorties-against-chinese-aircraft-will-the-japanese-air-self-defense-force-conduct-by-april-30th-2020. 13 Forecast data is available on Foretell, https://www.cset-foretell.com/questions/61-how-much-will-the-u-s-department-of-defense-spend-on-ai-research-contracts-between-july-1-and-december-31-2020-inclusive. 14 Forecast data is available on Foretell, https://www.cset-foretell.com/questions/40-how-much-will-dod-spend-on-ai-grants-between-june-1-and-december-31-2020. 15 Forecast data is available on Foretell, https://www.cset-foretell.com/questions/74-what-will-the-combined-revenue-of-alphabet-amazon-apple-facebook-and-microsoft-be-in-the-first-two-quarters-january-1-through-june-30-of-2021. Data on a related question covering an earlier time period is also available, https://www.cset-foretell.com/questions/63-what-will-the-revenue-of-alphabet-amazon-apple-facebook-and-microsoft-be-in-the-third-quarter-of-2020-ending-september-30. 16 Forecast data is available on Foretell, https://www.cset-foretell.com/questions/69-how-much-new-funding-will-private-u-s-tech-companies-raise-between-july-1-and-december-31-2020-inclusive. 17 Forecast data is available on Foretell, https://www.cset-foretell.com/questions/82-how-many-postings-for-u-s-jobs-requiring-machine-learning-skills-will-be-published-between-january-1-and-march-31-2021-inclusive. Data on a related question covering an earlier time period is also available, https://www.cset-foretell.com/questions/19-how-many-new-u-s-jobs-requiring-machine-learning-skills-will-be-advertised-in-q3-2020.

Center for Security and Emerging Technology | 18 18 Forecast data is available on Foretell, https://www.cset-foretell.com/questions/23-how-many-new-h-1b-applications-submitted-by-the-big-5-tech-companies-will-the-u-s-grant-in-fy-2020. 19 Forecast data is available on Foretell, https://www.cset-foretell.com/questions/72-what-percentage-of-software-engineer-job-postings-between-january-1-and-march-31-2021-inclusive-will-allow-for-remote-work. Data on a related question covering an earlier time period is also available, https://www.cset-foretell.com/questions/65-what-percentage-of-software-engineer-job-postings-between-july-1-and-september-30-2020-inclusive-will-allow-for-remote-work. 20 See https://github.com/georgetown-cset/public-foretell. 21 See the examples in the introduction, notes 1-3. 22 See, e.g., Naomi Klein, “Screen New Deal: Under Cover of Mass Death, Andrew Cuomo Calls in the Billionaires to Build a High-Tech Dystopia, Intercept, May 8, 2020, https://theintercept.com/2020/05/08/andrew-cuomo-eric-schmidt-coronavirus-tech-shock-doctrine/. To see how Klein’s forecast can be operationalized on Foretell, see Maia Nikoladze, “Tech Companies Threaten Democracy,” Foretell Blog, July 30, 2020, https://www.cset-foretell.com/blog/tech-threat. 23 See, e.g., Cady, Etzioni, “China May Overtake US in AI Research,” Medium, March 13, 2019, https://medium.com/ai2-blog/china-to-overtake-us-in-ai-research-8b6b1fe30595. 24 See, e.g., Castro et al., “Who Is Winning the AI Race: China, the EU, or the United States?,” Center for Data Innovation, August 19, 2020, https://www.datainnovation.org/2019/08/who-is-winning-the-ai-race-china-the-eu-or-the-united-states. 25 CSET works with Cultivate Labs to recruit and maintain a pool of forecasters. To learn more about Foretell’s forecasters, see our fall cohort of forecaster ambassadors at https://www.cset-foretell.com/our-ambassadors.


Hacking AI

Hacking AI
A PRIMER FOR POLICYMAKERS
ON MACHINE LEARNING CYBERSECURITY
DECEMBER 2020AUTHOR
Andrew J. Lohn
Center for Security and Emerging Technology2Established in January 2019, the Center for Security and
Emerging Technology (CSET) at Georgetown’s Walsh
School of Foreign Service is a research organization fo-
cused on studying the security impacts of emerging tech-
nologies, supporting academic work in security and tech-
nology studies, and delivering nonpartisan analysis to the
policy community. CSET aims to prepare a generation of
policymakers, analysts, and diplomats to address the chal-
lenges and opportunities of emerging technologies. During
its first two years, CSET will focus on the effects of progress
in artificial intelligence and advanced computing.
CSET.GEORGETOWN.EDU | CSET@GEORGETOWN.EDU

Hacking AIDECEMBER 2020
AUTHOR
Andrew J. LohnA PRIMER FOR POLICYMAKERS
ON MACHINE LEARNING CYBERSECURITY
ACKNOWLEDGMENTS
The author would like to thank Ben Buchanan and John Bansemer for
their support and guidance. Thanks are also due to Chris Rohlf, Jeff Alstott, and Tantum Collins for helpful critiques and comments.
The author is solely responsible for the views expressed in this piece
and for any errors or omissions.

PRINT AND ELECTRONIC DISTRIBUTION RIGHTS
© 2020 by the Center for Security and Em erging Technology.
This work is licens ed under a Creative C ommons Attribution-
NonCommercial 4.0 In ternational License .
To view a copy of thi s license, visit:
https://creativecommo ns.org/licenses/by -nc/4.0/.
Document Identifier: doi: 10.51593/2020CA006
Cover background: Pixabay/ims66.
Cover Icon: Noun Project/Chameleon Design, IN.
Iconography: Noun Project.
Center for Security and Emerging TechnologyiEXECUTIVE SUMMARY
INTRODUCTION
1 | THE BASICS OF MACHINE LEARNING
2 | COMMON TYPES OF ATTACKS
3 | THE RANGE OF POSSIBLE OF ATTACKS
4 | ASSESSING THE THREAT: EASE, PERVASIVENESS,
 AND DEFENSES
5 | CONCLUSION AND RECOMMENDATIONS
ENDNOTES III
V
1
5
11
13
17
21Contents
Center for Security and Emerging Technologyiv
Center for Security and Emerging Technologyiiirtificial intelligence is vulnerable to cyber attacks. Machine
learning systems—the core of modern AI—are rife with vulnera-
bilities. Attack code to exploit these vulnerabilities has already
proliferated widely while defensive techniques are limited and struggling
to keep up. Machine learning vulnerabilities permit hackers to manipu-
late the machine learning systems’ integrity (causing them to make mis-
takes), confidentiality (causing them to leak information), and availability
(causing them to cease functioning).
These vulnerabilities create the potential for new types of privacy
risks, systemic injustices such as built-in bias, and even physical harms.
Developers of machine learning systems—especially in a national security
context—will have to learn how to manage the inevitable risks associated
with those systems. They should expect that adversaries will be adept at
finding and exploiting weaknesses. Policymakers must make decisions
about when machine learning systems can be safely deployed and when
the risks are too great.
Attacks on machine learning systems differ from traditional hacking ex-
ploits and therefore require new protections and responses. For example,
machine learning vulnerabilities often cannot be patched the way tradi-
tional software can, leaving enduring holes for attackers to exploit. Even
worse, some of these vulnerabilities require little or no access to the victim's
system or network, providing increased opportunity for attackers and less
ability for defenders to detect and protect themselves against attacks.
Accordingly, this paper presents four findings for policymakers’
consideration:
• Machine learning introduces new risks: Using machine learn-
ing means accepting new vulnerabilities. This is especially true in Executive Summary
A
Center for Security and Emerging Technologyivthe context of national security, but also in critical infrastructure, and even in
the private sector. However, this does not mean machine learning should be
prohibited. Rather, it is incumbent upon policymakers to understand the risks
in each case and decide whether they are outweighed by the benefits.
• New defenses may only offer short-term advantage: Attackers and
defenders of machine learning systems are locked in a rapidly evolving cat-
-and-mouse game. Defenders appear to be losing; their techniques are cur-
rently easily defeated and do not seem well-positioned to keep pace with
advances in attacks in the near future. Still, defensive measures can raise the
costs for attackers in some narrow instances, and a proper understanding of
machine learning vulnerabilities can aid defenders in mitigating risk. None -
theless, the effectiveness of defensive strategies and tactics will vary for
years and will continue to fail at thwarting more sophisticated attacks.
• Robustness to attack is most likely to come from system-level defens-
es: Given the advantages that attackers have, for machine learning systems
to function in high-stakes environments, they must be built in with greater
resilience than is often the case today. To aid this effort, policymakers should
pursue approaches for providing increased robustness, including the use of
redundant components and ensuring opportunities for human oversight and
intervention when possible.
• The benefits of offensive use often do not outweigh the costs: The
United States could employ the types of attacks described in this primer to
good effect against adversaries’ machine learning systems. These offensive
techniques could provide another valuable arrow in the U.S. national se -
curity community’s quiver and might help prevent adversaries from fielding
worrisome AI weapons in the first place. On the other hand, the United
States can lead by setting norms of restraint. The United States must also be
cautious to ensure its actions do not alienate the community that is develop-
ing these technologies or the public at large who rely on machine learning.
Machine learning has already transformed many aspects of daily life, and it is
easy to see all that the technology can do. It likewise offers the allure of reshaping
many aspects of national security, from intelligence analysis to weapons systems and
more. It can be hard, however, to perceive machine learning’s limitations, especially
those—like its susceptibility to hacking—that are most likely to emerge in highly con-
tested environments. To better understand what the technology can and cannot do,
this primer introduces the subject of machine learning cybersecurity in a detailed but
non-technical way. It provides an entry point to the concepts and vocabulary need-
ed to engage the many important issues that arise and helps policymakers begin the
critical work of securing vital systems from malicious attacks.
Center for Security and Emerging TechnologyvIntroduction
s he removes his hands from the steering wheel and leans back,
the driver becomes a passenger. Now under its own control, the
car accelerates toward the skyscrapers in the distance, yet to
notice a small, innocuous-looking sticker on the road ahead. If spotted
at all, the sticker might be confused for a paint smudge. Suddenly the car
swerves left. Alarms sound and warnings flash. Then, a voice speaking in
Chinese backed by ominous music sheds light on what is happening: the
machine learning system in the car has been hacked.
In 2019 Tencent, a leading Chinese technology company unveiled a
set of three attacks against Tesla automobiles and posted a video demon -
strating them.1 Two of the attacks were directed at machine learning com-
ponents, the second of which made the car veer while driving. The fact that
AI can be hacked in this way comes as little surprise to researchers who
study machine learning cybersecurity, but the subject receives insufficient
attention in national security circles. That situation must end.
Machine learning is starting to deliver on promises of enhanced
support to the warfighter, to reconnaissance teams, and in streamlined
operations and logistics.2 It is increasingly becoming a predominant, albeit
hidden, force in the daily lives of many Americans. It will increasingly
route and control the vehicles on our roads and secure and manage our
homes by interpreting our voice commands. Lying dormant in those systems
are vulnerabilities that are different from the traditional flaws with which
we have decades of experience. These vulnerabilities are pervasive and
inexpensive to exploit using tools that have proliferated widely and against
which there is often little defense.A
Center for Security and Emerging TechnologyviThis report summarizes and contextualizes machine learning vulnerabilities for
policymakers, providing a starting point for familiarizing themselves with the broad
set of concepts and potential concerns. These concepts have broad applicability,
since machine learning affects society in many ways; its vulnerabilities create the
potential for new types of privacy leaks, injustices, and even physical harm. This
report briefly describes some of the most popular types of attacks and then discuss-
es the range of possibilities. It also highlights the pervasiveness of the vulnerabilities,
the ease of exploiting them, and the state of defenses. First, though, it offers a primer
on the basics of machine learning.
Center for Security and Emerging Technology1achine learning systems use computing power to execute algo-
rithms that learn from data. These systems learn patterns that they
then use to make classifications or predictions. Together, the com-
ponents of machine learning enable systems that have proven remarkably
adept in a wide variety of fields, including automated imagery intelligence
analysis important for national security. Understanding the threats against
machine learning requires only a cursory understanding of the technology.
THE MODEL
The centerpiece of machine learning is the “model” itself. The model could
be a neural network or a list of yes/no questions or a variety of other
possible techniques, some of which have not been invented yet. The model
is composed of anywhere from a few to hundreds of billions of parameters
that can each be adjusted to make it more accurate. In one type of model
the parameters might be basic yes/no questions, like “Did the number
of tweets with #overthrowthegovernment exceed 40,000 last week?” In
complex models, such as neural networks, the parameters can instead rep-
resent the strength of connections between neurons. For our purposes, only
one principle matters: the machine learning model will be more accurate if
the parameters are tuned well and inaccurate if they are not. This is where
the process of training comes in.
TRAINING
Data fuels machine learning systems; the process of training shows how.
This process is shown on the left side of Figure 1, illustrating explicitly
that the data has to be collected or mined from somewhere, such as a
surveillance drone, a Twitter feed, or computer data. During training, the
machine learning system extracts patterns from this data. The system learns The Basics of
Machine Learning1
M
Center for Security and Emerging Technology2by adjusting the parameters of its model to correspond to these patterns. Different
kinds of machine learning systems learn from data in distinct ways, but the idea of
matching the model to the training data generally holds. Crucially, the system does
not know which patterns are desirable to learn and which, like those corresponding to
human biases, are not. It simply learns everything it can from the data. Once training
is complete, the model can be used as a component in a larger system, as shown to
the right in Figure 1. In some cases, the model can continue to be updated and trained
while deployed for use, but in other cases it is frozen as is before it is deployed.
USE AS PART OF A DEPLOYED SYSTEM
The deployed system may use many models to perform similar tasks and could
have humans involved at various stages. Autonomous cars, for example, collect
data from the environment through video or radar and use it to decide whether to
brake, accelerate, or turn. There might be one machine learning model analyzing
video and another analyzing the data from the car’s laser or radar sensors, while
still another model synthesizes information from several data sources and makes
a decision. And there may be a human driver who can choose to accept or reject
the decision before it becomes an action. In essence, the machine learning system
uses the models to convert real-world input data, which is hopefully similar to the
training data, into decisions and then actions.
INPUTS
DATA STORAGEDATA COLLECTION
TRAININGMODEL A
HUMAN-IN-THE-LOOPDECISION MODEL B ACTION
MACHINE LEARNING DEPLOYED SYSTEMFIGURE 1
Once a machine learning model is trained, it becomes part of a larger
system that converts inputs to decisions and subsequently into actions.
Center for Security and Emerging Technology3The notion that the machine learning model is part of a larger system highlights
the main challenges attackers face. A properly designed system creates redundan-
cies that guard against bad outcomes. For example, if attackers cause the computer
vision system to interpret a stop sign as a 45 mph speed limit sign, system-level
defenses could avert a catastrophe. The car might still decide to stop at the sign if its
laser or radar detects crossing traffic or if the car has been instructed to never cross
a blind intersection at high speed.3 Moreover, if a human is present, he or she may
be able to notice and override peculiar inputs or abnormal decisions. Even with
failsafes, though, attackers can successfully compromise multiple systems or identify
single points of failures. To figure out how to stop such threats, it is essential to under-
stand how they work.
Center for Security and Emerging Technology4
Center for Security and Emerging Technology5n cybersecurity, possible harms are typically grouped into three
broad categories represented by the acronym CIA: confidentiality,
integrity, and availability. All three categories also apply to ma-
chine learning. Integrity attacks alter data to cause machines to make er-
rors and have attracted the most attention. Confidentiality attacks extract
information meant to remain hidden; they also garner notable research
focus. Availability attacks cause the machine learning component to run
slowly or not at all. While availability attacks are starting to attract more
attention, they have been the least popular.4 This section will only discuss
integrity and confidentiality attacks.
INTEGRITY ATTACKS
While there are many ways to cause the machine learning model to make
errors,\* two approaches stand out as the most popular: data poisoning
and evasion. Both target different parts of the machine learning process,
as shown in Figure 2. In “data poisoning,” attackers make changes to the
training data to embed malicious patterns for the machine to learn. This
causes the model to learn the wrong patterns and to tune its parameters
in the wrong way. In “evasion,” attackers discover imperfections in the
model—the ways in which its parameters may be poorly tuned—and then
exploit these weaknesses in the deployed model with carefully crafted in-
puts.† Even the most well-trained models have a seemingly infinite supply
of these imperfections that allow the model to be turned against itself.Common Types
of Attacks2
I
\*Attackers could retune the model’s parameters individually or they could create bugs in
parts of the software that are used to tune the parameters, for example.
† This report only discusses evasion and data poisoning attacks in the context of integrity
attacks, but they can also be used in confidentiality and availability attacks.
Center for Security and Emerging Technology6Neither data poisoning nor evasion require directly breaking into the ma-
chine learning system. This means that attackers can manipulate machine learn-
ing systems even if they are unable to tamper with the system itself. For example,
attackers might not need to get their hands on a spy drone to cause it to misiden-
tify its targets. Instead, they might make educated guesses about the model of
the drone’s machine learning system and break into the company that designs
the drone to uncover the model. Attackers might even alter the publicly available
data that software developers often use as the foundation or starting points for
their models.5 The range of possibilities underscores the fact that are many options
for attackers to manipulate machine learning systems that do not require directly
observing or breaking into the target.
Data Poisoning Attacks
A machine learning model tries to find patterns in the data; if an attacker can
control the data, they can control what the model learns. In some cases, just a few
changes to the data can implant something akin to a Pavlovian bell, causing the
machine learning system to respond to a particular input in a certain way.6 Further,
the poisoned data patterns do not have to make sense to a human, making them
INPUTS
DATA STORAGEDATA COLLECTION
TRAININGMODEL A
HUMAN-IN-THE-LOOPDECISION MODEL B ACTION
MACHINE LEARNING DEPLOYED SYSTEMDATA POISONING EVASIONFIGURE 2
Data poisoning attacks manipulate the training data from which a model
learns, while evasion attacks control the inputs to the deployed system to
exploit pre-existing weaknesses.
Center for Security and Emerging Technology7easy for attackers to hide or disguise. As a hypothetical example, imagine an auto-
matic order placement system for a manufacturing company that uses past data on
monthly demand to send the right number of parts to each factory the next month. An
adversary could poison the dataset so that when more than 10,000 screws are sent
to Kazakhstan then in the next month, only half the usual amount of oil for tanks is sent
to Syria. If the attacker can send extra screws to Kazakhstan, he or she could cause a
shortage of tank oil in Syria, which could reduce the effectiveness of operations.
There are many opportunities for attackers to supply the system with data
intended to subvert the model. They could hack into the victim’s servers to change
the database or they could trick the victim into downloading a malicious datapoint
when they are updating their model. These methods often take advantage of tradi-
tional offensive cyber techniques. For example, attackers could break into a victim’s
network and manipulate the data stored within it.
A less intrusive means of data poisoning involves creating false precedent. For
example, if an attacker would like to use a piece of malware but is worried that a
machine learning-based antivirus program will detect them, the attacker can first
distribute a similar but benign piece of code. The antivirus might learn that code with
those characteristics is safe and therefore think that the malware is also safe once it
is released. Data poisoning attacks like these are possible for nearly all models.
Evasion Attacks
The most common type of attack against machine learning systems is known as an
evasion attack. In these operations, the attacker makes changes to the inputs that
are so subtle humans have trouble noticing them but are significant enough for a
machine to change its assessment; these inputs are often “adversarial examples.”
To demonstrate how easy it is to perform this type of attack, we created one.
We began with the picture of Georgetown University’s iconic Healy Hall (Figure
3). A common image recognition system identified with 85.8 percent confidence that
the picture was of a “castle.” This is a good guess because the system, which is basic,
was not programmed to recognize schools or universities. Using openly published
techniques, our attack program made a series of small changes to the image to trick
the machine learning system into identifying Healy Hall as a triceratops dinosaur.7
Human eyes would find the changes difficult to notice, but they were tailored to trick
the machine learning system. Once all the changes were made, the picture looked the
same to the human eye, but the machine was 99.9 percent sure the picture was of a
triceratops (Figure 3). This is the power of adversarial examples in action.\*
\*The image classifier was MobileNetV2 attacked using projected gradient descent with a l∞-norm
bound run on Google Colab’s free GPU. Many other techniques and models and runtimes were also
tried and could have been used interchangeably.
Center for Security and Emerging Technology8Image classification algorithms are among the greatest triumphs of machine
learning, so attacks like these are striking. While these kinds of attacks are easiest to
visualize when they manipulate images, they also affect machine learning systems
that perform other tasks such as voice recognition. Evasion attacks do not need to
be as subtle as the Healy Hall example above, but can more significantly manipu-
late the input given to a machine learning system.
CONFIDENTIALITY ATTACKS
In a confidentiality attack, attackers observe how machine learning systems re -
spond to different kinds of inputs. From this observation, attackers can learn infor-
mation about how the model works and about its training data. If the training data is
particularly sensitive—such as if the model is trained on classified information—such
an attack could reveal highly sensitive information. In essence, the machine learning
system learns from the training data and might unintentionally reveal what it knows
to others. There are three main kinds of confidentiality attacks.
Model Extraction
The easiest type of attack to understand is “model extraction.” By recording the
inputs and outputs of the victim model enough times, the attacker can build a close
facsimile of the model to be attacked. Model extraction poses two risks. First, steal-
ing the model provides the attacker with a copy that the victim may not have want-
FIGURE 3
Classification of Georgetown’s Healy Hall unperturbed on top and
attacked to appear to a machine learning system to be a triceratops
on bottom. To human eyes, the two images look identical.
ORIGINAL IMAGE
Castle: 85.8%
Palace: 3. 17%
Monastery: 2.4%
ATTACKED IMAGE
Triceratops: 99.9%
Barrow: 0.005%
Sundial: 0.005%
Center for Security and Emerging Technology9ed to share, revealing information about how the machine learning system works.
Second, and more significantly for the purposes of cybersecurity, stealing a model
facilitates all the other attacks discussed in this report. The understanding of how a
system works makes it easier to determine how a system may be compromised.
Membership Inference
In a membership inference attack, the attacker studies the machine learning sys-
tem’s inputs and outputs and learns details about the data on which the model was
trained. For example, imagine a company that offers customers a medical diagno-
sis after they answer a list of questions about themselves and their symptoms. The
company would want to protect the data used to build their model both for intellec-
tual property reasons and because it contains sensitive medical information about
the participants. The company could delete its copies of patient data after training
the model, but this may not be enough to guarantee confidentiality, since the model
itself has learned information about the patients and the model is subject to mem-
bership inference attacks.
To carry out such an attack, attackers often consider a model’s confidence
rating—how sure it is that its output is correct. Machine learning models are often
overconfident when they see real world examples that match those provided during
training. For example, a machine learning system is likely to be more confident
about John Doe’s medical data if his information was used to train the model; it will
be less confident about Jane Doe, whose data was not used to train the model,
even if the symptoms of the two patients are similar. Based on the higher confidence
rating, the attacker might conclude that John was in the original dataset and thereby
learn his sensitive medical history.
Model Inversion
Instead of looking for individual pieces of data, with model inversion attackers try to
understand more about the model’s output categories. For a facial recognition sys-
tem that takes a facial image as an input, the output categories are people’s names.
In model inversion, the attacker tries to do the opposite. In the facial recognition
case, that means starting with a target’s name and trying to produce images of the
corresponding face.8
Attackers do not run the model in reverse. Instead, they start with a randomly
generated image and make small changes to that image that make the model a little
more likely to label the image as the target’s face. These machine learning attacks
are the rough equivalent of a police sketch artist slowly building a composite image
of a suspect.
With enough small adjustments—and continual feedback from the model’s
evaluation of each iteration of the model—attackers can eventually draw a com-
Center for Security and Emerging Technology10plete picture of the face. Model inversion is not limited to faces or pictures, though.
Models of all types can be inverted, such as inferring a person’s purchasing tenden-
cies from a fraud detection model.
INPUTS
MODEL INVERSIONMEMBERSHIP
INFERENCE
TRAININGMODEL A
HUMAN-IN-THE-LOOPDECISION MODEL EXTRACTION ACTION
MACHINE LEARNING DEPLOYED SYSTEMATTACKEROBSERVE INPUTSOBSERVE
ACTIONSFIGURE 4
Popular confidentiality attacks require only that the attacker observe the
inputs and the outputs of the deployed system. They can then extract
models or invert the model to learn about the output categories, such as by
exploiting a facial recognition system and a name to draw a person’s face.
They can also perform a membership inference attack to learn specific
traits of the data.
Center for Security and Emerging Technology11ttackers must make a range of choices about when and how
to carry out their operations. They can direct different kinds of
attacks at different stages of the machine learning process, from
data collection to deployment. For example, attackers can use both data
poisoning and evasion attacks to cause misclassification, but poisoning
attacks target the training process whereas evasion targets the model
after deployment.
Attackers can also choose how aggressive or stealthy to be. For
example, evasion attacks have different degrees of subtlety.9 At one ex-
treme, the attackers can create whatever inputs they want, such as random
squiggles to try to bypass a cellphone’s facial recognition system. Those
attacks are obvious to a human inspecting the images either in real-time or
forensically after an attack. At the stealthier extreme, attackers may restrict
themselves to changes that are indistinguishable to humans, as the Healy
Hall image demonstrates. Even just within the category of indistinguishable
attacks, there are many options. For example, to attack an image detection
system, attackers may limit the number of pixels that can be changed, the
amount of change per pixel, or the average change across all pixels, and
so forth. The technical details of how these attacks can be most efficient are
not necessary to gain a conceptual understanding; we mention them only
to suggest that there are many options for the attackers to choose from.The Range
of Possible Attacks3
A
Center for Security and Emerging Technology12As Figure 5 shows, in the Healy Hall example we chose to carry out an attack
on integrity that forced a system to misclassify an image, and we carried out the
attack against a machine learning system that was deployed by indistinguishably
manipulating an image and only slightly changing a large number of pixels. In re -
sponse to different operational priorities, we might have used an alternative attack
design, such as making larger changes to a smaller number of pixels or worrying
less about making the original image and the manipulated image look more alike.
We could have achieved a similar effect by data poisoning instead. Still other kinds
of attacks would have targeted confidentiality and extracted information about the
model rather than manipulating its outputs.
Adversarial attacks are powerful, counterintuitive, and subtle. There are many
tools available online for carrying out these kinds of operations, which are expect-
ed to become increasingly common. That said, there is a risk of focusing too much
on adversarial examples and neglecting other serious types of attacks against
machine learning systems. Recognizing the range of possible threats should lead to
a broader and more robust defense of machine learning systems; to best inform this
defense, we need to examine the ease, persuasiveness, and severity of the threats. EFFECT
CATEGORYSPECIFIC
EFFECTML STAGE
ATTACKEDCONSTRAINT
CATEGORYSPECIFIC
CONSTRAINT
Confidence
Reduction
Max change
per pixelAvailability
Integrity
Confidentiality...
...
.........Misclassification
DeploymentIndistinguishable
Quantitative
ErrorData Collection
Training
TestingContent-
Preserving
Content-
Constrained
Non-Suspicious
UnconstrainedSum of
all changes
Number of
pixels changedFIGURE 5
Attackers have many options to choose from when they target machine learning
systems. The blue path in Figure 5 shows the attack used to make Figure 3, in
which Georgetown University’s Healy Hall was turned into a triceratops (at least
according to a machine learning system).
Center for Security and Emerging Technology13he attacks described in this report are likely to become common in
the future. They are easy to conduct, the vulnerabilities they exploit
are pervasive, and the attacks are difficult to defend against. The
combination of these three characteristics means that managing machine
learning vulnerabilities is a complex problem, even when compared to
other problems in cybersecurity.
EASE OF ATTACK
Conducting attacks on machine learning systems often requires less exper-
tise than it takes to design the models and fewer resources than it takes to
train them; it is easier to destroy than to create. Just as the offense has long
held an advantage in traditional cyber operations, it appears to also have
the edge in machine learning for the time being.
The tools for conducting the common attacks discussed in this report
have already proliferated widely. They can be found and downloaded
freely from the internet and are not difficult to build. We were able to make
many versions of the attack shown in Figure 3 over the course of a single
afternoon. None of the versions took more than 20 lines of code and each
could run in about a second.
The attacks in Figure 3 executed quickly because they avoided the hard
part of machine learning: training the model. Not all attacks avoid the train-
ing stage, so some can require substantially more than a second of effort. At-
tackers who cannot steal the model either have to build one of their own or
use trial-and-error on the victim’s deployed system.10 For example, attackers
often need large amounts of time to train a model for data poisoning attacks
to confirm that the poison will have the desired effect. Assessing the Threat:
Ease, Pervasiveness,
and Defenses4
T
Center for Security and Emerging Technology14Even when the attacks are sometimes time-consuming, the number of actions an
attacker must take can be surprisingly small.11 Changing as little as a single data-
point can sometimes be enough to have the desired negative effects on the model’s
performance.12 The same is true for confidentiality attacks, where observing the in-
puts and outputs of a machine learning system just a few hundred or thousand times
can be enough to determine how machine learning models work.13 And, as we have
seen, evasion attacks can be successful with only imperceptible changes.14
Of course, coding the attacks is only one part of an operation. The attacker
needs information to conduct the attack and a way to launch it against the target. To
do this, the three confidentiality attacks previously discussed only need to observe
the inputs and outputs of the model or the deployed system. Similarly, neither data
poisoning nor evasion require direct access to the target. In short, there are many
opportunities for attackers to achieve their goals and the attacks themselves do not
require much expertise to create. However, the difficulty of introducing them to the
victim and ensuring they cause the intended malicious outcome will vary on a case-
by-case basis.
Some attacks do require substantial information about the target, and obtaining
that information can be somewhat complicated. For the Healy Hall attack, there
was an exact copy of the model to be attacked. It was as if a crashed spy drone
had been recovered and could be used to design camouflage to fool a similar
drone’s machine learning systems. Depending on our objective, we may not need
such direct access to the target; oftentimes information about something similar to
the targeted model will work. The techniques used in the Healy Hall attack can be
used to simultaneously fool many different image classifiers.15 Without knowing
what model the other classifiers used, we could send them our doctored image of
Healy Hall and there is a good chance they would misclassify it. Absent having the
model, though, it would be harder to force these classifiers to misidentify the build-
ing as a triceratops.16 In general, causing specific failures—such as getting a facial
recognition system to misclassify someone as a specific other person—requires more
information about the target model.
PERVASIVENESS
All machine learning models are susceptible to attack; different kinds of models
are vulnerable in different ways. In models with few inputs, for example, the victim
stands a better chance of noticing data poisoning attacks. Similarly, models with
few inputs are often more robust against evasion attacks because there are fewer
ways for an attacker to manipulate those inputs. In contrast, more complex models
offer more opportunities for attackers. That partly explains why image recognition
systems are so vulnerable: each pixel is an input, creating many manipulation op-
portunities for the attackers.
Center for Security and Emerging Technology15But simpler models that are trained on just a few data points have drawbacks,
too. For example, they are more vulnerable to confidentiality attacks. If there are
only five people in a training database, then each of those people contributes a lot
to the tuning of parameters in the model. In models trained on millions of people,
each person contributes only a little to any parameter and so information can be
harder to extract via confidentiality attack.
As a result of these and other systemic weaknesses, all machine learning sys-
tems have vulnerabilities. Some of the most common examples of machine learning
models are vision systems; evasion attacks against vision systems receive significant
attention, as this report has shown, but there are prominent examples of attacks
against audio and text systems, as well. Subtle changes can be made so that the
computer “hears” whatever the attacker chooses. In voice-controlled homes or
phones, attackers may gain unauthorized access. Systems that process text are also
vulnerable to manipulation and evasion. For example, Twitter’s AI for identifying
misleading tweets about COVID-19 flagged one reading, “Do not give oxygen
to the idea, which comes up with great frequency, that we are approaching some
kind of strong AI”—a statement that has nothing to do with COVID-19. These fail-
ures offer a reminder of the shortcomings of machine learning systems.17 It is easy to
imagine that governments will benefit from the ability to manipulate an adversary’s
machine learning systems and will perceive an imperative to defend their own.
DEFENSES
Reliable defenses against these types of attacks are hard to come by, but some
developments are more promising than others. Protecting information about the
data sources used to train a model—to guard against membership inference, for ex-
ample—is an area of comparative promise. A technique called differential privacy
can mathematically limit how much information can be gleaned about any individ-
ual person or datapoint.\* The designers of machine learning systems can use those
techniques to manage their degree of risk and constrain the information available
to those seeking to breach confidentiality.18 Most of these techniques today force
the defender to sacrifice performance for privacy and so they are not widely imple -
mented, but future privacy-preserving techniques may be more efficient.19 In mission
critical systems or cases of extreme data sensitivity, however, these performance
tradeoffs may be more acceptable. Differential privacy has other limits, too. For
example, it only protects an individual’s contributions to the training data and will
not help obscure traits common among groups of contributors.
\*Other mathematical techniques such as secure multi-party computing, homomorphic encryption,
and federated learning are also promising. Those three techniques, though, solve the different
problem of keeping the developer from accessing private data rather than keeping the developer’s
data private.
Center for Security and Emerging Technology16Defenses against other attacks are less promising. To guard against the other
confidentiality attacks of model inversion and model extraction, the defender can
reduce their vulnerability by limiting the number of times customers can use their
model or by intentionally decreasing its accuracy.20 But those steps can interfere
with the business case and limit the value of machine learning; if a company makes
money each time the model is used, then limiting the number of uses is not very
appealing. Other approaches, like keeping the model on a classified server and
making sure cleared analysts are the only ones who see its outputs, limit the risk of
model stealing and model inversion but at the cost of restricting the model’s use and
adding security constraints.
Defending against attacks on integrity is harder still; it is a game of Whack-a-
Mole where new attacks are invented and defenses are developed, and then those
defenses are defeated and so on.21 This dynamic applies both to defenses that try to
detect attacks and those that try to make the models immune to them. And defend-
ing against one attack can invite others. For example, freezing a model and cutting
off its access to new information means no additional data poisoning is possible,
but letting it continue to update its defenses can pressure evasion attacks to evolve
to keep pace.
These defenses are typically only somewhat effective and only for very highly
constrained attacks of specific types.22 For example, to guard against attacks that
make imperceptible changes to a picture, a defense might be effective against
attackers who limit the average change per pixel while not protecting against those
who limit the maximum change per pixel.23 A subtle change in an attacker’s opera-
tions can change how effective the defense is.†
This is a vital and perhaps alarming point: machine learning vulnerabilities are
hard to fix. Fixing them is more akin to addressing hardware vulnerabilities—which
are notoriously challenging—than it is to the relative ease of patching traditional
software vulnerabilities.24 For some of the attacks discussed in this paper there is no
clear solution on the horizon. The persistence of these weaknesses should prompt
caution when using machine learning in national security contexts against sophisti-
cated adversaries.
† There are actually many ways to calculate the average and there are many more interesting ways
to keep the changes small that are more esoteric, such as using the Wasserstein distance.25
Center for Security and Emerging Technology17istorically, where vulnerabilities have existed in traditional cyber
systems, attackers have often exploited them for nefarious or
destructive ends. The same will likely be true of vulnerabilities in
machine learning. This is not a call to eliminate machine learning from on-
going modernization, as vulnerabilities do exist in non-AI systems as well.
Rather, it is a wake-up call: machine learning brings with it new vulnera-
bilities that must be understood well enough to make informed decisions
about risks and investments. A few findings follow from our analysis.
MACHINE LEARNING INTRODUCES RISK IN
ACQUISITION AND MODERNIZATION
Machine learning is deeply integrated into various facets of society
and will likely continue to gain traction. In some cases, such as integrity
attacks against movie recommendation systems, there may be relatively
little incentive to attack and so the risk of using machine learning is low. In
contrast, the risks to national security systems are substantial, and there are
many well-resourced and highly motivated adversaries seeking to attack.
A first step in assessing the risks of deploying machine learning systems
in such a competitive context is understanding the range of options avail-
able to the potential attackers, which include the model stealing, model
inversion, membership inference, data poisoning and evasion attacks
discussed in this report. A subsequent step is understanding the defensive
options that exist and their effectiveness.Conclusion and
Recommendations
H5
Center for Security and Emerging Technology18NEW DEFENSES MAY ONLY OFFER SHORT-TERM ADVANTAGE
One of the perpetual questions in cybersecurity is whether the attacker or the de -
fender has the upper hand. It is hard to answer this question until the field of ma -
chine learning cybersecurity settles on specific offensive and defensive techniques.
Even then the answer may not be clear, as attackers and defenders engage one
another, both sides will discover new techniques. Currently, defenses do not look
promising, and many traditional cybersecurity techniques are not easily applied to
machine learning. In general, attackers can move more quickly than defenders and
the costs are higher to retrain a model than they are to find a new attack.
The offense-defense balance changes as machine learning systems reach
different levels of model complexity. Some techniques that appear to be effective or
ineffective at first behave differently when applied to more or less powerful systems.
For example, some defenses that are promising for securing imaging systems that
read low-resolution handwritten digits are not promising for imaging systems that
are powerful enough to recognize high-resolution pictures of cars and animals.
ROBUSTNESS TO ATTACK IS MOST LIKELY TO COME FROM
SYSTEM-LEVEL DEFENSES
Given the difficulty in finding reliable defenses against the wide range of attack
options, systemic defenses seem essential. Defenders should assume that attackers
will successfully compromise some parts of machine learning systems. To limit the
damage attackers can do, we should build redundancy and increase resilience.
Especially given how hard it is to fix underlying weaknesses in machine learning
systems, designing architectures that maximize robustness and prevent cascading
failures is key.
For instance, a commonly cited example of an attack involves placing a sticker
on a stop sign that makes it appear to autonomous vehicles to be a 45 mph sign.
Although this attack is possible and easy to perform, it only achieves a destructive
effect if the car drives into a busy intersection. If the car has many ways to decide
to stop, such as by knowing that intersections usually have stop signs, relying on
lasers for collision avoidance, observing other cars stopping, or noticing high speed
cross-traffic, then the risk of attack can remain low despite the car being made of
potentially vulnerable machine learning components. The systemic-level defense—
to not rely on just one input in making the decision to accelerate through the inter-
section—thwarts the attack.
Center for Security and Emerging Technology19THE BENEFITS TO OFFENSIVE USE OFTEN DO NOT OUTWEIGH
THE COSTS
The United States is not the only country fielding AI systems, and the opportunity
to exploit these vulnerabilities in adversaries’ systems may be tempting. There are
obvious military benefits of causing an enemy weapon to misidentify its targets or
send an adversary’s autonomous vehicles off course. There are also the obvious
intelligence benefits of stealing adversaries’ models and learning about the data
they have used.
On the other hand, the United States is among the countries best positioned to
benefit from progress in AI technologies. It has the most to lose if these technologies
are vulnerable. Demonstrating global norms of restraint against attacking AI may be
a wise stance. Even if nation-states do not adhere to global norms against attacking
military AI in conflict, there may be benefits to clarifying the lines against attacking
civilian systems or critical infrastructure. Clarifying rules and norms would help man-
age the problem of machine learning security both domestically and internationally.
A posture of restraint when it comes to attacking machine learning systems may also
help the United States government win the support of AI talent that national security
officials have been eager to court.
Norms alone will not solve this problem. Whether or not the United States
decides to pursue attacks on machine learning systems, adversaries will make their
own decisions about restraint, or lack thereof. Given that machine learning’s vulner-
abilities are pervasive, easy to exploit, and hard to defend, managing the risks they
pose is too large a task for the technology community to handle alone. It is incum-
bent upon policymakers to understand the threats well enough to assess the dangers
that the nation, its military and intelligence arms, and its civilians face when they use
machine learning. In some cases, that exposure may be acceptable, and in others,
it may not. But in all cases, the management of risk must be informed by technical
understanding. This primer is meant to help with that endeavor.
Center for Security and Emerging Technology16
Center for Security and Emerging Technology 21Endnotes
1. Tencent Keen Security Lab Experimental Security Research of Tesla Autopilot. (2019).
2. Morgan, F. E. et al. Military Applications of Artificial Intelligence: Ethical Concerns in an Uncertain World.
https://apps.dtic.mil/sti/citations/AD10973 1 3 (2020).
3. Eykholt, K. et al. Robust Physical-World Attacks on Deep Learning Models. arXiv [cs.CR] (2017).
4. Shumailov, I., Zhao, Y., Bates, D., Papernot, N., Mullins, R. & Anderson, R. Sponge Examples: Energy-
Latency Attacks on Neural Networks. arXiv [cs.LG] (2020).; Tabassi, E., Burns, K. J., Hadjimichael, M.,
Molina-Markham, A. D. & Sexton, J. T. A taxonomy and terminology of adversarial machine learning.
(2019) doi:10.6028/NIST.IR.8269-draft.
5. Gu, T., Dolan-Gavitt, B. & Garg, S. BadNets: Identifying Vulnerabilities in the Machine Learning Model
Supply Chain. arXiv [cs.CR] (2017).; Kurita, K., Michel, P. & Neubig, G. Weight Poisoning Attacks on Pre-
trained Models. arXiv [cs.LG] (2020).
6. Saha, A., Subramanya, A. & Pirsiavash, H. Hidden Trigger Backdoor Attacks. arXiv [cs.CV] (2019).
7. Madry, A., Makelov, A., Schmidt, L., Tsipras, D. & Vladu, A. Towards Deep Learning Models Resistant to
Adversarial Attacks. arXiv [stat.ML] (2017).
8. Zhang, Y. et al. The Secret Revealer: Generative Model-Inversion Attacks Against Deep Neural Networks.
arXiv [cs.LG] (2019).
9. Gilmer, J., Adams, R. P., Goodfellow, I., Andersen, D. & Dahl, G. E. Motivating the Rules of the Game for
Adversarial Example Research. arXiv [cs.LG] (201 8).
10. Shukla, S. N., Sahu, A. K., Willmott, D. & Zico Kolter, J. Hard Label Black-box Adversarial Attacks in
Low Query Budget Regimes. arXiv [cs.LG] (2020).; Ilyas, A., Engstrom, L., Athalye, A. & Lin, J. Black-box
Adversarial Attacks with Limited Queries and Information. arXiv [cs.CV] (201 8).
11 . Shafahi, A. et al. Poison Frogs! Targeted Clean-Label Poisoning Attacks on Neural Networks. in Advances
in Neural Information Processing Systems 3 1 (eds. Bengio, S. et al.) 6103–61 1 3 (Curran Associates, Inc.,
201 8).
1 2. Park, S., Weimer, J. & Lee, I. Resilient linear classification: an approach to deal with attacks on training data.
in Proceedings of the 8th International Conference on Cyber-Physical Systems 1 55–164 (Association for
Computing Machinery, 2017).
13 . Tramèr, F., Zhang, F., Juels, A., Reiter, M. K. & Ristenpart, T. Stealing machine learning models via prediction
apis. in 25th ${USENIX} Security Symposium ({USENIX}$ Security 16) 601–61 8 (2016).
1 4. Su, J., Vargas, D. V. & Sakurai, K. One Pixel Attack for Fooling Deep Neural Networks. IEEE Trans. Evol.
Comput. 23, 828–841 (2019).
15. Tramèr, F., Papernot, N., Goodfellow, I., Boneh, D. & McDaniel, P. The Space of Transferable Adversarial
Examples. arXiv [stat.ML] (2017).; Papernot, N., McDaniel, P. & Goodfellow, I. Transferability in Machine
Learning: from Phenomena to Black-Box Attacks using Adversarial Samples. arXiv [cs.CR] (2016).
16. Liu, Y., Chen, X., Liu, C. & Song, D. Delving into Transferable Adversarial Examples and Black-box Attacks.
arXiv [cs.LG] (2016).
17. Carlini, N. et al. Hidden voice commands. in 25th ${USENIX} Security Symposium ({USENIX}$ Security
16) 5 1 3–530 (2016). Tweet from Ludwig Yeetgenstein, June 26, 2020, https://twitter.com/yeetgenstein/
status/1 2765 1 8982565 1 46624.
1 8. Truex, S., Liu, L., Gursoy, M. E., Wei, W. & Yu, L. Effects of Differential Privacy and Data Skewness on
Membership Inference Vulnerability. in 20 19 First IEEE International Conference on Trust, Privacy and
Security in Intelligent Systems and Applications (TPS-ISA) 82–91 (2019).
Center for Security and Emerging Technology 2219. Jayaraman, B. & Evans, D. Evaluating differentially private machine learning in practice. in 28th ${USENIX}
Security Symposium ({USENIX}$ Security 19) 1 895–191 2 (2019).; Rahman, M. A., Rahman, T., Laganière,
R., Mohammed, N. & Wang, Y. Membership Inference Attack against Differentially Private Deep Learning
Model. Trans. Data Priv. 1 1, 61–79 (201 8).
20. Tramèr, F., Zhang, F., Juels, A., Reiter, M. K. & Ristenpart, T. Stealing machine learning models via prediction
apis. in 25th ${USENIX} Security Symposium ({USENIX}$ Security 16) 601–61 8 (2016).
21. Athalye, A., Carlini, N. & Wagner, D. Obfuscated Gradients Give a False Sense of Security: Circumventing
Defenses to Adversarial Examples. arXiv [cs.LG] (201 8).
22. Hartnett, G. S., Lohn, A. J. & Sedlack, A. P. Adversarial Examples for Cost-Sensitive Classifiers. arXiv [stat.
ML] (2019).; Xie, C., Wu, Y., van der Maaten, L., Yuille, A. & He, K. Feature Denoising for Improving
Adversarial Robustness. arXiv [cs.CV] (201 8).
23. Lecuyer, M., Atlidakis, V., Geambasu, R., Hsu, D. & Jana, S. Certified Robustness to Adversarial Examples
with Differential Privacy. arXiv [stat.ML] (201 8). Wong, E. & Zico Kolter, J. Provable defenses against
adversarial examples via the convex outer adversarial polytope. arXiv [cs.LG] (2017).
24. Lohn, A. What do Meltdown, Spectre and RyzenFall Mean for the Future of Cybersecurity? TechCrunch
(201 8).
25. Wong, E., Schmidt, F. R. & Zico Kolter, J. Wasserstein Adversarial Examples via Projected Sinkhorn Iterations.
arXiv [cs.LG] (2019).
Center for Security and Emerging Technology33

CSET.GEORGETOWN.EDU | CSET@GEORGETOWN.EDU