Translation: Measures for the Management of Generative Artificial Intelligence Services (Draft for Comment) – April 2023

*This translation is by (in randomized order), Seaton Huang, Helen Toner, Zac Haluza, and Rogier Creemers, and was edited by Graham Webster. During editing, an* [*alternative translation*](https://www.chinalawtranslate.com/en/gen-ai-draft/) *from China Law Translate was consulted.*

**[For analysis of this draft, please see our DigiChina Forum, compiling analysis from a group of invited specialists.](https://digichina.stanford.edu/work/how-will-chinas-generative-ai-regulations-shape-the-future-a-digichina-forum/)**

### Translation

April 11, 2023

Measures for the Management of Generative Artificial Intelligence Services (Draft for Comment)
----------------------------------------------------------------------------------------------

**Article 1:** In order to stimulate the healthy development and standardized application of generative artificial intelligence (AI), on the basis of the [Cybersecurity Law of the People’s Republic of China](https://digichina.stanford.edu/work/translation-cybersecurity-law-of-the-peoples-republic-of-china-effective-june-1-2017/), the [Data Security Law of the People’s Republic of China](https://digichina.stanford.edu/work/translation-data-security-law-of-the-peoples-republic-of-china/), the [Personal Information Protection Law of the People’s Republic of China](https://digichina.stanford.edu/work/translation-personal-information-protection-law-of-the-peoples-republic-of-china-effective-nov-1-2021/), and other such laws and administrative regulations, these Measures are formulated.

**Article 2:** These Measures apply to the research, development, and use of products with generative AI functions, and to the provision of services to the public within the [mainland] territory of the People’s Republic of China.

Generative AI, as mentioned in these Measures, refers to technologies generating text, image, audio, video, code, or other such content based on algorithms, models, or rules.

**Article 3:** The State supports [indigenous innovation](https://digichina.stanford.edu/work/lexicon-indigenous-innovation-or-independent-innovation-zizhu-chuangxin/), broad application, and international cooperation in foundational technologies such as AI algorithms and frameworks, and encourages the prioritized use of secure and reliable software, tools, computing, and data resources.

**Article 4:** The provision of generative AI products or services shall abide by the requirements of laws and regulations, respect social virtue and good public customs, and conform to the following requirements:

1. Content generated through the use of generative AI shall reflect the Socialist Core Values, and may not contain: subversion of state power; overturning of the socialist system; incitement of separatism; harm to national unity; propagation of terrorism or extremism; propagation of ethnic hatred or ethnic discrimination; violent, obscene, or sexual information; false information; as well as content that may upset economic order or social order.
2. In processes such as algorithm design, selecting training data, model generation and model optimization, service provision, etc., adopt measures to prevent the emergence of discrimination on the basis of race, ethnicity, religious belief, nationality, region, sex, age, or profession.
3. Respect intellectual property rights and commercial ethics; advantages in algorithms, data, platforms, etc., may not be used to engage in unfair competition.
4. Content generated through the use of generative AI shall be true and accurate, and measures are to be adopted to prevent the generation of false information.
5. Respect the lawful rights and interests of others; prevent harm to the physical and mental health of others, infringement of their likeness rights, reputation rights and personal privacy, as well as infringement of intellectual property rights. It is prohibited to illegally obtain, divulge or use personal information and private [information], as well as commercial secrets.

**Article 5:** Organizations or individuals that use generative AI to provide services such as chat, text, image, or audio generation (hereinafter referred to as “providers”); including providing programmable interfaces [i.e., APIs] and other means which support others to themselves generate text, images, audio, etc.; bear responsibility as the producer of the content generated by the product. Where personal information is involved, they bear legal responsibility as personal information handlers and are to fulfill personal information protection obligations.

**Article 6:** Before using generative AI products to provide services to the public, a security assessment must be submitted to the state cyberspace and information department [i.e., the Cyberspace Administration of China] in accordance with the [Provisions on the Security Assessment of Internet Information Services With Public Opinion Properties or Social Mobilization Capacity](https://digichina.stanford.edu/work/new-rules-target-public-opinion-and-mobilization-online-in-china-translation/), and the procedures of algorithm filing, modification, and cancellation of filing must be carried out in accordance with the [Internet Information Service Algorithmic Recommendation Management Provisions](https://digichina.stanford.edu/work/translation-internet-information-service-algorithmic-recommendation-management-provisions-effective-march-1-2022/).

**Article 7:** Providers shall bear responsibility for the legality of the sources of generative AI product pre-training data and optimization training data.

Data used for generative AI product pre-training and optimization training shall satisfy the following requirements:

1. Conforming to the requirements of the Cybersecurity Law of the People’s Republic of China and other such laws and regulations;
2. Not containing content infringing intellectual property rights;
3. Where data includes personal information, the consent of the personal information subject shall be obtained, or other procedures conforming with the provisions of laws and administrative regulations followed;
4. Be able to ensure the data’s veracity, accuracy, objectivity, and diversity;
5. Other supervision requirements of the state cybersecurity and informatization department concerning generative AI functions and services.

**Article 8:** When human annotation is used in the development of generative AI products, providers shall formulate clear, specific, and practicable annotation rules conforming to the requirements of these Measures; necessary training of annotation personnel shall be conducted; and the validity of annotation content shall be spot checked.

**Article 9:** When providing generative AI services, users shall be required to provide real identity information in accordance with the provisions of the Cybersecurity Law of the People’s Republic of China.

**Article 10:** Providers shall explicitly disclose the user groups, occasions, and uses for their services, and adopt appropriate measures to prevent users from excessive reliance on or addiction to generated content.

**Article 11:** In the process of providing services, providers have the duty to protect information input by users and usage records. They may not illegally preserve input information from which it is possible to deduce the identity of users, they may not conduct profiling on the basis of information input by users and their usage details, and they may not provide information input by users to others. Where laws or regulations provide otherwise, those provisions are to be followed.

**Article 12:** Providers may not engage in content generation that is discriminatory based on a user’s race, nationality, sex, etc.

**Article 13:** Providers shall establish mechanisms for receiving and handling user complaints and promptly handle individual requests concerning revision, deletion, or masking of their personal information; and when they discover or learn that generated text, images, audio, video, etc., infringe other persons’ likeness rights, reputation rights, personal privacy, or commercial secrets, or do not conform to the demands of these Measures, they shall adopt measures to cease generation and prevent the expansion of the harm.

**Article 14:** Providers shall, throughout the lifecycle, provide secure, stable and sustained services, and ensure users’ normal usage.

**Article 15:** When generated content that does not conform to the requirements of these Measures is discovered during operations or reported by users, aside from adopting content filtering and other such measures, repeat generation is to be prevented through such methods as optimization training within three months.

**Article 16:** Providers shall mark generated images, videos, and other content in accordance with the [Internet Information Service Deep Synthesis Management Provisions](https://digichina.stanford.edu/wp-admin/post.php?post=2005&action=edit).

**Article 17:** Providers shall, in accordance with the requirements of the state cybersecurity and informatization department and relevant responsible departments, provide necessary information that could influence users trust or choices, including descriptions of the source, scale, type, quality, etc., of pre-training and optimization training data; rules for human annotation; the scale and type of human-annotated data; and foundational algorithms and technological systems.

**Article 18:** Providers shall guide users to scientifically understand and rationally use content generated by generative AI; not to use generated content to damage others’ image, reputation, or other lawful rights and interests; and not to engage in commercial hype or improper marketing.

When users discover generated content that does not meet the requirements of these measures, they have the right to report this to cybersecurity and informatization departments or relevant responsible departments.

**Article 19:** If a provider finds that a user has used generative AI products to violate laws or regulations; violate business ethics or social virtue, including engaging in online hype, malicious posting and commenting, creating spam, or writing malicious software; or engage in improper business marketing; etc.; service shall be suspended or terminated.

**Article 20:** If a provider violates the provisions of the Measures, the cybersecurity and informatization department and relevant responsible departments are to impose penalties in accordance with the provisions of Cybersecurity Law of the People’s Republic of China, the Data Security Law of the People’s Republic of China, the Personal Information Protection Law of the People’s Republic of China, and other such laws and administrative regulations.

Where there are no provisions of law or administrative regulation, the cybersecurity and informatization department and relevant responsible departments are to, in accordance with their duties, issue warnings, circulate criticisms, and order corrections within a set period of time. Where corrections are refused or circumstances are grave, they are to order suspension or termination of their use of generative AI provider services, and a penalty more than 10,000 yuan and less than 100,000 yuan is to be imposed. Where behavior constitutes a violation of public security management, public security management penalties are to be imposed in accordance with the law. Where a crime is constituted, criminal responsibility shall be pursued in accordance with the law.

**Article 21:** These measures are effective beginning [day] [month], 2023.

### Chinese-language Original

[[Source]](http://www.cac.gov.cn/2023-04/11/c_1682854275475410.htm) [[Archived copy]](https://web.archive.org/web/20230411065823/http://www.cac.gov.cn/2023-04/11/c_1682854275475410.htm)

生成式人工智能服务管理办法（征求意见稿）
--------------------

**第一条** 为促进生成式人工智能健康发展和规范应用，根据《中华人民共和国网络安全法》《中华人民共和国数据安全法》《中华人民共和国个人信息保护法》等法律、行政法规，制定本办法。

**第二条** 研发、利用生成式人工智能产品，面向中华人民共和国境内公众提供服务的，适用本办法。

本办法所称生成式人工智能，是指基于算法、模型、规则生成文本、图片、声音、视频、代码等内容的技术。

**第三条** 国家支持人工智能算法、框架等基础技术的自主创新、推广应用、国际合作，鼓励优先采用安全可信的软件、工具、计算和数据资源。

**第四条** 提供生成式人工智能产品或服务应当遵守法律法规的要求，尊重社会公德、公序良俗，符合以下要求：

（一）利用生成式人工智能生成的内容应当体现社会主义核心价值观，不得含有颠覆国家政权、推翻社会主义制度，煽动分裂国家、破坏国家统一，宣扬恐怖主义、极端主义，宣扬民族仇恨、民族歧视，暴力、淫秽色情信息，虚假信息，以及可能扰乱经济秩序和社会秩序的内容。

（二）在算法设计、训练数据选择、模型生成和优化、提供服务等过程中，采取措施防止出现种族、民族、信仰、国别、地域、性别、年龄、职业等歧视。

（三）尊重知识产权、商业道德，不得利用算法、数据、平台等优势实施不公平竞争。

（四）利用生成式人工智能生成的内容应当真实准确，采取措施防止生成虚假信息。

（五）尊重他人合法利益，防止伤害他人身心健康，损害肖像权、名誉权和个人隐私，侵犯知识产权。禁止非法获取、披露、利用个人信息和隐私、商业秘密。

**第五条** 利用生成式人工智能产品提供聊天和文本、图像、声音生成等服务的组织和个人（以下称“提供者”），包括通过提供可编程接口等方式支持他人自行生成文本、图像、声音等，承担该产品生成内容生产者的责任；涉及个人信息的，承担个人信息处理者的法定责任，履行个人信息保护义务。

**第六条** 利用生成式人工智能产品向公众提供服务前，应当按照《具有舆论属性或社会动员能力的互联网信息服务安全评估规定》向国家网信部门申报安全评估，并按照《互联网信息服务算法推荐管理规定》履行算法备案和变更、注销备案手续。

**第七条** 提供者应当对生成式人工智能产品的预训练数据、优化训练数据来源的合法性负责。

用于生成式人工智能产品的预训练、优化训练数据，应满足以下要求：

（一）符合《中华人民共和国网络安全法》等法律法规的要求；

（二）不含有侵犯知识产权的内容；

（三）数据包含个人信息的，应当征得个人信息主体同意或者符合法律、行政法规规定的其他情形；

（四）能够保证数据的真实性、准确性、客观性、多样性；

（五）国家网信部门关于生成式人工智能服务的其他监管要求。

**第八条** 生成式人工智能产品研制中采用人工标注时，提供者应当制定符合本办法要求，清晰、具体、可操作的标注规则，对标注人员进行必要培训，抽样核验标注内容的正确性。

**第九条** 提供生成式人工智能服务应当按照《中华人民共和国网络安全法》规定，要求用户提供真实身份信息。

**第十条** 提供者应当明确并公开其服务的适用人群、场合、用途，采取适当措施防范用户过分依赖或沉迷生成内容。

**第十一条** 提供者在提供服务过程中，对用户的输入信息和使用记录承担保护义务。不得非法留存能够推断出用户身份的输入信息，不得根据用户输入信息和使用情况进行画像，不得向他人提供用户输入信息。法律法规另有规定的，从其规定。

**第十二条** 提供者不得根据用户的种族、国别、性别等进行带有歧视性的内容生成。

**第十三条** 提供者应当建立用户投诉接收处理机制，及时处置个人关于更正、删除、屏蔽其个人信息的请求；发现、知悉生成的文本、图片、声音、视频等侵害他人肖像权、名誉权、个人隐私、商业秘密，或者不符合本办法要求时，应当采取措施，停止生成，防止危害持续。

**第十四条** 提供者应当在生命周期内，提供安全、稳健、持续的服务，保障用户正常使用。

**第十五条** 对于运行中发现、用户举报的不符合本办法要求的生成内容，除采取内容过滤等措施外，应在3个月内通过模型优化训练等方式防止再次生成。

**第十六条** 提供者应当按照《互联网信息服务深度合成管理规定》对生成的图片、视频等内容进行标识。

**第十七条** 提供者应当根据国家网信部门和有关主管部门的要求，提供可以影响用户信任、选择的必要信息，包括预训练和优化训练数据的来源、规模、类型、质量等描述，人工标注规则，人工标注数据的规模和类型，基础算法和技术体系等。

**第十八条** 提供者应当指导用户科学认识和理性使用生成式人工智能生成的内容，不利用生成内容损害他人形象、名誉以及其他合法权益，不进行商业炒作、不正当营销。

用户发现生成内容不符合本办法要求时，有权向网信部门或者有关主管部门举报。

**第十九条** 提供者发现用户利用生成式人工智能产品过程中违反法律法规，违背商业道德、社会公德行为时，包括从事网络炒作、恶意发帖跟评、制造垃圾邮件、编写恶意软件，实施不正当的商业营销等，应当暂停或者终止服务。

**第二十条** 提供者违反本办法规定的，由网信部门和有关主管部门按照《中华人民共和国网络安全法》《中华人民共和国数据安全法》《中华人民共和国个人信息保护法》等法律、行政法规的规定予以处罚。

法律、行政法规没有规定的，由网信部门和有关主管部门依据职责给予警告、通报批评，责令限期改正；拒不改正或者情节严重的，责令暂停或者终止其利用生成式人工智能提供服务，并处一万元以上十万元以下罚款。构成违反治安管理行为的，依法给予治安管理处罚；构成犯罪的，依法追究刑事责任。

**第二十一条** 本办法自2023年 月 日起实施。


The Social Science of Computerized Brains – Review of The Age of Em: Work, Love, and Life When Robots Rule the Earth by Robin Hanson (Oxford University Press, 2016)

Introduction The Age of Em: Work, Love, and Life When Robots Rule the Earth by Robin Hanson discusses the possibility of "uploading" human minds into computers. An uploaded mind or "em" (short for "emulation) is able to think more or less like a regular biological human, but with properties of computer programs and data. Ems can be copied ad infinitum and sped up or slowed down by being placed on faster or slower hardware. These capabilities create radical possibilities, which is the book's focus. The book is by far the most detailed description of the em world available. Hanson, an economist with a background in physics and artificial intelligence (AI), brings a distinctive and valuable perspective to this important topic.  The Age of Em describes one em world scenario in detail, giving only brief attention towards the end of the book (Ch. 28) to variations of this scenario. The chosen scenario is rated as the most likely one, given that uploading occurs (p.34-37). The scenario is intended as a baseline that is representative of many possible scenarios; interested analysts can introduce variations to produce still other scenarios. The book states that "Such straightforward scenario construction seems taboo among many professional futurists today", who instead favor creating "a modest number of scenarios to cover a wide range of possibilities across key axes of uncertainty and disagreement, with each scenario being story-like, archetypical in representing clusters of relevant driving forces, and describing equilibrium rather than transitory situations" (p.35), citing  Schoemaker (1995) . It is certainly true that some futures studies feature sets of story-like equilibrium scenarios, and it may well be true that some futurists disprefer presenting a single scenario, covering transition periods. However, in my experience, futurists are open to a wide range of approaches, and I see no reason why they should reject that of The Age of Em. The bulk of the book consists of descriptions of the world in which the ems The scope is encyclopedic, with attention to everything from the social relations among ems to the resistance of em buildings to earthquakes. Indeed, the book may be best treated as an encyclopedia of a particular em scenario. Reading it straight through, it can get bogged down in details. Topics are also often presented in an unintuitive order. For example, the initial creation of the em world appears in subsections of chapters 12 and 28, instead of in its logical place at the beginning of the book. A more story-like structure would make it more readable and may reduce certain cognitive biases  (Steinhardt & Shapiro, 2015) . However, if you are wondering about what some aspect of an em world might look like, odds are good that a description can be found somewhere in the book. A central premise of the book's scenario is that the em world is organized around market activity. Ems exist because they provide valuable labor. Ems with different capabilities are assigned to different tasks; groups of ems are organized into functional teams; cities are built to collocate ems for enhanced productivity. Ems are paid for their services and can spend their earnings on leisure, though competition between ems drives wages down. While these basics are broadly similar to much of today's society, the nature of ems creates some radical differences. For starters, while some ems are housed in robotic bodies to perform physical-world tasks, most live and work disembodied in virtual reality worlds. This enables them to travel at great speeds via telecommunications lines. Additionally, ems can be sped up or slowed down as needed, causing them to experience the world (and act on it) thousands or millions of times faster or slower than biological humans. Finally, ems can be copied repeatedly, which enables em teams to perform extreme multitasking. The book even proposes the concept of "spurs", which are em copies that perform a brief task and are then terminated or retired. These novel capacities enable ems to vastly outperform biological human workers. Em labor thus displaces humans while creating rapid economic growth. The book proposes that those humans who own parts of the em economy "could retain substantial wealth", but everyone else is "likely to starve" (p.336). To my eyes, this seems overly optimistic: if ems are so much smarter than humans, and if they have such a profit motive, then surely they could figure out how to trick, threaten, or otherwise entice humans into giving up their wealth. Humans would likely die out or, at best, scrape by in whatever meager existence the ems leave them with. Given these dire prospects for humans, one might question whether it would be good to create ems in the first place. Unfortunately, the book does not consider this topic in any detail. The book is pro-em, even proposing to "subsidize the development of related technologies to speed the arrival of this transition [to the em era], and to subsidize its smoothness, equality, or transparency to reduce disruptions and inequalities in that transition" (p.375). But this position is tenuous at best and quite possibly dangerous. The book's methodology is notable for its emphasis on social science. Many people who write about topics like uploading are trained in physical science, computer science, or philosophy; they thus "don't appreciate that social scientists do in fact know many useful things" (p.34). Hanson's defense of social science in this context is welcome. The book's methodology is rooted in extrapolation of current social science to em world conditions. At times, the extrapolations seem strained. For example, a claim that economic benefits of urban agglomeration would continue in an em world (p.215) cites  Morgan (2014) , which is a popular media description of Amazon's large data centers. It is true that Amazon data centers are large, and it may well be true that em cities are large for similar reasons, but the latter does not necessarily follow from the former. One can sympathize with Hanson, who is trying to present the social science of a topic that virtually no social scientists have previously touched. Indeed, the breadth of research covered is impressive; the book quietly doubles as a good literature review. However, some of the language relating the literature to the em world could have been better finessed. In other stretches, Hanson's personal tastes are apparent. This is seen, for example, in discussions of combinatorial auctions and prediction markets (p.184-188), two schemes for adapting market mechanisms for social decision making. Prediction markets in particular are a longstanding interest of Hanson's. The book's discussion of these topics says little about the em world and seems mainly oriented towards promoting them for society today. The reader gets the impression that Hanson wishes society today was more economically efficient and rational in a certain sense, and that he has embedded his hopes for a better world into his vision of ems. In conclusion, The Age of Em is a thoughtful and original work of futures studies, bringing an insightful social science perspective to the topic of mind uploading. The book covers wide ground about the nature of the em world, offering a valuable resource for anyone interested in the topic, or about futures studies more generally. The book is accessibly written and could be read by undergraduates or even advanced high school students, though experts from any discipline will also find much of interest. The book is especially worthwhile for anyone who could benefit from an overview of contemporary social science research, which the book provides in abundance. The book is not flawless. One can disagree with some of the book's arguments, including some of the important ones. The author's personal perspective is apparent; different authors would often reach different conclusions about the same topic. The organization is often scattered, and some literature is misused. However, none of these flaws take away from the significant accomplishment of describing the em world in great detail. The book offers a starting point-indeed, a baselinefor the serious study of future uploading scenarios. One can only hope that additional scholars, especially in the social sciences, follow the book's lead in taking on this important future scenario. Futures j o u r n a l h o m e p a g e : w w w . e l s e v i e r . c o m / l o c a t e / f u t u r e s


Compute Accounting Principles Can Help Reduce AI Risks

*Krystal Jackson is a visiting AI Junior Fellow at Georgetown University’s Center for Security and Emerging Technology (CSET), where Karson Elmgren is a research analyst and Jacob Feldgoise is a data research analyst. Andrew Critch is an AI research scientist at UC Berkeley’s Center for Human-Compatible AI (CHAI), and also the CEO of Encultured AI, a small AI-focussed video game company.*

![](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%201024%20576'%3E%3C/svg%3E)![](https://techpolicy.press/wp-content/uploads/2022/11/Green-AI-1-1024x576.jpg)
Computational power, colloquially known as “compute,” is the processing resource required to do computational tasks, such as training artificial intelligence (AI) systems. Compute is arguably a key factor driving AI progress. Over the last decade, it has enabled increasingly large and powerful neural networks and ushered in the age of deep learning. Given compute’s role in AI advances, the time has come to develop standard practices to track the use of these resources.

Modern machine learning models, especially many of the most general ones, use orders of magnitude more compute than their predecessors. Stakeholders, including AI developers, policymakers, academia, and civil society organizations, all have reasons to track the amount of compute used in AI projects. Compute is at once a business resource, a large consumer of energy (and thus a potential factor in carbon emissions), and a rough proxy for a model’s capabilities. However, there is currently no generally accepted standard for compute accounting.

![](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%200%200'%3E%3C/svg%3E)![](https://lh3.googleusercontent.com/OF9OnABKasUH8jXLKWM4ZcFhT8yfl7G5CWN5vJzfAuQFj2TcEuQYe-xEpp3TgXU7weLiJmlH4cZAdjgyHjFp9OrSVe9yJBetBK1PtHYh_BKoYuf4Z-mFl86Le3RsTlKgDk_Ew2rjdzZWUh5FDIthIB81NiM8DNf07spy84VfnNxVBLC5U-eiql76ffLsqg)***Source***[*Epoch*](https://epochai.org/blog/compute-trends) *AI: An estimation of the total amount of compute used by various models, in floating point operations per second (FLOPs).*
There are two critical reasons for compute accounting standards: (1) to help organizations manage their compute budgets according to a set of established best practices and (2) to enable responsible oversight of AI technologies in every area of the economy. AI developers, government, and academia should work together to develop such standards. Among other benefits, standardized compute accounting could make it easier for company executives to measure, distribute, and conduct due diligence on compute usage across organizational divisions. Moreover, we need such standards to be adopted *cross-industry,* such that top-level line items on accounts can be compared between different sectors.

Many large companies already build substantial internal tracking infrastructure for logging, annotating, and viewing the project-specific usage of compute. Cloud-computing providers, such as Amazon Web Services (AWS), Google Cloud Platform, and Microsoft Azure, provide users with tools to track how their resources are spent. However, there is not yet an industry standard to document compute usage.

This absence of standardized compute accounting contrasts sharply with the situation for other essential resources and impacts that span across industry sectors, like financial assets, energy, and other utilities, as well as externalities such as carbon emissions, which are tracked using accounting standards. For instance, companies do not invent their own financial accounting software to keep track of money; they use ready-made solutions that work across banks and payment platforms. A single company can easily use multiple banks at once and consolidate all of its revenue and expenditures into a single standardized bookkeeping system using Generally Accepted Accounting Principles (GAAP). Standard practices enable apples-to-apples comparisons between organizations, which in turn fosters trust between investors, lenders, and companies. This trust adds significant economic value by facilitating well-informed transactions of all kinds.

In contrast, the absence of a compute accounting standard makes it challenging to exercise due diligence and audit compute usage. Both activities rely on consistent measurement and record-keeping, which currently does not exist across the industry or even, in some cases, between a large company’s divisions. This makes it more difficult for companies to conduct due diligence, for organizations to track and audit their use of these resources, and for governments and researchers to study how compute relates to AI progress, risks, and impacts. For example, without a compute accounting standard, measuring the environmental impact of AI training and inference has [proven to be challenging](https://arxiv.org/pdf/2104.10350.pdf).

There are many unanswered questions concerning the best approaches for compute accounting standards which further research should address:

1. **Tools for Companies**
--------------------------

With vendor-agnostic compute accounting tools, small companies would not need to invent their own compute accounting practices from scratch; they could simply employ publicly available best practices and tools. Furthermore, if compute providers offered usage reports in a standardized format, then consumers of compute — small and large businesses alike — could more easily track performance across their usage of multiple providers simultaneously. Instead of copying or reinvesting in these systems, companies could reduce costs by picking from a menu of accredited standards from the beginning. A mixture of copying and reinvention already happens to some degree, and there are efficiency gains to be made by standardizing the choices involved at start-up time.

*How researchers can help:* Continue to build and develop open-source tools for estimating and reporting compute usage. Several programming libraries and tools exist to calculate compute; however, many only estimate compute usage instead of measuring it, while others are vendor specific. Software developers could create general compute accounting tools to build a foundation for implementing practices and standards.

2. **Tracking Environmental Impacts**
-------------------------------------

Compute accounting standards could help organizations measure the environmental impacts of their business activities with greater precision. The cost of compute has decreased significantly, enabling many resource-intensive AI projects; however, the increase in compute accessibility has also increased the risk of high-carbon emission projects. Standards that facilitate tracking environmental impacts as part of a risk calculation could allow organizations to manage resources to meet their environmental goals and values. Tracking compute in a standardized way would help elucidate the relationships between energy use, compute, and performance in order to better manage tradeoffs in building AI systems.

*How researchers can help:* More research is needed to evaluate the environmental impacts of AI. We do not fully understand where and how energy is used in the AI development pipeline. When developers report final training information, they usually do not include previous training runs or consider how energy is sourced. Research into the environmental impact across the AI pipeline and how we can track that impact would help inform metrics and reporting practices.

3. **Critical Resource Tracking**
---------------------------------

A standard compute accounting measure would enable industry-wide tracking of this critical resource. Such a standard would make it easier for industry associations and researchers alike to study how compute is distributed. A standard would also help policymakers decide whether additional measures are needed to provide equitable access to compute—building, for example, on the mission of the National AI Research Resource.

*How researchers can help:* Determine what barriers exist to equitable access to computational resources. Identify the best ways to measure and track these disparities so the resulting data can be used to help remedy inequity.

4. **Assessment of Scaling Risk**
---------------------------------

In addition, careful tracking of compute could aid in risk assessment. As AI systems scale up in some domains, they can exhibit [emergent capabilities](https://arxiv.org/abs/2206.07682) – ones that were entirely absent in smaller models. Since models with emergent capabilities may pose new risks, organizations should consider imposing additional safeguards and testing requirements for larger AI systems. A consistent means of counting the compute used to train AI models would allow for scale-sensitive risk management within and between organizations.

*How researchers can help:* Additional research on the scaling properties of different model types would help determine the relationship between compute and capabilities across domains.

– – –

Standards development organizations should convene industry stakeholders to establish compute accounting standards. Specifically, standards bodies such as NIST, ISO, and IEEE should begin working with large companies that have already developed internal practices to track and report compute usage to establish readily-usable standards that are useful to businesses everywhere. Additionally, technology and policy researchers should conduct relevant research to help inform a compute accounting standard. These actions would help realize the benefits of compute accounting for all stakeholders and advance best practices for AI.

Krystal Jackson
Karson Elmgren
Jacob Feldgoise
Andrew Critch


Initial Draft
i AI Risk Management Framework: Initial Draft
March 17, 2022

This initial draft of the Artificial Intelligence Risk Management Framework (AI RMF, or
Framework) builds on the concept paper released in December 2021 and incorporates the
feedback received. The AI RMF is intended for voluntary use in addressing risks in the design,
development, use, and evaluation of AI products, services, and systems.
AI research and deployment is evolving rapidly. For that reason, the AI RMF and its companion
documents will evolve over time. When AI RMF 1.0 is issued in January 2023, NIST, working
with stakeholders, intends to have built out the remaining sections to reflect new knowledge,
awareness, and practices.
Part I of the AI RMF sets the stage for why the AI RMF is important and explains its intended
use and audience. Part II includes the AI RMF Core and Profiles. Part III includes a companion
Practice Guide to assist in adopting the AI RMF.
That Practice Guide which will be released for comment includes a dditional examples and
practices that can assist in using the AI RMF. The Guide will be part of a NIST AI Resource
Center that is being established.
NIST welcomes feedback on this initial draft and the related Practice Guide to inform further
development of the AI RMF.   Comments may be provided at a workshop on March 29 -31, 2022 ,
and also are strongly encouraged to be shared via email. N IST will produce a second draft for
comment, as well as host a third workshop, before publishing AI RMF 1.0 in January 2023.
Please send comments on this initial draft to AIframework@nist.gov  by April 29, 2022 .

Initial Draft
ii Comments are especially requested on:
1. Whether the AI RMF appropriately covers and addresses AI risks, including with the right level of
specificity for various use cases.
2. Whether the AI RMF is flexible enough to serve as a continuing resource considering  evolving
technology and standards landscape.
3. Whether the AI RMF enables decisions about how an organization can increase understanding of,
communication about , and efforts to manage AI risks.
4. Whether the functions,  categories, and subcategories  are complete, appropriate, and clearly stated.
5. Whether the AI RMF is in a lignment with or leverag es other frameworks and standards such as those
developed or being deve loped by IEEE or ISO/IEC SC42.
6. Whether the AI RMF is in a lignment with existing practices, and broader risk management practices.
7. What might be  missing from the AI RMF .
8. Whether the soon to be published draft companion document citing AI risk management pr actices is
useful as a complementary resource and what practices or standards should be added.
9. Others?
Note: This first draft does not include Implementation Tiers as considered in the concept paper.
Implementation Tiers may be added later if stakeholders consider them to be a helpful feature in the AI
RMF.  Comments are welcome.

Initial Draft
iii Table of Contents
Part 1: Motivation
1 OVERVIEW  1
2 SCOPE  2
3 AUDIENCE  3
4 FRAMING RISK  5
4.1 Understanding Risk and Adverse Impacts  5
4.2 Challenges for AI Risk Management  6
5 AI RISKS AND TRUSTWORTHINESS  7
5.1 Technical Characteristics  8
5.1.1 Accuracy  9
5.1.2 Reliability  9
5.1.3 Robustness  10
5.1.4 Resilience or ML Security  10
5.2 Socio -Technical Characteristics  10
5.2.1 Explainability  11
5.2.2 Interpretability  11
5.2.3 Privacy  11
5.2.4 Safety  12
5.2.5 Managing Bias  12
5.3 Guiding Principles  12
5.3.1 Fairness  13
5.3.2 Accountability  13
5.3.3 Transparency  13
Part 2: Core and Profiles
6 AI RMF CORE  14
6.1 Map  15
6.2 Measure  16
6.3 Manage  17
6.4 Govern  18
7 AI RMF PROFILES  20
8 EFFECTIVENESS OF THE AI RMF  20
Part 3: Practical Guide
9 PRACTICE GUIDE  20

Initial Draft
1 AI Risk Management Framework: Initial Draft -  1
Part 1: Motivation  2
1 Overview  3
Remarkable surges in artificial intelligenc e (AI)  capabilities have led to a wide range of 4
innovations with the potential to benefit nearly all aspects of our society and economy – 5
everything from commerce and healthcare to transportation and cybersecurity. AI systems are 6
used for tasks s uch as informing and advising people and taking actions where they can have 7
beneficial  impact, such as safety  and housing.  8
AI systems sometimes do not operate as intended because they are making inferences from 9
patterns observed in data rather than a true understanding of what causes those patterns. Ensuring 10
that these inferences are helpful and not harmful in particular use cases – especially when 11
inferences are rapidly scaled and amplified – is fundamental to trustworthy AI. While answers to 12
the question of what makes an AI technology trustworthy differ, there are certain key 13
characteristics which support trustworthiness, including accuracy, explainability and 14
interpretability, privacy, reliability, robustness, safety, security (resilience) and mitigation of 15
harmful bias.  There also are key guiding principles to take into account such as accountability, 16
fairness, and equity.  17
Cultivating trust and communication about how to understand and manage the risks of AI 18
systems will help create opportunities for inno vation and realize the full potential of this 19
technology.  20
Many activities related to managing risk for AI 21
are common to managing risk for other types of 22
technology. An AI Risk Management Framework 23
(AI RMF, or Framework) can address challenges 24
unique to AI systems. This AI RMF is an initial 25
attempt to describe how  the risks from AI -based 26
systems differ from other domains and to 27
encourage and equip many different stakeholders 28
in AI to address those risks purposefully.  29
This voluntary framework provides a flexible, 30
structured, and measurable process to address AI ris ks throughout the AI lifecycle, offering 31
guidance for the development and use of trustworthy and responsible AI. It is intended to 32
improve understanding of and to help organizations manage both enterprise and societal risks 33
related to the development, depl oyment, and use of AI systems. Adopting the AI RMF can assist 34
organizations, industries, and society to understand and determine their acceptable levels of risk.  35 It is important to note that the AI RMF is
neither a checklist nor should be used in
any way to certify an AI system. Likewise,
using the AI RMF does not substitute fo r
due diligence and judgment by
organizations and individuals in deciding
whether to design, develop, and deploy AI
technologies – and if so, under what
conditions.
Initial Draft
2 In addition, it can be used to map compliance considerations beyond those addressed by this 1
framework, including existing regulations, laws, or other mandatory guidance.  2
Risks to any software or information -based system apply to AI; that includes important concerns 3
related to cybersecurity, privacy, safety, and infrastructure. This framework aims  to fill the gaps 4
related specifically to AI. Rather than repeat information in other guidance, users of the AI RMF 5
are encouraged to address those non -AI specific issues via guidance already available.  6
Part 1 of this framework establishes the context for the 7
AI risk management process. Part 2 provides guidance 8
on outcomes and activities to c arry out that process to 9
maximize the benefits and minimize the risks of AI. 10
Part 3 [yet to be developed] assists in using the AI 11
RMF and offers sample practices to be considered in 12
carrying out this guidance, before, during, and after AI products, service s, and systems are 13
developed and deployed.  14
The Framework, and supporting resources, will be updated and improved based on evolving 15
technology and the standards landscape around the globe. In addition, as the AI RMF is put into 16
use, additional lessons will  be learned that can inform future updates and additional resources.  17
NIST’s development of the AI RMF in collaboration with the private and public sectors is 18
consistent with its broader AI ef forts called for by the National AI Initiative Act of 2020 (P.L. 19
116-283), the  National Security Commission on Artificial Intelligence  recommendations, and the 20
Plan for Federal Engagement in AI Standards and Related Tools . Engagement with the broad AI 21
community during this Framework’s development also informs AI research and development 22
and evaluation by NIST and others.  23
2 Scope  24
The NIST AI RMF offers a process for managing risks related to AI systems across a wide 25
spectrum of types, applications, and maturity. This framework is organized and intended to be 26
understood and u sed by individuals and organizations, regardless of sector, size, or level of 27
familiarity with a specific type of technology. Ultimately, it will be offered in multiple formats, 28
including online versions, to provide maximum flexibility.  29
The AI RMF serves a s a part of a broader NIST resource center containing documents, 30
taxonomy, suggested toolkits, datasets, code, and other forms of technical guidance related to the 31
development and implementation of trustworthy AI. Resources will include a knowledge base of  32
terminology related to trustworthy and responsible AI and how those terms are used by different 33
stakeholders.  34
The AI RMF is not a checklist nor a compliance mechanism to be used in isolation. It should be 35
integrated within the organization developing and  using AI and be incorporated into enterprise 36
For the purposes of the NIST AI RMF
the term artificial intelligence  refers
to algorithmic processes that learn
from data in an automated or semi -
automated manner.
Initial Draft
3  risk management; doing so ensures that AI will be treated along with other critical risks, yielding 1
a more integrated outcome and resulting in organizational efficiencies.  2
Attributes of the AI RMF  3
The AI RMF strives to:  4
1. Be risk -based, resource efficient, and voluntary.  5
2. Be consensus -driven and developed and regularly updated through an open, transparent  process. 6
All stakeholders should have the opportunity to contribute to the AI RMF’s development.   7
3. Use clear and plain language that is understandable by a broad audience, including senior 8
executives, government officials, non -governmental organization lea dership, and those who are 9
not AI professionals – while still of sufficient technical depth to be useful to practitioners. The AI 10
RMF should allow for communication of AI risks across an organization, between organizations, 11
with customers, and to the publi c at large.  12
4. Provide common language and understanding to manage AI risks. The AI RMF should offer 13
taxonomy, terminology, definitions, metrics, and characterizations for AI risk.  14
5. Be easily usable and mesh with other aspects of risk management. Use of the Fr amework should 15
be intuitive and readily adaptable as part of an organization’s broader risk management strategy 16
and processes. It should be consistent or aligned with other approaches to managing AI risks.  17
6. Be useful to a wide range of perspectives, sectors , and technology domains. The AI RMF should 18
be both technology agnostic and applicable to context -specific use cases.   19
7. Be outcome -focused and non -prescriptive. The Framework should provide a catalog of outcomes 20
and approaches rather than prescribe one -size-fits-all requirements.  21
8. Take advantage of and foster greater awareness of existing standards, guidelines, best practices, 22
methodologies, and tools for managing AI risks – as well as illustrate the need for additional, 23
improved resources.  24
9. Be law - and regula tion-agnostic. The Framework should support organizations’ abilities to 25
operate under applicable domestic and international legal or regulatory regimes.  26
10. Be a living document. The AI RMF should be readily updated as technology, understanding, and 27
approaches  to AI trustworthiness and uses of AI change and as stakeholders learn from 28
implementing AI risk management generally and this framework in particular.  29
3 Audience  30
AI risk management is a complex and relatively new area, and the list of individuals, groups, 31
communities, and organizations that can be affected by AI technologies is extensive. Identifying 32
and managing AI risks and impacts – both positive and adverse – requires a broad set of 33
perspectives and stakeholders.  34
Initial Draft
4 1
Figure 1: Key stakeholder groups associ ated with the AI RMF.  2
As Figure 1 illustrates, NIST has identified four stakeholder groups as intended audiences of this 3
Framework: AI system stakeholders, operators and evaluators, external stakeholders, and the 4
general public. Ideally, members of all sta keholder groups would be involved or represented in 5
the risk management process, including those individuals and community representatives that 6
may be affected by the use of AI technologies.  7
AI system  stakeholders are those  who have the most control and responsibility over the design, 8
development, deployment, and acquisition of AI systems,  and the implementation of AI risk 9
management practices. This group comprises the primary adopters of the AI RMF. They may 10
include individ uals or teams within or among organizations with responsibilities to commission, 11
fund, procure, develop, or deploy an AI system: business teams, design and development teams, 12
internal risk management teams, and compliance teams. Small to medium -sized organ izations 13
face different challenges in implementing the AI RMF than large organizations.  14
Operators and evaluators  provide monitoring and formal/informal test, evaluation, validation, 15
and verification (TEVV) of system performance, relative to both technical and socio -technical 16
requirements. These stakeholders, which include organizations which operate or employ AI 17
systems, use the output for decisions or to evaluate their performance.  This group can include 18
users who interpret or incorporate the output of AI  systems in settings with a high potential for 19
adverse impacts. They might include academic, public, and private sector researchers; 20
professional evaluators and auditors; system operators; and expert end users.  21
External stakeholders provide formal and/or q uasi-formal norms or guidance for specifying and 22
addressing AI risks. External to the primary adopters of the AI RMF, they can include trade 23

Initial Draft
5 groups, standards developing organizations, advocacy groups, and civil society organizations. 1
Their actions can des ignate boundaries for operation (technical or legal) and balance societal 2
values and priorities related to civil liberties and rights, the economy, and security.  3
The general public  is most likely to directly experience positive and adverse impacts of AI 4
technologies. They may provide the motivation for actions taken by the other stakeholders and 5
can include individuals, communities, and consumers in the context where an AI system is 6
developed or deployed.  7
4 Framing Risk  8
AI systems hold the potential to advanc e our quality of life and lead to new services, support, 9
and efficiencies for people, organizations, markets, and society. Identifying, mitigating, and 10
minimizing risks and potential harms associated with AI technologies are essential steps towards 11
the acc eptance and widespread use of AI technologies. A risk management framework should 12
provide a structured, yet flexible, approach for managing enterprise and societal risk resulting 13
from the incorporation of AI systems into products, processes, organizations,  systems, and 14
societies. Organizations managing an enterprise’s AI risk also should be mindful of larger 15
societal AI considerations and risks. If a risk management framework can help to effectively 16
address and manage AI risk and adverse impacts, it can lea d to more trustworthy AI systems.  17
4.1 Understanding Risk and Adverse Impacts  18
Risk is a measure of the extent to which an entity is negatively influenced  by a potential 19
circumstance or event. Typically, risk is a function of 1) the adverse impacts that could arise if 20
the circumstance or event occurs; and 2) the likelihood of occurrence. Entities can be individuals, 21
groups, or communities as well as systems , processes, or organizations.  22
The impact of AI systems can be positive, negative, or both and can address, create, or result in 23
opportunities or threats. According to the International Organization for Standardization (Guide 24
73:2009; IEC/ISO 31010), certa in risks can be positive. While risk management processes 25
address adverse impacts, this framework intends to offer approaches to minimize anticipated 26
negative impacts of AI systems and identify opportunities to maximize positive impacts. 27
Additionally, this framework is designed to be responsive to new risks as they emerge rather than 28
enumerating all known risks in advance. This flexibility is particularly important where impacts 29
are not easily foreseeable, and applications are evolving rapidly. While AI ben efits and some AI 30
risks are well -known , the AI community is only beginning to understand and classify incidents 31
and scenarios that result in harm.  Figure 2 provides examples of potential harms from AI 32
systems.  33
Risk management can also drive AI developers a nd users to understand and account for the 34
inherent uncertainties and inaccuracy of their models and systems, which in turn can increase the 35
Initial Draft
6 overall performance and trustworthiness of those models. Managing risk and adverse impacts 1
contributes to building trustworthy AI technologies and applications  2
 3
Figure 2:  Examples of potential harms from AI systems.  4
4.2 Challenges for AI Risk Management  5
4.2.1 Risk Measurement   6
AI risks and impacts that are not well -defined or adequately understood are difficult to measure 7
quantitatively or qualitatively. The presence of third -party data or systems may also complicate 8
risk measurement. Those attempting to measure the adverse impact on a population may not be 9
aware that certain demographics may experience harm differe ntly than others.  10
AI risks can have a temporal dimension. Measuring risk at an earlier stage in the AI lifecycle 11
may yield different results than measuring risk at a later stage. Some AI risks may have a low 12
probability in the short term but have a high l ikelihood for adverse impacts. Other risks may be 13
latent at present but may increase in the long term as AI systems evolve.  14
Furthermore, inscrutable AI systems can complicate the measurement of risk. Inscrutability can 15
be a result of the opaque nature of AI technologies (lack of explainability or interpretability), 16
lack of transparency or documentation in AI system development or deployment, or inherent 17
uncertainties in AI systems.  18
4.2.2 Risk Thresholds   19
Thresholds refer to the values used to establish concrete decision points and operational limits 20
that trigger a response, action, or escalation. AI risk thresholds (sometimes referred to as Key 21
Risk Indicators)  can involve both technical factors  (such as error rates for determining bias) and 22
human values (such as  social or legal norms for appropriate levels of transparency). These 23

Initial Draft
7 factors and values can establish levels of risk (e.g., low, medium, or high) based on broad 1
categories of adverse impacts or harms.  2
Thresholds and values can also determine where AI syst ems present unacceptable risks to certain 3
organizations, systems, social domains, or demographics. In these cases, the question is not how 4
to better manage risk of AI, but whether an AI system should be designed, developed, or 5
deployed at all.  6
The AI RMF d oes not prescribe risk thresholds or values. Risk tolerance – the level of risk or 7
degree of uncertainty that is acceptable to  organizations or society – is context and use case - 8
specific. Therefore, risk thresholds should be set through policies and norms that can be 9
established by AI system owners, organizations, industries, communities, or regulators (who 10
often are acting on behalf of individuals or societies). Risk thresholds and values are likely to 11
change and adapt over time as policies and norms chang e or evolve. In addition, different 12
organizations may have different risk thresholds (or tolerance) due to varying organizational 13
priorities and resource considerations. Even within a single organization there can be a balancing 14
of priorities and tradeoffs  between technical factors and human values. Emerging knowledge and 15
methods for better informing these decisions are being developed and debated by business, 16
governments, academia, and civil society. To the extent that challenges for specifying risk 17
thresh olds or determining values remain unresolved, there may be contexts where a risk 18
management framework is not yet readily applicable for mitigating AI risks and adverse impacts. 19
The AI RMF provides the opportunity for organizations to specifically define th eir risk 20
thresholds and then to manage those risks within their tolerances.  21
4.2.3 Organizational Integration  22
The AI RMF is not a checklist nor a compliance mechanism to be used in isolation. It should be 23
integrated within the organization developing and using AI  technologies and be incorporated into 24
enterprise risk management; doing so ensures that AI will be treated along with other critical 25
risks, yielding a more integrated outcome and resulting in organizational efficiencies.  26
Organizations need to establish an d maintain the appropriate accountability mechanisms, roles 27
and responsibilities, culture, and incentive structures for risk management to be effective. Use of 28
the AI RMF alone will not lead to these changes or provide the appropriate incentives. Effective  29
risk management needs organizational commitment at senior levels and may require significant 30
cultural change for an organization or industry.  31
Small to medium -sized organizations face different challenges in implementing the AI RMF than 32
large organizations.  33
5 AI Risks and Trustworthiness  34
The AI RMF uses a three -class taxonomy, depicted in Figure 3, to classify characteristics that 35
should be considered in comprehensive approaches for identifying and managing risk related to 36
AI systems: technical characteristics, socio -technical characterist ics, and guiding principles . 37
Initial Draft
8 This AI RMF taxonomy frames AI risk using characteristics that are aligned with trustworthy AI 1
systems, in conjunction with contextual norms and values. Since AI trustworthiness and risk are 2
inversely related, approaches which enhance trustworthiness can contribute to a reduction or 3
attenuation of related risks. The AI RMF taxonomy articulates several key building blocks of 4
trustworthy AI within each category, which are particularly suited to the examination of potential 5
risk.  6
7
Figure 3: AI Risks and Trustworthiness. The three -class taxonomy to classify characteristics that should 8
be considered in comprehensive approaches for identifying and managing risk related to AI systems. The 9
taxonomy articulates several key building blo cks of trustworthy AI within each category, which are 10
particularly suited to the examination of potential risk.  11
Figure 4 provides a mapping of the AI RMF taxonomy to the terminology used by the 12
Organisation for Economic Co -operation and Development (OECD)  in their Recommendation 13
on AI , the European Union (EU)  Artificial Intelligence Act , and U nited States Executive Order 14
(EO) 13960.  15
5.1 Technical Characteristics  16
Technical characteristics in the AI RMF taxonomy refer to factors that are under the direct 17
control of AI system designers and developers, and which may be measured using standard 18
evaluation criteria. Technical characteristics include the tradeoff between convergent - 19
discriminant validity (whether the data reflects what the user intends to measure and not  other 20
things) and statistical reliability (whether the data may be subject to high levels of statistical 21
noise and measurement bias). Validity of AI, especially machine learning (ML) models, can be 22
assessed using technical characteristics. Validity for de ployed AI systems is often assessed with 23
ongoing audits or monitoring that confirm that a system behaves as intended. It may be possible 24
to utilize and automate explicit measures based on variations of standard statistical or ML 25
techniques and specify thre sholds in requirements. Data generated from experiments that are 26
designed to evaluate system performance also fall into this category and might include tests of 27
causal hypotheses and assessments of robustness to adversarial attack.  28

Initial Draft
9 1
Figure 4: Mapping of AI RMF taxonomy to AI policy documents.  2
The following technical characteristics lend themselves well to addressing AI risk: accuracy, 3
reliability, robustness, and resilience (or ML security).  4
5.1.1 Accuracy  5
Accuracy  indicat es the degree to which the  ML model is correctly capturing a relationship that 6
exists within training data. Analogous to statistical conclusion validity, accuracy is examined  via 7
standard ML metrics (e.g., false positive and false negative rates, F1 -score, precision, and recall) ,   8
as well as assessment of  model underfit or overfit (high testi ng errors irrespective of error rates 9
in training).  It is widely acknowledged that current ML methods cannot guarantee that the 10
underlying model is capturing a causal relationship.  Establishing internal (causal) validity in ML 11
models is an active area of research. AI risk management processes should take into account the 12
potential risks to the enterprise and society if the underlying causal relationship inferred by a 13
model is not valid, calling i nto question decisions made on the basis of the model. Determining a 14
threshold for accuracy that corresponds with acceptable risk is fundamental to AI risk 15
management and highly context -dependent . 16
5.1.2 Reliability  17
Reliability indicates whether a model consisten tly generates the same results, within the bounds 18
of acceptable statistical error. Techniques designed to mitigate overfitting (e.g., regularization) 19
and to adequately conduct model selection in the face of the bias/variance tradeoff can increase 20
model rel iability. The definition of reliability is analogous to construct reliability in the social 21
sciences, albeit without explicit reference to a theoretical construct. Reliability measures may 22
give insight into the risks related to decontextualization, due to the common practice of reusing 23

Initial Draft
10 ML datasets or models in ways that cause them to become disconnected from the social contexts 1
and time periods of their creation. As with accuracy, reliability provides an evaluation of the 2
validity of models, and thus can be  a factor in determining thresholds for acceptable risk.  3
5.1.3 Robustness  4
Robustness is a measure of model sensitivity, indicating whether the model has minimum 5
sensitivity to variations in uncontrollable factors. A robust model will continue to function 6
despite  the existence of faults in its components. The performance of the model may be 7
diminished or otherwise altered until the faults are corrected. Measures of robustness might 8
range  from sensitivity of a model’s outputs to small changes in its inputs, but mig ht also include 9
error measurements on novel datasets. Robustness contributes to sensitivity analysis in the AI 10
risk management process.  11
5.1.4 Resilience or ML Security  12
A model that can withstand adversarial attacks, or more generally, unexpected changes in its 13
environment or use, may be said to be resilient or secure. This attribute has some relationship to 14
robustness except that it goes beyond the provenance of the data to encompass unexpected or 15
adversarial use of the model or data. Other common ML security con cerns relate to the 16
exfiltration of models, training data, or other intellectual property through AI system endpoints.  17
5.2 Socio -Technical Characteristics  18
Socio -technical characteristics in the AI RMF taxonomy refer to how AI systems are used and 19
perceived in individual, group, and societal contexts. This includes mental representations of 20
models, whether the output provided is sufficient to evaluate compliance (transparency), whether 21
model operations can be easily understood (explainability), whet her they provide output that can 22
be used to make a meaningful decision (interpretability), and whether the outputs are aligned 23
with societal values. Socio -technical factors are inextricably tied to human social and 24
organizational behavior, from the dataset s used by ML processes and the decisions made by 25
those who build them, to the interactions with the humans who provide the insight and oversight 26
to make such systems actionable.  27
Unlike technical characteristics, socio -technical characteristics require sig nificant human input 28
and cannot yet be measured through an automated process. Human judgment must be employed 29
when deciding on the specific metrics and the precise threshold values for these metrics. The 30
connection between human perceptions and interpretat ions, societal values, and enterprise and 31
societal risk is a key component of the kinds of cultural and organizational factors that will be 32
necessary to properly manage AI risks. Indeed, input from a broad and diverse set of 33
stakeholders is required throug hout the AI lifecycle to ensure that risks arising in social contexts 34
are managed appropriately.  35
The following socio -technical characteristics have implications for addressing AI risk: 36
explainability, interpretability, privacy, safety, and managing bias.  37
Initial Draft
11 5.2.1 Explainability  1
Explainability seeks to provide a programmatic, sometimes causal, description of h ow model 2
predictions are generated. Even given all the information required to make a model fully 3
transparent, a human must apply technical expertise if they want to understand how the model 4
works. Explainability refers to the user’s perception of how the model works – such as what 5
output may be expected for a given input. Explanation techniques tend to summarize or visualize 6
model behavior or predictions for technical audiences. Explanations can be useful in promoting 7
human learning from machine learning, for addressing transparency requirements, or for 8
debugging issues with AI systems and training data. However, risks due to explainability may 9
arise for many reasons, including, for example, a lack of fidelity or consistency in explanation 10
methodologies, or  if humans incorrectly infer a model’s operation, or the model is not operating 11
as expected. Risk from lack of explainability may be managed by descriptions of how models 12
work to users’ skill levels. Explainable systems can be more easily debugged and moni tored, and 13
lend themselves to more thorough documentation, audit, and governance.  14
Explainability is related to transparency. Typically the more opaque a model is, the less it is 15
considered explainable. However, transparency does not guarantee explainabilit y, especially if 16
the user lacks an understanding of ML technical principles.  17
5.2.2 Interpretability  18
Interpretability seeks to fill a meaning deficit. Although explainability and interpretability are 19
often used interchangeably, explainability refers to a represen tation of the mechanisms 20
underlying an algorithm’s operation, whereas interpretability refers to the meaning of its output 21
in the context of its designed functional purpose. The underlying assumption is that perceptions 22
of risk stem from a lack of ability to make sense of, or contextualize, model output appropriately. 23
Model interpretability refers to the extent to which a user can determine adherence to this 24
function and the consequent implications of this output upon other consequential decisions for 25
that user. Interpretations are typically contextualized in terms of values and reflect simple, 26
categorical distinctions. For example, a society may value privacy and safety, but individuals 27
may have different determinations of safety thresholds. Risks to interp retability can often be 28
addressed by communicating the interpretation intended by model designers, although this 29
remains an open area of research. The prevalence of different interpretations can be readily 30
measured with psychometric instruments.  31
5.2.3 Privacy  32
Privacy refers generally to the norms and practices that help to safeguard values such as human 33
autonomy and dignity. These norms and practices typically address freedom from intrusion, 34
limiting observation, or individuals’ control of facets of their identit ies (e.g., body, data, 35
reputation). Like safety and security, specific technical features of an AI system may promote 36
privacy, and assessors can identify how the processing of data could create privacy -related 37
problems. However, determinations of likelihoo d and severity of impact of these problems are 38
contextual and vary among cultures and individuals.  39
Initial Draft
12 5.2.4 Safety  1
Safety as a concept is highly correlated with risk and generally denotes an absence (or 2
minimization) of failures or conditions that render a system dangerous. As AI systems interact 3
with humans more directly in factories and on the roads, for example, the safety of these systems 4
is a serious consideration for AI risk management. Safety is often – though not always – 5
considered through a legal lens. Pr actical approaches for AI safety often relate to rigorous 6
simulation and in -domain testing, real -time monitoring, and the ability to quickly shut down or 7
modify misbehaving systems.  8
5.2.5 Managing Bias  9
NIST has identified three major categories of bias in AI: s ystemic, computational, and human.  10
Managing bias in AI systems requires an approach that considers all three categories.  11
Bias exists in many forms, is omnipresent in society, and can become ingrained in the automated 12
systems that help make decisions about our lives. While bias is not always a negative 13
phenomenon, certain biases exhibited in AI models and systems can perpetuate and amplify 14
negative impacts on individuals, organizations, and society, and at a speed and scale far beyond 15
the traditional discrim inatory practices that can result from implicit human or systemic biases. 16
Bias is tightly associated with the concepts of transparency and fairness in society. See NIST 17
publication “Towards a Standard for Identifying and Managing Bias in Artificial Intelli gence.”  18
When managing risks in AI systems it is important to understand that the attributes 19
of the AI RMF risk taxonomy are interrelated. Highly secure but unfair systems, 20
accurate but opaque and uninterpretable systems, and inaccurate, but fair, secure, 21
privacy -protected, and transparent systems are all undesirable. It is possible for 22
trustworthy AI systems to achieve a high degree of risk control while retaining a 23
high level of performance quality. Achieving this difficult goal requires a 24
comprehensive ap proach to risk management, with tradeoffs among the technical 25
and socio -technical characteristics.  26
5.3 Guiding Principles  27
Guiding principles in the AI RMF taxonomy  refer to broader societal norms and values that 28
indicate societal priorities. While there is no  objective standard for ethical values, as they are 29
grounded in the norms and legal expectations of specific societies or cultures, it is widely agreed 30
that AI technologies should be developed and deployed in ways that meet contextual norms and 31
ethical val ues. When specified as policy, guiding principles can enable AI stakeholders to form 32
actionable, low -level requirements. Some requirements will be translated into quantitative 33
measures of performance and effectiveness, while some may remain qualitative in nature.  34
Guiding principles that are relevant for AI risk include fairness, accountability, and transparency.  35
Fairness in AI systems includes concerns for equality and equity by addressing socio -technical 36
issues such as bias and discrimination. Individual human operators and their organizations 37
should be answerable and held accountable for the outcomes of AI systems, particularly adverse 38
Initial Draft
13 impacts stemming from risks. Absent transparency, users are left to guess about these factors and 1
may make unwarranted an d unreliable  assumptions regarding model provenance. Transparency 2
is often necessary for actionable redress related to incorrect and adverse AI system outputs.  3
5.3.1 Fairness  4
Standards of fairness can be complex and difficult to define because perceptions of fairness 5
differ among cultures. For one type of fairness, process fairness, AI developers assume that ML 6
algorithms are inherently fair because the same procedure applies r egardless of user. However, 7
this perception has eroded recently as awareness of biased algorithms and biased datasets has 8
increased. Fairness is increasingly related to the existence of a harmful system, i.e., even if 9
demographic parity and other fairness measures are satisfied, sometimes the harm of a system is 10
in its existence. While there are many technical definitions for fairness, determinations of 11
fairness are not generally just a technical exercise. Absence of harmful bias is a necessary 12
condition fo r fairness.  13
5.3.2 Accountability  14
Determinations of accountability in the AI context are related to expectations for the responsible 15
party in the event that a risky outcome is realized. Individual human operators and their 16
organizations should be answerable and held accountable for the outcomes of AI systems, 17
particularly adverse impacts stemming from risks. The relationship between risk and 18
accountability associated with AI and technological systems more broadly differs across cultural, 19
legal, sectoral, and soci etal contexts. Grounding organizational practices and governing 20
structures for harm reduction, like risk management, can help lead to more accountable systems.   21
5.3.3 Transparency  22
Transparency seeks to remedy a common information  imbalance between AI system operators 23
and AI system consumers. Transparency reflects the extent to which information is available to a 24
user when interacting with an AI system. Its scope spans from design decisions  and training data  25
to model training, the structure of the model, its intended use case, how and when deployment 26
decisions were made and by whom, etc. Absent transparency, users are left to guess about these 27
factors and may make unwarranted and unreliable assumptions regarding model provenance. 28
Transparency is often necessary for actionable redress related to incorrect and adverse AI system 29
outputs. A transparent system is not necessarily a fair, priva cy-protective , secure, or robust 30
system. However, it  is difficult to determine whether an opaque s ystem possesses such 31
desiderata, and to do so over time as complex systems evolve.  32
  33
Initial Draft
14 Part 2: Core and Profiles  1
6 AI RMF Core  2
The AI RMF Core provides outcomes and actions that enable dialogue, understanding, and 3
activities to manage AI risks. The Core is  composed  of three elements: functions, categories, and 4
subcategories. As illustrated in Figure 5, functions organize AI risk management activities at 5
their highest level to map, measure, manage, and govern AI risks. Within each function, 6
categories and subcategories subdivide the function into specific outcomes and actions . 7
 8
Figure 5: Functions organize AI risk management activities at their highest level to map, measure, 9
manage, and govern AI risks. Governance is a cross -cutting function that is infused throughout and 10
informs the other functions of the process.  11
Govern is a cross -cutting function that is infused throughout and informs the other functions of 12
the process. Aspects of Govern, especially those related to compliance or evaluation, shoul d be 13
integrated into each of the other functions. Assuming a governance structure is in place, 14
functions may be performed in any order across the AI lifecycle as deemed to add value by a 15
user of the framework. In most cases, it will be more useful and effe ctive to begin with Map 16
before Measure and Manage. Regardless, the process should be iterative, with cross -referencing 17
between functions as necessary. Similarly, there are categories and subcategories with elements 18
that apply to multiple functions.  19
Technic al and socio -technical characteristics and guiding principles of AI trustworthiness are 20
essential considerations for each function . AI RMF core functions should be carried out in a way 21
that reflects diverse and multidisciplinary perspectives, potentially i ncluding the views of 22
stakeholders from outside the organization. Risk management should be performed throughout 23
the AI system life cycle (Figure 6) to ensure it is continuous and timely.  24

Initial Draft
15 On the following pages, Tables 1 through 4 provide the Framework Cor e listing.  1
 2
Figure 6:  Risk management should be performed throughout the AI system life cycle to ensure it is 3
continuous and timely.  Example activities for each stage of the AI lifecycle follow. Pre-Design : data 4
collection, curation or selection, problem formulation, and identification of stakeholders . Design & 5
Development : data analysis, data cleaning, model training, and requirement analysis. Test & Evaluation : 6
technical validation and verification. Deployment : user feedback and override, post deployment  7
monitoring, and decommissioning.  8
6.1 Map  9
The Map function  establishes the context and applies the attributes of the AI RMF taxonomy 10
(Figure 3) to frame risks related to an AI system. The information gathered while carrying out 11
this function informs decisions about model management, including an initial decision about 12
appropriateness or the need for an AI solution. Determination of whether AI use is appropriate or 13
warranted can be considered in comparison to the status quo per a qualitative or more formal 14
quantitative analysis of benefits, costs, and risks.  15
A com panion document describes practices related to mapping AI risks. Table 1 lists the Map 16
function’s categories and subcategories.  17
Table 1: Example of categories and subcategories for Map function  18
ID Category  Subcategory
Map: Context is recognized  and risks related to the context are identified
1
 Context is established and
understood.  Intended purpose, setting in which the AI system will be deployed,
the specific set of users along with their expectations, and impacts
of system use are understood and documented as appropriate.
The business purpose or context of use ha s been clearly defined or
– in the case of assessing existing AI systems – re-evaluated .
The organization’s mission and relevant goals for the AI
technology are understood.
Stakeholders are defined, a plan for continuous
engagement/communication is developed, and outreach is
conducted.

Initial Draft
16 System requirements are elicited and understood from relevant
stakeholders (e.g., “the system shall respect the privacy of its
users”). Design decisions take socio -technical implications into
account for addressing AI risks.
Risk tolerances are determined.
2 Classification of AI system
is performed.  The specific task that the AI system will support is defined (e.g.,
recommendation, classification, etc.).
Considerations related to data collection and selection are
identified. (e.g., availability, representativeness, suitability).
Detailed information is provided about the operational context in
which the AI system will be deployed (e.g., human -machine
teaming, etc.) and how output will be utilized.
3 AI capabilities, targeted
usage, goals, and expected
benefits and costs over
status quo are understood.  Benefits of intended system behavior are examined.
Cost (monetary or otherwise) of errors or unintended system
behavior is examined.
Targeted application scope is specified and narrowed to the extent
possible based on established context and AI system classification.
4
 Risks and harms to
individual, organizational,
and societal perspectives are
identified.  Potential business and societal (positive or adverse) impacts of
technical and socio -technical characteristics for potential users, the
organizations, or society as a whole are understood.
Potential harms of the AI system are elucidated along  technical
and socio -technical characteristics and aligned with guiding
principles.
Likelihood of each harm is understood based on expected use, past
uses of AI systems in similar contexts, public incident reports or
other data.
Benefits of the AI system outweigh the risks,  and risks can be
assessed and managed. Ideally, this evaluation should be
conducted by an independent third party or by experts who did not
serve as front -line developers for the system, and who consults
experts, stakeholders, and impacted communities.
6.2 Measure  1
The Measure function provides knowledge relevant to the risks associated with attributes of the 2
AI RMF taxonomy in Section 5. This includes analysis, quantitative or qualitative assessment, 3
and tracking the risk and its impact.  Risk analysis and measurement may involve a detailed 4
consideration of uncertainties, tradeoffs, consequences, likelihood, events, controls, and their 5
effectiveness. An event can have multiple causes and consequences and can affect multiple 6
objectives.  7
Methods a nd metrics for quantitative or qualitative measurement rapidly evolve. Both qualitative 8
and quantitative methods should be used to track risks.  9
A companion document describes practices related to measuring AI risks. Table 2 lists the 10
Measure Function’s ca tegories and subcategories.  11
Initial Draft
17 Table 2: Example of categories and subcategories for Measure function  1
ID Category  Subcategory
Measure: Identified risks are assessed, analyzed, or tracked
1 Appropriate methods and
metrics are identified and
applied.

 Elicited system  requirements are analyzed .
Approaches and metrics for quantitative or qualitative measurement of the
enumerated risks, including technical measures of performance for specific
inferences, are identified and selected for implementation.
The appropriateness of metrics and effectiveness of existing controls is
regularly assessed and updated.
2 Systems are evaluated.  Accuracy, reliability, robustness, resilience (or ML security), explainability
and interpretability, privacy, safety, bias, and other system performance or
assurance criteria are measured, qualitatively or quantitatively.
Mechanisms for tracking identified risks over time are in place, particularly
if potential risks are difficult to assess using currently available
measurement techniques, or are not yet available.
3 Feedback from a ppropriate
experts and stakeholders is
gathered and assessed.  Subject matter experts assist in measuring and validating whether the
system is performing consistently with their intended use and as expected
in the specific deployment setting.
Measurable performance improvements (e.g., participatory methods) based
on consultations are identified.
6.3 Manage   2
This function addresses risks which have been mapped and measured and are managed in order 3
to maximize benefits and minimize adverse impacts.  These are  risks associated with the 4
attributes of the AI RMF taxonomy (Section 5). Decisions about this function take into account 5
the context and the actual and perceived consequences to external and intern al stakeholders. That 6
includes interactions of the AI system with the status quo world and potential benefits or costs.  7
Management can take the form of deploying the system as is if the risks are deemed tolerable; 8
deploying the system in production environ ments subject to increased testing or other controls; 9
or decommissioning the system entirely if the risks are deemed too significant and cannot be 10
sufficiently addressed. Like other risk management efforts, AI risk management must be 11
ongoing.  12
Practices rel ated to AI risk management are discussed in the companion document. Table 3 lists 13
the Manage function’s categories and subcategories.  14
Table 3: Example categories and subcategories for Manage function  15
ID Category  Subcategory
Manage: Risks are prioritized and acted upon based on a projected impact
1 Assessments of potential
harms and results of analyses
conducted via the map and
measure functions are used to
respond to and manage AI
risks.  Assessment of whether the AI is the right tool to solve the given
problem (e.g., if the system should be further developed or
deployed).
Identified risks are prioritized based on their impact, likelihood,
resources required to address them, and available methods to
address them.
Initial Draft
18 Responses to enumerated risks are identified and planned.
Responses can include mitigating, transferring or sharing, avoiding,
or accepting AI risks.
2 Priority  actions to maximize
benefits and minimize harm
are planned , prepared,
implemented, and
communicated to internal and
external stakeholders as
appropriate (or required) and
to the extent practicable.  Resources required to manage risks are taken into account , along
with viable alternative systems, approaches, or methods, and related
reduction in severity of impact or likelihood of each potential
action.
Plans are in place, both performance and control -related, to sustain
the value of the AI system once deployed.
Mechanisms are in place and maintained to supersede, disengage,
or deactivate existing applications of AI that demonstrate
performance or outcomes that are inconsistent with their intended
use.
3 Responses to enumerated and
measured risks are
documented and monitored
over time.  Plans related to post depl oyment monitoring of the systems are
implemented, including mechanisms for user feedback, appeal and
override, decommissioning , incident response, and change
management.
Measurabl e performance improvements (e.g., participatory
methods) based on consultations are integrated into system updates.
6.4 Govern  1
The Govern function cultivates and implements a culture of risk management within 2
organizations devel oping, deploying, or acquiring AI systems. Governance is designed to ensure 3
risks and potential imp acts are identified, measured, and managed effectively and consistently.  4
Governance processes focused on potential impacts of AI technologies are the backbon e of risk 5
management. Governance focuses on technical aspects of AI system design and development as 6
well as on organizational practices and competencies that directly impact the individuals 7
involved in training, deploying, and monitoring such systems. Gov ernance should address supply 8
chains, including third -party software or hardware systems and data as well internally developed 9
AI systems.  10
Governance is a function that has relevance across all other functions, reflecting the importance 11
of infusing governa nce considerations throughout risk management processes and procedures. 12
Attention to g overnance is a continual and intrinsic requirement for effective AI risk 13
management over an AI system ’s entire lifespan . For example, compliance with internal and 14
externa l policies or regulations is a universal aspect of the governance function in risk 15
management. Similarly, governance provides a structure through which AI risk management 16
functions can align with organizational policies and strategic priorities, including those not 17
directly related to AI systems.  18
A companion document describes p ractices related to governance of AI risk management.  Table 19
4 lists Govern function’s categories and subcategories.  20
  21
Initial Draft
19 Table 4: Example categories and subcategories for Govern functio n 1
ID Category  Subcategory
Govern: A culture of risk management is cultivated and present
1 Policies, process es, procedures  and
practices across the organization
related to the development, testing,
deployment, use and auditing of AI
systems  are in place, transparent,
and implemented effectively.  The risk management process and its outcomes are documented
and traceable  through  transparent mechanisms , as appropriate
and to the extent practicable.
Ongoing monitoring and periodic review of the risk
management process and its outcomes are planned, with
responsibilities clearly defined.
Methods for ensuring all dimensions of trustworthy AI are
embedded into policies, processes, and procedures.
2 Accountability structures are in
place to ensure that the appropriate
teams and individuals are
empowered, responsible, and
trained for managing the risks of
AI systems.  Roles and responsibilities and lines of communication related
to identifying and addressing AI risks are clear to individuals
and teams throughout the organization.
The organization’s personnel and partners are provided AI risk
management awareness education and training to enable them
to perform their duties and responsibilities consistent with
related policies, procedures, and agreements.
Executive leadership of the organization considers decisions
about AI system development and deployment ultimately to be
their responsibility.
3 Workforce diversity, equity and
inclusion processes are prioritized.  Decision making throughout the AI lifecyc le is informed by a
demographically and disciplinarily diverse team, including
internal and external personnel. Specifically, teams that are
directly engaged with identifying design considerations and
risks include a diversity of experience, expertise and
backgrounds to ensure AI systems meet requirements beyond a
narrow subset of users.
4 Teams are committed to a culture
that considers and communicates
risk. Teams are encouraged to consider and document the impacts of
the technology they design and to dev elop and communicate
about these impacts more broadly.
Organizational practices are in place to ensure that teams
actively challenge and question steps in the design and
development of AI systems to minimize harmful impacts.
5 Processes are in place to ensure
that d iversity, equity, inclusion,
accessibility, and cultural
considerations  from potentially
impacted individuals and
communities are fully taken into
account.  Organizational policies and practices are in place that prioritize
the consideration and adjudication of external stakeholder
feedback regarding the potential individual and societal harms
posed by AI system deployment.
Processes are in place to empower teams to make decisions
about if and how to develop and deploy AI systems based on
these considerations, and define periodic reviews of impacts,
including potential harm.
6 Clear policies and procedures are
in place to address AI risks arising
from supply chain issues, including
third -party software and data.  Policies and procedures include guidelines for ensuring supply
chain and partner involvement and expectations regarding the
value and trustworthiness of third -party data or AI systems.
Contingency processes are in place to address potential issues
with third -party data or AI systems.
 2
Initial Draft
20 7 AI RMF Profiles  1
Profiles are instantiations of the AI RMF Core for managing AI risks for context -specific use 2
cases. Using the AI RMF, profiles illustrate how risk can be managed at various stages of the AI 3
lifecycle or in sector, technology, or end -use applications. Profiles may state an “as is” and 4
“target” state of how an organization addresses AI risk management.  5
NOTE: Development of profiles is deferred until later drafts of the AI RMF are developed with  6
the community. NIST welcomes contributions of AI RMF profiles. These profiles will inform 7
NIST and the broader community about the usefulness of the AI RMF and likely lead to 8
improvements which can be incorporated into future versions of the framework.  9
8 Effectiveness of the AI RMF  10
The goal of the AI RMF is to offer a resource for improving the ability of organizations to 11
manage AI risks in order to maximize benefits and to minimize AI -related harms. Organizations 12
are encouraged to periodically evaluate whet her the AI RMF has improved their ability to 13
manage AI risks, including but not limited to their policies, processes, practices, implementation 14
plans, indicators, and expected outcomes.  15
NOTE: NIST is deferring development of this section until later draft s of the AI RMF are 16
developed with the community.  17
Part 3: Practice Guide  18
9 Practice Guide  19
NOTE: NIST is developing a companion Practice Guide which will include additional 20
examples and practices that can assist in using the AI RMF. That Guide, which will res ide 21
online only and will be updated regularly with contributions expected to come from many 22
stakeholders, will be part of the NIST AI Resource Center that is being established.  23


Testimony
Artificial Intelligence: Challenges and
Opportunities for the Department of
Defense

Jason Matheny
CT-A2723 -1
Testimony presented  to the  U.S. Senate Committee on Armed Services, Subcommittee on Cybersecurity , on April 19, 2023

CORPORATION
For more information on this publication, visit www.rand.org/t/CTA2723-1
Testimonies
RAND testimonies record testimony presented or submitted by RAND associates to federal,
state, or local legislative committees; government-appointed commissions and panels; and private review and oversight bodies.
Published by the RAND Corporation, Santa Monica, Calif.
© 2023 RAND Corporation
RA®  is a registered trademark.
Limited Print and Electronic Distribution Rights
This publication and trademark(s) contained herein are protected by law. This representation of
RAND intellectual property is provided for noncommercial use only. Unauthorized posting of
this publication online is prohibited; linking directly to its webpage on rand.org is encouraged. Permission is required from RAND to reproduce, or reuse in another form, any of its research products for commercial purposes. For information on reprint and reuse permissions, please visit
www.rand.org/pubs/permissions.
www.rand.org

 1 Artificial Intelligence: Challenges and Opportunities for the Department of Defense
Testimony of Jason Matheny1
The RAND Corporation2
Before the Committee on Armed Services
Subcommittee on Cybersecurity
United States Senate
April 19, 2023
hairman Manchin, Ranking Member Rounds, and members of the committee: Good
morning, and thank you for the opportunity to testify today. I’m the president and CEO of RAND, a nonprofit and nonpartisan research organization. Before RAND, I served in
the White House National Security Council and Office of Science and Technology Policy, as a
commissioner on the National Security Commission on Artificial Intelligence, as assistant
director of national intelligence, and as director of the Intelligence Advanced Research Projects Activity, which develops advanced technologies for the U.S. intelligence community.
For the past 75 years, RAND has conducted research in support of U.S. national security, and
we currently manage four federally funded research and development centers (FFRDCs) for the federal government: one for the Department of Homeland Security (DHS) and three for the Department of Defense (DoD). Today, I’ll focus my comments on how DoD can best ensure that progress in artificial intelligence (AI) benefits U.S. national security instead of degrading it.
Among a broad set of technologies, AI stands out for both its rate of progress and its scope of
applications. AI holds the potential to broadly transform entire industries, including ones critical

1 The opinions and conclusions expressed in this testimony are the author’s alone and should not be interpreted as
representing those of the RAND Corporation or any of the sponsors of its research.
2 The RAND Corporation is a research organization that deve lops solutions to public policy challenges to help make
communities throughout the world safer and more secure, healthier and more prosperous. RAND is nonprofit,
nonpartisan, and committed to the public interest.  RAND’s mission is enabled through its core values of quality and
objectivity and its commitment to integrity and ethical behavior. RAND subjects its research publications to a robust and exacting quality -assurance process; avoids financial and other conflicts of interest through staff training, pro ject
screening, and a policy of mandatory disclosure; and pursues transparency through the open publication of research findings and recommendations, disclosure of the source of funding of published research, and policies to ensure intellectual independence.  This testimony is not a research publication, but witnesses affiliated with RAND
routinely draw on relevant research conducted in the organization.  C

 2 to our future economic competitiveness and our national security. Integrating AI into our
national security plans poses special challenges for several reasons:
• The technologies are driven by commercial entities that are frequently outside our
national security frameworks.
• The technologies are advancing quickly, typically outpacing policies and organizational
reforms within government.
• Assessments of the technologies require expertise that is concentrated in the private sector and that has rarely been used for national security.
• The technologies lack conventional intelligence signatures that distinguish benign from
malicious use.
The United States is currently the global leader in AI;
3 however, this may change as the
People’s Republic of China seeks to become the world’s primary AI innovation center by
2030—an explicit goal of China’s AI national strategy.4 In addition, both China and Russia are
pursuing militarized AI technologies,5 intensifying the challenges I just outlined. In response, I
will highlight four sets of actions that DoD could take:
1. Ensure that DoD cybersecurity strategies and cyber Red team activities track
developments in AI that could affect cyber defense and cyber offense, such as the
automated development of cyber weapons.
2. To prevent bad actors from having access to advanced AI systems, (1) ensure strong export controls of leading-edge AI chips and chip-making equipment while licensing
benign uses of chips that can be remotely throttled if need be; (2) use Defense Production
Act authorities to require companies to report the development or distribution of large AI computing clusters, training runs, and trained models (e.g. >1,000 AI chips, >10
27 bit
operations, and >100 billion parameters, respectively); (3) include in DoD contracts with cloud-computing providers a requirement that they employ “know your customer”
screening for all customers before training large AI models; and (4) include in DoD
contracts with AI developers “know your customer” screening, as well as strong
cybersecurity requirements to prevent the theft of large AI models.
3. Work with the intelligence community to significantly expand the collection and analysis of information on key foreign public- and private-sector actors in adversary states
involved in AI, including assessments of key foreign public and private entities; their
infrastructure, investments, and capabilities; and their supply chains of tools, material,
and talent. Strengthen DoD’s institutional capacity for such activities by (1) creating new
partnerships and information-sharing agreements among U.S. and allied government

3 Although there are many ways to measure this, the Stanford Global AI Vibrancy Tool has consistently ran ked the
United States at the top. See Stanford University , “Global AI Vibrance Tool: Who’s Leading the Global AI Race?”
undated,  https://aiindex.stanford.edu/vibrancy/.
4 Graham Webster, Rogier Creemers, Elsa Kania, and Paul Triolo, “Full Translation: China’s ‘New Generation
Artificial Intelligence Development Plan,’” DigiChina, August 1, 2017, https://digichina.stanford.edu/work/full -
translation -chinas- new-generation -artificial- intelligence- development -plan- 2017/ .
5 Forrest E. Morgan, Benjamin Boudreaux, Andrew J. Lohn, Mark Ashby, Christian Curriden, Kelly Klima, and
Derek Grossman, Military Applications of Artificial Intelligence: Ethical Concerns in an Uncertain World , RAND
Corporation, RR -3139- AF, 2020,  https://www.rand.org/pubs/research_reports/RR3139 -1.html .

 3 agencies, academic labs, and industrial firms and (2) recruiting private-sector AI experts
to serve in the government on short-term or part-time appointments.
4. Invest in potential moon shots for AI security, including (1) microelectronic controls embedded in AI chips to prevent the development of large AI models without security
safeguards and (2) generalizable approaches to evaluate the security and safety of AI systems before they are deployed.
I thank the committee for the opportunity to testify, and I look forward to your questions.