Background Ballistic Missile Threats Ballistic missiles, which foreign adversaries generally use as a deterrent or instrument of coercion, are becoming increasingly important weapons to support military and political objectives. These weapons continue to proliferate and show advances in mobility, reliability, in-flight maneuverability, accuracy, and ability to reach longer distances. According to the defense intelligence community, there has been a dramatic increase in ballistic missile capabilities over the last decade, and the over 20 countries that already possess ballistic missiles are likely to pursue further expansions in their quantities and capabilities. Figure 1 shows the lineup of operational ballistic missiles from North Korea and Iran, two of the various countries that pose threats to the United States and its allies and are of concern to the BMDS. Ballistic missile threats are generally categorized by their range (i.e., ground distance covered between the launch point and impact of the missile) as shown in figure 2 below. The configuration of a ballistic missile is also largely determined by the range a missile is expected to travel. For example, longer range ballistic missiles typically have two or three distinct sections, known as stages, that separate during flight and each has an independent propulsion system to ensure the warhead reaches its target. Shorter range ballistic missiles generally only have one section, or a single stage, that remains intact until the warhead reaches its intended target and detonates. Ballistic missiles may also carry countermeasures or adversaries may employ tactics and techniques, both of which are intended to confuse missile defense systems. For example, countermeasures can include penetration aids that are released during flight, such as decoys, which are intended to complicate the ability of missile-tracking sensors and missile defense interceptors to identify the warhead among the multiple objects. Challenging tactics and techniques can include structured attacks, such as simultaneously launching a number of missiles or outfitting a single missile with multiple warheads. In addition, some newer missiles are capable of traveling at greater speeds, performing maneuvers during all phases of flight, and remaining in the atmosphere for longer durations of their flight. These newer missiles, generally referred to as hypersonics, possess a combination of high speed, maneuverability, and relatively low altitude that can make them a challenging target for missile defense systems to track and engage. According to a publicly released intelligence assessment, nearly all adversaries that possess ballistic missiles have devised various means to confuse missile defense systems. Defense Intelligence Community’s Roles in Assessing Missile Threats and Supporting MDA Acquisitions In November 2010, the Defense Intelligence Agency (DIA) established the Defense Intelligence Ballistic Missile Analysis Committee (DIBMAC) to oversee and coordinate intelligence analysis and threat assessment production activities pertaining to foreign ballistic missile developments. Under the leadership of this committee, the defense intelligence community performs important stakeholder, advisor, and oversight functions in support of MDA’s acquisitions by (1) producing threat assessments; (2) providing advice on important threat-related issues pertaining to BMDS acquisition; and (3) validating threat models and reports. Table 1 provides further explanation of these roles and additional information on the defense intelligence community is in appendix I. In November 2013, DOD’s acquisition leadership issued a memorandum that requested DIA work with the acquisition community to produce more timely, relevant, and dynamic defense intelligence community threat assessments for DOD acquisition programs. The memorandum notes that DOD acquisition program officials expressed concerns about the timeliness of threat assessments due to the lengthy process and varying timelines that sometimes left them with threat assessments that did not contain the most up-to-date information. In addition, the defense intelligence community noted its concerns with the significant duplication in producing certain threat assessments, which placed a huge burden on its manpower and resources. Consequently, DOD leadership directed the acquisition customers and defense intelligence community to work together to improve threat assessments and in 2016 the defense intelligence community set forth its planned revisions to threat assessment processes and products. Subsequent revisions include creating a library of threat modules and replacing a former type of threat assessment with a new Validated Online Lifecycle Threat (VOLT) report, among others. These revisions were codified in the defense intelligence community’s policies in September 2016 and in DOD policy in August 2017. However, defense intelligence community officials noted that they are still in the process of implementing these revisions. MDA’s Responsibility for Defending Against Ballistic Missile Threats MDA is developing a variety of missile defense systems, known as elements, including sensors, interceptors, and battle management and communication capabilities. The ultimate goal is to integrate these various elements to function as a layered system called the Ballistic Missile Defense System (BMDS). The BMDS elements, when integrated, are designed to destroy enemy missiles of various ranges, speeds, sizes, and performance characteristics in different phases of flight, as seen in figure 3 below. When MDA was established in 2002, the agency was granted exceptional flexibilities to diverge from DOD’s traditional acquisition lifecycle and defer the application of acquisition policies and laws designed to facilitate oversight and accountability until a mature capability is ready to be handed over to a military service for production and operational use. In particular, MDA was exempted from DOD’s standard requirements-setting process and instead uses a unique and flexible requirements-setting process that is intended to enable MDA to quickly develop and field useful but limited capabilities, which can be incrementally improved over time and adapted to address changes in the threat. MDA also implemented a tailored process that is intended to use defense intelligence community threat assessments in a way that enables the BMDS to defend against a broad range of uncertain and evolving threats. MDA uses defense intelligence community threat assessments as the foundation for developing threat models and establishing wide-ranging critical threat parameters upon which to design, develop, and test the BMDS. Specifically, MDA’s process includes the following: Design: MDA uses threat assessments to select a set of threat models in which it incrementally designs BMDS capabilities to defend against. MDA combines the capabilities from the selected threat models into parameters, forming what MDA refers to as the “parametric threat space.” MDA assigns subsets of the threat space to each of the BMDS elements to inform the design of their respective systems. Development: MDA assigns specific threat models to each of the elements for use in simulations as they are undergoing development. MDA uses these threat models to verify that the element’s system design has the capability necessary to defend against its assigned threat space. Test: Toward the end of BMDS element development, MDA coordinates with the warfighter and test and evaluation communities to select specific threat models for use in testing to assess the performance of the BMDS elements. MDA also uses its threat models to prepare for flight tests to help ensure that the BMDS elements have a high probability of achieving their test objectives, such as successfully intercepting the target. Operational capability: MDA uses its threat models as the foundation for algorithms, which are embedded into the BMDS to enable its sensors and interceptors to determine which object(s) amongst a group of objects (e.g., countermeasures, debris, etc.) is lethal. This capability is referred to as “discrimination.” Mounting Challenges Are Delaying the Availability of Threat Assessments, but Opportunities Exist to Help MDA Receive the Information It Needs Various challenges have recently emerged that have affected the availability of the threat assessments MDA needs to inform the agency’s acquisition decisions. Challenges include an upsurge in threat missile activity, which has increased the overall demand for threat assessments; a transition period as the defense intelligence community works through how to implement recent revisions to its processes and products; and MDA’s request for accelerated support from the defense intelligence community. Defense intelligence community officials say they are contending with all of these challenges without the provision of additional manpower or resources. Consequently, defense intelligence community officials have stated that their manpower and resources are constrained, which can affect the timely delivery of threat assessments to customers, such as MDA. If MDA does not have the threat information it needs when it is needed, the delay of information could result in setbacks for the agency’s weapon system design, development, and testing, or could put the agency in the position of moving forward without the requisite information, thereby increasing the risk of performance shortfalls and costly retrofits. However, MDA has opportunities to mitigate these challenges by collectively prioritizing its threat assessment requests and working through existing venues with the defense intelligence community to determine what additional resources may be needed to secure the accelerated support that it needs. Various Challenges Are Delaying the Availability of Defense Intelligence Community Threat Assessments MDA Uses for Acquisition Design and Testing Decisions Increased Threat Activity One challenge for the defense intelligence community is a recent upsurge in threat missile activity, which has increased MDA’s requests for threat assessments. For example, ballistic missile flight testing has more than doubled from 2005 to 2016, from about 70 tests in 2005 to nearly 180 tests in 2016, and the most notable increases have occurred since 2010 (see figure 4). This upsurge of threat missile activity increases the urgency for the defense intelligence community to provide the requisite type of threat assessments to MDA to enable the agency to counter and defeat such threats; however, defense intelligence community officials have said that manpower and resource constraints have limited their ability to do so. In 2016, we reported on how the defense intelligence community’s manpower and resource constraints have impacted its ability to provide threat assessments. Since then, defense intelligence community officials have said that the manpower and resource constraints have not been resolved, but threat missile activity has increased. For example, some countries have recently displayed or flight tested new threat missiles capable of reaching the United States. When new threat missiles emerge, MDA requests missile-specific threat assessments—known as reference documents—from the defense intelligence community to understand their size, performance characteristics, and signature when detected by a sensor. This detailed information on the threat missiles enables MDA to build the threat models used to design, develop, and test BMDS weapon systems. Defense intelligence community officials have said that, although important, missile-specific threat assessments utilize considerable manpower and resources because they can be labor-intensive, lengthy, and take months, and at times a year or longer, to prepare. According to these officials, one way to minimize the workload and shorten the preparation timeframe is for MDA to differentiate the specific information that it needs from anything that might be extraneous. As a simplified and hypothetical example, defense intelligence community officials explained that MDA may only need some simple, general information about a missile or conversely it may need complex, highly-detailed information on everything about the missile from tip to tail. The amount of time and effort it would take defense intelligence community officials to gather the information in these two scenarios would vary significantly. MDA officials have acknowledged that some extraneous information may be gathered and included in these threat assessments but noted that, at the time they request a threat assessment, they may not yet fully understand what information is essential for their purposes. Therefore, they prefer to have as much information as possible, with the ability to determine whether and how to use it. Defense intelligence community officials, however, told us that they believe this is an inefficient use of their manpower and resources, especially given current constraints. Revisions to Processes and Products Another challenge for the defense intelligence community is the implementation of recent revisions to its threat assessment processes and products, which apply to all DOD acquisition programs. In 2016, in response to the November 2013 memorandum from DOD’s acquisition leadership, the defense intelligence community began overhauling its threat assessment processes and products to produce more timely, efficient, and relevant information. See table 2 for an overview of these revisions. While each of these revisions has potential benefits, defense intelligence community officials have said that implementing the revisions has been more time-consuming and difficult than anticipated, which has affected their ability to provide certain threat assessments to MDA when needed. For example, MDA and the defense intelligence community were initially uncertain about the responsibilities and processes for creating a VOLT report for the BMDS. Although it took some time to resolve these uncertainties, MDA is now compiling its own country-specific threat assessments—known as the BMDS VOLT report—which DIA then validates. The military services generally have their own defense intelligence production centers, and therefore, a means for compiling VOLT reports. MDA, however, uses information from multiple defense intelligence production centers and does not possess its own production center. In September 2017, MDA reached out to DIA on this matter and DIA responded that, per the DOD policy update, it does not see anything that would preclude MDA, as a DOD component, from compiling VOLT reports. DIA stated that MDA compiling its own VOLT report aligns the agency with the rest of the DOD acquisition community. MDA is waiting on threat modules from the defense intelligence community to prepare its preliminary BMDS VOLT report, which MDA will use to inform acquisition decisions. MDA needs specific threat modules from the defense intelligence community, including those for six specific countries, in order to compile its preliminary BMDS VOLT report. However, defense intelligence community officials have said that they are still in the process of creating some of the digitized threat modules MDA needs, because it has taken more time and effort than they expected to standardize the threat modules’ content and coordinate production across multiple defense intelligence community production centers. Consequently, MDA is planning to publish its preliminary BMDS VOLT report in 2019 (table 3). In the meantime, without the preliminary BMDS VOLT report or digitized threat modules used to compile the BMDS VOLT report, MDA is reliant on threat assessments written between 2014 and 2016 for some of its acquisition decisions. For example, MDA recently made design decisions for certain BMDS elements using these threat assessments, although these threat assessments have not yet been updated. Consequently, these weapon systems that MDA recently made design decisions for could have capability gaps or performance shortfalls that present risks for the warfighter. MDA has attempted to fill the void for digitized threat modules and the preliminary BMDS VOLT report by submitting ad hoc requests for threat assessments to the defense intelligence community, but this has only added to the defense intelligence community’s workload and exacerbated delays. Request for Accelerated Delivery of Threat Modules Moving forward, MDA has asked the defense intelligence community to provide the digitized threat modules on an accelerated schedule to ensure the agency can compile BMDS VOLT reports in a timely manner to inform its acquisition decisions; however, some defense intelligence production centers have said that an accelerated schedule will be difficult, if not impossible, without additional manpower and resources. Specifically, MDA wants the defense intelligence community to provide the digitized threat modules every year, as opposed to every two years as required by DOD policy. MDA has stressed the importance of having these digitized threat modules on an accelerated schedule in order to be responsive to threat advancements and mitigate the potential for capability gaps or performance shortfalls in its weapon systems. Defense intelligence community officials have acknowledged MDA’s need to have the digitized threat modules on an accelerated schedule but are concerned about their ability to provide them due to personnel and resourcing issues at some defense intelligence production centers. For example, two defense intelligence production centers have said that MDA’s request for an accelerated schedule is currently unrealistic due to their manpower and resource levels. Defense intelligence officials have said that once the initial digitized threat modules are created, the threat modules will be easier and quicker to update, but whether they can provide them annually is still being determined. Opportunities Exist That Could Help MDA and the Defense Intelligence Community Address Threat Assessment Availability Challenges Although MDA has the capability to centrally and collectively prioritize its threat assessment requests submitted to the defense intelligence community, it currently prioritizes its threat assessment needs through two distinct, individual lanes—country-specific and missile-specific— supplemented by informal discussions with the defense intelligence community. According to MDA, the individual lanes are as follows: 1. Country-specific threat assessments (i.e., threat modules for BMDS VOLT reports) are prioritized via the VOLT Threat Steering Group, which is co-chaired by MDA and DIA. The VOLT Threat Steering Group’s objectives are to determine MDA’s threat module requirements, to achieve concurrence on the threat modules used in the BMDS VOLT report, and to review the BMDS VOLT production schedule. The first VOLT Threat Steering Group meeting was held in April 2018 and during that meeting, MDA presented its prioritized list of threat assessments by adversary country to the defense intelligence community personnel in attendance. 2. Missile-specific threat assessments (i.e., reference documents used to build threat models) are prioritized via an annual intelligence mission data process managed by the Joint Staff. Through the intelligence mission data process, MDA prioritizes the data it needs for threat missiles by most to least critical—119 total threat missiles in 2018. With these two individual lanes for prioritization, MDA treats each type of threat assessment as independent and unrelated. According to MDA, the agency maintains these individual lanes for prioritizing its threat assessment requests because the requests can be more easily managed by the defense intelligence community components that develop the threat assessments. For example, MDA stated that requests for missile-specific threat assessments are often routed to intelligence production centers while requests for country-specific threat assessments are often routed to DIA’s regional centers (see appendix I for more information on defense intelligence community components). According to MDA, the vast majority of new requirements submitted to the defense intelligence community are also accompanied by an informal verbal discussion and if MDA’s priorities shift because a new threat emerges, MDA stated that it can convey that shift to the defense intelligence community in an effort to work out the best path forward. If the defense intelligence community cannot meet MDA’s needs, MDA stated that it works with the defense intelligence community to determine the best course of action for resolving prioritization issues. For example, MDA cited a recent example where it had worked with the U.S. Navy’s Office of Naval Intelligence to develop a threat model production schedule for two threat systems; however, the emergence of a new threat shifted MDA’s priorities. MDA was able to understand the effect of choosing one system ahead of the others based on the priority and projected production timelines. MDA cited another recent example where it had similarly worked with the U.S. Air Force’s National Air and Space Intelligence Center to prioritize production of a threat model for a new, unique threat. After some initial informal discussions and questions about whether the threat model production effort was a top priority for MDA, both agreed in a meeting in January 2019 to lower the priority for the model production effort. The specific threats referenced in the examples above have been omitted because they are classified. However, MDA’s approach of prioritizing its threat assessment needs through individual lanes creates the potential for unresolved, competing priorities because the defense intelligence community produces threat assessments collaboratively rather than disparately. Defense intelligence community officials told us that the underlying analyses that support both country-specific and missile-specific threat assessments are developed and reviewed by many of the same subject matter experts and managers within the defense intelligence community. Defense intelligence community officials told us that they have no way of knowing whether the information to build a specific threat model is a greater or lesser priority than updating a particular threat module needed to support the BMDS VOLT report. Our prior best practices work found that successful commercial companies employ a formal process for prioritizing their investments collectively rather than as independent and unrelated initiatives. MDA instead stovepipes its threat assessment prioritization through individual lanes and informally discusses its collective priorities with the defense intelligence community. Consequently, MDA’s requests, and resulting output from the defense intelligence community, may not be based on the collective order of importance, as depicted in figure 5. MDA relies on both country- and missile-specific threat assessments for its acquisitions, as each characterizes threats in unique ways and for different purposes, and it uses other requests to fill information gaps, as needed. Thus, all of MDA’s requests are important, but one among them may be the most important or urgent due to the timing of an upcoming design or testing decision. In the example illustrated above in figure 5, the most important request is for a country-specific threat assessment; however, it will not likely be the next one out of the defense intelligence community’s queue because there is a missile-specific request ahead of it. Hence, MDA may have the information it needs to build the threat model used to test one weapon system’s performance, but it may delay the country-specific information it needs to make design decisions for another. This delay in the country-specific information could put MDA in a position of moving forward with design decisions without the requisite information or relying on outdated information, which increases the risk for performance shortfalls and costly retrofits. One opportunity that MDA has to address the availability of threat assessments from the defense intelligence community is to collectively prioritize its threat assessment requests based on the order of importance. We have previously identified collective prioritization as a best practice—specifically, that it is important for an agency to regularly evaluate the totality of its needs or tasks, to determine whether specific ones should be prioritized ahead of others, based on the costs, benefits, and risks. While MDA has no formal requirement to collectively prioritize its threat assessment requests, defense intelligence community officials said that they have had discussions with MDA through existing venues and requested that it do so to ensure it has the most urgently needed information. MDA has the capability to collectively prioritize its threat assessment requests because all of the requests go through a centralized intelligence requirements group within the agency’s engineering directorate. This group has insight into the totality of the agency’s threat assessment requests and is uniquely positioned to make determinations about the order of importance among them. As the group submits requests to the defense intelligence community, the defense intelligence community responds to the requests in the order that they were received, because, as we previously found, the defense intelligence community is not required to prioritize the requests, does not currently possess the capability to do so, and would not be in a position to dictate to an agency what is most important. Another opportunity for MDA to address the availability of threat assessments is through further collaboration with the defense intelligence community to determine the extent of additional resources that would be needed to enable accelerated support. When intelligence support requirements exceed the defense intelligence community’s responsibilities, DOD acquisition programs are generally required to account for resources to augment intelligence support. For example, according to defense intelligence community officials, the Air Force is providing one of the defense intelligence community’s production centers with additional resources to collect data and devise tools primarily to support a specific major defense acquisition program via a military interdepartmental purchase request because the program’s request exceeds the defense intelligence community’s responsibilities. According to MDA, intelligence mission data shortfalls are currently identified through an annual departmental review process. MDA stated that in fiscal year 2019 DOD approved budgeting additional funding in the future to help address intelligence mission data shortfalls for all of the military services, including MDA. MDA has not provided the defense intelligence community with additional resources for an accelerated schedule to update threat modules more frequently. MDA has requested that the defense intelligence community update the digitized threat modules it needs to compile a BMDS VOLT report every year to ensure that it has the updated threat information needed for acquisition decisions; however, the defense intelligence community is only required to update the digitized threat modules every two years. Some defense intelligence community officials have acknowledged MDA’s need to have an accelerated schedule, but have communicated to MDA that given its current manpower and resource constraints, the accelerated schedule is unrealistic without additional resources. Thus, MDA’s request for the defense intelligence community to update the digitized threat modules faster exceeds what the defense intelligence community is currently able to do given its manpower and resource constraints. With existing venues, like the VOLT Threat Steering Group, MDA and the defense intelligence community have a forum to further collaborate and identify what additional resources are needed and the potential funding scenarios to support an accelerated schedule for threat module production. Without collaboration through these existing venues, MDA and the defense intelligence community may not be utilizing an available method to ensure their individual needs are met. According to our best practices for inter-governmental agency collaboration, it is important for the inter-reliant agencies to collaboratively identify the resources— information, manpower, and funding—needed to accomplish their respective missions. Doing so enables the agencies to have a common understanding and explore opportunities to leverage each other’s resources; thus, realizing benefits that would not be available if they were working separately. Therefore, working together, MDA and the defense intelligence community would be better positioned to determine how to best meet their respective needs. Opportunities Exist for MDA to Further Engage the Defense Intelligence Community on BMDS Acquisition to Address the Challenges of Keeping Pace with the Threat MDA uses defense intelligence community threat assessments to inform its acquisitions, but the agency has not fully engaged the defense intelligence community on challenges in preparing the BMDS for existing and emerging threats. According to MDA, the rapid pace of threat evolution presents significant challenges for the agency to sufficiently plan for emerging threats. Although the defense intelligence community is uniquely positioned to assist MDA in addressing these challenges, the agency generally limits the defense intelligence community’s insight into and input on critical threat-related BMDS acquisition processes and decisions, such as establishing the BMDS threat space and assigning threat parameters and threat models to BMDS elements. Major defense acquisition programs are generally required to engage the defense intelligence community on how to design and test weapon systems, but MDA generally does not, due to the acquisition flexibilities DOD has granted to the agency. Moreover, DIA is currently unable to validate MDA’s threat models, as required by DOD policy, because MDA does not follow the department’s best practices on models and simulations. MDA has steadily increased its outreach to the defense intelligence community and other stakeholders over the past few years, but opportunities remain for more comprehensive engagement on key challenges the agency faces with keeping pace with the threat. MDA Faces Challenges in Preparing the BMDS for Existing and Emerging Threats According to MDA, the rapid pace of threat evolution presents significant challenges for the agency to sufficiently plan for emerging threats. MDA currently faces some difficult choices regarding what steps it needs to take and in what order to address recent threat advancements. In making these decisions, MDA has an opportunity to engage the defense intelligence community on whether and how it should make changes to the BMDS threat space, threat parameters, and threat models the agency uses as design requirements and test cases for BMDS elements. As previously noted, the defense intelligence community plays important stakeholder, advisor, and oversight roles for MDA’s acquisitions. Although the department has provided MDA with flexibilities on following many of the requirements that specifically define when and how major defense acquisition programs are to engage the defense intelligence community, DOD policy requires MDA to vet its threat models and consult with the defense intelligence community on threat-related acquisition matters. DOD, senior defense officials, and expert panels supported by DOD have consistently maintained that the defense intelligence community’s direct involvement in MDA’s acquisitions is critical to staying ahead of the threat: In a written response following a 2002 congressional hearing, a senior defense official stated that every effort was being made to coordinate development of the document establishing the BMDS threat space with the defense intelligence community and that the defense intelligence community’s participation was critical to the agency’s success. In 2010, DOD’s Ballistic Missile Defense Review similarly found the need to maintain a strong focus by the defense intelligence community on the ballistic missile threat and that accurate and timely intelligence should play a vital role in informing BMDS planning. In 2010, an expert panel known as JASON (not an acronym) found that MDA lacked sufficient plans for improving discrimination and that the agency risked falling behind the evolution of the threat’s countermeasure capabilities. The study recommended that DOD form stronger two-way connections between MDA and defense intelligence agencies. In 2012, the National Research Council found that MDA did not follow through on efforts to improve discrimination and that much of the agency’s expertise on discrimination was lost in the late 2000s. The study recommended that MDA seek assistance from experts with experience in understanding sensor data for threat missiles. In 2018, DOD’s National Defense Strategy stated that modernizing missile defense, among other items, was necessary to keep pace with adversaries and that the department must expand the role of intelligence analysis throughout the acquisition process in order to streamline rapid, iterative approaches for delivering performance at what DOD refers to as “the speed of relevance.” During a 2018 congressional hearing, the Under Secretary of Defense for Research and Engineering stated that catching up to near-peer adversaries in missile defense can be achieved by exceeding their technical capabilities and that the intelligence community was critical to making sure that we are outpacing our adversaries. MDA Limits the Defense Intelligence Community’s Insight Into and Input on Some Critical Threat- Related BMDS Acquisition Processes and Decisions Although MDA uses defense intelligence community threat assessments to inform BMDS acquisition, the defense intelligence community generally has limited insight into the BMDS, which is unprecedented among major defense acquisition programs. When MDA was established in 2002, DOD granted the agency exceptional flexibilities to diverge from the standard acquisition framework that most major defense acquisition programs follow. These flexibilities enable MDA to forego obtaining the defense intelligence community’s input on some critical threat-related BMDS acquisition processes and decisions, such as how MDA establishes the: threat space that informs overall BMDS design and development; threat parameters assigned to each BMDS element as design requirements; and threat models assigned to each BMDS element as test cases for design reviews and testing. However, according to MDA, the new BMDS VOLT report will serve as the source document for specific details on the BMDS threat space, threat parameters, and threat models. Although MDA may leverage the defense intelligence community’s threat assessments, MDA has not included the defense intelligence community in these key threat-related BMDS acquisition processes and decisions. For example, in response to a questionnaire we sent to MDA in May 2018, agency officials stated that decisions related to the threat parameters it assigns to the different BMDS elements should be left to MDA, as it is within the agency’s purview and authority to design threats as it deems necessary for research, development, test, and evaluation purposes. Moreover, MDA indicated that the defense intelligence community should provide the agency with the best intelligence information on adversary missile capabilities, in a timely manner, to support the agency’s mission. As such, MDA stated it does not support obtaining the defense intelligence community’s concurrence on the threat parameters it assigns to the BMDS elements. MDA has provided the defense intelligence community with some insight into the BMDS but not to the same extent DOD generally requires of major defense acquisition programs. For example, MDA has held a number of “immersion days” over the past nine years, which allow the defense intelligence community to receive briefings from MDA programs on priorities, future developments, and weapon system operations. According to MDA, it also assigns intelligence portfolio managers to BMDS elements and their mission, among other items, is to keep the defense intelligence community informed on key program developments and how intelligence feeds into the agency’s threat-related acquisition processes and decisions. In addition, MDA has briefed the DIBMAC on how it uses threat assessments to inform BMDS acquisition. However, defense intelligence community officials stated that they generally lack fundamental information on the BMDS and have no visibility into the BMDS threat space, threat parameters, or test cases MDA assigns to the BMDS elements. In contrast, for most major defense acquisition programs, the defense intelligence community is integrally involved in determining the: threat(s) of record upon which requirements of the weapon system are based; key performance parameters and attributes of the weapon system; threat parameters that could critically degrade or negate the weapon system; and operational threat environment the weapon system is tested against. These insights, enabled by DOD’s standard requirements-setting process and acquisition framework, are intended to provide the defense intelligence community with in-depth knowledge of the design and performance requirements for most major DOD weapon systems. Officials from other various organizations we met with, such as the Joint Staff, contractors, warfighters, and test and evaluation, expressed concerns about MDA’s ability to unilaterally define the threats it designs the BMDS against. As one MDA prime contractor told us, what really matters is how the BMDS would perform in the real world against real threats. Defense intelligence community officials acknowledged that MDA, as the BMDS developer, has a legitimate need to explore threat capabilities beyond those that the intelligence community has observed from specific adversaries. However, defense intelligence community officials rejected a sentiment expressed to us by MDA officials that the defense intelligence community lacks expertise in understanding the bounds of threat capabilities. To the contrary, according to defense intelligence community officials, this is exactly the type of analysis at which the defense intelligence community excels. In choosing not to engage the defense intelligence community on these key threat-related BMDS acquisition processes and decisions, MDA runs the risk of not sufficiently planning for existing and emerging threats. MDA’s reluctance to provide the defense intelligence community with insight into or input on some threat-related BMDS acquisition processes and decisions is consistent with how MDA has engaged other DOD stakeholders and oversight groups. Our prior work on defense acquisitions has shown that establishing buy-in from decision makers is a key enabler of achieving better acquisition outcomes because DOD components provide varying perspectives due to their unique areas of expertise and experience. However, in May 2017, we found that MDA generally limits the warfighter’s input on the requirements it pursues and overlooked stakeholder concerns on the acquisition strategy for a redesigned kill vehicle for the Ground-based Midcourse Defense system. We made recommendations aimed at increasing stakeholder engagement and oversight in BMDS acquisition, such as coordinating operational requirements with the warfighter and obtaining input from DOD’s Office for Cost Assessment and Program Evaluation (CAPE) on acquisition strategies for new efforts. DOD’s acting Assistant Secretary of Defense (Acquisitions) did not concur with the recommendations, stating that warfighters lacked the skillset to determine operational BMDS requirements and existing DOD policy does not require MDA to obtain CAPE’s concurrence on acquisition policies. We continue to maintain that DOD should implement the recommendations. DIA Is Currently Unable to Validate MDA’s Threat Models, as Generally Required by DOD Policy MDA builds its own threat models to support BMDS design, development, and testing but it does not validate its threat models with DIA, which is inconsistent with DOD policy and best practices. Although the defense intelligence community builds threat models, MDA cannot currently use those models as-is because they are generally not compatible with MDA’s modeling and simulation framework. Even with MDA using its own threat models, DOT&E has found that integrating the various BMDS models and presenting them with a common threat scene has been an extremely challenging task for MDA. Moreover, MDA’s BMDS modeling and simulation architecture requires highly detailed threat models for simulations to function properly. Defense intelligence community officials stated that they generally do not need the same level of detail MDA requires for the types of analyses the defense intelligence community performs. In addition, according to a March 2018 MDA memorandum, the agency was previously told by representatives of the DIBMAC that they do not have the staff or resources to produce the high volumes of detailed threat models that MDA needs to support BMDS development and testing. Therefore, MDA continues to build its own threat models for use in BMDS development and testing. MDA uses defense intelligence community threat assessments to build its threat models, but independent evaluators have not been able to fully trace MDA’s threat models to defense intelligence community threat assessments. According to a briefing MDA presented to the defense intelligence community in September 2018, every target, model, and test can be traced back to defense intelligence data. However, in August 2018, the U.S. Army issued a memorandum for MDA stating that the BMDS Operational Test Agency (OTA)—the agency responsible for independently analyzing the verification and validation data for models used in operational testing—was only able to certify some of the threat models used in a recent ground test. In other ground tests, though, the BMDS OTA was able to trace MDA’s threat models back to defense intelligence community threat assessments. In February 2019, DOT&E reported that (a) credible threat models are the linchpins of BMDS models and simulation; (b) reducing threat model uncertainty is a high priority; and (c) MDA and the BMDS OTA should ensure that MDA-developed threat models are representative of the defense intelligence community’s understanding of the threat. MDA also has not implemented best practices established by DOD’s Models and Simulation Coordination Office that would enable DIA to be in a position to validate MDA’s threat models. According to DOD best practices on modeling and simulation, the validation agent should: (1) be brought on in the beginning of the modeling and simulation development process; (2) work closely with the model developers as the models are built and tested; and (3) perform validation as a continuing activity of the overall process of developing and preparing a model for use or reuse in a simulation. Conversely, defense intelligence community officials stated that they lack sufficient insight into and input on how MDA builds and uses threat models. For example, the defense intelligence community has emphasized to MDA that caveats need to be carried through with the model data and voiced concerns about the engineering judgments the agency makes in its threat models, because these judgments could lead to the BMDS performing well or poorly for reasons not based on the actual threat. Given these uncertainties and the defense intelligence community’s lack of insight into the purposes for which MDA uses its threat models, DIA lacks the insight and input necessary to validate MDA’s threat models. Although MDA has previously expressed interest in validating its threat models with the defense intelligence community, long-standing obstacles remain. During a May 2018 meeting between MDA and the DIBMAC, defense intelligence community officials identified the lessons they have learned from working with other acquisition programs to validate threat models. Model validation can be achieved if the acquisition program: establishes a partnership with the defense intelligence community; prioritizes its threat modeling needs; recognizes there are limits to how many threat models can be built in a given time; provides in-depth insight into its threat modeling needs and weapon system’s capabilities; discusses how the models will be applied; jointly defines model acceptance criteria early in the process; provides resources, including funding and staff; and invests in the defense intelligence community’s capability and capacity. MDA officials stated that the agency desires to have its threat models validated but noted that the defense intelligence community does not validate models produced by other organizations. MDA officials also emphasized that the defense intelligence community cannot meet MDA’s timeline for building threat models, whereas the agency can. In addition, MDA officials indicated to us that they do not believe it is practical to provide the amount of insight defense intelligence community officials told us they would need in order to validate MDA’s threat models. MDA officials told us that the only way in which the defense intelligence community could obtain such insight is by being co-located with MDA’s threat modelers as the models are being built. However, the 2010 JASON study found that this type of close working arrangement between MDA engineers and defense intelligence analysts is necessary to effectively plan for emerging threats. Defense intelligence community officials also clarified for MDA that the defense intelligence community can validate models produced by another agency but it would require the defense intelligence community having detailed knowledge of everything used to produce the model. As a result, although DOD policy generally requires that threat models used to support acquisition decisions be validated by DIA, MDA has yet to validate any of the numerous threat models it has developed since 2004. Without independent validation, MDA runs the risk that DOD and congressional decisionmakers may not have confidence that the agency’s plans and proposals for developing the BMDS are appropriate and sufficient to address the threat because any flaws or bias in MDA’s threat models can have significant implications on the BMDS’s overall performance. According to a Federally Funded Research and Development Center publication describing its efforts supporting MDA threat modeling, acquisition influences can place pressure on MDA threat modelers to tailor the missile threats to suit the currently feasible BMDS design. In May 2017, we found a parallel circumstance where, in the absence of warfighter validation of MDA-established requirements, the agency made critical design choices for three new BMDS efforts. These design choices reflected the needs and preferences of MDA ahead of the warfighter, potentially compromising performance to the extent of not being able to defeat current and future threats. MDA Has Steadily Increased Its Outreach to DOD Stakeholders Over the Past Few Years, but Opportunities Remain for Further Engagement MDA has undertaken a number of efforts over the past few years to generally increase stakeholder involvement in BMDS acquisition. The engagement efforts, in large part, are a result of efforts led by MDA’s previous director to improve the agency’s relationship with department stakeholders. In addition to previously serving as the Deputy Director for MDA, the Director also held a variety of assignments in operational, acquisition, and staff units within DOD. When we met with the MDA Director in March 2018, he told us that he wanted to change the agency’s culture of limiting stakeholder input, noting that he had recently provided updated guidance to his leadership team and agency personnel on bringing stakeholders in early, engaging them more frequently and substantively, and ensuring that the agency has obtained their buy-in on major undertakings. The MDA Director also stated that he was willing to take some actions that could effectively address a recommendation we made in May 2017 intended to provide the warfighter with greater input on operational requirements for ballistic missile defense. Officials from several DOD organizations we met with over the course of our review observed that MDA’s engagement with their respective organizations was improving. In 2018, MDA began working with the defense intelligence community to determine a more appropriate level of involvement for the defense intelligence community throughout MDA’s acquisition activities. MDA and defense intelligence community officials agreed during a May 2018 meeting that processes could be put in place to develop intelligence-based countermeasure assessments if adequate resources are provided. MDA officials also acknowledged that the defense intelligence community would benefit from having a better understanding of how the BMDS responds to threats and agreed to work towards providing such information. Defense intelligence community officials stated that increased insight would allow them to better focus their intelligence collection, analysis, and production by knowing which threat parameters MDA most often uses and the specificity of those parameters. The defense intelligence community and MDA also agreed that providing defense intelligence community engineers with MDA program-level access would improve the support the defense intelligence community provides to MDA. MDA has also recently increased its outreach to the defense intelligence community on some early BMDS planning decisions, although opportunities for more comprehensive engagement remain. For example, MDA engaged the defense intelligence community on an analysis of alternatives the agency completed in February 2017 that assessed future sensor options for the BMDS. According to MDA officials, they are also engaging the defense intelligence community on another analysis of alternatives pertaining to defense against hypersonic missiles. In addition, MDA worked with the defense intelligence community to establish threat space parameters for some specific threat systems. Also, as noted earlier, over the last nine years, MDA has held 18 “immersion day” events with the defense intelligence community, half of which occurred in the last two years. Moving forward, MDA has opportunities to more comprehensively engage the defense intelligence community on updating the BMDS threat space and determining threat parameters and threat models assigned as design requirements and test cases for BMDS elements. In addition, MDA has recently begun placing greater emphasis on ensuring its models are credible. According to an internal MDA memorandum signed by the MDA Director in April 2018, a culture exists within the agency that generally tolerates the use of models that have not been sufficiently vetted and is too willing to accept the associated risk. The memorandum states that the agency’s goal is for all MDA personnel to help address this culture problem and that model verification, validation, and accreditation is a high priority for MDA. During a meeting with the BMDS OTA in October 2018, officials confirmed that MDA is taking steps to address the challenges raised in the memorandum. MDA also increased its outreach to the defense intelligence community in 2016 to coordinate on threat modeling efforts. In the past three years, MDA and the defense intelligence community have collaborated to quickly model several newly-observed threat missiles, according to MDA. Figure 6 below shows that MDA held 93 threat model coordination meetings with the defense intelligence community over the last four years, with more frequent meetings occurring in early 2016 and again in early-to-mid 2018. In addition, MDA is working with the defense intelligence community to address compatibility issues that currently prevent MDA from directly using the defense intelligence community’s threat models in BMDS ground testing. MDA plans to include a few missile trajectory models produced by the defense intelligence community in the models and simulation framework for the agency’s upcoming Ground Test-08 campaign. The Technical Interchange Meetings and pathfinder efforts for MDA directly using defense intelligence community threat models are improving collaboration between MDA and the defense intelligence community on threat modeling efforts. However, they do not provide MDA with a pathway for validating its threat models with DIA. Even if compatibility issues that currently prevent MDA from using defense intelligence community threat models could be resolved, the defense intelligence community is currently not resourced to build threat models for MDA. Moreover, although MDA has indicated that the Technical Interchange Meetings can include any topic of interest, the meetings do not provide defense intelligence officials with sufficient insight into how MDA builds its models, including the assumptions, caveats, or intended use of the models. According to MDA, the agency continues to hold discussions with the defense intelligence community and explore process improvements, as well as technical and resource requirements, to ensure the creation of valid, threat-representative models for BMDS development. In March 2018, the MDA Director told us that one of his priorities was to ensure that the agency was using appropriately validated models and acknowledged the importance of ensuring its threat models are sufficiently representative. In April 2018, MDA subsequently began holding meetings with the DIBMAC to define the issues preventing the defense intelligence community from validating MDA’s threat models. MDA and the defense intelligence community met five times in 2018 to identify actions that would facilitate working together to develop threat models the defense intelligence community would be comfortable validating. During these meetings, both organizations agreed on specific actions intended to increase the defense intelligence community’s involvement in MDA’s threat modeling process. To achieve threat model validation, an initial plan was developed that included a combination of (a) MDA directly using aspects of defense intelligence community threat models; and (b) MDA partnering with the defense intelligence community to build threat models. MDA and the defense intelligence community plan to hold follow-on meetings in 2019 to further discuss the plan and review actions. Conclusions MDA is reliant on threat assessments from the defense intelligence community, as they inform what weapon systems the agency pursues, the design of those systems, and how those systems are tested prior to being delivered to the warfighter for operational use. However, the defense intelligence community has been facing a variety of challenges that are affecting its ability to provide MDA the threat assessments it needs, when it needs them. If MDA does not have the threat assessments it needs, when needed, the agency’s weapon systems are at risk of being designed or tested against irrelevant or outdated information, which could result in performance shortfalls and costly retrofits. MDA has opportunities to mitigate these challenges and risks by collectively prioritizing its threat assessment requests and working through existing venues with the intelligence community to determine if and to what extent additional resources may be needed to secure the support that it needs. If MDA does not take advantage of these opportunities, the defense intelligence community’s challenges will likely continue, which will impact the availability of threat assessments and increase the likelihood that MDA’s weapon systems will not be designed or tested against the most up-to-date threat information. In addition, MDA faces a steep challenge in developing the BMDS and fielding capabilities at a rate that keeps pace with the threat. MDA was previously informed by expert panels and senior defense leaders that it needed to work more closely with the defense intelligence community to better prepare for future threats or risk falling behind the threat. Given these challenges, it is imperative for MDA to make the most out of its available resources. Aside from providing MDA with threat assessments, the defense intelligence community is a resource MDA has yet to fully tap into. The defense intelligence community is uniquely qualified to assist MDA on fundamental and critically important BMDS acquisition processes and decisions, such as establishing the BMDS threat space and the threat parameters and models it assigns to the BMDS elements. Moreover, after nearly 15 years of building numerous threat models, MDA has yet to fully implement a plan for DIA to validate these threat models, as generally required by DOD policy. However, MDA has recently begun laying the groundwork for more comprehensive engagement with the defense intelligence community through efforts which have the potential to address long-standing obstacles that have prevented DIA from validating MDA’s threat models. Resolving these issues would help MDA keep pace with emerging threats and improve the BMDS’s viability to defend against the complex missile threats of the future. Recommendations for Executive Action We are making a total of three recommendations to DOD: The Director, MDA should coordinate with the defense intelligence community on the agency’s collective priorities for threat assessments and work with the defense intelligence community to determine if additional resources are needed to support the agency’s threat assessment needs. (Recommendation 1) The Director, MDA should fully engage the defense intelligence community on key threat-related missile defense acquisition processes and decisions, including providing insight into and obtaining input from the defense intelligence community on the threat space MDA establishes for the BMDS and the threat parameters and threat models MDA assigns to BMDS elements as design requirements and test cases. (Recommendation 2) The Secretary of Defense should require the Director, MDA and the Director, DIA to coordinate on establishing a process for MDA to obtain validation of its threat models. (Recommendation 3) Agency Comments and Our Evaluation DOD provided written comments in response to the classified version of this report (GAO-19-92C), indicating that the department concurred with all three of our recommendations. An edited version of DOD’s comments is reprinted in appendix II as some information had to be omitted due to classification. In addition, the summarized version of DOD’s comments below is reflective of the content in the classified version. DOD provided us with technical comments and a significant amount of new information in response to the classified version of this report. We incorporated this information into our report, as appropriate, but the new information did not substantively change our findings and did not alter our recommendations. Although DOD concurred with our third recommendation, DOD also raised concerns about statements in our report related to our third recommendation that the department believes are inaccurate. We do not believe DOD’s concerns are warranted because our findings are based on evidence we obtained during our review—evidence that we believe is sufficient and appropriate and provides a reasonable basis for our findings and conclusions. We address this in further detail below. DOD concurred with our first recommendation that the Director, MDA should coordinate with the defense intelligence community on the agency’s collective priorities for threat assessments and determine whether additional resources are needed. In its response, DOD stated that MDA will continue to follow established processes to identify threat assessment needs and to determine if additional resources are required. However, our review found that these established processes—prioritizing exclusively through distinct, individual threat assessment lanes—have not proven entirely effective. In addition, although MDA has participated in the department’s intelligence mission data review process since 2016, the agency has yet to provide the defense intelligence community with additional resources to address known funding and manpower shortages. Moreover, this review process is limited to intelligence mission data and does not cover all of the other types of threat assessments that MDA needs. As such, we maintain that MDA should take additional steps beyond continuing existing processes to address the challenges MDA currently faces in obtaining the threat assessments it needs, when it needs them. DOD also concurred with our second recommendation that the Director, MDA should provide insight into and obtain input from the defense intelligence community on the threat space MDA establishes for the BMDS and the threat parameters and threat models the agency assigns to BMDS elements as design requirements and test cases. DOD stated in its response that MDA has and will continue to fully engage the defense intelligence community on key threat-related missile defense acquisition processes and decisions. The efforts MDA has recently undertaken to expand its outreach to the defense intelligence community are positive steps. However, we have yet to see MDA provide the defense intelligence community with further insight into or input on the threat space the agency has established for the BMDS or the assignment of threat models and threat parameters to BMDS elements. We will continue to monitor MDA’s ongoing efforts to see whether it takes this next step toward more fully engaging the defense intelligence community. DOD concurred with our third recommendation that the Secretary of Defense should require the MDA and DIA Directors to coordinate on establishing a process for MDA to obtain validation of its threat models. In its response, DOD stated that the department will re-examine the most cost-effective approach to meet the intent of DIA validation to support development and fielding of effective BMDS elements. More specifically, DOD stated that MDA and the DIBMAC are currently having extensive discussions regarding how the defense intelligence community can best support MDA’s threat modeling requirements. As noted in our report, the discussions MDA has had with the defense intelligence community over the course of 2018 demonstrate that the department is beginning to consider substantive measures to address the long-standing issue of MDA not using DIA-validated threat models. However, MDA and defense intelligence community officials have also cautioned that obstacles remain and that alternative solutions may need to be explored. We will continue to monitor these ongoing discussions and any results that emerge. DOD also stated in its response that it was concerned that statements in our report pertaining to our third recommendation imply that MDA has not coordinated with DIA on validating its threat models and that our report could be interpreted as saying MDA does not internally conduct threat model validation. To be clear, our review did, in fact, find that, until recently, MDA did not sufficiently coordinate with DIA on establishing a process for creating valid threat models for use in MDA simulations. Furthermore, we explain in our report that MDA was told that the defense intelligence community can validate MDA’s threat models if it has sufficient insight into how MDA builds its models—insight which MDA officials previously told us was unnecessary. Additionally, although MDA may internally validate its threat models for each ground test, the BMDS OTA was not able to certify many of those threat models, in part, because some models could not be traced back to the defense intelligence community’s threat assessments. We therefore excluded MDA’s internal threat model validation process from our report, as it is not a comparable substitute for DIA threat model validation. We are sending copies of this report to the appropriate congressional committees and the Secretary of Defense. In addition, the report is available at no charge on the GAO website at http://www.gao.gov. If you or your staff have any questions about this report, please contact me at (202) 512-4841 or chaplainc@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are listed in appendix Ill. Appendix I: Defense Intelligence Components Responsible for Assessing Foreign Ballistic Missiles In its entirety, the intelligence community is a federation of 17 agencies and organizations that span the executive branch of the U.S. government. The defense intelligence components responsible for assessing foreign ballistic missile threats are headed by the Defense Intelligence Agency and overseen and coordinated by the Defense Intelligence Ballistic Missile Analysis Committee. Table 4 below identifies each component and its respective focus areas. Appendix II: Comments from the Department of Defense Appendix III: GAO Contact and Staff Acknowledgments GAO Contact Staff Acknowledgements In addition to the contact named above, LaTonya Miller (Assistant Director), Rose Brister, Lori Fields, Laura Greifner, Kurt Gurka, Helena Johnson, Kevin O’Neill, Jay Tallon, Brian Tittle, Hai Tran, Alyssa Weir, and Robin Wilson made key contributions to this report.