Background ONDCP’s Responsibilities ONDCP was established by the Anti-Drug Abuse Act of 1988 as a component of the Executive Office of the President, and its Director is to assist the President in the establishment of policies, goals, objectives, and priorities for the National Drug Control Program. ONDCP is responsible for (1) leading the national drug control effort, (2) coordinating and overseeing the implementation of national drug control policy, (3) assessing and certifying the adequacy of National Drug Control Programs and the budget for those programs, and (4) evaluating the effectiveness of national drug control policy efforts. About a dozen National Drug Control Program agencies, as identified by ONDCP, have responsibilities for drug prevention, treatment, and law enforcement activities. Developing the National Drug Control Strategy Among other responsibilities, the Director of ONDCP is required to develop and promulgate the National Drug Control Strategy. The National Drug Control Strategy is to set forth a comprehensive plan to reduce illicit drug use and the consequences of such illicit drug use in the United States by limiting the availability of and reducing the demand for illegal drugs. Many of the SUPPORT Act’s requirements for the National Drug Control Strategy are the same as, or similar to, those that applied under the ONDCP Reauthorization Act of 2006. For example, both laws require the National Drug Control Strategy to include a 5-year projection for the National Drug Control Program and budget priorities. However, there are certain differences, and the SUPPORT Act includes a wide range of detailed new requirements that were not included under the ONDCP Reauthorization Act of 2006. One of these is that the National Drug Control Strategy include a description of how each comprehensive, research-based, long-range quantifiable goal established in the Strategy for reducing illicit drug use and the consequences of illicit drug use in the United States will be achieved. Other examples of new requirements include creating plans to increase data collection and expand treatment of substance use disorders. The SUPPORT Act also requires the Director to release a statement of drug control policy priorities in the calendar year of a presidential inauguration (but not later than April 1). The President is then required to submit to Congress a National Drug Control Strategy not later than the first Monday on February following the year in which the term of the President commences, and every two years thereafter. Certifying Agency Drug Control Budgets The Director of ONDCP is also responsible for developing a consolidated National Drug Control Program budget proposal for each fiscal year, which is designed to implement the National Drug Control Strategy and inform Congress and the public about total federal spending on drug control activities. As part of this effort, the Director of ONDCP is required to assess and certify National Drug Control Program agencies’ drug control budgets on an annual basis to determine if they are adequate to meet the goals and objectives of the National Drug Control Strategy. Figure 1 illustrates ONDCP’s budget certification process. ONDCP Did Not Fully Address Selected Statutory Requirements Related to the National Drug Control Strategy in 2017, 2018, or 2019 For 2017 and 2018, ONDCP Did Not Issue a National Drug Control Strategy ONDCP did not issue a National Drug Control Strategy for 2017 or 2018. Pursuant to the ONDCP Reauthorization Act of 2006, the Director of ONDCP was required to promulgate the National Drug Control Strategy annually and the President was to submit the National Drug Control Strategy to Congress by February 1 of each year. According to ONDCP officials, ONDCP did not issue a National Drug Control Strategy for these years because (1) ONDCP did not have a Senate-confirmed Director during those years; and (2) 2017 was the administration’s inaugural year, and previous administrations also did not issue a Strategy during their first years. By statute, in the absence of a Director, the Deputy Director of ONDCP is to perform the functions and duties of the Director temporarily in an acting capacity. ONDCP had officials serving as Acting Director beginning in January 2017. The current Director of ONDCP was appointed Deputy Director beginning in February 2018 and served as Acting Director from February 2018 until April 2018. As of April 2018, the current Director continued in his role as Deputy Director until he was confirmed by the Senate as Director of ONDCP in January 2019. The previous administration also did not issue a National Drug Control Strategy in its inaugural year—2009—but it did issue a National Drug Control Strategy in its second year, as shown in table 1. On January 31, 2019, ONDCP issued its National Drug Control Strategy for 2019, which we discuss in more detail later in the report. Without a National Drug Control Strategy, ONDCP Could Not Complete the Drug Control Budget Certification Process in Accordance with Statutory Requirements in 2017 and 2018 The ONDCP Reauthorization Act of 2006 required the Director of ONDCP to issue drug control funding guidance to the heads of departments and agencies with responsibilities under the National Drug Control Program by July 1 of each year. ONDCP is to issue funding guidance for agency budget proposals for the fiscal year two years in the future. For example, ONDCP was to issue funding guidance to agencies in 2017 for development of the 2019 budget, and issue funding guidance in 2018 for development of the 2020 budget. Such funding guidance was required to address funding priorities developed in the National Drug Control Strategy. National Drug Control Program agencies are to submit their budget requests to ONDCP in the summer of each year (before submission to the Office of Management and Budget) and in the fall of each year (at the same time as submission to the Office of Management and Budget).The Director of ONDCP then determines whether National Drug Control Program agencies’ summer budget requests are adequate to meet the goals of the National Drug Control Strategy and certifies whether fall budget submissions include the funding levels and initiatives identified during the summer budget review. Since ONDCP did not issue a Strategy in 2017 or 2018, ONDCP could not develop and issue funding guidance, nor could it review and certify budget requests and submissions of National Drug Control Program agencies, in accordance with the statutory requirement. ONDCP officials stated that—in lieu of a Strategy—they used other sources to formulate the administration’s priorities, which served as the basis for drug control funding guidance in 2017 and 2018. For example, for the development of the fiscal year 2019 drug control budget in calendar year 2017, ONDCP officials stated that they relied upon the following sources for drug policy guidance: Initial development of the President’s Initiative to Stop Opioid Abuse and Reduce Drug Supply and Demand; Draft recommendations from the President’s Commission on Combating Drug Addiction and the Opioid Crisis; policy statements made by the President as a candidate; and policy priorities identified in the fiscal year 2018 President’s Budget. Additionally, for the development of the fiscal year 2020 funding guidance in calendar year 2018, ONDCP officials stated that they relied upon the following sources for drug policy priorities: the interim and final Report of the President’s Commission on Combating Drug Addiction and the Opioid Crisis; the President’s Initiative to Stop Opioid Abuse and Reduce Drug Supply and Demand; the draft National Security Council Strategic Framework; and a draft 2018 National Drug Control Strategy that ONDCP officials told us they drafted but did not issue. These sources may have provided ONDCP officials with some information about policy priorities and actions. However, ONDCP officials stated they did not consider these documents to be the National Drug Control Strategy, and none of the sources fulfill the statutory requirements under the ONDCP Reauthorization Act of 2006, which require funding guidance to address priorities from the National Drug Control Strategy. ONDCP officials told us that they provided drug control funding guidance to the heads of departments and agencies with responsibilities under the National Drug Control Program in 2017 and 2018. As described by ONDCP officials, drug control funding guidance identifies key program goals and the programs and activities that require agency funding to achieve the objectives of the National Drug Control Strategy. ONDCP has since issued the 2019 National Drug Control Strategy which states that it establishes the administration’s drug control priorities. The Strategy also states that the priorities provide federal drug control departments and agencies strategic guidance for developing their own drug control plans and strategies, and that the Strategy is intended to ensure federal drug control budget dollars are allocated in a manner consistent with the administration’s priorities. ONDCP officials told us that the agency intends to issue the next National Drug Control Strategy in February 2020 in accordance with the SUPPORT Act. ONDCP Issued a 2019 National Drug Control Strategy that Addresses Some, But Not All, Selected Requirements The 2019 National Drug Control Strategy and companion documents include information to address some but not all selected requirements under the ONDCP Reauthorization Act of 2006. ONDCP issued multiple documents that together were intended to address the requirements for the National Drug Control Strategy. The first document, the 2019 National Drug Control Strategy, was issued January 31, 2019, with three companion documents issued later in April and May 2019. These companion documents were the 2019 Data Supplement, the 2019 Performance Reporting System, and the 2019 Budget and Performance Summary. In our March 2019 testimony, we reported that the first document—the National Drug Control Strategy, which was the only one of the four documents available at the time of our testimony—did not include certain information required under the ONDCP Reauthorization Act of 2006. These selected requirements included: annual quantifiable and measurable objectives and specific targets; a 5-year projection for program and budget priorities; specific drug trend assessments; and a description of a performance measurement system. Following our March 2019 testimony, we reviewed the three companion documents and found that while they provide some additional information to address these same selected requirements, they do not completely address the requirements. As stated earlier, we based our analysis of the 2019 National Drug Control Strategy and companion documents on the ONDCP Reauthorization Act of 2006, which was the applicable law at the time ONDCP began drafting the Strategy. Current law is reflected in the SUPPORT Act, which includes some of the same requirements from the ONDCP Reauthorization Act of 2006 and some new or different requirements. In the paragraphs below, we identify which selected requirements from the ONDCP Reauthorization Act of 2006 were retained under the SUPPORT Act, and therefore represent current law, and which selected requirements were not retained. For those selected requirements that were not retained, we identify comparable current requirements in the SUPPORT Act. Annual quantifiable and measurable objectives and specific targets. Pursuant to the ONDCP Reauthorization Act of 2006, the National Drug Control Strategy was required to include “annual quantifiable and measurable objectives and specific targets to accomplish long-term quantifiable goals that the Director determines may be achieved during each year beginning on the date on which the National Drug Control Strategy is submitted.” The SUPPORT Act retained this requirement. We testified in March 2019 that while the 2019 National Drug Control Strategy lists seven items it designates as measures of performance or effectiveness, the document did not indicate how these would be quantified or measured. The document also did not include targets to be achieved each year. Our subsequent analysis of the three companion documents showed that one additional document provided more information related to this requirement. The 2019 Performance Reporting System includes 9 goals and 17 quantifiable and measurable objectives with specific targets for certain years. Specifically, the goals and objectives identified in the 2019 Performance Reporting System included educating the public about the dangers of drug use; expanding access to evidence-based treatment; decreasing the over-prescribing of opioid medications; and reducing the availability of illicit drugs in the United States through reduced production, increased seizure trends, and increased prices and reduced drug purity, among other things. The document states that each goal “is accompanied by aggressive, but achievable, objectives with two- and five-year targets from a baseline of 2017.” However, the 2019 Strategy does not meet the statutory requirement because it does not have annual targets that may be achieved each year. Instead, the Performance Reporting System states that 16 of the 17 objectives in the Strategy have 2-year targets to be achieved in 2019, and 14 of the 17 objectives have 5-year targets to be achieved in 2022. The objectives do not include annual targets for the other intervening years— 2018, 2020, and 2021, as required. The Performance Reporting System states that while ONDCP assumes a linear progression from the baseline year—2017, in most cases—to the 2022 target, the trajectory may not actually be linear, “but rather it may occur at varying rates over the 5-year period due to multiple factors which influence the ability to achieve each of the stated goals and objectives.” In contrast, other information ONDCP provided to us stated that annual targets can readily be determined from the linear paths between the 2- and 5-year targets. Without identifying annual targets, the 2019 National Drug Control Strategy and companion documents do not meet the statutory requirement. Further, annual targets would better position ONDCP to monitor progress in intervening years and make any needed changes to achieve its goals and objectives. The SUPPORT Act continues to require ONDCP to establish annual quantifiable and measurable objectives and specific targets in future Strategy iterations. By taking steps to address this requirement ONDCP could further demonstrate whether it is making meaningful progress every year toward the targets it sets. A 5-year projection for program and budget priorities. Pursuant to the ONDCP Reauthorization Act of 2006, the National Drug Control Strategy was required to include “a 5-year projection for program and budget priorities.” The SUPPORT Act retained this requirement. As we testified in March 2019, the 2019 National Drug Control Strategy did not include this information. Our subsequent analysis of the three companion documents showed that one document—the 2019 Performance Reporting System—provided more information about ONDCP’s program priorities but not ONDCP’s budget priorities. Specifically, 14 of the 17 objectives ONDCP included in the 2019 Performance Reporting System contain various 5-year targets, such as to reduce the rates of illicit drug and opioid use among youth by 15 percent. According to ONDCP officials, the objectives and targets in the 2019 Performance Reporting System satisfy the requirement for 5-year program and budget priorities. However, the document does not include how these objectives and targets relate to 5-year budget priorities. The SUPPORT Act continues to require ONDCP to include a 5-year projection of program and budget priorities in future Strategy iterations. By taking steps to address this requirement, ONDCP and National Drug Control Program agencies will be better positioned to plan for the resources needed to achieve the efforts that will have the greatest impact. Specific drug trend assessments. Pursuant to the ONDCP Reauthorization Act of 2006, the National Drug Control Strategy was required to include assessments of the reduction of the consequences of illicit drug use and availability and the reduction of illicit drug availability. We testified in March 2019 that the 2019 National Drug Control Strategy did not include these assessments. Our subsequent analysis of the three companion documents showed that the 2019 Data Supplement provided more information to address the required assessments but did not address all of the requirements. For example, the assessment of the reduction of the consequences of illicit drug use and availability was to include, among other things, the annual national health care cost of illicit drug use. However, the most recent national health care cost data in the 2019 Data Supplement is from 2007, and ONDCP did not indicate in the supplement whether more recent data were available. In another example, the assessment of the reduction of illicit drug availability was to be measured by, among other things, the number of illicit drug manufacturing laboratories seized and destroyed and the number of hectares of marijuana, poppy, and coca cultivated and destroyed domestically and in other countries. The 2019 Data Supplement provided data for marijuana and poppy until 2016 and for the quantity of coca eradicated until 2015. The SUPPORT Act no longer requires these specific assessments. However, the SUPPORT Act does include a new requirement that the National Drug Control Strategy provide “ description of the current prevalence of illicit drug use in the United States, including both the availability of illicit drugs and the prevalence of substance use disorders.” The SUPPORT Act also contains a new requirement—which we describe later in this report—for ONDCP to describe how each comprehensive, research-based, long-range quantifiable goal in the National Drug Control Strategy was determined, including data, research, or other information used to inform the determination. We address ONDCP’s implementation of this new requirement under the SUPPORT Act later in the report. A description of a performance measurement system. Pursuant to the ONDCP Reauthorization Act of 2006, the National Drug Control Strategy was required to include a “description of a national drug control performance measurement system” that: develops 2-year and 5-year performance measures and targets; describes the sources of information and data that will be used for identifies major programs and activities of the National Drug Control Program agencies that support the goals and annual objectives of the National Drug Control Strategy; evaluates the contribution of demand reduction and supply reduction activities implemented by each National Drug Control Program agency in support of the Strategy; monitors consistency between the drug-related goals and objectives of the National Drug Control Program agencies and ensures that each agency’s goals and budgets support and are fully consistent with the National Drug Control Strategy, among others; and coordinates the development and implementation of national drug control data collection and reporting systems to support policy formulation and performance measurement, including certain assessments. We testified in March 2019 that the 2019 National Drug Control Strategy did not include a description of a performance measurement system pursuant to the ONDCP Reauthorization Act of 2006. Our subsequent analysis of the three companion documents showed that the 2019 Performance Reporting System provides information about some of the elements the performance measurement system is required to do. For example, the 2019 Performance Reporting System includes 2-year and 5- year targets for many of its objectives and describes some of the sources of data that will be used to measure each target. However, it does not include a description of the system that will accomplish each of the requirements in the ONDCP Reauthorization Act of 2006. For example, it does not describe a performance measurement system that identifies major programs and activities of the National Drug Control Program agencies that support the goals and annual objectives of the National Drug Control Strategy. Such programs and activities could indicate how ONDCP expects to achieve these objectives, such as how to educate the public about the dangers of drug use, or how to expand access to evidence-based treatment. Additionally, it does not describe how the performance measurement system monitors consistency between the drug-related goals and objectives of the National Drug Control Program agencies and ensures that each agency’s goals and budgets support and are fully consistent with the National Drug Control Strategy. ONDCP officials stated they believe the 2019 Performance Reporting System meets the statutory requirement for a description of a performance measurement system. The SUPPORT Act, as originally enacted in October 2018, no longer required a description of a performance measurement system. However, the ONDCP Technical Corrections Act of 2019, enacted in November 2019, amended the SUPPORT Act to reinstate the requirement for a description of a performance measurement system. Therefore, this requirement will apply to the 2020 National Drug Control Strategy and future Strategy iterations. ONDCP Has Met Some SUPPORT Act Requirements That GAO Reviewed but Its Approach to Meeting Others Does Not Incorporate Key Planning Elements ONDCP Has Addressed Requirements for New Coordinator Positions As of August 2019, ONDCP filled all five coordinator positions described in the SUPPORT Act, two of which are substantively new positions. Specifically, ONDCP officials stated that they have designated officials for the new positions of performance budget coordinator and emerging and continuing threats coordinator. By filling each of these positions, ONDCP is better positioned to fulfill the responsibilities for which each position is accountable, as described in figure 2 below. ONDCP’s Approach to Meeting Selected New Requirements for the National Drug Control Strategy and the Drug Control Data Dashboard Does Not Incorporate Key Planning Elements As of October 2019, ONDCP officials could not provide in writing or otherwise describe key planning elements to ensure ONDCP can meet selected new requirements in the SUPPORT Act related to the development of the 2020 and future National Drug Control Strategy iterations, and related to the development and implementation of the Drug Control Data Dashboard. Figure 3 outlines the selected requirements for the Strategy, which were effective upon enactment of the SUPPORT Act in October 2018. Each of the four selected SUPPORT Act requirements described in Figure 3 requires ONDCP to include specific information in the 2020 and future National Drug Control Strategy iterations. For example, for each comprehensive, research-based, long-range, quantifiable goal, the National Drug Control Strategy must contain (1) a description of how each goal will be achieved; (2) a performance evaluation plan for each goal; and (3) a description for how each goal was determined. The National Drug Control Strategy must also include a plan to expand treatment for substance use disorders. Officials from ONDCP and selected agencies told us that in spring 2019 ONDCP requested that the National Drug Control Program agencies determine how their existing programs and activities align with the 2019 National Drug Control Strategy, including the goals and objectives articulated in the 2019 Performance Reporting System. In October 2019, ONDCP officials told us that the 2020 Strategy would be issued in accordance with the SUPPORT Act, by the first Monday in February (February 3, 2020). ONDCP also provided us with two documents to describe its approach for meeting this deadline. One document includes a table that lists SUPPORT Act requirements along with the ONDCP component(s) responsible for implementation and the deadline. The other document provides a high-level summary of the National Drug Control Strategy development and interagency review process. For example, the plan to monitor progress on the drafting of components’ sections of the Strategy notes that it is to occur through “as- needed (but frequent)” meetings with the deputy chief of staff and the components and their heads. The extensive nature of the new SUPPORT Act requirements, as described above, indicates that significant implementation steps may be necessary, such as, a description of the specific steps necessary to accomplish this overarching task, identification of who will be responsible for each step, and a schedule of interim milestones. However, neither of these documents describes such critical implementation steps. Further, neither specifies what resources or processes, for example, would be needed and by what specific milestone date ONDCP would accomplish any particular step to complete the overall work in a timely manner. For example, the document that includes the table indicates that the deadline for all requirements related to the National Drug Control Strategy is February 2020. However, some requirements associated with the development of the Strategy, such as consultation requirements, would need to be completed before the Strategy’s due date—February 2020. According to Standards for Internal Control in the Federal Government under Internal Control Principle 6, to achieve an entity’s mission, management should define objectives in specific terms so they are understood at all levels of the entity. This involves clearly defining what it is to be achieved, who is to achieve it, how it will be achieved, and the time frames for achievement—in other words, key planning elements. Standards for project management also state that managing a project involves developing a plan with specific actions and milestone dates. Defining these key planning elements will help provide assurance that ONDCP’s efforts will result in a National Drug Control Strategy—for 2020 and future years—that fully addresses the requirements of the SUPPORT Act. In addition, developing and documenting these planning elements would help ONDCP structure its planning efforts through consideration of resource investments, time frames, and any necessary processes, policies, roles, and responsibilities to address each requirement. Furthermore, implementing these planning elements will help ensure that ONDCP follows a routine planning process going forward, and that future iterations of the National Drug Control Strategy that ONDCP develops are consistent with the law. Additionally, as of December 2019, ONDCP has not documented key planning elements to ensure it will meet the SUPPORT Act’s requirements for the Drug Control Data Dashboard, to make timely information publicly available on the scope and complexity of drug use and drug control activities. The SUPPORT Act includes requirements for what data is to be included in the Drug Control Data Dashboard as well as its functionality, to ensure it is searchable and sortable. Figure 4 outlines the requirements for the Drug Control Data Dashboard, which were effective upon enactment of the SUPPORT Act in October 2018. In August 2019, ONDCP posted a public version of the Drug Control Data Dashboard that included information from the 2019 Data Supplement in spreadsheet format, but did not provide all of the data required by the SUPPORT Act. For example, the Drug Control Data Dashboard does not include required data on the extent of the unmet need for substance use disorder treatment. ONDCP officials shared information regarding potential data sources they may use to fulfill the additional required data elements. In addition, ONDCP officials told us that some data requirements listed in the statute do not exist at this time. For example, ONDCP officials stated that data do not exist regarding the known and estimated flow of substances into the United States for the current calendar year and each of the three previous years. ONDCP officials stated that there was more work necessary to ensure all the required data are incorporated into the Drug Control Data Dashboard. At that time, they also stated that they expected to address all required elements by the end of 2019. However, we found that they do not have key planning elements, such as a specific timeline with interim milestones or documented plans for when and how they would complete this work. ONDCP subsequently posted an updated version of the Drug Control Data Dashboard, which we reviewed in December 2019. While the updated Drug Control Data Dashboard identifies required data elements that are unavailable, ONDCP has not addressed how or when ONDCP planned to provide them, such as by identifying alternative data sources or identifying additional resources that may be necessary for enhanced data collection efforts. The SUPPORT Act also requires the Drug Control Data Dashboard to be machine-readable and searchable by year, agency, drug, and location, to the extent practicable. Officials stated in September 2019 they planned to add this functionality to the Drug Control Data Dashboard in the fall of 2019. In written comments on a draft of this report in December 2019, ONDCP indicated that the data have been posted in a machine-readable, sortable, and searchable format. However, as of December 2019, we found that the Drug Control Data Dashboard is still not fully searchable by year, agency, drug and location. We have previously reported on key practices for agencies to follow when reporting government data. These practices describe, for example, that agencies should ensure their website’s data search functions and overall interface is intuitive to users. While effective implementation of such functions can be a significant undertaking, ONDCP does not have plans to account for timing, content, functionality, or any additional resources required to fully implement this requirement. ONDCP officials stated in September 2019 they may need to consult Congress about additional resources to fulfill all of the requirements related to the Drug Control Data Dashboard, but stated that they do not have specific plans for what resources they may request. Internal control standards call for agencies to define key planning elements, including how a task will be accomplished and associated timeframes. Developing and documenting key planning elements— including resource investments, time frames, and any necessary processes, policies, roles, and responsibilities—will better position ONDCP to fully implement all of the law’s requirements for the Drug Control Data Dashboard. Once implemented, the Drug Control Data Dashboard will help enable ONDCP to capitalize on available data to better understand the scope and nature of the drug crisis. Conclusions ONDCP is responsible for leading the nation’s fight against a persistent drug epidemic that continues to devastate Americans’ lives. However, the 2019 National Drug Control Strategy does not fully comply with the law, and the agency has not developed key planning elements to help ensure it will meet its significant additional responsibilities under the SUPPORT Act. These responsibilities include issuing the National Drug Control Strategy in accordance with statutory requirements to help prioritize and measure key efforts to address the drug epidemic and creating a Drug Control Data Dashboard that contains timely information about the scope and complexity of the drug epidemic. These responsibilities also extend beyond the upcoming 2020 Strategy, with requirements to complete future Strategy iterations on a regular basis. Developing and documenting key planning elements, such as resource investments, time frames, and any necessary processes, policies, roles, and responsibilities—will help ONDCP structure its ongoing efforts. Implementing this approach will then better position ONDCP to meet statutory requirements for the next Strategy, due in February 2020, and satisfy all requirements related to the Drug Control Data Dashboard. Implementing this approach over time will also help ONDCP ensure it is meeting statutory requirements for future iterations of the National Drug Control Strategy. Recommendations for Executive Action We are making 4 recommendations to ONDCP. The Director of ONDCP should develop and document key planning elements to help the agency meet the SUPPORT Act requirements for the 2020 National Drug Control Strategy and future Strategy iterations. These planning elements should include descriptions of resource investments, time frames, and any processes, policies, roles, and responsibilities needed to address each requirement. (Recommendation 1) The Director of ONDCP should—after developing and documenting key planning elements to meet the SUPPORT Act requirements—routinely implement an approach, based on these planning elements, to meet the requirements for the 2020 National Drug Control Strategy and future Strategy iterations. (Recommendation 2) The Director of ONDCP should develop and document key planning elements to help the agency meet the SUPPORT Act requirements to establish a Drug Control Data Dashboard that would include descriptions of resource investments, time frames, and any processes, policies, or roles, and responsibilities needed to address this requirement. (Recommendation 3) The Director of ONDCP should—after developing and documenting key planning elements—implement an approach, based on these planning elements, to meet the SUPPORT Act requirements to establish a Drug Control Data Dashboard. (Recommendation 4) Agency Comments and Our Evaluation We provided a draft of this report for review and comment to ONDCP, DHS, DOJ, and HHS. ONDCP provided written comments, which are summarized below and reproduced in appendix I. ONDCP, DHS, and DOJ also provided technical comments, which we incorporated, as appropriate. In an email, an HHS official stated that HHS did not have any comments on the report. In its written comments, ONDCP stated that it accepted the first two recommendations regarding the need for a robust internal planning process for National Drug Control Strategies. Specifically, the first recommendation is for ONDCP to develop and document key planning elements to help the agency meet the SUPPORT Act requirements for the 2020 National Drug Control Strategy and future Strategy iterations. The second recommendation is for ONDCP to routinely implement an approach to meet these requirements. In particular, ONDCP agreed to implement key planning elements for future Strategies that will include detailed descriptions of planned steps, identifying which ONDCP component will be responsible for each step, resource investments, interim milestones, and overall time frames. If implemented as planned, these actions would address the intent of these recommendations. Regarding the third and fourth recommendations related to the Drug Control Data Dashboard, ONDCP noted that these recommendations have been rendered moot because the agency has already fully complied with posting the Data Dashboard to its website. ONDCP also stated that it has posted to the Data Dashboard all of the drug-related data required by ONDCP’s statute that currently exists. Further, ONDCP stated that the data has been posted in machine-readable, sortable, and searchable format as required and it will be updated on a continuous basis throughout the year as new data become available. While ONDCP has included additional information on the Dashboard, the two recommendations are to develop, document, and implement key planning elements for the Dashboard to fully meet the law’s requirements, which ONDCP has not yet done. For example, ONDCP identifies in the Dashboard which of the required data elements are unavailable, such as required data on the extent of the unmet need for substance use disorder treatment. However, as stated in the report, ONDCP has not documented key planning elements for how it will address these missing data. Such planning elements could include approaches for collecting the missing data, such as articulating a plan to work with Congress to identify alternative data sources or to identify additional resources that may be necessary for enhanced data collection efforts. Furthermore, ONDCP has not developed or implemented key planning elements to ensure the Drug Control Data Dashboard has the search features noted in the statute. In its current format, the Dashboard is not fully searchable by year, agency, drug, and location. While the statute indicates that search features should have been implemented “to the extent practicable,” ONDCP did not explain why it was not practical to implement them. Therefore, we continue to believe that developing, documenting, and implementing key planning elements for the Dashboard to fully meet the law’s requirements will help enable ONDCP to capitalize on available data to better understand the scope and nature of the drug crisis. ONDCP also noted several points related to our specific findings, as discussed below. First, ONDCP noted that it did issue robust drug budget guidance to National Drug Control Program agencies during 2017 and 2018. The report acknowledges that ONDCP provided this guidance. However, as explained in the report, the guidance is statutorily required to address funding priorities developed in the National Drug Control Strategy. Since ONDCP did not issue a Strategy in 2017 or 2018, it could not meet this statutory requirement. In addition, ONDCP stated that it maintains that the 2019 National Drug Control Strategy met all statutory requirements, and therefore does not agree with our analysis of its adherence to those requirements. ONDCP also noted that the four requirements we assessed constitute only a small portion of the many requirements for the 2019 National Drug Control Strategy and that the report gives the misleading impression that ONDCP did not comply with some significant number of requirements. We recognize that there are a number of requirements for the Strategy; however, as stated in the report, our review focused on these four provisions because we determined them to be significant to ONDCP’s role in setting a strategic direction to oversee and coordinate national drug control policy, and because they are critical to ensuring a framework for measuring results. Specifically, the provisions related to including information in the Strategy related to annual quantifiable and measurable objectives and specific targets, a 5-year projection for program and budget priorities, specific drug trend assessments, and a description of a performance measurement system. As detailed in the report, we found that the 2019 Strategy addressed some—but not all—of these four statutory requirements. For example, we found that the Strategy did not include a 5-year projection for budget priorities and included only some information related to specific drug trend assessments. In its written comments, ONDCP provided additional explanation for why it did not agree with our characterization of the requirements. For example, ONDCP stated that it is not able to provide quantitative fiscal year projections for future years because this would go against long-standing Office of Management and Budget policy. Related to drug trend assessments, ONDCP noted that it reports data generated by other government agencies, and that policy research funding for ONDCP has not been appropriated since fiscal year 2011. We made recommendations, which ONDCP agreed to implement, focused on developing and implementing key planning elements such as descriptions of resource investments; timeframes; and processes, policies, and responsibilities needed to address each requirement. Implementing these planning elements could, for example, help ensure that ONDCP addresses any policy considerations or additional resources needed to help ensure that future iterations of the Strategy fully meet all statutory requirements. We are sending copies of this report to the appropriate congressional committees, the Director of the Office of National Drug Control Policy, the Secretary of the Department of Health and Human Services, the Acting Secretary of the Department of Homeland Security, the Attorney General, and other interested parties. In addition, this report is available at no charge on GAO’s website at http://www.gao.gov. If you or your staff has any questions concerning this testimony, please contact Triana McNeil at (202) 512-8777 or McNeilT@gao.gov or Mary Denigan-Macauley at (202) 512-7114 or McNeilT@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are listed in appendix II. Appendix I: Comments from the Office of National Drug Control Policy Appendix II: GAO Contacts and Staff Acknowledgments GAO Contacts Staff Acknowledgments In addition to the contacts named above, Joy Booth (Assistant Director), Will Simerl (Assistant Director), Michelle Loutoo Wilson (Analyst-in- Charge), Billy Commons, Wendy Dye, Jane Eyre, Kaitlin Farquharson, Susan Hsu Michael, Amanda Miller, and Jan Montgomery made key contributions to this report.


Background OSS is software distributed under a license that provides broad rights to use, modify, and redistribute the original source code. Open source licenses impose certain obligations on users who exercise these rights. Specific obligations vary among the many different open source licenses. Common obligations include making the source code available, publishing a copyright notice, or giving any recipient of the program a copy of the license. Certain restrictive open source licenses allow users to copy, modify and distribute software provided that modified versions (i.e., derivatives) are subject to the same license terms and conditions as the original code. This is intended to prevent software that is derived from or contains code issued under such a license from becoming a closed- source product that can be marketed and sold exclusively. The reuse of OSS is viewed as a promising means to reduce development costs while improving software quality. According to software experts, software reuse has the potential to: increase reliability because systems will be developed with thoroughly tested and proven components, increase productivity by reducing the time and effort needed to develop software, reduce costs by enabling the sharing of knowledge and practices needed to develop and maintain software, and establish a more standard and consistent approach to software development and maintenance by using common components and procedures. OMB Memorandum on Federal Source Code Policy In August 2016, OMB issued a memorandum to the heads of departments and agencies to ensure that new custom-developed source code be made available for reuse across the federal government. The memorandum was intended to improve the way federal agencies buy, build, and deliver information technology and software, and required that all agencies establish a pilot program under which at least 20 percent of new custom-developed code would be released as OSS for 3 years. OMB also required that agencies develop a metric to calculate the percentage of code released as OSS to gauge its progress on implementing the pilot program. OMB’s memorandum also identified four supporting requirements, among others, needed to implement an OSS pilot program: Issue an OSS policy that ensures code is available for government- wide reuse. Conduct an OSS three-step software solutions analysis that includes: (1) a strategic analysis and analysis of alternatives, (2) consideration of existing commercial solutions, and (3) consideration of custom development. In addition, agencies must consider several factors throughout each of the three-steps of the analysis such as cloud computing and open standards. Secure data rights to government-wide reuse and inventory new custom code, in accordance with the guidance provided by the code.gov website. Facilitate the OSS community by developing and releasing the code in a manner that (1) fosters communities around shared challenges; (2) improves the ability of the OSS community to provide feedback on, and make contributions to, the source code; and (3) encourages federal employees and contractors to contribute back to the broader OSS community by adding value to existing projects. In doing so, agencies should comply with the following principles: (1) leverage existing communities, (2) engage in open development, (3) adopt a regular release schedule, (4) consider code contributions, and (5) document source code to facilitate use and adoption. DOD Implementation of OMB’s Open Source Software Requirements DOD’s CIO is responsible for implementing OMB’s requirements for the department’s OSS pilot program. The CIO reports directly to the Secretary of Defense, and is responsible for the department’s information technology (including national security systems and defense business systems), information resources management, and efficiencies. The CIO is also responsible for developing strategies and policy on the operation and protection of all of the department’s information technology and information systems. Other responsibilities include maintaining a consolidated inventory of mission critical and mission essential information systems, evaluating and monitoring performance measurements, and other duties to manage the information environment throughout the department. In addition, the Defense Digital Service is responsible for assisting the CIO in implementing the OSS pilot program, among other initiatives. The Defense Digital Service is composed of commercially experienced software developers, software designers, product managers, and problem solvers within DOD. The organization works on specific projects or programs in support of the DOD in a hands-on way to materially improve digital services across the department. The Defense Digital Service also works with the CIO to monitor the identified programs, facilitate the process to open source the code, and populate the source code inventory located on Code.mil. DOD’s Efforts to Increase Use of Open Source Software In June 2018, DOD’s CIO issued a report to Congress as directed by section 875(b) of National Defense Authorization Act for Fiscal Year 2018 (Public Law 115-91). The report provided Congress with the department’s plan to implement the OSS pilot program established by OMB’s memorandum. In the report, the CIO committed to sharing its unclassified, custom-developed source code as widely as possible in four ways: (1) review and select software programs that have self-identified to the Defense Digital Service as ready to open source its code; (2) query its contracts database to identify contracts that contain appropriate data rights language; (3) determine if source contained within existing source code repositories can be made available; and (4) issue a department- wide data call to identify and select programs where new, custom code is being developed. The CIO also reported that the department would prioritize and assess identified software programs and work with components to develop mechanisms to report progress. The report included selection criteria for identifying candidate software programs: (1) programs with contractually secured government data rights; (2) programs with contractual rights to enable, support, and enforce DOD and government-wide sharing and reuse of custom-developed code; (3) programs that have appropriate administration to ensure that government’s rights are maintained; and (4) programs that utilize best practices to ensure custom-developed code, documentation, and other associated materials are delivered in a reusable manner. The CIO also reported that the Defense Digital Service would assist programs. Specifically, the Defense Digital Service is to develop guidelines, processes, and answers to frequently asked questions regarding the use of OSS. In October 2018, the CIO issued a memorandum to the Chief Management Officer, secretaries of the military departments, Chairman of the Joint Chiefs of Staff, under secretaries of Defense, chiefs of the military services, general counsel, Director of Cost Assessment and Program Evaluation, Director of Operational Test and Evaluation, and the Assistant Secretary of Defense for Legislative Affairs notifying them of the need to implement an OSS pilot program in accordance with OMB’s 2016 memorandum. The CIO required these organizations to take four actions within 30 days of issuing the memorandum: Identify all unclassified custom-developed source code created or paid for by the department—regardless of data rights and open source status—created on or after August 2016 and provide the CIO information required by the guidelines on the code.gov website; Identify and provide a point of contact that can participate in open source efforts; Direct authorizing officials to rapidly review and approve unclassified code for public, open source release after appropriate security, code, and policy review; and Direct contracting officers to secure the least restrictive data rights to custom-developed source code in all future contracts in accordance with DOD federal acquisition regulations. DOD Has Not Fully Implemented an Open Source Software Pilot Program and Related OMB Requirements DOD was mandated by law to initiate the OSS pilot program established by OMB memorandum M-16-21 which required (1) releasing at least 20 percent of newly custom-developed code each year for the term of the pilot program, and (2) collecting additional data concerning new custom software to inform measures to gauge the performance of the pilot program. Further, the OMB memorandum also required DOD to (1) issue an OSS policy, (2) conduct an OSS analysis, (3) secure data rights and inventory custom code, and (4) facilitate the OSS community. As of late April 2019, DOD had not fully implemented the OSS pilot program mandated by law. DOD had partially implemented the requirement of releasing at least 20 percent of newly custom-developed code as OSS. Specifically, as of July 2019, the Code.gov website reported that the department had released less than 10 percent of its custom developed code. The department was in the early stages of its pilot program and had not determined when the pilot would be fully implemented. The CIO reported that the size of the department makes it nearly impossible to inventory all of its source code custom developed since August 2016. As such, the CIO stated that it would be difficult to meet the OMB memorandum’s goal of releasing at least 20 percent of its new custom code as OSS. In addition, DOD had not implemented the requirement to develop a consistent measure to gauge the performance of the department’s pilot program. DOD had not developed such a measure due to a lack of consensus in the department about what data should be collected. According to the CIO, if the measure is “lines of code,” then it unfairly discounts projects that invest a significant amount on research, but are small otherwise. If the measure is “project hours,” then it discounts those projects that came about from sparks of innovation that took little time to develop. If the measure is “project count,” then it ignores the other two possible measures. The CIO noted that since the components will be expected to collect and report the required data, the CIO office plans to reach out to them to facilitate consensus around data needs and what measure should be used to calculate and monitor performance. Regarding the four OMB memorandum requirements supporting the implementation of the pilot program, the department partially implemented three and did not implement the remaining one. Table 1 describes the extent to which DOD implemented the OMB supporting requirements, and is followed by a description of DOD’s efforts on each requirement. Issue OSS policy. DOD had not implemented the requirement to issue an OSS policy. According to DOD’s CIO, the department has existing acquisition policies applicable to OSS, such as the 5000 series and a memorandum issued in October 2009 that clarifies OSS. However, according to DOD officials, these policies are outdated and do not comply with OMB’s memorandum. For example, while the department’s policies indicated that programs must conduct an analysis of alternatives, trade studies, or a business case analysis prior to initiating any technology acquisition or custom code development, they did not require programs to consider the value of publishing custom code as OSS and negotiate data rights reflective of its value. The department acknowledged that it does not have a policy that addresses the OMB memorandum’s requirement. According to the CIO, the department had been slow to develop a policy because these types of changes require significant resources, coordination, and buy-in across the department that will take additional time to address. The CIO also stated that the department plans to update its existing OSS memorandum by the end of the 2019 calendar year and issue it as policy. In particular, DOD intends to work with acquisition and program management officials to define approaches, processes, and best practices to expand software reuse. If the department effectively implements this intended step consistent with the OMB memorandum, DOD should be able to fully address this requirement. Conduct OSS analyses. DOD had partially implemented the requirement to conduct analyses to consider alternative software solutions. According to the DOD CIO, of the three elements required by OMB for a three-step analysis, the department’s current 5000 series policy addresses some of these elements. For example, as previously mentioned, the policy required programs to conduct an analysis of alternatives, trade studies, or a business case analysis prior to initiating any technology acquisition or custom code development. However, according to the CIO, DOD’s policy did not require programs to consider the value of publishing custom code as OSS and negotiate data rights reflective of its value-consideration. According to the CIO, the department has plans to address gaps in its existing policy by the end of the 2019 calendar year. If the department effectively implements this intended step consistent with the OMB memorandum, DOD should be able to fully address this requirement. Secure data rights and inventory new custom code. DOD had partially implemented this requirement by initiating the process to secure data rights for government-wide reuse and inventory its new custom code. Specifically, the October 2018 memorandum called for defense organizations to direct contracting officers to secure the least restrictive data rights for custom-developed source code in all future contracts, and identify all unclassified custom-developed source code created after August 2016. However, when we discussed this matter with DOD information technology and software officials responsible for the management of software in December 2018—2 months after the October 2018 memorandum had been issued— seven of 11 officials were unaware of the statutory mandate to initiate the pilot program in compliance with the OMB memorandum. Further, DOD has not provided a milestone for when it expects its inventory to be completed. Facilitate OSS community. DOD had partially implemented the requirement to establish an OSS community. According to the DOD CIO’s Custom Developed Source Code Data Call memorandum, dated October 2018, the DOD CIO is working with Defense Digital Service to develop guidelines and processes on when and how to open source code. In February 2017, DOD announced the launch of Code.mil, an open source initiative led by Defense Digital Service, that allows software developers around the world to collaborate on unclassified code written by federal employees in support of DOD projects. Finally, the Defense Information Systems Agency established a website (Forge.mil) where community members can collaborate on open source and DOD community source software. The Forge.mil website also enables collaborative development through services such as software version control, requirements management, discussion forums and document repositories. However, DOD had not yet fully engaged in open source code development, established a regular release schedule for its software code, or fully documented its source code to facilitate use and adoption department-wide. According to the CIO, the department is in the early stages of implementing the OSS pilot program and had not yet published a revision to the existing OSS policy memorandum. The CIO stated that the office plans to request collaboration and input from organizations throughout DOD for improvement initiatives and identifying specific processes and expectations for improving custom-developed software within the Department. However, DOD has not provided a milestone for when the requirements will be fully implemented and stated that achieving 100 percent compliance is not a realistic expectation. Until DOD fully implements the OSS pilot program mandated in the National Defense Authorization Act for Fiscal Year 2018 including the requirements of OMB memorandum M-16-21, the department will likely miss opportunities to achieve related cost savings and efficiencies. Further, the department will not be effectively positioned to ensure management oversight and implementation of the pilot program. DOD Officials Shared Views of Expected Benefits and Risks Associated with the Use of Open Source Software DOD Officials Agree That Using Open Source Software Could Result in Financial Benefits and Increased Efficiency DOD officials representing 11 components reported that OSS can potentially yield financial benefits and increase efficiency. Officials provided the following examples of financial benefits: Officials in the office of the Navy Chief Information Officer, the Army Communications-Electronics Command, the Office of the Under Secretary of Defense for Acquisition and Sustainment, and the Office of the Assistant Secretary of the Air Force for Acquisition, Technology, and Logistics stated that OSS is generally less expensive than commercial off-the-shelf (COTS) leading to cost savings. An Air Force official we spoke with stated that the increased use of OSS may potentially result in cost savings. Further, the official noted general criteria used by the Air Force to identify software projects that may potentially be appropriate for the use of OSS. Specifically, OSS should be considered if, among other things, the required maintenance would not result in a reduction of cost savings or efficiency if the maintenance is performed in-house. An official we spoke with at the Defense Advanced Research Projects Agency stated that OSS has positive benefits in terms of reducing costs by reducing duplicative efforts. In addition, this official also stated that OSS allows institutions of any size and budget to partake in shared investments providing access to software capabilities at a much lower cost. A program manager from the Defense Information Systems Agency reported that the agency had identified an OSS solution that provided more functionality at less cost than the commercial solution provided through a vendor. The program manager explained that when the agency implemented the new OSS solution, it realized $20 million in annual savings over the commercial solution that had been maintained by a vendor. Officials also shared examples of how OSS can increase efficiency in software development. For example, Officials from the offices of the Navy CIO and the US Marine Corps CIO stated that OSS solutions may increase efficiency by providing a rapid resolution to the needs and requirements of users. In contrast, rapid development efforts are not conducive for COTS solutions because of the long process required to obtain solutions that are needed quickly. Similarly, an Air Force official noted that the increased use of OSS may result in increased efficiency by providing rapid responses to user requirements. A program manager from the Defense Information Systems Agency reported that the selection of an OSS solution rather than a COTS solution contracted through a vendor had resulted in increased efficiency. The official explained that the use of the OSS solution allowed the agency to develop and maintain in-house skills that would not have been available had they opted to contract with a vendor providing a skilled workforce. DOD Officials Expressed Differing Views on the Cybersecurity Risk Posed by Open Source Software Officials from the 11 components expressed mixed views on managing cybersecurity risks that could be posed by using OSS. Specifically, officials from three components expressed their views that security concerns and the lack of a cybersecurity governance process could result in the sporadic use of OSS. For example: A Navy CIO official viewed insider threats, such as a disgruntled employee embedding malicious code, as a factor that could significantly limit the use and sharing of OSS. According to Navy officials, without a process to verify that the software is free of malicious code, the Navy would risk the assurance it requires to increase the use of OSS. The official said that, in contrast, such concerns are not an issue when it comes to COTS software because of the test and verification process to ensure it is free from malicious code. An official in the office of the Marine Corps CIO stated that OSS is used sporadically in their software development efforts because some cybersecurity officials within the Marine Corps discourage its use due to security concerns. An official from the Army’s Communications-Electronics Command noted that DOD lacks a governance process once the originating entity releases the source code as open source. The originating entity no longer retains control over redistributed versions of the source code. According to Communications-Electronics Command officials, Army project managers may be hesitant to utilize OSS because of this perceived security risk. On the other hand, DOD officials from eight components stated that the potential cybersecurity risks posed by the use of OSS were manageable and that the use of OSS should not be limited. For example: The policy advisor from the Office of the Under Secretary of Defense for Acquisition and Sustainment noted that scanning tools to analyze and identify safe and reliable open source code are not being used. Employing available scanning tool options could result in discovering available OSS. The policy advisor also noted that building security into software operations, rather than through the development of software, would enable users to know if code has been subverted and to react appropriately. A program management official from the Office of the Under Secretary of Defense for Acquisition and Sustainment suggested that security concerns may be mitigated by establishing a secure repository for trusted code. An official in the Office of the Assistant Secretary of the Air Force for Acquisition, Technology, and Logistics reported that, as long as OSS is properly vetted to ensure it is secure and free from malware, it offers an opportunity for the department to achieve cost savings and efficiencies. Conclusions Pilot testing the use of OSS is an important way to ascertain and improve the way DOD buys, builds, and delivers information technology and software solutions. However, the department is in the early stages of implementing its pilot program and had not determined when the pilot would be fully implemented. Specifically, DOD has not made 20 percent of its new code available for reuse nor has it identified a measure to gauge the performance of its pilot program. Moreover, DOD has not yet established milestones for securing data rights and conducting an inventory or facilitating community. Until DOD fully implements its pilot program and establishes milestones for completing the OMB requirements, the department will not be positioned to take advantage of significant cost savings and efficiencies. Recommendations for Executive Action We are making the following four recommendations to DOD: The Secretary of Defense should ensure the department implements the pilot program by releasing at least 20 percent of newly custom- developed code as OSS. (Recommendation 1) The Secretary of Defense should ensure the department identifies a measure to calculate the percentage of code released to gauge its progress on implementing the pilot program. (Recommendation 2) The Secretary of Defense should ensure the department establishes milestones for completing the requirements of OMB memorandum M- 16-21 of securing data rights and conducting an inventory. (Recommendation 3) The Secretary of Defense should ensure the department establishes a milestone for completing the OMB memorandum’s requirement of facilitating an OSS community. (Recommendation 4) Agency Comments and Our Evaluation DOD provided written comments on a draft of this report, which are reproduced in appendix II. In its comments, the department did not concur with our first recommendation, partially concurred with our second recommendation, and concurred with the third and fourth. DOD did not concur with the first recommendation on ensuring that the department implements the pilot program by releasing at least 20 percent of newly custom-developed code as OSS. The department stated that it does not believe that the pilot program as described in the OMB memorandum is implementable as proposed. For example, DOD asserts that most of DOD’s custom developed software is created for weapons systems and releasing the associated code is sensitive for national security reasons. In addition, the size and complexity of DOD presents unique challenges for the department compared to other federal agencies such as inventorying all software development projects to establish a baseline. DOD added, however, that the OMB memorandum explicitly states that national security exceptions do not apply to the pilot program. DOD also stated that it recognizes the value of collaborative software development and has plans to release additional guidance on releasing OSS and procedures for maintaining its inventory. Once DOD establishes a baseline inventory of custom-developed software and the procedures for maintaining it, the department states it will be able to determine if the 20 percent is an appropriate goal. We understand the potential constraints DOD faces and that national security considerations are to be factored into decisions DOD will need to make about which custom developed software to include in the pilot. However, DOD is mandated by law to implement the OSS pilot program established by OMB memorandum M-16-21. Further, the OMB memorandum instructs agencies to refrain from selecting code for release that would fall under exceptions such as national security risk. As such, DOD has flexibility on making decisions about which custom-developed code to include in the pilot. While we agree that a baseline inventory is needed, DOD must include at least 20 percent of new custom-developed code each year for the term of the program to satisfy the mandate. DOD partially concurred with the second recommendation on ensuring that the department identifies a measure to calculate the percentage of code released to gauge its progress on implementing the pilot program. Specifically, the department stated that the additional guidance it plans to release before the end of 2019 on OSS will include measures to gauge how much code has been developed and how much has been released. In addition, DOD noted that these measures will support good management of the overall portfolio of information technology, even in the absence of the mandated pilot program established by the OMB memorandum. We believe that the measure to calculate the percentage of code should be used to assist the department in meeting the OMB memorandum’s requirements. We also agree with the benefits of developing a measure to manage the portfolio of information technology. DOD concurred with the third and fourth recommendations related to establishing milestones for completing the OMB memorandum’s requirements of securing data rights, conducting an inventory, and facilitating an OSS community. According to the department, it has issued a memorandum that directed contracting officers to secure the least restrictive data rights to custom-developed source code, in furtherance of the OMB requirements, and also included a data call that forms an initial basis of an inventory of custom-developed software. Regarding facilitating an OSS community, DOD stated that it has formed a community of practice called DevSecOps that is open to all software development organizations in the department and plans to use this forum to facilitate collaboration on the use of OSS. We are sending copies of this report to the appropriate congressional requesters and the Secretary of Defense. In addition, the report is available at no charge on the GAO website at http://www.gao.gov. If you or your staffs have any questions about this report, please contact me at (202) 512-4456 or at harriscc@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are listed in appendix III. Appendix I: Objectives, Scope, and Methodology Our objectives were to: (1) assess the extent to which the Department of Defense (DOD) has implemented the open source software (OSS) pilot program and other related requirements established by the Office of Management and Budget (OMB), and (2) describe the views of responsible DOD officials on the use of OSS to achieve efficiency at the department. To address the first objective, we selected six requirements from the OMB memorandum titled the Federal Source Code Policy: Achieving Efficiency, Transparency, and Innovation through Reusable and Open Source Software (M-16-21, Aug. 8, 2016) as criteria to assess the extent to which DOD has implemented the OSS pilot program. Two requirements establish the OSS pilot program: (1) releasing at least 20 percent of newly custom-developed code each year for the term of the pilot program, and (2) developing a metric to gauge the performance of the pilot program. The other four requirements support the implementation of the pilot program: (1) issuing an OSS policy, (2) conducting an OSS analysis, (3) securing data rights and inventorying custom code, and (4) facilitating the OSS community. We met with officials from OMB to collect background information on the selection of requirements for the pilot program established in memorandum M-16-21. We also met with officials from the Office of the DOD Chief Information Officer (CIO) and the Defense Digital Service to discuss the status of the department’s implementation of the OSS pilot program. We reviewed DOD’s June 8, 2018 report to Congress and its October 2018 memorandum that details the department’s plans to implement the pilot program and compared them to the six requirements. To determine the extent to which the pilot program had been implemented, we evaluated DOD’s efforts to address each of the requirements using a 3- stage gradient scale (implemented, partially implemented, and not implemented). The requirement was assessed to be fully implemented if DOD provided us with sufficient evidence that the requirement had been fully met. We assessed a requirement to be partially implemented if DOD provided us with documentation of initial plans or had initiated action towards implementing the requirement. We determined that a requirement was not implemented when DOD did not provide us with documentation of planned or initiated actions to implement the requirement. To address the second objective on views of various responsible DOD officials, using professional judgement, we selected components across the department responsible for the management and development of OSS. The scope of stakeholders selected represent department-wide nongeneralizable views including military components, defense agencies, and other offices. At least one representative was selected from the following components: (1) Office of the Under Secretary of Defense for Acquisition and Sustainment, (2) Office of the DOD CIO, (3) office of a military service CIO, (4) Military Service Software Center or Command Center, (5) the Defense Information Systems Agency, and (6) the Defense Advanced Research Projects Agency. We conducted interviews with DOD officials from the following entities: Office of the Under Secretary of Defense for Acquisition and Sustainment; Office of the DOD CIO; Offices of the Navy and Marine Corps CIOs; Office of the Air Force Chief Technology Officer; Army Communications Electronics Command; the Defense Information Systems Agency; the Defense Advanced Research Projects Agency; and the Office of the Assistant Secretary of the Air Force for Acquisition, Technology, and Logistics. In order to summarize and report the views of the responsible DOD officials, we conducted structured interviews with representatives from the selected components. Each interview consisted of the same discussion topics based on the pilot program requirements established in OMB’s memorandum. The scope of this objective represents individual thoughts, views, and opinions and is not intended to convey an official or department response. Prior to each interview, participants were provided with OSS discussion topics, and the OMB memorandum, titled the Federal Source Code Policy: Achieving Efficiency, Transparency, and Innovation through Reusable and Open Source Software (M-16-21, Aug. 8, 2016). The contents of each interview were reviewed and summarized to identify the general views of OSS and on the anticipated implementation of the pilot program requirements established in OMB’s memorandum. We noted similarities and differences in the responses provided by the officials in the use of OSS including, but not limited to, potential benefits of using OSS, managing associated risk, and opinions on implementing the pilot program in compliance with the OMB memorandum. Discussions were split into two topic areas: practices on the use of OSS, and OMB’s memorandum to establish an OSS pilot program. Specifically, the discussion topics were presented during each meeting as follows: Part I: Practices on the use of OSS Your experience or your organization’s history with the practice of using OSS as a means to achieve cost reduction or efficiencies when buying, building, or delivering information technology and software solutions. The processes or practices that you or your organization perform to leverage open source code for projects that require the acquisition or development of custom source code. The extent to which you or your organization shares or uses open source software. For example, do you share or use open source software: (1) within your organization only, (2) across the DOD with other military services or defense agencies, (3) with other federal agencies, or (4) outside the federal government with the public. Also, how is the source code shared, leveraged, catalogued, stored, and accessed. Policies or guidance currently in use for OSS. General views and opinions on the use of open source code. Part II: OMB’s Memorandum to Establish an OSS Pilot Program When and how you or your organization became aware of OMB’s memorandum on Federal Source Code Policy. Your opinions and views about the OMB memorandum. Any specific concerns or reservations about the requirements contained in the OMB memorandum. The extent to which you or your organization may already be performing the steps in OMB’s proposed Three-Step Software Solutions Analysis. Discuss the feasibility of the pilot program requirement to release at least 20 percent of new custom-developed code as OSS. We conducted this performance audit from August 2018 to September 2019 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives. Appendix II: Comments from the Department of Defense Appendix III: GAO Contact and Staff Acknowledgments GAO Contact Staff Acknowledgments In addition to the contact named above, Eric Winter (Assistant Director), John Ortiz (Analyst-in-Charge), Rebecca Eyler, Franklin Jackson, and Kate Nielsen made key contributions to this report.


Background Air Force pilots are required to complete various phases of training before they are considered to be mission ready. To assess pilot candidates’ flying aptitude for both traditional (i.e., manned) and remotely-piloted aircraft, the Air Force first requires these candidates to attend initial flight training. Successful candidates for traditional aircraft then attend one of two schools: Euro-NATO Joint Jet Pilot Training, the graduates of which become Specialized Undergraduate Pilot Training, the graduates of which become fighter, bomber, airlift, or tanker aircraft pilots, depending on pilot strengths and Air Force needs. During or following this training, a pilot is assigned to a specific aircraft, and the flight training program proceeds through three stages: Initial Qualification Training. This stage of training qualifies a pilot for basic flying duties associated with the type of aircraft (e.g., an F-16). The pilot accomplishes this stage at a formal training unit before moving to an assigned squadron. Graduates of initial qualification training courses have basic aircraft qualification status. Mission Qualification Training. This stage of training occurs once the basic aircraft qualified pilot is at the assigned unit. The pilot undergoes mission qualification training in the type of aircraft assigned to qualify for the specific missions the unit is required to perform. Continuation Training. This stage of pilot training has two components. Pilots participate in continuation training to (1) in some instances upgrade their qualifications to fill certain positions, such as flight lead, instructor, or forward air controller through specialized continuation training; and (2) in all instances maintain proficiency and improve their capabilities to perform their units’ assigned missions. The Ready Aircrew Program establishes the minimum number of live training events, or “sorties,” and virtual simulator training events, or “simulator missions,” that aircrews of a particular combat aircraft must complete during the annual training cycle to maintain mission readiness. These sorties and simulator missions are aligned with the units’ primary missions, for which the units must maintain “proficiency,” and secondary missions, for which they must maintain “familiarity.” The Air Combat Command, as lead command for the Ready Aircrew Program, with the assistance of other major commands (including the Air Force Global Strike Command, Pacific Air Forces, and U.S. Air Forces in Europe) and associated subordinate organizations (i.e., air wings and squadron commanders), develops tasking memorandums for the Ready Aircrew Program that delineate and specify the annual continuation training requirements for personnel assigned to each of the subordinate combat units. The RAND Report Addressed Statutory Requirements, Followed Generally Accepted Research Standards, and Identified Deficiencies The RAND Report on the Ready Aircrew Program Addressed Section 351 Requirements by Assessing Training Requirements and Making Recommendations On the basis of our analysis, we found that the RAND report addressed each of the three statutory elements required by Congress in Section 351. First, the RAND report addressed two statutory elements by reviewing and assessing the assumptions underlying annual continuation training requirements for the Ready Aircrew Program and the overall effectiveness of the Ready Aircrew Program in managing aircrew training requirements. These two statutory elements focus on issues raised in our prior report recommendations, which we discuss in more detail later in this report. Table 1 provides detailed information about these statutory elements, our assessment of RAND’s findings, and RAND’s findings associated with each element. The RAND report addressed the third statutory element by making recommendations for the improved management of training requirements. Specifically, the RAND report made nine recommendations, listed in table 2. The RAND Report on the Ready Aircrew Program Was Consistent with Generally Accepted Research Standards The RAND report and its underlying analysis is consistent with generally accepted research standards for design, execution, and presentation. Table 3 summarizes our assessment of the extent to which the RAND report conformed with these standards. The RAND Report on the Ready Aircrew Program Identified Similar Deficiencies as Reported by Us in 2016 The RAND study and our previous audit work identified similar deficiencies in the management and operation of the Ready Aircrew Program. Specifically, in September 2016, we reported that the Air Force had used the same underlying assumptions to establish its annual training requirements in the Ready Aircrew Program from 2012 through 2016, which may not reflect current and emerging training needs. We concluded that without fully reassessing the assumptions underlying its training requirements, the Air Force could not be certain that its annual training plans are aligned with its stated goals to ensure a full-spectrum- capable force that can successfully achieve missions across a broad range of current and emerging threats. We recommended that the Air Force comprehensively reassess the assumptions underlying its annual training requirements—and make any appropriate adjustments in future aircrew training plans to ensure that its forces can accomplish a full range of missions. Additionally, in our September 2016 report, we also reported that the Air Force did not systematically evaluate the effectiveness of training performed as part of the Ready Aircrew Program. We recommended that the Air Force establish desired learning objectives and training support elements needed to accomplish the training expectations identified by the Ready Aircrew Program and develop a process to collect data to assess the effectiveness of annual training against these features. We discuss these recommendations and their status in more detail later in this report. The Air Force Reported Some General Actions to Link Readiness to Training but Plans No Further Actions in Response to RAND Recommendations Section 351 of the NDAA for Fiscal Year 2017 required the Air Force to report on any actions it plans to take in response to RAND’s recommendations and to estimate the resources required to implement the recommendations. On August 30, 2018, the Air Force provided its report—a one-page transmittal letter from the Secretary of the Air Force with the RAND report incorporated as an enclosure—to congressional committees in fulfillment of the Section 351 requirements. In its report, the Air Force agreed with RAND that more investment is needed in data collection because its current system does not lend itself to analysis that could be used to gain efficiencies. The Air Force also stated that it is addressing the RAND recommendations by working to link readiness to Ready Aircrew Program training requirements. Specifically, the Air Force stated that it was taking the following three actions: building training matrices to help commanders assess their units’ establishing common data architecture through the Air Force’s Chief Data Officer–led effort, and evaluating aspects of the Ready Aircrew Program to increase lethality and improve readiness as the Air Force shifts to executing the mandates of the 2018 National Defense Strategy. However, in its August 2018 report, the Air Force did not provide any additional details to further describe or link these broad actions to the RAND findings and recommendations. Therefore, the extent to which these three actions are responsive to the RAND recommendations is unclear. In October 2019, upon completion of our audit work, Air Force officials provided us with a briefing they described as a corrective action plan that further elaborated on the Air Force’s position with respect to each of RAND’s nine recommendations. Air Force officials conceded that the actions described in that briefing and plan were already underway at the time of the study and were not initiated in response to the study’s recommendations. They stated that, though they generally agreed with the recommendations, the Air Force lacked the manpower, resources, and means to implement them. As such, the Air Force considers each of the recommendations closed and plans no further actions. Further, the Air Force has no plans for future follow-up on implementation of the RAND recommendations. Accordingly, we did not further assess the actions described by the Air Force in relation to the RAND study. Table 4 summarizes the Air Force’s position as provided in its corrective action plan briefing. As described in table 4, the Air Force concurred with three recommendations, partially concurred with five, and did not concur with one. The following summarizes the Air Force position by concurrence category: Concur: In concurring with RAND Recommendations 2 and 8—to invest in data systems for the collection, access, and storage of data to correct deficiencies in current systems and improve analysis and readiness reporting—the Air Force stated that, before the recommendations were made, it had made available $5.15 million in fiscal year 2020 funding to develop an Aircrew Readiness Training Management Module as an upgrade to the Air Force’s Aviation Resource Management System. The Air Force expects that this module will centralize management of the Air Force Ready Aircrew Program Tasking Memorandum at the command level, transfer aircrew training data whenever a member moves to a new station, and improve capability to track the types of flight simulators used for training. Further, because the Air Force does not plan to take additional actions, it estimated no resources are required, beyond the $5.15 million already funded, prior to RAND making these recommendations. In concurring with RAND Recommendation 5—to document training quality to support requests for training resources—the Air Force explained that it is documenting the quality of training prescribed in its annual Air Force Ready Aircrew Program Tasking Memorandum at the squadron level. However, this is not a change based on the RAND recommendation and was being done prior to RAND making this recommendation. The Air Force factors unit training, accomplishments, and readiness inputs into the Air Force Ready Aircrew Program Tasking Memorandum and overall flying hour program. As the flying hour program is the basis for resource training requests and the Air Force factors unit training and accomplishments into its flying hour program, the Air Force explained that it plans to take no additional actions based on this recommendation. Further, because the Air Force does not plan to take additional actions, it estimated no resources were required to implement this recommendation. Partially Concur: In partially concurring with RAND recommendations 1, 3, 4, 6, and 9, Air Force officials explained that the mechanisms reflected in its comments are sufficient to address the intent of the RAND recommendations even though these mechanisms predated RAND’s recommendations. Consequently, according to the Air Force officials who briefed us, the Air Force plans no additional actions based on the recommendations, obviating the need to estimate resources in the case of these five recommendations. Nonconcur: In not concurring with RAND Recommendation 7—to consider changing how Air Force Flying Hour Program requirements affect the flying hour program—the Air Force explained that the flying hour program determination is standard across the Total Air Force, affecting more than combat aircrews alone, and that the centrality of the flying hour program to readiness and combat capability cannot be overemphasized and must be defendable and auditable. Further, based on its not concurring with this recommendation, the Air Force did not estimate resources for this recommendation. Notwithstanding the RAND study, the actions taken by the Air Force may not fully implement our prior recommendations as described previously. Specifically, we recommended that the Air Force comprehensively reassess the assumptions underlying its annual training requirements— including for example the total annual training requirements by aircraft, the criteria for designating aircrews as experienced or inexperienced, and the mix between live and simulator training—and make any appropriate adjustments in future aircrew training plans to ensure that its forces can accomplish a full range of missions. While RAND accomplished such an analysis as part of its review (see table 1, items 1(a) through 1(e) above), RAND concluded in its analysis that the Air Force did not have the objective measures of proficiency needed to determine the minimum and optimum number of sorties and that the combat air forces aviation community lacks consensus on how to define and measure an aircrew member’s proficiency. While the Air Force has defined proficiency in an Air Force manual issued in September 2019, the Air Force has not reassessed its assumptions underlying training requirements and made appropriate adjustments to future training plans per our recommendation. Further, we recommended that the Air Force establish desired learning objectives and training support elements needed to accomplish the training expectations in its annual Ready Aircrew Program tasking memorandums, and develop a process to collect data to assess the effectiveness of annual training. In commenting on RAND’s recommendations 2 and 8, the Air Force stated that, before the recommendations were made, it had made available $5.15 million in fiscal year 2020 funding to develop an Aircrew Readiness Training Management Module as an upgrade to the Air Force’s Aviation Resource Management System. The Air Force’s effort to upgrade this system, while not a result of the RAND recommendations, may meet the intent of our recommendation. For example, the Air Force’s development of the Aviation Resource Management System is expected to centralize management of the Air Force’s Ready Aircrew Program training requirements data at the command level, transfer aircrew training data whenever a member moves to a new station, and improve capability to track the types of flight simulators used for training. When fully implemented, these improvements may ultimately allow the Air Force to assess the effectiveness of annual training, as we recommended. However, it is too early to tell as the actions were under development or had just begun at the end of September 2019, and sufficient data to evaluate the results have not been collected. Fully implementing both of our recommendations would better position the Air Force to ensure that its aircrews receive effective training to achieve a range of missions for current and emerging threats. Agency Comments and Our Evaluation We provided a draft of this report to DOD for review and comment. In its written comments (reprinted in appendix II), DOD stated that it saw great value in our discussion. DOD added that its position on the two recommendations we made in our 2016 report—with which DOD did not concur—is fundamentally unchanged. However, we continue to believe the recommendations should be implemented by the Air Force, as previously discussed in this report. In its comments, DOD also stated that it is addressing training infrastructure and aircrew proficiency through two initiatives. First, in response to the 2018 National Defense Strategy, the department's Joint Operational Infrastructure Plan is framing the modernization effort of DOD-wide Operational Infrastructure. The Joint Operational Infrastructure Plan specifically addresses areas such as Live Virtual Constructive and aircrew training. Second, the department is pursuing efforts to align training events with the range of current and evolving threats. According to DOD, both efforts will address the underlying assumptions for aircrew training and proficiency with reportable readiness metrics. However, the Joint Operational Infrastructure Plan is a draft and not yet officially issued, according to an official at the Office of the Under Secretary of Defense for Personnel and Readiness. Therefore, details needed to assess this plan are not yet available for us to consider in determining whether the change will help to address the recommendations we made in 2016. Regarding efforts the department is pursuing to better align training events, the DOD comments did not include sufficient details to allow us to state whether the change could be helpful in addressing our recommendations. Nonetheless, to the extent that DOD is successful in completing, issuing, and implementing its new Joint Operational Infrastructure Plan, or takes further actions related to our prior recommendations, we will consider them as we continue to analyze DOD efforts to address our recommendations. We are sending copies of this report to the appropriate congressional committees, the Secretary of Defense, the Under Secretary of Defense for Personnel and Readiness, and the Secretary of the Air Force. In addition, the report is available at no charge on the GAO website at http://www.gao.gov. If you or your staff have any questions about this report, please contact me at (202) 512-5431 or russellc@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made major contributions to this report are listed in appendix III. Appendix I: GAO’s Criteria for Generally Accepted Research Standards To determine whether the RAND report followed generally accepted research standards, we chose the following criteria, which are based on a review of research literature and Department of Defense (DOD) guidance, from which we identified frequently occurring, generally accepted research standards that are relevant for defense studies, including those related to the presentation of results. These standards have been used in a number of our prior reports, modified as appropriate for each situation. For the purposes of this engagement, we assessed the RAND study against three major elements, listed below, that fall under generally acceptable research standards, as based on our prior work. These generally accepted research standards are consistent with Office of Management and Budget (OMB) Guidelines and DOD guidance on ensuring and maximizing the quality of information disseminated by federal agencies to the public. We discussed these standards with RAND officials, who agreed that they were generally consistent with their own quality standards for research and were applicable to this study. We determined that these standards are still current and relevant for the purposes of this report, based on their consistency with OMB and DOD guidance, discussions with RAND officials, and consideration of prior GAO work applying generally accepted research standards with the assistance of GAO’s Applied Research and Methods Team. The standards include the following: Design—Study is well designed. For the RAND study, we focused on the following elements for design: The study plan, scope, and objectives follow existing guidance. Assumptions and constraints are reasonable and consistent. Execution—Study is well executed. For the RAND study, we focused on the following elements for execution: The methodology is successfully executed. Data used to support study and analyses are validated. Presentation—Results are well presented. For the RAND study, we focused on the following elements for presentation: Timely, complete, accurate, concise, and relevant to stakeholders. Presentation of results supports findings. Appendix II: Comments from the Department of Defense Appendix III: GAO Contact and Staff Acknowledgments GAO Contact Staff Acknowledgments In addition to the contact named above, Beverly Schladt (Assistant Director), John Strong (Analyst in Charge), John Beauchamp, Vincent Buquicchio, Martin De Alteriis, and Lillian Moyano Yob made key contributions to this report.