Background OMB Guidelines In January 2018, OMB released (M-18-04) Monitoring and Evaluation Guidelines for Federal Departments and Agencies that Administer United States Foreign Assistance (the “Guidelines”) in response to the 2016 FATAA legislation. (See appendix III for additional information on the requirements in the legislation). The Guidelines provide direction to federal departments and agencies that administer foreign assistance on monitoring the use of resources, evaluating the outcomes and impacts of the foreign assistance projects and programs, and applying the findings and conclusions of such evaluations to proposed project and program design. The goals of the Guidelines are to set forth key principles to guide each agency and to specify requirements, where appropriate, that agencies must cover in their own policies on M&E of foreign assistance. The Guidelines define monitoring and evaluation as follows: Monitoring is the ongoing and systematic tracking of data and information relevant to policies, strategies, programs, projects, and/or activities and is used to determine whether desired results are occurring as expected during program, project, or activity implementation. Monitoring often relies on indicators, quantifiable measures of a characteristic or condition of people, institutions, systems, or processes that may change over time. Evaluation is the systematic collection and analysis of information about the characteristics and outcomes of the program, including projects conducted under such program, as a basis for making judgments and evaluations regarding the program; improving program effectiveness; and informing decisions about current and future programming. Table 1 lists OMB’s M&E requirements and key excerpts of the descriptions as noted in the OMB M-18-04. GAO Leading Practices In 2016, we reported on leading practices for foreign assistance program M&E. We identified 28 leading practices—14 for monitoring and 14 for evaluation. Table 2 lists and defines these monitoring practices. Table 3 lists the evaluation practice and corresponding definition. OMB’s Foreign Assistance Monitoring and Evaluation Guidelines Incorporate Most but Not All of GAO’s Leading Practices Based on our review, the Guidelines incorporate most of GAO’s leading practices for monitoring and evaluation. However, they do not incorporate practices on developing monitoring plans that are based on risks, ensuring that staff are appropriately qualified to conduct monitoring, establish procedures to close out programs, developing staff skills for evaluation, and following up on evaluation recommendations. OMB indicated that it intended the Guidelines to focus on elements required by the FATAA legislation. Nevertheless, incorporating these leading practices in the Guidelines can help ensure that all agencies address impediments, effectively manage foreign assistance, and meet their assistance goals. The Guidelines Incorporate Most of the GAO’s Leading Practices for Monitoring, but Do Not Include Risk Assessments, Staff Qualifications, or Close-Out Procedures Based on our review, OMB incorporates 11 of 14 GAO’s leading practices. Figure 1 shows our assessment of the Guidelines with regard to monitoring foreign assistance. The OMB Guidelines do not incorporate practices on developing monitoring plans that are based on risks, ensuring that staff are appropriately qualified to conduct monitoring, and establishing close-out procedures for projects and programs. Developing monitoring plans based on an assessment of risk. The Guidelines do not incorporate GAO’s leading practice of developing monitoring plans based on as assessment of risks related to achieving the defined objectives. Identifying and assessing risks can help agencies determine if impediments exist that they might need to mitigate in order to manage their foreign assistance more effectively. Additionally, determining which activities warrant greater oversight and which require less can also help agencies ensure the appropriate allocation of foreign assistance. Ensuring Staff qualifications for monitoring. The Guidelines do not incorporate GAO’s leading practice for agencies to ensure that staff members responsible for monitoring programs or projects have the relevant knowledge, skills, and training. By having qualified staff for monitoring programs or projects, agencies can help ensure they meet their foreign assistance goals. By hiring qualified staff and providing them the right training, tools, structure, incentives, and responsibilities, agencies can make operational success possible. Establishing close out procedures for projects and programs. The Guidelines do not incorporate GAO’s leading practice for agencies to establish program closeout procedures for all required work and administrative actions completed by the implementing partner. By establishing such procedures, agencies can help ensure their foreign assistance is less susceptible to fraud, waste, and mismanagement; addresses increases to potential costs in fees for maintaining foreign assistance; and increases their ability to redirect foreign assistance to other projects. The Guidelines Incorporate Most of the GAO’s Leading Practices for Evaluation, but Do Not Include Developing Staff Skills and Following Up on Recommendations Based on our review, OMB incorporates 12 of 14 GAO’s leading practices. Figure 2 shows our assessment of the Guidelines with regard to evaluating foreign assistance. The OMB Guidelines do not incorporate some practices such as developing staff skills for evaluation and following up on evaluation recommendations. Developing staff skills regarding evaluating. The Guidelines do not incorporate GAO’s leading practice for agencies to establish requirements that the staff responsible for overseeing and using evaluations should continually undertake the relevant education, training, or supervised practice needed to learn new concepts, techniques, and skills. By having their staff continually undertake such education, training, or supervised practice, agencies can benefit more fully from program evaluations. Following up on recommendations. The Guidelines do not incorporate GAO’s leading practice for agencies to determine whether management or programs have accepted the recommendations made in evaluation reports and taken the actions needed to address them. By developing mechanisms to track recommendations, agencies can better address inefficient, mismanaged, or costly programs or projects. OMB Notes the Guidelines for Monitoring and Evaluation Include Elements Required in the FATAA Legislation The FATAA requires the President to set forth guidelines “according to best practices of monitoring and evaluation” but does not define these best practices. Specifically, FATAA states, “the President shall set forth guidelines, according to best practices of monitoring and evaluation studies and analyses, for the establishment of measurable goals, performance metrics, and monitoring and evaluation plans that agencies can apply with reasonable consistency to covered United States foreign assistance.” OMB staff told us that the Guidelines were intended to focus on elements required by the FATAA legislation but noted that agencies are free to add additional requirements to their own M&E policies. However, we have previously reported that while some of these agencies have incorporated these leading practices, others have not. Furthermore, agencies that have incorporated these practices would not necessarily continue to include them if they are not required in the Guidelines. Regarding leading practices, officials noted that while these practices are important, there is no singular established standard for best monitoring practices. Nevertheless, both OMB’s circulars and recent legislation note the importance of leading practices for M&E. For example, Circular A-123 notes that management should identify internal and external risks that may prevent the organization from meeting its objectives. Additionally, the Foundations for Evidence-Based Policymaking Act of 2018 requires OPM, in consultation with the OMB, to identify skills and competencies needed for program evaluation, establish a new occupational series or update an existing one for program evaluation, and establish a new career path for program evaluation. Most Agencies Have Incorporated OMB’s Guideline Requirements in Their Policies, and All Have Taken Initial Steps to Implement Them Based on our review, most agencies incorporated all of OMB’s Guidelines for monitoring in their policies. However, DOD did not include the requirement to establish roles and responsibilities among agencies that participate in funding transfers or ensure that verifiable, reliable, and timely information is collected and available to monitoring personnel. We also found that agencies incorporate most of OMB’s Guideline requirements for evaluation in their policies, but some did not include the requirement to conduct impact evaluation on all pilot programs. Without incorporating these Guideline requirements, agencies risk losing accountability over their funding and monitoring and evaluating activities. They also risk replicating programs without fully understanding their effectiveness. We also found that all of the agencies we reviewed have taken initial steps to implement their M&E policies. Most Agencies Have Incorporated OMB’s Guideline Requirements for Monitoring Based on our review of agency monitoring policies, all the agencies except DOD incorporated relevant Guideline requirements. All six agencies we reviewed incorporated the requirement to establish monitoring policies that apply to their major foreign assistance programs. For example, State, USAID, and MCC have agency-wide policies for foreign assistance M&E. USDA and HHS have policies relevant to their major foreign assistance programs—for USDA, the Foreign Agriculture Service’s food aid programs, and for HHS, the President’s Emergency Plan for AIDS Relief (PEPFAR). All of the agencies with relevant monitoring policies—DOD, HHS, MCC, State, USAID, and USDA— incorporate the requirement to develop, collect, analyze, and report data on performance indicators. These policies help ensure the measurement of project implementation and progress, and promote the timely analysis and reporting of results that could identify any needed corrections. DOD did not incorporate Guideline requirements to establish agencies’ roles and responsibilities and ensure verifiable data for monitoring activities. Establishing agencies’ roles and responsibilities when funds are transferred. DOD did not include the Guideline requirement for agencies to establish roles and responsibilities in funding transfers. Without defined roles and responsibilities, agencies risk losing accountability over funding and monitoring activities. In addition, agencies could miss opportunities to collaborate and leverage interagency efforts to facilitate decision-making and address barriers across agency boundaries. Ensuring verifiable, reliable, and timely data. DOD did not include the Guideline requirement for agencies to ensure they collect and provide verifiable, reliable, and timely data to monitoring personnel. Without ensuring that such data are available to monitoring personnel, agencies risk employing inappropriate methods, continuing ineffective programs or projects, and making uninformed decisions. DOD officials told us these practices are currently not required because they are still in the process of fully aligning their policy with the Guidelines. Officials explained that working on prioritizing and directing resources towards M&E efforts has been a challenge. Officials noted they expect to update the policy to include these requirements in the future, but they have no specific timelines in place. Agencies Incorporate Most but Not All of OMB’s Guideline Requirements on Evaluation The agencies we reviewed incorporated nearly all relevant Guideline requirements on evaluation. Three of the six agencies—DOD, HHS and USDA—did not include a requirement to conduct impact evaluations on all pilot programs or projects. Figure 4 shows our assessment of agencies’ evaluation policies against the Guidelines. All the agencies we reviewed have established project-specific evaluation plans. For example, HHS implements PEPFAR’s evaluation plan which indicates specific requirements for describing the evaluation component, strategy, or intervention, the reason for the evaluation, the type of evaluation, the key evaluation questions, the data sources, the methods by question, and the dissemination and utilization plan. All the agencies we reviewed also had policies on distributing their evaluation reports internally and publicly reporting them. For example, State and USAID have a web-based, customized Evaluation Registry system that they jointly maintain for bureaus and independent offices to record and track planned, ongoing, and completed evaluations. Conduct impact evaluations for pilot programs or projects. DOD, HHS, and USDA did not include the Guideline requirement for agencies to conduct impact evaluations for pilot programs or to conduct only a performance evaluation and to provide a justification for not conducting an impact evaluation. Without a requirement to conduct impact evaluations of pilot programs, agencies risk duplicating or scaling up programs without fully understanding the factors that could lead to their success or failure. DOD. DOD officials told us they do not require this practice because they are still in the process of fully aligning their policy with the Guidelines. According to DOD, it has determined that impact evaluations are impractical and inappropriate for the planned evaluations; instead, it plans to conduct only performance evaluations and provide justifications for not conducting impact evaluations, as required by the OMB Guidelines. DOD plans to address the evaluation methodology of pilot programs in future updates, according to officials. However, DOD has no specific timelines in place for these updates. HHS. PEPFAR’s M&E documents indicate that PEPFAR teams are encouraged but not required to evaluate all current pilot programs to see which should be taken to scale for specific populations. Officials from HHS and the Office of the U.S. Global Aids Coordinator noted that they conduct their own evaluation of pilot programs and use routine program data to inform scaling of programs. However, PEPFAR policies do not specifically require that such evaluations be like the impact evaluations described in the Guidelines. USDA. FAS’s M&E documents indicate that when selecting projects to undergo impact evaluation the agency will consider pilot projects. USDA officials told us they have no requirement to conduct impact evaluations on all pilot projects because impact evaluations may be cost prohibitive and project lifecycles are short (i.e., 3 to 5 years). Officials further noted that implementing partners can conduct an impact evaluation on pilot programs, but are not required to do so. Although the Guideline requirement indicates that agencies can forgo impact evaluations, they must provide a justification in their M&E policy. USDA officials have not provided such a justification provided in their M&E policy. Establish agencies’ roles and responsibilities for evaluation activities when funds are transferred. DOD did not include the Guideline requirement for agencies to define roles and responsibilities when there are funding transfers between or among U.S. government agencies to ensure accountability for evaluation activities. Without defined roles and responsibilities, agencies risk losing accountability over funding and evaluation activities. In addition, they could miss opportunities to collaborate and leverage interagency efforts to facilitate decision-making and address barriers across agency boundaries. Evaluate all programs at least once whose dollar value equals or exceeds that of a median sized program within the agency. DOD did not include the Guideline requirement for agencies to evaluate all programs, at least once during their existence, whose dollar value equals or exceeds that of a median sized program in the agency. Without a mechanism to evaluate all these types of programs, agencies risk continuing inefficient, mismanaged, or costly projects. DOD officials told us they do not currently require these practices because they are still in the process of fully aligning their policy with the Guidelines. They noted that they expect to update the policy to include these requirements, but they have no specific timelines in place. Agencies Have Taken Initial Steps to Implement Their M&E Policies Since the six agencies we reviewed recently updated their M&E policies to align with the OMB Guidelines, many existing assistance projects and programs may not be governed by these requirements. Nonetheless, the agencies we reviewed have taken initial steps to help ensure implementation of agency M&E policies. In interviews, agencies provided us with the following examples of such steps. State. State developed a guidance document and tool-kit to operationalize and oversee its M&E policy to ensure it implements the Guidelines. According to State officials, they provide classroom training on the M&E policy and are piloting a revised online and classroom evaluation courses for staff. Officials also noted that they have dedicated staff to assist bureaus in implementing the Guidelines, among other agency policies. USAID. USAID has an approval process to ensure key deliverables include Activity plans that meet Guideline requirements. Additionally, USAID’s policy requirements indicate that each mission program office must identify a point of contact for monitoring and evaluation to ensure that USAID and its partners are complying with the agencies policies and foreign assistance M&E guidelines. MCC. MCC also has an approval process through their Department of Policy and Evaluation to ensure implementation of the Guidelines. As part of the process, the MCC Board of Directors or the appropriate partner country must approve initial M&E plans. HHS. Within HHS, the Centers for Disease Control and Prevention (CDC) are responsible for implementing the monitoring and evaluation guidance for their PEPFAR programs. CDC officials told us that they have existing mechanisms and supervisory structures in place to ensure that the Guidelines’ requirements are met in PEPFAR programs. USDA. USDA officials told us that the current M&E policy applies only to food assistance programs within FAS and not for other USDA programs. Officials explained they are trying to develop a structure that allows FAS to ensure all USDA components are implementing the OMB Guidelines. DOD. DOD developed guidance for fiscal year 2020 on implementing its M&E policy. DOD officials we spoke to noted they are working on identifying resources, skills, and capabilities to fully implement DOD’s M&E policy. Conclusions OMB’s Guidelines set forth key principles to guide agencies and to specify requirements, where appropriate, which they must cover in their own policies on M&E of foreign assistance. However, they do not include key leading practices for M&E that GAO identified for ensuring agencies meet their foreign assistance goals and objectives. While OMB allows agencies discretion to include these or other best practices, it is unknown if the agencies will do so. By ensuring that OMB’s government-wide Guidelines include these best practices, agencies can help address impediments, effectively manage foreign assistance, and meet their goals. Although all agencies we reviewed developed or updated their M&E policies to align with the Guidelines, not all of them include important requirements. DOD, HHS, and USDA did not include the requirement for agencies to conduct impact evaluations for pilot programs or to conduct performance evaluations and provide a justification for not doing an impact evaluation. Without a requirement to conduct impact evaluations of pilot programs, agencies risk duplicating or scaling up programs without fully understanding the causes that could lead to their success or failure. Recommendations for Executive Action We are making the following seven recommendations, including one to OMB, four to DOD, one to State, and one to USDA. The Director of the Office of Management of Budget should update the Guidelines to include GAO’s leading practices of developing monitoring plans that are based on risks, ensuring that monitoring staff have appropriate qualifications, establishing procedures to close-out programs, developing staff skills regarding evaluations, and establishing mechanisms for following up on evaluation recommendations. (Recommendation 1) The Secretary of Defense should update the Department’s monitoring and evaluation policies to define roles and responsibilities among agencies that participate in interagency funding transfers. (Recommendation 2) The Secretary of Defense should update the Department’s monitoring and evaluation policies to ensure verifiable, reliable, and timely data are available to monitoring personnel. (Recommendation 3) The Secretary of Defense should update the Department’s monitoring and evaluation policies to ensure that it evaluates all programs, at least once in their lifetimes, whose dollar value equals or exceeds that of the median program in the agency. (Recommendation 4) The Secretary of Defense should update the Department’s monitoring and evaluation policies to require the agency to conduct impact evaluations on all pilot programs before replicating or expanding, or conduct performance evaluations for those programs and provide a justification for not conducting an impact evaluation. (Recommendation 5) The Department of State’s U.S. Global AIDS Coordinator, in collaboration with HHS and other implementing agencies, should update the PEPFAR monitoring and evaluation policies to require these agencies to conduct impact evaluations on all pilot programs before replicating or expanding, or conduct performance evaluations for those programs and provide a justification for not conducting an impact evaluation. (Recommendation 6) The Secretary of Agriculture, in collaboration with the Foreign Agriculture Service, should update their monitoring and evaluation policies to require USDA to conduct impact evaluations on all pilot programs before replicating or expanding, or conduct performance evaluations for those programs and provide a justification for not conducting an impact evaluation. (Recommendation 7) Agency Comments and Our Evaluation We provided a draft of this product to the DOD, HHS, MCC, OMB, State, USDA, and USAID for comment. OMB commented on the draft report in an email from the staff responsible for economic policy, federal financial management, and international affairs. In the email, OMB disagreed with the recommendation to revise the Guidelines. It emphasized that an interagency group had developed the Guidelines and had consulted a number of expert sources on monitoring and evaluation policies and practices, including GAO’s leading practices. OMB also developed the guidelines to achieve the objectives contained in the Foreign Aid Transparency and Accountability Act of 2016 within the context of other existing OMB guidance. OMB suggested that it would be more effective to remind agencies that, in addition to the Guidelines specified in M-18-04, they should follow all guidance OMB had issued affecting monitoring and evaluation activities. This guidance includes policies for closeout procedures in the Uniform Guidance, for the Enterprise Risk Management and Internal Control in A-123, and for the Foundations for Evidence- Based Policymaking Act on using evaluation information and monitoring and evaluation staff skills and qualifications. We acknowledge that relevant monitoring and evaluation guidance is available to agencies in other forms beyond the Guidelines. However, we believe it is important for OMB to incorporate this guidance into its Guidelines, if only by reference, in order to emphasize the importance of these practices in the context of monitoring and evaluation of foreign assistance. This step would help ensure that OMB had integrated this guidance into the management of foreign assistance programs as appropriate. DOD concurred with our recommendations and indicated that it would address many of them in the next iteration of its M&E policy for security assistance (see appendix IV for written comments). DOD noted that two of our recommendations had limited applicability to DOD for security assistance, but described how it would implement them. First, DOD stated that it has not used its authority to transfer funds for security cooperation assistance to other departments and agencies. However, DOD indicated it would implement our recommendation to define roles and responsibilities among agencies that participate in interagency funding transfers, should such transfers become necessary. Second, DOD stated that conducting impact evaluations was not a feasible in the context of security assistance. Instead, DOD plans to conduct only performance evaluations, but it would provide justifications for not conducting impact evaluations, as required by the Guidelines. By documenting these approaches in its M&E policies, DOD would help ensure that those departments conducting M&E for DOD security assistance initiatives implement them as required. State agreed with the intent of the recommendation (see appendix V for written comments). State explained that impact evaluations are often not feasible in the context of assistance provided under PEPFAR and described its alternative approach to evaluating new initiatives. State indicated it would update appropriate PEPFAR policies to clarify when agencies should conduct impact and/or performance evaluations. These clarifications will reflect how State evaluates PEPFAR programs in practice in accordance with OMB guidance and legislation, according to State. USAID provided written comments (see appendix VI). HHS and USDA provided technical comments, which we incorporated as appropriate. MCC did not provide comments. We are sending copies of this report to the appropriate congressional committees, the Director of the Office of Management and Budget; Secretaries of Agriculture, Defense, Health and Human Services, and State; Administrator of the U.S. Agency for International Development; and the Executive Officer of the Millennium Challenge Corporation and other interested parties. In addition, this report will be available at no charge on the GAO website at http://www.gao.gov. If you or your staff have any questions about this report, please contact me at (202) 512-3149 or GootnickD@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made major contributions to this report are listed in appendix VI. Appendix I: Objectives, Scope, and Methodology This report examines the extent to which (1) the Office of Management and Budget’s (OMB) monitoring and evaluation (M&E) Guidelines incorporate GAO leading practices and (2) agencies incorporate the OMB Guidelines in their M&E policies and plans. To address objective one, we examined the OMB Guidelines against GAO’s 28 leading practices—14 for monitoring and 14 for evaluation— identified in GAO-16-861R. In 2016, GAO developed the 28 leading practices. In 2019, we provide specific definitions for each of the practices noted. We made slight modifications to the language to align with the definitions provided. For monitoring, we developed this list of leading practices based on our review of the GPRA Modernization Act of 2010; OMB’s Uniform Administrative Requirements, Cost Principles, and Audit Requirements for Federal Awards; GAO’s Standards for Internal Control in the Federal Government (Greenbook); and others. The list of leading practices for monitoring includes developing monitoring plans; collecting, reviewing, and analyzing monitoring data; and establishing roles and responsibilities of personnel responsible for monitoring. For evaluation, we developed a list of leading practices based on the American Evaluation Association’s (AEA) 2016 An Evaluation Roadmap for a More Effective Government (AEA Roadmap) and Preface to Evaluators’ Ethical Guiding Principles. The list of leading practices for evaluation include developing evaluation plans; ensuring evaluator independence; developing staff skills regarding evaluation and use of evidence; and establishing roles and responsibilities of personnel responsible for evaluation. To perform these analyses, two analysts assessed if the Guidelines incorporated specific GAO leading practices. The analysts worked iteratively, comparing notes and reconciling differences at each stage of the analysis. In addition, GAO staff independent of the two analysts reviewed the final analysis, and made modifications as appropriate. We also interviewed relevant OMB officials in Washington D.C. involved in developing the memorandum and inquired about specific requirements and plans to ensure the implementation of these Guidelines. To address our second objective, we examined U.S. agency M&E policies against the requirements noted in the OMB Guidelines. We identified the six major agencies administering the most foreign assistance funds. The six agencies are the U.S. Agency for International Development (USAID), the Department of State (State), the Millennium Challenge Corporation (MCC), the Department of Health and Human Services (HHS), the U.S. Department of Agriculture (USDA) and the Department of Defense (DOD). We asked these agencies to identify or provide all relevant policies and guidance relating to foreign assistance M&E, including, where appropriate, standard operating procedures or other guidance. For USDA, we reviewed the Foreign Agricultural Service’s food assistance; for HHS, the President’s Emergency Program for AIDS Relief; and for DOD, security cooperation programs. To perform these analyses, two analysts assessed agency M&E policy documents against the requirements in the OMB Guidelines. We identified requirements as phrases that included the following language “required,” “must,” “mandatory,” or “should.” The analysts worked iteratively, comparing notes and reconciling differences at each stage of the analysis. In addition, other GAO staff independent of the two analysts reviewed the final analysis, and made modifications as appropriate. We also interviewed relevant OMB staff and agency officials in Washington D.C. involved in developing and implementing the M&E policies and inquired about specific requirements, and plans to ensure their M&E policies are implemented. We conducted this performance audit from July 2018 to July 2019 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives. Appendix II: The Office of Management and Budget Monitoring and Evaluation Guidelines (OMB Memorandum M-18-04) Appendix II: The Office of Management and Budget Monitoring and Evaluation Guidelines (OMB Memorandum M-18-04) In January 2018, the Office of Management and Budget (OMB) released (M-18-04) Monitoring and Evaluation Guidelines for Federal Departments and Agencies that Administer United States Foreign Assistance (the “Guidelines”) in response to the Foreign Aid Transparency and Accountability Act of 2016 (FATAA). Table 4 shows the complete description of the requirements noted in the Guidelines. Appendix III: Assessment of the Foreign Aid Transparency and Accountability Act of 2016 and the Office of Management and Budget Monitoring and Evaluation Guidelines The Foreign Aid Transparency and Accountability Act of 2016 (FATAA) has required objectives on monitoring and evaluation for the Office of Management and Budget (OMB) to include in the Guidelines. We compared the 13 required objectives for the Guidelines set forth in the FATAA legislation with those in the OMB Guidelines. We found that all of the monitoring and evaluation requirements set forth in the legislation are included in the OMB Guidelines. Table 5 shows the FATAA legislation requirements, OMB Guidelines, and our assessment of the alignment between the legislation and OMB’s Guidelines. Appendix IV: Comments from the Department of Defense Appendix V: Comments from the Department of State Appendix VI: Comments from the United States Agency of International Development Appendix VII: GAO Contact and Staff Acknowledgments GAO Contact Staff Acknowledgments In addition to the individual named above, James B. Michels (Assistant Director), Farahnaaz Khakoo-Mausel (Analyst-in-Charge), Paulina Maqueda-Escamilla, Mark Dowling, Martin De Alteriis, Benjamin Licht, John Hussey, Neil Doherty, Aldo Salerno, Carolina Morgan and Michael Simon made key contributions to this report. Related GAO Products Government Auditing Standards 2018 Revision (Supersedes GAO-12-331G. GAO-18-568G. Washington, D.C.: July 17, 2018. Foreign Assistance: Agencies Can Improve the Quality and Dissemination of Program Evaluations. GAO-17-316. Washington, D.C.: March 3, 2017. Foreign Assistance: Selected Agencies’ Monitoring and Evaluation Policies Generally Address Leading Practices. GAO-16-861R. Washington, D.C.: September 27, 2016. Program Evaluation: Some Agencies Reported that Networking, Hiring, and Involving Program Staff Help Build Capacity. GAO-15-25. Washington, D.C.: November 13, 2014. Government Efficiency and Effectiveness: Inconsistent Definitions and Information Limit the Usefulness of Federal Program Inventories. GAO-15-83. Washington, D.C.: October 31, 2014. Standards for Internal Control in the Federal Government. GAO-14-704G. Washington, D.C.: September 10, 2014. State Department: Implementation of Grants Policies Needs Better Oversight. GAO-14-635. Washington, D.C.: July 21, 2014. Program Evaluation: Strategies to Facilitate Agencies’ Use of Evaluation in Program Management and Policy Making. GAO-13-570. Washington, D.C.: June 26, 2013. President’s Emergency Plan for AIDS Relief: Agencies Can Enhance Evaluation Quality, Planning, and Dissemination. GAO-12-673. Washington, D.C.: May 31, 2012. Grants Management: Action Needed to Improve the Timeliness of Grant Closeouts by Federal Agencies. GAO-12-360. Washington, D.C.: April 16, 2012. Designing Evaluations: 2012 Revision. GAO-12-208G. Washington, D.C.: January 31, 2012. International School Feeding: USDA’s Oversight of the McGovern-Dole Food for Education Program Needs Improvement. GAO-11-544. Washington, D.C.: May 19, 2011. Program Evaluation: Experienced Agencies Follow a Similar Model for Prioritizing Research. GAO-11-176. Washington, D.C.: January 14, 2011. Managing for Results: Enhancing Agency Use of Performance Information for Management Decision Making. GAO-05-927. Washington, D.C.: September 9, 2005. Program Evaluation: An Evaluation Culture and Collaborative Partnerships Help Build Agency Capacity. GAO-03-454. Washington, D.C.: May 2, 2003. Managing For Results: Federal Managers’ Views Show Need for Ensuring Top Leadership Skills. GAO-01-127. Washington, D.C.: October 20, 2000. Performance Plans: Selected Approaches for Verification and Validation of Agency Performance Information. GAO/GGD-99-139. Washington, D.C.: July 30, 1999. Agency Performance Plans: Examples of Practices That Can Improve Usefulness to Decisionmakers. GAO/GGD/AIMD-99-69. Washington, D.C.: February 26, 1999. Executive Guide: Effectively Implementing the Government Performance and Results Act. GAO/GGD-96-118. Washington, D.C.: June 1, 1996.


Background OMB’s ERM Requirements and Guidance OMB provides guidance to federal managers on how to improve accountability and effectiveness of federal programs and operations by identifying and managing risks. OMB updated its Circular No. A-123 in July 2016 to establish management’s responsibilities for ERM. As part of the overall governance process, ERM calls for the consideration of a risk across the entire organization and how it may interact with other identified risks. When used appropriately, ERM is a decision-making tool that allows agency leadership to view risks across an organization and helps management understand an organization’s portfolio of top risk exposures, which could affect achievement of the agency’s goals and objectives. In December 2016, we issued a report that provided an overall framework for agencies to build an effective ERM program. In July 2016, OMB also updated Circular No. A-11, Preparation, Submission, and Execution of the Budget. In Circular No. A-11, OMB referred agencies to Circular No. A-123 for requirements related to ERM implementation, including for developing a risk profile as a component of the agency’s annual strategic review. A risk profile is a prioritized inventory of the most significant risks identified and assessed through the risk assessment process. It considers risks from a portfolio perspective, identifies sources of uncertainty that are both positive (opportunities) and negative (threats), and facilitates the review and regular monitoring of risks. Together, these two OMB circulars constitute the ERM policy framework for executive agencies by integrating and operationalizing specific ERM activities and helping to modernize existing risk management efforts. Internal Control Requirements and Guidance Standards for Internal Control in the Federal Government describes internal control as a process put in place by an entity’s oversight body, management, and other personnel that provides reasonable assurance that objectives related to performing operations effectively and efficiently, producing reliable internal and external reports, and complying with applicable laws and regulations will be achieved. Internal control serves as the first line of defense in safeguarding assets. Its importance to federal agencies is further reflected in permanent requirements enacted into law. The internal control processes required by FMFIA and the Standards for Internal Control in the Federal Government help to form an integrated governance structure designed to improve mission delivery, reduce costs, and focus corrective actions toward key risks. OMB Circular No. A-123 precludes agencies from concluding that their internal control is effective if there are one or more material weaknesses identified from its assessment. Air Force’s Annual Statement of Assurance and Financial Audit As a component of DOD, the Air Force is required to (1) identify and manage risks, (2) establish and operate an effective system of internal control, (3) assess and correct control deficiencies, and (4) report on the effectiveness of internal control through an annual Statement of Assurance. In addition, the Chief Financial Officers Act of 1990 (CFO Act), as amended by the Government Management Reform Act of 1994 and implemented by guidance in OMB Bulletin No. 19-03, Audit Requirements for Federal Financial Statements (August 27, 2019), requires the Air Force to annually undergo a financial statement audit. However, since 1990, the Air Force has continued to be unable to demonstrate basic internal control that would allow it to pass a financial statement audit, which has contributed to DOD’s financial management remaining on the GAO High-Risk List since 1995. For fiscal year 2018, the Air Force reported 11 material weaknesses in internal control over operations and 14 material weaknesses in internal control over reporting in its Statement of Assurance. For fiscal year 2019, it reported the same number of operations-related material weaknesses, and its reporting-related material weaknesses increased to 25. During the Air Force’s fiscal years 2018 and 2019 financial statement audits, independent auditors specifically considered the Air Force’s internal control over financial reporting in order to determine appropriate audit procedures to perform in order to express an opinion on the financial statements. The independent auditors disclaimed an opinion on the Air Force’s fiscal years 2018 and 2019 financial statements, stating that the Air Force continued to have unresolved accounting issues, and for each year, the auditors reported 23 material weaknesses in internal control over financial reporting. These material weaknesses included control deficiencies in processes related to the Air Force’s mission-critical assets and involved a lack of policies and procedures, inadequate financial information systems and reporting, and inaccurate and incomplete information in its accountability records and financial reports. Air Force Has Not Fully Integrated ERM into Its Management Practices The Air Force’s efforts to implement ERM are in the early stages, and accordingly, it has not fully incorporated ERM into its management practices. Since the July 2016 update to OMB Circular No. A-123 required agencies to implement ERM, the Air Force has been leveraging and relying on its existing risk management practices. To date, these practices have focused on the organizational unit level and not at the entity level, as required by OMB Circular No. A-123. The Air Force plans to integrate ERM increasingly into its management practices over the next several years, with expectations of a fully developed ERM approach after fiscal year 2023. The Air Force has taken the initial steps to establish an ERM governance structure, define risk classifications, and develop its ERM framework. For instance, the Air Force has drafted charters updating responsibilities for two senior management advisory councils—(1) the Enterprise Productivity Improvement Council (EPIC) and (2) the Executive Steering Committee (ESC)—to implement OMB Circular No. A-123. EPIC will oversee the agency’s risk management function, with a specific emphasis on overseeing the regular assessment of risk and approving risk responses and the Air Force’s risk profile. ESC will lead the implementation, assessment, and documentation of risk management over financial reporting, financial systems, all associated activities, and oversight with respect to the Air Force’s internal control program. EPIC is designed to focus exclusively on potential operational material weaknesses, and ESC will focus on potential financial reporting and financial systems material weaknesses. Air Force officials informed us that both councils would share responsibility for compliance objectives and resulting material weaknesses. During our audit, we analyzed the Air Force’s financial reports beginning with those for fiscal year 1999 and noted that the agency and the external auditors have generally reported material weaknesses each year involving the tracking, reporting, location, accountability, and cost of certain mission-critical assets. These weaknesses identified risks that decreased the Air Force’s ability to perform operations efficiently, prepare reliable financial reports, and comply with applicable laws and regulations. EPIC and ESC currently assess proposed material weaknesses that the primary reporting elements (PRE) submit and determine whether to recommend them to the Secretary of the Air Force for reporting in the annual Statement of Assurance. However, the Air Force’s governance structure does not include a mechanism for EPIC or ESC to oversee the management of risk associated with material weaknesses and consider its effect across the entire agency. Based on our review of the draft charters and documentation from governance meetings, the Air Force included provisions for ESC to identify material weaknesses related to financial reporting and financial systems and EPIC to identify material weaknesses related to operations objectives. However, there were no charter provisions for either council to identify, assess, respond to, and report on the risks associated with those material weaknesses or material weaknesses identified through external audits. A material weakness, reported by either the agency or an external auditor, by definition indicates a significant decrease in an agency’s ability, during the normal course of operations, to achieve objectives and address related risks. Under OMB Circular No. A-123, an agency’s risk management governance structure helps ensure that the agency identifies risks that have the most significant effect on the mission outcomes of the agency. Without a thorough and integrated ERM governance structure that includes oversight responsibilities managing risks associated with material weaknesses in internal control, there is an increased risk that the Air Force will not properly identify, assess, and respond to significant entity-level risks. Air Force Has Not Designed a Comprehensive Approach for Assessing Internal Control, Including Processes Related to Mission-Critical Assets The Air Force’s current internal control assessment process is not designed to facilitate the timely identification and correction of internal control deficiencies or to be used to support the Air Force’s annual Statement of Assurance. Specifically, Air Force management has not designed an adequate process for assessing internal control. Further, the process does not focus on areas with the greatest risk, such as mission- critical assets. In addition, the reviews of mission-critical assets in fiscal years 2018 and 2019 in support of the financial statement audit did not result in adequate assessments of internal control. The Air Force’s policy for assessing the effectiveness of its internal control system and for preparing the agency’s annual Statement of Assurance is based on DOD Instruction 5010.40, Managers’ Internal Control Program Procedures, dated May 2013. The Air Force’s policy is outlined in Air Force Policy Directive 65-2, Managers Internal Control Program. This policy is supported by the procedures outlined in Air Force Instruction (AFI) 65-201, Managers Internal Control Program Procedures, dated February 2016, which the Air Force currently is revising to address the July 2016 OMB Circular No. A-123 update. The Air Force provides additional guidance to supplement AFI 65-201 in its Statement of Assurance Handbook and its Internal Control Playbook. The Air Force’s OMB Circular No. A-123 program comprises 17 designated PREs, including the Secretariat and Air Force staff offices, major commands, the Army and Air Force Exchange Service, and direct- reporting units. The Air Force subdivides each PRE along organizational lines into more than 6,500 organizational assessable units (organizational units), such as a squadron or wing, and other specific programs and functions, where it evaluates internal controls per AFI 65-201. Each of the organizational units has an assessable unit manager (unit manager) who has authority over the unit’s internal control, including continual monitoring, testing, and improvement. Figure 1 illustrates how the Air Force’s organizational structure informs its overall annual Statement of Assurance. The Air Force requires each unit manager to submit an annual supporting statement of assurance providing the manager’s opinion on whether the unit has reasonable assurance that its internal controls are effective. The units submit the statements to the Assistant Secretary of the Air Force, Financial Management and Comptroller (SAF/FM), the office responsible for OMB Circular No. A-123 implementation and compilation of the annual Statement of Assurance. Based on discussions with Air Force officials, SAF/FM uses the unit managers’ supporting statements of assurance to develop the overall Air Force annual Statement of Assurance. Air Force Has Not Designed an Adequate Process for Assessing Internal Control The Air Force’s internal control assessment process does not require (1) an assessment of all required elements of an effective internal control system; (2) test plans that specify the nature, scope, and timing of procedures to conduct; and (3) management validation of results. In addition, existing policies and procedures that staff follow to perform the assessments do not fully implement OMB Circular No. A-123. Further, the Air Force provided inadequate training to those responsible for conducting and concluding on the internal control assessments. Assessment of Internal Control Not Designed to Evaluate All Required Elements Although not required by policy, the Air Force performed its first assessment of the five components of internal control during fiscal year 2019 through an SAF/FM review of entity-level controls, which are controls that have a pervasive effect on an entity’s internal control system and may pertain to multiple components. Based on this assessment, SAF/FM concluded in the Air Force’s Statement of Assurance for fiscal year 2019 that three components of internal control (i.e., risk assessment, control activities, and information and communication) were not designed, implemented, or operating effectively. Although SAF/FM performed this assessment in 2019, the assessment did not include a determination of whether each internal control principle was designed, implemented, and operating effectively. Also, there was no indication that the Air Force designed the assessment of entity-level controls to be pertinent to all Air Force objectives, such as those related to operations, reporting, or compliance. In addition, SAF/FM did not provide the assessment results to the unit managers for input or consideration in their unit-specific control assessments and supporting statements of assurance. The Air Force’s Internal Control Playbook directs unit managers to assess the design and operating effectiveness of the relevant entity-level controls within their purview. However, for fiscal year 2019, SAF/FM performed this assessment, and officials informed us that it was not their intent for unit managers to assess entity-level controls. According to OMB Circular No. A-123, management must summarize its determination of whether each of the five components and 17 principles from Standards for Internal Control in the Federal Government are designed, implemented, and operating effectively and components are operating together in an integrated manner. The determination must be a “yes/no” response. If one or more of the five components are not designed, implemented, and operating effectively, or if they are not operating together in an integrated manner, then an internal control system is ineffective. AFI 65-201 states, as part of its discussion on assessing internal control over financial reporting, that OMB Circular No. A-123 prescribes a process to evaluate controls at the entity level for the five components of internal control (i.e., control environment, risk assessment, control activities, information and communication, and monitoring). The Air Force’s assessment lacked required determinations related to internal control principles because the Air Force lacked policies or procedures for the following: Clearly delineating who within the Air Force (e.g., unit managers or SAF/FM) is responsible for assessing the components and principles of internal control, how often assessments are performed, at what level (e.g., entity or transactional) components and principles are to be evaluated, what objectives are covered in the assessment of entity-level controls, to whom to communicate the results if the results are relevant to others performing assessments of internal control, and what Air Force guidance to follow. Documenting management’s summary, whether performed by the unit managers as outlined in the guidance or by SAF/FM as performed during fiscal year 2019, of its determination of whether each component and principle is designed, implemented, and operating effectively and whether components are operating together in an integrated manner. By not ensuring that management is assessing whether each internal control component and principle is designed, implemented, and operating effectively, the Air Force cannot determine whether internal control is effective at reducing the risk of not achieving its stated mission and objectives to an acceptable level. Moreover, given the entity-wide relevance of SAF/FM’s conclusions, unit managers may not be aware of all the necessary information with which to draw conclusions about the effectiveness of their organizational units’ internal control. Further, management’s assurances on internal control effectiveness, as reported in the Statement of Assurance, may not appropriately represent the effectiveness of the Air Force’s internal control. Assessment of Internal Control Not Designed to Use Consistent Test Plans The Air Force did not have a process in place to base its annual assessment of internal control and Statement of Assurance preparation on uniform testing performed across its agency. Although the Air Force had standard test plans for reviews associated with financial reporting objectives, SAF/FM could not demonstrate what procedures are performed to support its assessment of internal control over its operational, internal reporting, and compliance objectives. Specifically, for these objectives, the Air Force did not develop guidance for those responsible for assessing internal controls on which tests to conduct to obtain the best evidence of whether controls are designed, implemented, and operating effectively; how much testing is needed in each area; when to conduct the tests; how to ensure that current year conclusions are based on current year how assessment procedures are to be adjusted or amended to reflect a consideration of prior year self-identified control deficiencies and internal and external audit results. Additionally, standard test plans for the reviews conducted as part of the Air Force’s financial statement audit remediation efforts did not include guidance on how to consider prior year self-identified control deficiencies and internal and external audit results in determining the nature, timing, and extent of procedures to be conducted for the current year. Further, although the Air Force outlines 20 overall objectives in its 2019 through 2021 Business Operations Plan (dated January 2019), it did not document the specific procedures the Air Force planned and performed to support an evaluation of its internal control over these 20 objectives. According to Standards for Internal Control in the Federal Government, management should establish and operate activities to monitor the internal control system and evaluate the results and should remediate identified internal control deficiencies on a timely basis. For example, as part of its monitoring activities, agency management responsible for the OMB Circular No. A-123 program could design a test plan or establish a baseline to monitor the current state of the internal control system and compare that baseline to the results of its internal control tests. The Air Force’s assessment of internal control and Statement of Assurance are not clearly supported by completed test plans or other documented monitoring activities because SAF/FM does not have a policy or procedures for conducting internal control assessments that require documented test plans that (1) tie back to specific objectives included in the Business Operations Plan; (2) specify the nature, scope, and timing of procedures to conduct under the OMB Circular No. A-123 assessment process; and (3) reflect a consideration of prior year self- identified control deficiencies and results of other internal and external audits. By not ensuring that its more than 6,500 unit managers are evaluating internal control based on the agency’s established baseline, the Air Force cannot ensure that it is consistently and effectively assessing its internal control in order to timely identify and correct deficiencies or that its design of internal control reduces, to an acceptable level, the risk of not achieving agency operational, reporting, and compliance objectives. As a result, Air Force management’s assurances on internal control, as reported in the overall agency Statement of Assurance, may not appropriately represent its internal control effectiveness. Assessment of Internal Control Not Designed to Include Management Validation of Results Air Force management did not have a process to validate whether its unit managers appropriately performed and documented their internal control assessments. During our review, Air Force management was uncertain about how many internal control assessments were being performed or by whom. SAF/FM officials initially stated that there were 5,567 organizational units responsible for assessing internal control, but officials later informed us that the actual number was more than 6,500. Furthermore, Air Force officials were unable to provide information on how many organizational unit managers failed to report on their specific internal control assessments or received waivers from performing such assessments. Finally, management lacked a process to ensure that results used to compile the current year Statement of Assurance are based upon current fiscal year assessments. The Air Force requires unit managers to assess internal control and submit results to SAF/FM through the automated statement of assurance submission system. SAF/FM then compiles the supporting statements of assurance submissions and prepares the Air Force’s annual Statement of Assurance. However, we found that the automated system that collects the annual assessments from more than 6,500 unit managers allows these managers to import internal control testing activities from the prior fiscal year. Air Force officials were unable to provide information about how they ensure that unit managers were not importing prior year results without performing current year testing. OMB Circular No. A-123 requires documentation to demonstrate and support conclusions about the design, implementation, and operating effectiveness of an entity’s internal control system, and requires agencies to consider carefully whether systemic weaknesses exist that adversely affect internal control across organizational or program lines. The Air Force’s process lacks management validation of results because it has not developed a documented policy or procedures to ensure that management can readily review and validate the results of its internal control testing. The Air Force has not required SAF/FM to validate (1) the number of organizational units reporting for its overall internal control assessment; (2) how it tested control procedures, what results it achieved, and how it derived conclusions from those results; and (3) whether it based the results used to compile the current year Statement of Assurance on current fiscal year assessments. Additionally, when PRE management waives assessments, SAF/FM does not have a process to track waivers and assess how they affect the current year assessment of internal control, determination of systemic weaknesses, and compilation of the Air Force’s overall Statement of Assurance. By not validating the internal control assessment results, Air Force management cannot ensure that the assessment was performed as expected to support related conclusions and timely identify internal control deficiencies. Further, management’s assurance on internal control, as reported in the overall Statement of Assurance, may not appropriately represent the internal control effectiveness. Guidance for Assessment of Internal Control Does Not Properly Define Material Weaknesses and Internal Control Air Force guidance for its assessment of internal control neither accurately nor completely reflects definitions included in OMB Circular No. A-123. For example, AFI 65-201 and the Statement of Assurance Handbook provided to unit managers for conducting internal control assessments, and the Internal Control Playbook that the Air Force developed in August 2019 to address internal control over reporting objectives, do not include the complete definitions of the four material weakness categories for deficiencies related to (1) operations, (2) reporting, (3) external financial reporting, and (4) compliance objectives, consistent with guidance in OMB Circular No. A-123. Additionally, the handbook does not define internal control as a process that provides reasonable assurance that objectives will be achieved or an internal control system as a continuous built-in component of operations, affected by people, that provides reasonable assurance that an entity’s objectives will be achieved. Although the playbook does adequately define internal control and a system of internal control, the Air Force developed this guidance after we initiated our review, and the guidance only addresses internal control over reporting objectives and not operational and compliance objectives. These inaccuracies and incomplete descriptions occurred because the Air Force did not provide its internal control assessment guidance preparers or reviewers with training to assist them in writing and reviewing the guidance to ensure proper application of the fundamental concepts of internal control and OMB Circular No. A-123, such as those related to definitions of internal control and material weakness. By not ensuring that Air Force guidance reflects accurate and complete definitions included in OMB Circular No. A-123, the Air Force is at increased risk that its officials performing internal control assessments will not properly conclude on the results; therefore, management’s assurances on internal control, as reported in the Statement of Assurance, may not appropriately represent the effectiveness of internal control. Air Force Lacks Adequate Training for Employees on How to Perform Assessments of Internal Control Among other things, OMB Circular No. A-123 requires staff to identify objectives, assess related risks, document internal controls, evaluate the design of controls, conduct appropriate tests of the operating effectiveness of controls, report on the results of these tests, and appropriately document the assessment procedures. However, the Air Force’s training provided to unit managers responsible for assessing internal control lacks sufficient instructions on how to perform such assessments. Specifically, the current annual training provided by SAF/FM lacks instruction on how to prepare documentation to adequately support conclusions, identify and test the key internal controls, and evaluate and document test results; limits discussion of OMB Circular No. A-123 internal control assessments to internal control over external financial reporting objectives and does not cover internal control over operational, compliance, and internal reporting objectives; lacks adequate definitions of material weaknesses included in OMB Circular No. A-123; lacks instruction on how to interpret, respond to, and correct self- identified deficiencies (control deficiencies, significant deficiencies, and material weaknesses); and is not required for individuals performing reviews related to external financial reporting. SAF/FM officials informed us that the definitions of material weakness and instructions on how to interpret, respond to, and correct deficiencies were included in other guidance documents, such as the newly created Internal Control Playbook. However, the Air Force did not provide the playbook to PREs during the fiscal year 2019 training, and it is not officially named as guidance in the Air Force’s policy for assessments of internal control. Although the Air Force has described the playbook as supplemental guidance, it does not refer to the playbook as such in its policy for assessing the effectiveness of its system of internal control to provide reasonable assurance that operational, reporting, and compliance objectives are achieved. These inadequacies occurred because SAF/FM has not fully evaluated and incorporated the requirements for assessing an internal control system into its training and has not designed training that (1) enhances skills in evaluating an internal control system and documenting the results; (2) reflects all OMB Circular No. A-123 requirements, such as those related to assessing controls for all objectives and determining material weaknesses; and (3) is provided to all who are responsible for performing internal control assessments. According to federal internal control standards, management should demonstrate a commitment to developing competent individuals. For example, management could provide training for employees to develop skills and competencies needed for key roles and responsibilities in assessing internal control. Without appropriate training, those responsible for assessing internal control may not do so adequately enough to identify internal control deficiencies timely and support the agency’s internal control assessments with appropriate documentation and summarization of the results. Air Force Has Not Designed a Process for Assessing Internal Control Based on Risk OMB Circular No. A-123 requires an agency to evaluate whether a system of internal control reduces the risk of not achieving the entity’s objectives using a risk-based assessment approach. However, the Air Force’s current AFI 65-201 approach calls for assessing internal control at more than 6,500 organizational units without regard to quantitative or qualitative risks. As previously discussed, the Air Force lacks procedures to verify whether its unit managers are performing internal control assessments as intended and does not provide guidance for uniform testing across the organization. Therefore, the Air Force’s current approach for assessing internal control does not ensure that areas of greatest risk are addressed, such as mission-critical assets, and instead may unnecessarily focus on areas of lower risk. As a result, the Air Force may not be using resources efficiently. The Air Force’s current design of assessing internal control does not ensure, at a minimum, the evaluation of internal control over areas key to meeting its mission. Specifically, the Air Force does not have a policy requiring evaluation of whether its internal control over processes related to areas of highest risk—such as processes related to mission-critical assets, including equipment, government-furnished equipment, and weapons-system spare parts managed and held by contractors and working capital fund inventory—reduces the risk of not achieving specific operation, reporting, or compliance objectives to an acceptable level. The Acting Secretary of Defense, during fiscal year 2019, emphasized two of these areas—government property in the possession of contractors, which includes government-furnished equipment, and working capital fund inventory—as high priority for corrective actions related to financial statement audit remediation. The Air Force’s current approach for assessing internal control calls for more than 6,500 organizational units to perform assessments without regard to risk because the Air Force has not developed a policy or procedures providing guidance on how to perform the assessment using a risk-based approach. A risk-based approach provides a methodology for Air Force management to focus and prioritize its internal control assessments on areas and activities of greater risk and importance to accomplishing mission and strategic objectives. By not evaluating internal control with a risk-based approach, Air Force management lacks the assurance that resources are used efficiently to assess key controls associated with achieving Air Force objectives subject to the highest risks along with those designated as high priority by agency management, such as controls over accounting for, managing, and reporting on mission-critical assets. Current Reviews Do Not Adequately Assess Internal Control over Processes Related to Mission-Critical Assets Although the Air Force has not designed a process for performing OMB Circular No. A-123 internal control assessments based on risk, it did review certain business process assessable units, such as mission-critical assets, as part of its financial statement audit remediation efforts. However, Air Force’s reviews of internal control over processes related to mission-critical assets did not meet OMB Circular No. A-123 requirements or federal internal control standards for evaluating a system of internal control. During fiscal years 2018 and 2019, the Air Force engaged the Air Force Audit Agency (AFAA) to review control activities for five processes related to mission-critical assets and instructed business process assessable unit leads to conduct additional internal control reviews for select mission-critical asset areas during fiscal year 2019. However, the organizational unit managers did not formally consider the results of these reviews when concluding on their assessments of internal control. For fiscal year 2018, AFAA performed certain agreed-upon procedures to confirm current transactional processes and related internal control over external financial reporting for five mission-critical asset areas as documented in the related business process cycle memorandums. In order to perform the procedures, AFAA used SAF/FM-prepared templates to confirm certain processes and key controls included in the respective process cycle memorandums. However, the procedures SAF/FM instructed AFAA to perform in 2018 did not meet the requirements of an assessment of an internal control system as prescribed in OMB Circular No. A-123. Specifically: Procedures to test design of controls did not include steps for evaluating whether the controls individually or in combination with other controls would achieve objectives or address related risks. Instead, SAF/FM instructed AFAA to confirm whether the process cycle memorandums accurately reflected the controls and processes in place. Procedures to test operating effectiveness of controls were conducted even though there was no determination of whether the controls were designed to achieve objectives or address related risks. Procedures performed involved the use of process cycle memorandums as a baseline, which, as noted by the Air Force’s auditor, did not always reflect the current process, and there was no process in place for management to assess whether the differences related to an inaccurate cycle memorandum or improper implementation of the process. For fiscal year 2019, tests continued to (1) address operating effectiveness without first determining if the controls were designed to meet objectives and reduce risks and (2) involve the use of process cycle memorandums as a baseline that did not always reflect the current business process. For fiscal year 2019, business process assessable unit leads conducted the additional internal control reviews for select processes related to mission-critical assets based on the templates for tests of design and tests of operating effectiveness in Internal Control Playbook appendixes. Similar to the procedures developed for AFAA, the Air Force did not devise the fiscal year 2019 playbook’s template procedures to support conclusions on the design, implementation, and operating effectiveness of internal control over processes that are key to achieving Air Force operational, internal reporting, and compliance objectives. For example, the procedures that the Air Force used to assess the design of internal control over a process related to spare engines at one air base only considered controls related to external financial reporting objectives. The Air Force did not provide evidence that it tested additional controls key to achieving internal reporting, operating, and compliance objectives, such as improving and strengthening business operations and harnessing the power of data for timely decision-making and mission success, or evidence that the Air Force would test such controls during future reviews. Additionally, the Air Force lacked a process for the organizational unit managers or PREs to consider the results of internal control reviews performed at the business process assessable unit level in assessing internal control when they assess and report on the status of internal control for the overall Air Force Statement of Assurance (see fig. 2). Specifically, the current and draft AFI 65-201 and Statement of Assurance Handbook do not include procedures for how information gathered from AFAA agreed-upon procedures or business process unit leads’ testing of internal control over processes related to mission-critical assets is considered in the conclusions reported through the organizational unit managers’ supporting statements of assurance. OMB Circular No. A-123 requires that management, in accordance with federal standards for internal control, evaluate whether a system of internal control reduces the risk of not achieving the entity’s objectives related to operations, reporting, or compliance to an acceptable level. According to the federal internal control standards, when evaluating the design of internal control, management determines if controls individually and in combination with other controls are capable of achieving an objective and addressing related risks. A control cannot be effectively operating if it was not properly designed and implemented. Further, management should establish and operate monitoring activities to monitor the internal control system and evaluate the results. For example, once established, management can use the baseline, or current state of the internal control system, as criteria in evaluating the internal control system and make changes to reduce the difference between the criteria (what is expected) and condition (what Air Force staff did do instead of what was expected). Also, per OMB Circular No. A-123, an agency may document its assessment of internal control using a variety of information sources, such as management reviews conducted expressly for the purpose of assessing internal control (e.g., AFAA agreed-upon procedures and Internal Control Playbook procedures). Air Force reviews of internal control over processes related to mission- critical assets were inadequate because SAF/FM did not include in the agreed-upon procedures or the Internal Control Playbook tests of design to determine if controls individually and in combination with other controls are capable of achieving an objective and addressing related risks, tests of implementation and operating effectiveness only after a favorable assessment of the design of control, and a baseline that has accurate descriptions of business processes and identifies key internal controls as designed by management to respond to risks. Further, SAF/FM did not document its approach for using results from the AFAA agreed-upon procedures in assessing the Air Force’s internal control over processes related to mission-critical assets because the Air Force did not provide guidance establishing the process and reporting lines of all the sources of information that it considered in preparing its overall Statement of Assurance. Also, SAF/FM did not have a documented process for integrating the results of internal control reviews performed at the business process assessable unit level into the organizational units’ assessment of internal control. Moreover, Air Force did not have guidance describing how often, through which conduit, or when the results from the business process internal control reviews were to be provided to relevant organizational units, or how this information would affect conclusions made in a unit’s respective assurance statement. By not comprehensively evaluating internal control over processes related to mission-critical assets, the Air Force is at increased risk that it may not timely identify internal control deficiencies and may lack reasonable assurance over the effectiveness of internal control over processes accounting for mission-critical assets. In addition, without performing internal control assessments in accordance with requirements or having a formal process to consider the results of the AFAA agreed-upon procedures and the Internal Control Playbook procedures in the organizational unit managers’ assessment process, the Air Force increases the risk that its assessment of internal control and related Statement of Assurance may not appropriately represent the effectiveness of internal control. Conclusions Air Force senior leaders work to achieve complex and inherently risky objectives across the agency, while managing over $230 billion in mission-critical assets available to carry out its mission. To reduce the risk of not achieving its objectives or efficiently managing its resources, the Air Force needs to implement an ERM capability that is integrated with an effective system of internal control, as outlined in OMB Circular No. A-123 and federal standards for internal control. Although the Air Force has been working to improve its risk management and internal control practices, including remediation of deficiencies in its internal control over financial reporting related to mission-critical assets, it still faces significant challenges. For example, the agency continues to have difficulties with tracking and reporting, with reasonable accuracy, financial information about its mission-critical assets that directly affect its ability to efficiently support the warfighter, achieve its objectives, and accomplish its mission through reliable, useful, and readily available information. Without an effective ERM governance structure, there is an increased risk that the Air Force will not properly identify, assess, and respond to significant entity-level risks. In addition, by not comprehensively implementing and evaluating its internal control system, the Air Force cannot ensure that it is timely identifying and correcting internal control deficiencies or effectively reducing, to an acceptable level, the risk of not achieving its objectives. Further, Air Force management’s assurances on internal control, as reported in the overall agency Statement of Assurance, may not appropriately represent its internal control effectiveness. Recommendations for Executive Action We are making the following 12 recommendations to the Air Force: The Secretary of the Air Force should develop and implement procedures for an ERM governance structure that includes oversight responsibilities for identifying, assessing, responding to, and reporting on the risks associated with agency material weaknesses from all relevant sources. These procedures should clearly demonstrate that risks associated with material weaknesses are considered by Air Force governance, as a whole, and are mitigated appropriately to achieve goals and objectives. (Recommendation 1) The Secretary of the Air Force should develop policies or procedures for assessing internal control to require (1) clearly delineating who within the Air Force is responsible for evaluating the internal control components and principles, how often they are to perform the evaluation, the level (e.g., entity or transactional) of the evaluation, what objectives are covered in the assessment, to whom to communicate the results if they are relevant to others performing assessments of internal control, and what guidance to follow; (2) documenting management’s determination of whether each component and principle is designed, implemented, and operating effectively; and (3) documenting management’s determination of whether components are operating together in an integrated manner. (Recommendation 2) The Secretary of the Air Force should develop policies or procedures for assessing internal control to require the use of test plans that (1) tie back to specific objectives to be achieved as included in the Business Operations Plan; (2) specify the nature, scope, and timing of procedures to conduct under the OMB Circular No. A-123 assessment process; and (3) reflect a consideration of prior year self-identified control deficiencies and results of internal and external audits. (Recommendation 3) The Secretary of the Air Force should develop policies or procedures for assessing internal control to require SAF/FM to validate (1) the number of organizational units reporting for its overall internal control assessment; (2) how control procedures were tested, what results were achieved, and how conclusions were derived from those results; and (3) whether the results used to compile the current year report are based on current fiscal year’s assessments. (Recommendation 4) The Secretary of the Air Force should develop policies or procedures for assessing internal control to require SAF/FM to assess how waivers affect the current year assessment of internal control, the determination of systemic weaknesses, and the compilation of the Air Force’s overall Statement of Assurance. (Recommendation 5) The Secretary of the Air Force should require that developers of the policy and related guidance associated with designing the procedures for conducting OMB Circular No. A-123 assessments receive recurring training and are appropriately skilled in conducting internal control assessments and are familiar with Standards for Internal Control in the Federal Government. (Recommendation 6) The Secretary of the Air Force should analyze all definitions included in Air Force ERM and internal control assessment policy and related guidance to ensure that all definitions and concepts are defined correctly. (Recommendation 7) The Secretary of the Air Force should require SAF/FM to design recurring training for those who will assess internal control that (1) includes enhancing their skills in evaluating the internal control system and documenting results; (2) reflects all OMB Circular No. A-123 requirements, such as those related to identifying objectives, evaluating deficiencies, and determining material weaknesses; and (3) is provided to all who are responsible for performing internal control assessments. (Recommendation 8) The Secretary of the Air Force should develop policy or procedures consistent with OMB Circular No. A-123 to assess the system of internal control using a risk-based approach. (Recommendation 9) The Secretary of the Air Force should develop procedures to assess internal control over processes related to mission-critical assets, including (1) tests of design that evaluate whether controls are capable of achieving objectives, (2) tests of effectiveness only after a favorable assessment of the design of the control, and (3) a baseline that has accurate descriptions of business processes and identifies key internal controls as designed by management to respond to risks. (Recommendation 10) The Secretary of the Air Force should establish a process and reporting lines of all the sources of information, including reviews performed of internal control processes related to mission-critical assets, that will be considered in the Secretary’s Statement of Assurance. (Recommendation 11) The Secretary of the Air Force should develop procedures to require coordination between business process leads and the Air Force’s unit managers to ensure that mission-critical asset–related internal control deficiencies are considered in the unit managers’ assessments of internal control and related supporting statements of assurance. These procedures should include how, when, and with what frequency the results from the business process internal control reviews should be provided to relevant organizational units for consideration in their respective assurance statements. (Recommendation 12) Agency Comments We provided a draft of this report to the Air Force for review and comment. In written comments, the Air Force concurred with all 12 of our recommendations and cited actions to address them. Air Force’s comments are reproduced in appendix I. We are sending copies of this report to the appropriate congressional committees, the Secretary of Defense, the Under Secretary of Defense (Comptroller)/Chief Financial Officer, the Secretary of the Air Force, the Assistant Secretary of the Air Force (Financial Management and Comptroller), and other interested parties. In addition, the report is available at no charge on the GAO website at https://www.gao.gov. If you or your staff have any questions about this report, please contact me at (202) 512-2989 or kociolekk@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are listed in appendix II. Appendix I: Comments from the Department of the Air Force Appendix II: GAO Contact and Staff Acknowledgments GAO Contact Staff Acknowledgments In addition to the contact named above, John Sawyer (Assistant Director), Russell Brown, Anthony Clark, Oliver Culley, Eric Essig, Patrick Frey, Jason Kelly, Aaron Ruiz, and Vanessa Taja made key contributions to this report.