Background OECA has a range of compliance monitoring, compliance assistance, and enforcement tools available to elicit compliance with laws and regulations from regulated entities, as shown in table 1. Enforcement actions can result in, among other things, the imposition of penalties, requirements to remedy the violation of law or regulation, or both. OECA has developed policies and guidance for EPA staff that describe the agency’s recommended responses to noncompliance based on a number of factors and the escalation of enforcement responses to continuing noncompliance. EPA guidance on informal and formal enforcement actions provides an example related to the Resource Conservation and Recovery Act. In that example, if a regulated entity does not return to compliance or notify the state or EPA that it cannot return to compliance within a certain number of days after an informal enforcement action, the state or EPA may take a formal enforcement action. Generally, according to this same 2010 EPA guidance, informal enforcement actions address small or isolated problems, and formal enforcement actions can address bigger problems. OECA stores and manages a range of compliance monitoring and enforcement data in ICIS. For example, ICIS includes descriptive information about regulated entities, violations, and the outcome of enforcement actions. ECHO, the public access website that integrates data from multiple agency databases, has an internal component for staff and other federal agencies and publicly available components. Staff in EPA’s 10 regional offices, OECA headquarters staff, and states input data into ICIS, which feeds data into ECHO. Regional office staff and OECA headquarters staff also use statute-specific databases to maintain data on compliance with a particular law or office-specific databases built to maintain data, according to the preferences of a particular regional or headquarters office. EPA Collects a Range of Compliance Monitoring and Enforcement Data, but Does Not Maintain Data on Informal Enforcement Actions and Compliance Assistance EPA requires regional offices to collect and enter a range of information on its compliance monitoring and enforcement activities—such as permit, inspection, and violations data—into the agency’s national databases. The agency uses these data to manage its oversight efforts and assess how well the efforts are meeting the agency’s strategic objectives. In addition, EPA is piloting an effort to collect data on coordination with states. However, EPA regional offices do not consistently collect or maintain data on informal enforcement actions. In addition, EPA does not require regional offices to collect and maintain data on their compliance assistance activities; therefore, it has no requirements for regional offices to enter data into the agency’s national databases. EPA Requires Regional Offices to Collect Some Data to Manage and Assess Its Oversight Efforts EPA requires regional offices to collect information from various data sources and enter it into national databases to monitor regulated entities’ compliance with environmental laws and track the agency’s enforcement actions. The information generally includes permit data on limits on emissions or for discharge of pollutants into waters, inspection or other evaluation data, violations data (e.g., failure to take or submit results for drinking water samples); informal enforcement actions, and formal enforcement actions, as shown in figure 1. OECA uses the data in its databases to manage the overall enforcement and compliance program and assess how well its efforts are meeting the objectives outlined in the agency’s strategic plan, according to EPA officials. For example, officials in one regional office told us that regional managers typically review ICIS data (for example, the number of inspections conducted) to monitor their progress toward meeting strategic objectives at the regional level. These regional officials said that staff in their office conduct monthly reviews of ICIS data to understand how their current efforts on certain indicators compare to prior years. OECA headquarters officials told us that the agency had begun to pilot a mechanism to collect data that can help measure agency progress in coordinating with states, one of the agency’s strategic objectives. Specifically, OECA officials told us that in 2018 the agency began a pilot effort to track instances in which regional office staff provide assistance with state enforcement actions, also characterized as “state assists.” According to agency guidance issued in June 2019, a state assist is defined as any instance in which the state could not or would not take the action without OECA’s help or any instance in which a state explicitly requests that OECA take over a case after OECA has identified a violation. During the pilot effort, state assists are documented as such when a regional office has expended substantial resources to identify a violation, develop the injunctive relief, or help the state take an action to obtain a remedy for the violation. According to OECA guidance, the pilot effort, which OECA officials expect to continue through 2021, will help the agency better track its efforts in this area. As of June 2019, according to our analysis of written responses, officials in eight of the 10 regional offices described having documented a state assist as defined by OECA. For example, officials in one regional office stated in their written response to our questions that one specific case against a company located in three different states would have been handled by the regional office. Instead, the regional office agreed to let two of the states take the lead for the cases in those states, and the regional office handled the case in the third state and documented this as two state assists. EPA’s Regional Offices Do Not Consistently Collect or Maintain Data on Informal Enforcement Actions OECA collects data on some informal enforcement actions, such as the number of warning letters sent to regulated entities, but EPA regions do not always collect data about these actions, according to EPA headquarters officials. As a result, the data do not tell the full story of OECA’s enforcement efforts, according to OECA’s Assistant Administrator in testimony during a February 26, 2019, congressional hearing. Furthermore, OECA headquarters officials we interviewed said that data on EPA and state informal enforcement actions are incomplete in EPA’s ECHO website in part because EPA policy and related guidance for each of the various programs defines informal enforcement differently and these definitions can differ from the definitions in ECHO. In a 2010 document, EPA explained how the various agency policy guidance and ECHO define formal and informal enforcement actions differently. For example, the document states that policy guidance for the Clean Air Act defines notices of violation as formal enforcement actions, but that policy guidance for the Clean Water Act and the Resource Conservation and Recovery Act defines notices of violation as informal enforcement actions. Similarly, this same 2010 document states that administrative penalty orders of field citations are considered informal enforcement actions in the policy guidance for the Clean Water Act, but formal enforcement actions in the policy guidance for the Clean Air Act and Resource Conservation and Recovery Act. In addition, the document states that ECHO characterizes notices of violations under the Clean Air Act as informal enforcement actions even though the policy guidance defines them as formal enforcement actions. OECA headquarters officials highlighted two issues that affect the agency’s ability to consistently maintain data on informal enforcement actions: (1) using different definitions of informal enforcement actions across programs and (2) maintaining data on such actions inconsistently. OECA headquarters officials said that they were addressing the first issue of not having one clear definition of informal enforcement actions that applies across all of the air, water, and hazardous waste programs. In September 2019, OECA headquarters officials said EPA was finalizing a single definition of informal enforcement actions for the purpose of collecting more consistent information. In January 2020, EPA provided us with a September 30, 2019, memorandum that defines enforcement response tools, including a definition of informal enforcement action across all programs. Regarding maintaining data inconsistently, while most of the regional offices collect data on some informal enforcement actions, they use different mechanisms to maintain these data. According to our analysis of written responses, officials in nine of the 10 regional offices stated that their offices collect data on some informal enforcement actions such as warning letters, notices of noncompliance, notices of violation, and notices of determination. However, the officials described using different mechanisms for maintaining the data they collect on informal enforcement. For example, officials in five of the nine regional offices that collect data on some informal enforcement actions stated that they maintain the data in ICIS. As we described, ICIS data feeds into ECHO, which has components available to the public. In three of the nine regional offices that collect data on some informal enforcement actions, staff collect data on such actions in a database other than ICIS, such as a statute- or office-specific database, according to our analysis of written responses. Finally, one of the nine regional offices that collect data on some informal enforcement actions maintains those data in paper records, according to an official in that office. In our October 2017 report on key considerations for agency enforcement decisions, we reported that transparency and availability of data are important to promoting compliance and achieving regulatory objectives. As described earlier, EPA changed the focus of its national priorities from enforcement to compliance and increased its use of informal enforcement actions to achieve its regulatory objectives. Having complete information about informal enforcement actions is essential because EPA has elevated the role of such activities in its overall enforcement efforts. EPA often works informally with regulated entities to help them comply with environmental laws and regulations, according to its 2018 EPA Enforcement Annual Results report. However, the agency does not have complete information on those actions for evaluating its compliance monitoring and enforcement performance. Moreover, more complete and consistent information about OECA’s informal enforcement actions would provide a fuller picture of EPA’s overall enforcement efforts. This, in turn, would better enable EPA and OECA to assess whether they are achieving the agency’s regulatory objectives and improve the transparency of OECA’s informal enforcement actions for Congress and the public. Guidance can help agencies communicate expectations and ensure consistency with a standard. While EPA has issued guidance on how various agency policies and ECHO define formal and informal enforcement actions, the agency has not provided guidance to regional offices on how they should collect or maintain data on informal enforcement actions. According to federal standards for internal control, management should design control activities to achieve objectives and respond to risks, such as by clearly documenting internal control in management directives, administrative policies, or operating manuals. On September 30, 2019, EPA issued a memorandum that provides definitions for enforcement response tools, including informal enforcement actions, and instructions on how to report such actions. Now that the agency has finalized its definition of informal enforcement actions and specified which mechanisms to use to maintain data on such actions, by clearly documenting in guidance to the regional offices how they should use the definition to collect data on these actions, EPA would have better assurance that the regional offices consistently collect and maintain these data. EPA Does Not Require Its Regional Offices to Collect and Maintain Data on Compliance Assistance Activities According to EPA headquarters officials, OECA stopped requiring regional offices to collect data and report on their compliance assistance activities around 2012. Prior to that time, each regional office had a full- time staff member dedicated to coordinating compliance assistance activities, according to these officials. However, the staff member’s activities were the only compliance assistance data that regional offices collected and maintained. EPA officials stated that the regional offices stopped collecting the compliance assistance activities associated with this position when the agency redirected the funding for the full-time staff position to compliance monitoring and other enforcement efforts. As a result, EPA officials told us that the agency does not have consistent data about its compliance assistance activities. EPA officials told us that the agency made a policy decision to stop dedicating funding to compliance assistance but encouraged staff to continue conducting compliance assistance activities as part of the agency’s outreach for other programs. EPA headquarters officials said that as of September 2019, the agency had no plans to require regional offices to collect and report data on compliance assistance. However, according to these officials, although the agency stopped funding the compliance assistance coordinator position, regional staff continue to conduct a range of compliance assistance activities as part of their regular enforcement duties. Figure 2 shows the types of compliance and enforcement data that EPA collects, including that the agency does not require regional offices to collect information about compliance assistance. According to our analysis of written responses, officials in nine of the 10 regional offices reported that they collect some data on the compliance assistance activities their offices conduct. Officials in one office said that they do not collect data on compliance assistance activities because it is not required. The types of data on compliance assistance that the nine regional offices collect and the methods those offices use for maintaining the data differ, according to our analysis of written responses. For example, some regional officials described collecting data on compliance assistance provided over the telephone, and other officials described collecting data on on-site compliance assistance provided during inspections. According to our analysis of written responses, officials in two regional offices described providing on-site compliance assistance for minor issues during inspections and tracking the number of times such assistance was provided. Officials in the nine regional offices that still collect data on some compliance assistance activities described storing the data differently, either in region-specific databases or in paper files. Officials in two of these regional offices said that regional staff decide how to document telephone calls from regulated entities for assistance. Officials in one region stated that they no longer conduct large-scale compliance assistance activities such as conducting workshops or developing informational materials because EPA eliminated the reporting requirement. Having complete information about its compliance assistance activities is essential because EPA has elevated the role of such activities in its overall enforcement efforts. However, EPA does not have complete information on its compliance monitoring and enforcement activities, partly because the agency does not require the collection of data on compliance assistance activities. EPA’s lack of complete information on its compliance assistance activities is inconsistent with its change in policy. In addition, in our October 2017 report on key considerations for agency enforcement decisions, we reported that transparency and availability of data are important to promoting compliance and achieving regulatory objectives. Having complete information about its compliance assistance activities may provide more complete information on those activities for evaluating its compliance monitoring and enforcement performance. As discussed earlier, most of the regional offices continue to collect some information on compliance assistance even though they are not required to do so and use varying mechanisms to maintain the information. Because EPA does not direct the regional offices to collect data on compliance assistance activities, the agency would not have issued guidance instructing regional offices to collect such data and specifying which mechanism to use to maintain them. However, according to federal standards for internal control, management should design control activities to achieve objectives and respond to risks, such as by clearly documenting internal control in management directives, administrative policies, or operating manuals. Without clearly documenting in guidance to the regional offices that they should collect data on compliance assistance activities and specifying which mechanism to use to maintain the data, such as ICIS, EPA will not have the information it needs to track progress toward its strategic objective of increasing the agency’s use of compliance assistance activities to help regulated entities comply with laws and regulations. EPA Communicates the Results of Its Compliance Monitoring Activities and Enforcement Actions to the Public and Congress through a Website and Annual Reports EPA communicates the results of its compliance monitoring activities and enforcement actions by making data available to the public and Congress through its website and annual reports. EPA’s ECHO website allows the public to view data over time, such as the number of facilities inspected by an authorized state or EPA from fiscal years 2011 to 2019. To help the public understand the data presented on its ECHO and other websites, EPA websites list a number of national and state-specific known data limitations concerning the data collected for its environmental programs. For example, the ECHO website identifies whether certain years of data are not appropriate for analyzing trends, such as its data on penalties under the Clean Water Act prior to 2015. EPA issues annual performance reports that include data on compliance monitoring and enforcement to fulfill requirements under the Government Performance and Results Act and other requirements. These reports describe progress toward the three strategic goals and related objectives in EPA’s Fiscal Year 2018-2022 Strategic Plan. In addition, since 2017, EPA has published a Year in Review report that outlines the agency’s accomplishments, including in the area of enforcement, using data on its compliance and enforcement actions to present the results of its efforts. In addition, at the end of each fiscal year, OECA publishes a Fiscal Year EPA Enforcement and Compliance Annual Results report and companion data graphs that provide enforcement data over a selected time period on such topics as the number of EPA inspections conducted, cases initiated, and value of fines and penalties collected. Environmental groups and media outlets have used EPA’s data to develop analyses, conclusions, and inferences about changes in EPA’s enforcement results. In December 2018, we reported that providing information about a dataset—for example, known limitations of the data in that dataset— allows users to determine whether the database is suitable for their intended purpose and make informed decisions about whether and how to use it. For example, EPA’s 2000 EPA Quality Manual for Environmental Programs states that published reports with environmental data shall be accompanied by a readily identifiable section or appendix that discusses the quality of the data and any limitations on the use of the data with respect to their original intended application. It also states that the agency’s reports should include applicable statements about possible misuse of the data for other purposes. EPA’s Fiscal Year 2018 Annual Performance Report includes a link to companion reports on its website that describe, among other things, the sources of the data used in the report and the known limitations of those data. Specifically, the companion reports include information such as the definition of terms used, units of measurement, data sources, method for analyzing the data, and the known limitations of the data. However, neither of EPA’s other 2018 annual reports we reviewed fully disclosed known limitations to the data the agency included in each report: Year in Review 2018. OECA’s Year in Review 2018 report, the most recent report available at the time of our review, includes a range of data—such as number of actions taken, monetary results, the reduction of emissions in tons, and data over selected time periods— to accompany its statements about the agency’s accomplishments. However, the report does not include any information about data sources or known limitations of the data. Fiscal Year 2018 EPA Enforcement and Compliance Annual Results. EPA’s Fiscal Year 2018 EPA Enforcement and Compliance Annual Results report, also the most recent at the time of our review, includes data sources and some known limitations of the data. For example, the report states that the data on results do not include state and local inspections or enforcement actions. Additionally, the report includes statements about changes in how the agency stores data that may prevent the data from being comparable across years. The report lists the various sources of the data used to create the report’s charts and graphs. EPA has published known limitations of these data on its ECHO website and indicated that broad data issues may affect the completeness, timeliness, or accuracy of the data in its various systems. However, based on our review of the report, it does not include information about known limitations of all of the data in the report. In addition, neither the Year in Review 2018 report nor the Fiscal Year 2018 EPA Enforcement and Compliance Annual Results report includes a readily identifiable section or appendix that discusses the known limitations of the data, as called for by leading practices for transparently reporting government data and as exemplified in EPA’s manual governing environmental data quality. In commenting on our assessment of the annual reports, EPA officials did not provide a reason why the reports do not discuss known data limitations but told us in a prior meeting that the current documentation on the ECHO website includes the current known data limitations. Furthermore, EPA’s Fiscal Year 2018 EPA Enforcement and Compliance Annual Results report does not fully describe how the data in the report should be interpreted given the known data limitations the report contains. For example, the 2018 annual results report provides a partial picture of overall enforcement of environmental laws because the data exclude state enforcement actions. In addition, for the yearly data across years (2008 through 2018 or 2012 through 2018), EPA does not fully provide information on any limitations in how the data should be analyzed; for example, whether the data are appropriate for the purpose of identifying trends or providing a snapshot of an activity for a single year. EPA does, however, include information on the impact of one or two large cases on the data presented for some data in the report such as the volume of contaminated soil and water to be cleaned up or the treatment and disposal of hazardous and nonhazardous waste. In our November 2019 report on data transparency, we concluded that without the transparent disclosure of known data limitations, users may view or analyze data without full knowledge of the extent to which the data are timely, complete, accurate, or comparable over time. Our November 2019 report also concluded that this could lead users to inadvertently draw inaccurate information or conclusions from the data. OECA’s Assistant Administrator has discussed the known limitations of EPA’s data in the annual reports. In a February 26, 2019, testimony before Congress, OECA’s Assistant Administrator stated that the averages for some of the metrics used in EPA’s annual results report cannot be interpreted to represent a statistical trend. OECA’s Assistant Administrator also stated that changes in the number of enforcement actions may be a function of changes in programmatic decisions and may not be reflective of changes in the underlying compliance of regulated entities with environmental statutes. By including the known limitations of data in its annual reports and providing information on the intended use of EPA’s data, as called for by leading practices for transparently reporting government data and as exemplified in existing EPA guidance for environmental data, EPA would have better assurance that Congress and the public are informed about the data presented and how the data should be interpreted. Conclusions EPA collects a range of information and uses the information to manage its enforcement and compliance program and assess how well its efforts are meeting the objectives outlined in the agency’s strategic plan and other documents. However, while most of the regional offices collect data on some informal enforcement actions, they use different mechanisms to maintain these data, and the agency has not provided guidance to regional offices on how they should collect or maintain the data. Without documenting in guidance to the regional offices how they should collect data on informal enforcement actions and specifying which mechanism to use to maintain the data, EPA lacks assurance that the regional offices will consistently collect and maintain these data. On September 30, 2019, EPA issued a memorandum that provides definitions for enforcement response tools, including informal enforcement actions and instructions on how to report such actions. We view this as a step in the right direction. Now that the agency has finalized its definition of informal enforcement actions and provided instructions on how regional offices should report such actions, by clearly documenting in guidance on how regional offices should use the definition to collect data on these actions, EPA would have better assurance that the regional offices consistently collect and maintain these data. Similarly, EPA does not have complete information on its compliance monitoring and enforcement activities because the agency does not require the collection of data on compliance assistance activities. As a result, the agency has not issued guidance instructing regional offices to collect such data and specifying which mechanism to use to maintain them. Without clearly documenting in guidance to the regional offices that they should collect data on compliance assistance activities and specifying which mechanism to use to maintain the data, such as ICIS, EPA will lack key information. Such information is needed to track progress toward its strategic objective of increasing the agency’s use of compliance assistance activities to help regulated entities comply with laws and regulations. While EPA communicates the results of its compliance monitoring activities and enforcement actions through its website and annual reports, neither of its 2018 annual reports includes a readily identifiable section or appendix that discusses the known limitations of the data. The 2018 annual results report also does not fully describe how the data in the report should be interpreted, given the known data limitations the report contains. By including the known limitations of the data in its annual reports and providing information on the intended use of EPA’s data, EPA would have better assurance that Congress and the public are informed about the data presented and how the data should be interpreted. Recommendations for Executive Action We are making the following three recommendations to EPA: The Assistant Administrator for EPA’s Office of Enforcement and Compliance Assurance should clearly document in guidance to the regional offices how they should use the definition of informal enforcement actions to collect data on these actions. (Recommendation 1) The Assistant Administrator for EPA’s Office of Enforcement and Compliance Assurance should clearly document in guidance to the regional offices that they should collect data on compliance assistance activities and specify which mechanism to use to maintain the data, such as ICIS. (Recommendation 2) The Assistant Administrator for EPA’s Office of Enforcement and Compliance Assurance should include the known limitations of data in its annual reports and provide information on the intended use of EPA’s data. (Recommendation 3) Agency Comments and Our Evaluation We provided a draft of this report to EPA for review and comment. In its written comments, reproduced in appendix I, EPA stated that it agreed with all three of our recommendations and many of our findings and conclusions. EPA also provided technical comments, which we incorporated into the report, as appropriate. In response to our first recommendation to clearly document in guidance how regional offices should use the definition of informal enforcement to collect data on these actions and specify a mechanism to maintain the data, EPA said that the agency issued a September 30, 2019, memorandum for headquarters and regional enforcement offices to implement. This memorandum provides guidance on EPA definitions for enforcement response tools, to promote consistency and clarity in the use of enforcement terms, according to EPA. EPA also said that the guidance defines “informal enforcement action.” The guidance includes instructions on how to report such actions. The guidance states that, with two exceptions, headquarters and regional offices are expected to report, in ICIS, all informal enforcement actions across all programs that meet the new definition. In addition, the guidance states that because it is only a definitional document and does not include guidance on appropriate use of the enforcement response policy tools, the agency will work to identify the specific changes in practice needed (i.e., changes in use and reporting). The guidance states that EPA anticipates that informal enforcement actions meeting the new definition will be included in the agency’s certified annual enforcement results beginning in fiscal year 2020. We view EPA’s guidance as a step in the right direction, and the guidance states that EPA will provide training and additional guidance for enforcement staff to ensure consistent implementation across regional offices and headquarters. Additional guidance will provide EPA with an opportunity to specify how regional offices are to use the definition of informal enforcement to collect data on these actions. We modified our recommendation because EPA’s recent guidance specifies mechanisms for EPA employees to maintain data on informal enforcement actions. In response to our second recommendation to clearly document in guidance that regional offices should collect data on compliance assistance activities and specify a mechanism to maintain the data, EPA said that it would collect data on compliance assistance for each of the National Compliance Initiatives and maintain those data in ICIS. In response to our third recommendation to include known data limitations in annual reports and provide information on intended use of its data, EPA stated that it acknowledges the importance of providing information about a dataset to facilitate proper interpretation. For that reason, EPA said that, in time for its fiscal year 2020 report, the agency will create a webpage to describe how best to interpret the data presented in the agency’s Fiscal Year EPA Enforcement and Compliance Annual Results report and include a reference to that webpage in the report itself as well as the Year in Review report. In technical comments related to our third recommendation, EPA stated that several of the limitations we identified in the report do not affect the data included in its Fiscal Year EPA Enforcement and Compliance Annual Results report. In considering EPA’s technical comments, we modified the text of the report concerning examples of the annual report’s data limitations, as appropriate. As agreed with your offices, unless you publicly announce the contents of this report earlier, we plan no further distribution until 30 days from the report date. At that time, we will send copies of this report to the appropriate congressional committees, the Administrator of EPA, and other interested parties. In addition, the report is available at no charge on the GAO website at http://www.gao.gov. If you or your staff members have any questions about this report, please contact me at (202) 512-3841 or gomezj@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. Key contributors to this report are listed in appendix II. Appendix I:Comments from the Environmental Protection Agency Appendix II: GAO Contact and Staff Acknowledgments GAO Contact Staff Acknowledgments In addition to the contact named above, Chad M. Gorman (Assistant Director); Tahra Nichols (Analyst in Charge); Mark Braza; Courtney Carroux; Tara Congdon; Jazzmin Cooper; Matthew Hunter; Caroline Prado; Dan Royer; Jeanette Soares; Kiki Theodoropoulos, Sonya Vartivarian, and Michelle R. Wong made key contributions to this report.


Background OJJDP Programs and Grants In fiscal year 2018, OJJDP administered 16 programs, which collectively included 53 grant solicitations and 295 associated awards. As shown in table 1, the 16 programs included the Title II Formula Grant Program, two discretionary grant programs for youth mentoring and missing and exploited children that awarded the most funding, and 13 other discretionary grant programs to support efforts such as helping opioid- affected youth, victims of child abuse, and girls in the juvenile justice system. For a complete list of OJJDP programs and grants in fiscal year 2018, see appendix I. On December 21, 2018, the Juvenile Justice Reform Act of 2018 (“Reform Act”) enacted various requirements to strengthen accountability and oversight in OJJDP grant programs, including grants to states under the Title II Formula Grant Program. The Reform Act expressed the sense of Congress that OJJDP must restore meaningful enforcement, and states must exercise vigilant oversight, to ensure compliance with core requirements of the Title II Formula Grant Program. Among the measures the Reform Act put into place to achieve this goal was a requirement that states maintain an “effective” system of monitoring compliance—a requirement that became effective for fiscal year 2020 awards. In contrast, the requirement prior to fiscal year 2020 was for states to maintain an “adequate” system of monitoring compliance, which was the requirement that applied to the grant performance period reviewed for this report (October 2015 through September 2018). This report is one of many called for by the Reform Act to improve oversight of OJJDP grant programs, including the Title II Formula Grant Program. Other oversight requirements in the Reform Act are addressed to OJJDP; OJP’s Office of Audit, Assessment, and Management; and the DOJ OIG. Appendix III provides information on selected Reform Act requirements that relate to accountability and oversight in the Title II Formula Grant Program, and the status of efforts to implement them. OJP Grant Monitoring and Compliance Auditing Activities All OJP grantees and awards are subject to “grant monitoring.” Grant monitoring consists of (1) programmatic and (2) financial monitoring, and according to OJP officials, helps ensure the programmatic and financial integrity and accountability of grantees. OJP policy requires programmatic desk reviews on all open awards each fiscal year and “in- depth” monitoring—consisting of enhanced programmatic desk reviews or site visits—on at least 10 percent of the total number and dollar amount of open and active awards annually. In addition, OJP’s Office of the Chief Financial Officer plans to financially monitor at least 10 percent of the award population annually. States awarded Title II Formula Grants are subject to an additional form of monitoring—“compliance auditing”—which is conducted by OJJDP annually to fulfill statutory requirements unique to the program. Specifically, “compliance auditing” refers to OJJDP’s process for (1) auditing the compliance monitoring systems used by states, and (2) evaluating states’ compliance with four core requirements specified in law. See appendix IV for more information on compliance auditing in the Title II Formula Grant Program associated with the performance period we reviewed. Both the grant monitoring and compliance auditing broadly consist of three parts: (1) desk reviews that occur annually, (2) risk assessments that assist officials in determining what additional monitoring or auditing activities to perform or how to prioritize them, and (3) additional monitoring or auditing activities, such as enhanced desk reviews or site visits (see figure 1). Fraud Risk Management and Related Guidance Fraud and “fraud risk” are distinct concepts. Fraud—obtaining something of value through willful misrepresentation—is challenging to detect because of its deceptive nature. Fraud risk (which is a function of likelihood and impact) exists when individuals have an opportunity to engage in fraudulent activity, have an incentive or are under pressure to commit fraud, or are able to rationalize committing fraud. Fraud risk management is a process for ensuring program integrity by continuously and strategically mitigating the likelihood and impact of fraud. When fraud risks can be identified and mitigated, fraud may be less likely to occur. Although the occurrence of fraud indicates there is a fraud risk, a fraud risk can exist even if actual fraud has not yet been identified or occurred. According to federal standards and guidance, executive-branch agency managers—including those at DOJ, OJP, and OJJDP—are responsible for managing fraud risks and implementing practices for addressing those risks. Federal internal control standards call for agency management officials to assess the internal and external risks their entities face as they seek to achieve their objectives. The standards state that as part of this overall assessment, management should consider the potential for fraud when identifying, analyzing, and responding to risks. We issued our Fraud Risk Framework in July 2015. The Fraud Risk Framework provides a comprehensive set of leading practices, arranged in four components, which serve as a guide for agency managers developing efforts to combat fraud in a strategic, risk-based manner. The Fraud Risk Framework is also aligned with Principle 8 (“Assess Fraud Risk”) of Standards for Internal Control in the Federal Government. The Fraud Reduction and Data Analytics Act of 2015 requires agencies to establish financial and administrative controls that are aligned with the Fraud Risk Framework’s leading practices. In addition, guidance under Office of Management and Budget Circular A- 123 affirms managers should adhere to the leading practices identified in the Fraud Risk Framework. OJJDP Has Goals and Measures to Assess the Performance of Individual Programs, But Has Not Set Corresponding Performance Targets OJJDP Has Established Goal Statements and Performance Measures for Individual Programs OJJDP has established goal statements and performance measures for individual programs. Goal statements broadly convey a program’s overall intent, and performance measures assess program outputs or outcomes. Figure 2 below shows current goal statements and performance measures for the Youth Mentoring Program, for example. OJJDP requires grantees to report on performance measures which are listed in grant solicitations. OJJDP has designated 15 of its measures as “core” measures that are generally applicable across most OJJDP programs. OJJDP aggregates data from some of these core measures across all applicable programs collectively to assess progress toward office-level targets. Targets represent outputs or outcomes expressed as a numeric goal. For example, OJJDP has office-level targets for the percent of youth who offend and reoffend, as well as the percent of grantees that have implemented an evidence-based program. OJJDP obtains, reports on, and uses OJJDP program performance data, as shown in figure 3 below. OJJDP Is Taking Steps to Address Limitations in Grantee-Submitted Program Performance Data We reviewed a selection of performance data that grantees provided to OJJDP for three programs: Title II Formula Grant Program, Youth Mentoring Program, and Gang Prevention Program. As a result of our review—which covered data from October 2015 through December 2018—we determined that these data were not sufficiently reliable for the purpose of providing examples of performance results in this report. For example, we found that grantees reported inconsistent information, such as different numbers of youth served for different performance measures within the same data collection period. Further, some performance measures double-count youth when presenting data by year. OJJDP collects data from discretionary grantees during two 6-month reporting periods and some measures are not designed to be aggregated across time periods. Nevertheless, we found that OJJDP was aware of most of the issues we identified and is taking steps to improve the reliability of grantee- submitted data. For instance, beginning October 2017 (for formula grant programs) and July 2018 (for discretionary grant programs), OJJDP implemented a process to identify inconsistent or otherwise questionable data and reach out to grantees for verification. Appendix II includes information about the issues we found with the data we reviewed and steps OJJDP is taking to improve the reliability of these data. OJJDP Has Not Set Program-Level Targets, Which Limits Its Ability to Assess Progress toward Program Goals While OJJDP has established goal statements and performance measures, it has not set numeric targets by which it can assess progress for each individual program. For example, one stated goal of the Title II Formula Grant Program is to prevent youth already in the juvenile justice system from reoffending. OJJDP’s annual Title II Formula Grant Program performance reports state the percentage of youth who reoffended, the performance measure for this goal; however, the reports do not provide a target against which to evaluate whether the result reflects progress toward the stated goal. Further, while OJJDP has set several office-wide targets for all programs collectively, these targets may not be appropriate for assessing the progress of individual programs because programs vary in size. For example, according to grantee-submitted performance data, the Title II Formula Grant Program served approximately 100,000 youth in fiscal year 2018, while the Gang Prevention Program served about 1,000 youth in calendar year 2018. Thus, office-level targets, while useful, may be more representative of the performance of OJJDP’s largest programs and obscure the results of individual programs. Since June 2019, OJJDP has been reviewing individual program goal statements and performance measures as part of an OJP-wide review, but this review does not include setting program-level targets. OJJDP officials said that they have not set program-level targets for two reasons: (1) there was uncertainty about whether OJJDP had the authority to do so (versus OJP’s Office of the Chief Financial Officer), and (2) setting such targets has not been a priority for OJJDP in the past, in part, due to a lack of resources. In October 2019, OJP clarified that the OJJDP Administrator has authority for OJJDP performance measurement, including setting program-level and office-level targets. OJJDP officials also stated that program oversight has recently become a higher priority and they plan to bring on new staff; and they agreed with the need to set program-level targets. Tracking performance measures against established numeric targets is a leading practice in performance management. Numeric targets establish standards against which federal programs can measure progress towards goals because comparisons can be easily made between projected performance and actual results. Thus, updating program goal statements and performance measures would be more effective with related numeric targets. One goal of the OJP-wide review is to increase accountability for achieving results. Setting program-level targets could help OJJDP meet this goal by establishing a clear means by which progress toward goals can be measured. DOJ Has Taken Steps to Consider Fraud Risks Relevant to OJJDP Grant Programs, but Has Not Determined a Fraud Risk Tolerance Tools Used in OJP’s Grant Monitoring and Compliance Auditing Efforts Provide Insight on Grant Risks Affecting OJJDP Grant Programs According to OJP officials, some of the tools it uses in grant monitoring and compliance auditing consider fraud risk affecting OJJDP grant programs. As previously discussed, OJP’s grant monitoring and compliance auditing broadly consist of three parts: (1) desk reviews that occur annually, (2) risk assessments that assist officials in determining what additional monitoring or auditing activities to perform or how to prioritize them, and (3) additional monitoring or auditing activities, such as enhanced desk reviews or site visits. To carry out these efforts, OJP relies on various tools to assess the overall risk of grantees and awards. Figure 4 provides additional information on the tools that provide insight on fraud risks, according to OJP officials. According to OJP officials, pre-award and ongoing risk assessment processes that apply to all OJJDP grantees are the primary way in which the office identifies fraud risks. The bullets below describe the nature of the primary tools used during these risk assessment processes for all OJJDP grantees and how they provide insights into potential fraud risks. Financial Capability Questionnaire: This questionnaire includes 28 questions designed to provide insight on the financial systems and internal controls a grantee has in place prior to receiving an award. OJP developed the current version of this questionnaire in part as a response to a 2013 DOJ OIG audit, according to OJP officials. All applicants for OJJDP awards are required to fill out this questionnaire, and new grantees’ pre-award risk scores and corresponding risk levels rely, in part, on the applicants’ responses to it. These include detailed questions related to the capabilities of the applicant’s financial management system, such as whether it has the capability to record expenditures by budget cost categories. According to OJP officials, if an applicant’s accounting system cannot do so, the opportunity for fraud increases because the commingling of funds between budget categories would make it difficult to determine whether federal funds were spent in accordance with the approved budget. The questionnaire also includes items related to procurement, travel policy, and subrecipient management and monitoring. These questions may similarly indicate increased fraud risk depending on how the applicant responds. For instance, according to OJP officials, questions related to procurement are designed to determine whether the applicant employs a fair, transparent, and competitive procurement process. If an applicant’s procurement standards do not meet these criteria, the likelihood of fraudulent activity may increase. Grant Assessment Tool: The Grant Assessment Tool helps assess open/active OJJDP awards and grantees against 38 risk criteria. OJP officials identified 14 of these criteria as being indicators of potential fraud risks, such as the results of recent audits, whether the award has subawards or subcontracts, and whether grantees have completed progress reports on time. Officials explained that progress report delinquencies, for example, may be an indicator that a recipient does not have adequate internal controls to handle federal awards, which may provide a greater opportunity for fraud. The Grant Assessment Tool generates a risk score and corresponding monitoring priority for each open/active grantee and award quarterly. According to OJP officials, Grant Assessment Tool criteria have evolved over time in response to common audit or monitoring findings, as well as ongoing coordination with the DOJ OIG, as discussed later. For the Title II Formula Grant Program, grantees are also assigned risk assessment scores and audited using the compliance auditing tools described in the bottom half of figure 4. OJP officials stated that certain responses to any of the questions in the tools used during compliance auditing may indicate inadequate program management or weak internal controls, which may increase the risk of fraudulent activity. OJP officials use a variety of other tools during grant monitoring to monitor each OJJDP grantee or award (see tools in the top half of figure 4). The tools used differ depending on the type and level of monitoring being performed. According to OJP officials, all of the monitoring and auditing tools include detailed questions that provide insight about the strength of a grantee’s internal controls. In cases where a grantee is unable to provide adequate documentation in response to these questions, OJP officials stated that they may have weak internal controls which may increase the risk of fraudulent activity. OJP Coordinates with the DOJ OIG OJP regularly coordinates with the DOJ OIG on issues related to fraud risk affecting OJJDP grant programs through meetings, trainings, and reviews of OIG audits. Specifically, according to officials, staff from OJP’s Office of Audit, Assessment, and Management meet with the OIG two to three times per year to discuss fraud allegations and ongoing fraud investigations related to OJJDP grant programs. These discussions assist OJP officials in determining their monitoring or auditing plans, but also provide insights about the types of issues that are being referred to the OIG for investigation, which can then inform needed changes to monitoring and auditing tools, according to OJP officials. The OIG also provides training to OJP staff every other year on how to identify and report potential fraud. Additionally, Office of Audit, Assessment, and Management officials regularly review OIG audit findings pertaining to individual grantees and compare them to grant monitoring findings for the same grantees. According to OJP officials, the purpose of these reviews is to evaluate where OJP may be able to improve its monitoring processes. Officials stated that one example of an outcome from such a review is the previously discussed Financial Capability Questionnaire. DOJ Has Not Determined the Department’s Fraud Risk Tolerance According to JMD officials, the department is implementing fraud risk management requirements through an iterative process that will be completed over multiple years, which leverages the department’s overall ERM processes. Leading practices for planning and conducting fraud risk assessments acknowledge that assessing fraud risks is an iterative process. According to JMD officials, as a result of this iterative approach, the fraud risk assessments JMD conducted in fiscal years 2017 and 2018 did not fully align with selected leading practices in the Fraud Risk Framework, which include documenting a fraud risk profile. Although the Department continued to implement additional leading practices from the Fraud Risk Framework by developing a fraud risk profile as part of the fiscal year 2019 fraud risk assessment, the fraud risk profile did not include any consideration of DOJ’s fraud risk tolerance—another leading practice identified by the Fraud Risk Framework. 2017 Fraud Risk Assessment In fiscal year 2017, JMD conducted an assessment of fraud risks that consisted of four facilitated discussion groups with relevant officials. One discussion group was tailored to the specific concerns of grant programs—which included grant programs managed by OJJDP—as recommended by leading practices of the Fraud Risk Framework. During the grant-focused facilitated discussion group, participants identified five fraud scenarios relevant to DOJ grant programs, such as misdirection of funds, which occurs when a recipient deliberately misdirects funds in a manner inconsistent with the purpose outlined in the award agreement. After participants reached consensus on the fraud scenarios, they then ranked the inherent risk of each scenario, in alignment with leading practices of the Fraud Risk Framework. DOJ officials also told us that participants ranked residual risk using a voting tool, which required the participants to understand the various management controls in place to address a particular fraud risk. However, the documented outcomes of these discussions did not identify specific fraud risk controls or the extent to which those controls mitigate specific fraud risks, and as a result, it is unclear how these residual risk values were determined. Further, according to officials responsible for managing the fiscal year 2017 effort, the facilitated discussion group did not determine a specific and measurable fraud risk tolerance for DOJ grant programs generally nor discuss specific fraud risk management activities or controls for any specific grant programs. As discussed in greater detail later in this report, leading practices for fraud risk management state that managers should determine a fraud risk tolerance and Standards for Internal Control in the Federal Government states that managers should define risk tolerances in specific and measurable terms so they are clearly stated and can be measured. 2018 Fraud Risk Assessment The fiscal year 2018 effort to assess fraud risk at DOJ consisted of a brief survey about fraud-related issues that was distributed to all DOJ components and was more limited in nature than the fiscal year 2017 effort. Specifically, the fiscal year 2018 questionnaire did not ask the components to identify inherent or residual risks, as was done in the prior year. Further, for the grants category, JMD included four of the five fraud risk areas that were addressed in fiscal year 2017. However, the fiscal year 2018 questionnaire did specifically ask components to identify key fraud risk management activities designed to prevent, detect, or respond to fraud, which had not been part of the fiscal year 2017 effort. Including information about control activities indicates additional maturation of JMD’s fraud risk management activities, but the questionnaire did not ask components to consider the extent to which these control activities mitigate the likelihood and impact of risk—as recommended by leading practices of the Fraud Risk Framework. The questions asked in fiscal year 2018 are shown in table 2 below. According to JMD officials, the components’ responses to this questionnaire were summarized for internal purposes, but no additional analysis or work, such as defining a fraud risk tolerance and documenting a fraud risk profile, was completed for any of the categories listed in table 2. Further, JMD officials stated that the department does not believe a full-scale fraud risk assessment is warranted annually, and the fiscal year 2018 effort was designed to build on the prior year’s assessment. According to officials, this is part of the department’s iterative approach to implementing fraud risk management requirements, which, consistent with leading practices, may not necessarily incorporate all relevant leading practices in each iteration. 2019 Fraud Risk Assessment and Profile In December 2019, JMD officials provided the final summary of the fiscal year 2019 fraud risk assessment, which included a fraud risk profile as recommended by leading practices. According to JMD officials, to conduct the fiscal year 2019 fraud risk assessment, JMD officials first created a fraud risk profile template using information from the 2017 and 2018 assessments. Figure 5 shows an excerpt of JMD’s draft fraud risk profile template, as of September 2019. After senior leadership reviewed the pre-populated template, JMD held a facilitated discussion with representatives from each component to evaluate the risk information presented in the template for each topic area. JMD identified several risks for each topic area, including the same five risks for the grants area that were identified in fiscal year 2017. Based on information provided, the fiscal year 2019 fraud risk assessment and resulting fraud risk profile incorporate many leading practices of the Fraud Risk Framework. These include consideration of inherent fraud risk, current fraud risk controls and their suitability (the extent to which control activities mitigate the likelihood and impact of risk), and residual fraud risk. However, the 2019 fraud risk profile did not determine a measurable fraud risk tolerance, or prioritize residual risk against that tolerance for any of the assessed categories, including grants. Managers’ defined risk tolerance may depend on the circumstances of individual programs and other objectives beyond mitigation of fraud risks. Leading practices for fraud risk management state that managers should define a fraud risk tolerance, examine the suitability of existing fraud controls, and then prioritize residual fraud risks. In doing so, managers should consider the extent to which existing control activities mitigate inherent risks and whether the remaining risks exceed managers’ tolerance. Based on this analysis and the defined risk tolerance, managers then rank residual risks in order of priority, and determine their responses, if any, to mitigate those risks that exceed their risk tolerance. JMD officials stated that they did not yet define the department’s fraud risk tolerance for any of the assessed categories because they view it as the next step in the maturation of DOJ’s fraud risk assessment processes. However, JMD did not provide details or documentation of its plans to develop a specific and measurable fraud risk tolerance for the next iteration of their fraud risk assessments. Although following an iterative approach to fraud risk management is consistent with leading practices, until DOJ defines a measurable fraud risk tolerance for the assessed categories, the department may not effectively allocate limited resources to address fraud risks—including those associated with OJJDP grant programs. Specifically, by determining a measurable fraud risk tolerance for the grants category and assessing identified residual fraud risks against that tolerance to prioritize these risks, the department will help ensure that OJJDP’s grant programs are not vulnerable to greater risks than DOJ is willing to tolerate. Doing so will also provide assurance that OJJDP does not unintentionally over-allocate limited funding to address fraud risks the department is willing to tolerate. Plans for Conducting Future Fraud Risk Assessments According to JMD officials, they are in the process of awarding a contract that will result in an implementation plan for addressing fraud risk management requirements in the future. Specifically, in July 2019, JMD released a Request for Quotes for a Blanket Purchase Agreement in support of DOJ’s implementation of OMB Circular A-123. One of the deliverables JMD expects to order under this agreement is an implementation plan for addressing fraud risk management requirements, which will include developing a plan for conducting regular fraud risk assessments consistent with leading practices for fraud risk management. According to officials, they expect to award the agreement by the end of calendar year 2019, after which the contractor will perform task orders issued by JMD that will include details related to the methodology, timeframes, and staffing associated with each deliverable. Because neither the award nor the task orders were in place at the time of our review, we cannot determine whether DOJ’s planned efforts will fully align with the leading practices of the Fraud Risk Framework, but we will continue to monitor DOJ’s efforts during related ongoing work. Conclusions In fiscal year 2018, OJJDP made 295 awards totaling over $290 million to support programs intended to ensure youth are held appropriately accountable and empower youth to live productive lives. Performance measurement helps ensure funding achieves such outcomes and fraud risk management helps ensure funding is not improperly diverted from this intended purpose. Both of these management principles facilitate stewardship and accountability for federal funds. Since June 2019, OJJDP has been reviewing and updating goal statements and performance measures for individual programs. While OJJDP has office-level targets, it does not have program-level targets. Program-level targets (specific numeric goals) would help OJJDP assess progress toward individual program goals and increase accountability for achieving positive outcomes. Over the past few years, DOJ has taken steps to consider fraud risk for all DOJ grants, including OJJDP’s. Determining a fraud risk tolerance—and assessing residual fraud risk against that tolerance to prioritize these risks—would help OJP calibrate resources to address grant fraud risk for OJJDP programs, helping ensure that resources are not under- or over- allocated. Recommendations for Executive Action We are making two recommendations, including one to OJJDP and one to JMD. Specifically: The OJJDP Administrator should set performance targets for individual grant programs. (Recommendation 1) The Assistant Attorney General for Administration should ensure that future department-level fraud risk profiles (1) determine the department’s fraud risk tolerance for DOJ grants—which include OJJDP grant programs, and (2) prioritize residual fraud risks based on an assessment against that tolerance, consistent with leading practices in GAO’s Fraud Risk Framework. (Recommendation 2) Agency Comments and Our Evaluation We provided a draft of this report to DOJ for review and comment. In an email message, an official within JMD stated that the Department concurred with our recommendations. In written comments provided by OJP, which are reproduced in appendix V, the agency concurred with our recommendation that it set performance targets for individual OJJDP programs. Specifically, OJP stated that the OJJDP Administrator will set performance targets, to the extent practical, for OJJDP's current and new grant programs. Further, the OJJDP Administrator will ensure that the performance targets are reviewed annually. We believe this action, if implemented, would address our recommendation. DOJ also concurred with our second recommendation to include a fraud risk tolerance for DOJ grants in future department-level fraud risk profiles, but did not provide details as to how they will address it. DOJ also provided technical comments, which we have incorporated as appropriate. We are sending copies of this report to the appropriate congressional committees, the Attorney General, and other interested parties. In addition, the report is available at no charge on GAO’s website at https://www.gao.gov. If you or your staff have any questions about this report, please contact Gretta Goodwin at (202) 512-8777 GoodwinG@gao.gov or Rebecca Shea at (202) 512-6722 or SheaR@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made major contributions to this report are listed in appendix VI. Appendix I: Office of Juvenile Justice and Delinquency Prevention Programs and Grants in Fiscal Year 2018 Under the Department of Justice Appropriations Act, 2018, the Office of Juvenile Justice and Delinquency Prevention (OJJDP) administered 16 grant programs in fiscal year 2018, corresponding with a total of 53 grant solicitations and 295 awards. OJJDP issued certain grant solicitations to carry out the purposes of more than one program. Table 3 shows OJJDP’s fiscal year 2018 programs and associated information, including the primary source of authority for each program, as identified by OJJDP. Appendix II: GAO Review of Performance Data and Steps Being Taken to Improve Data Reliability To provide examples of performance results from three of the Office of Juvenile Justice and Delinquency Prevention’s (OJJDP) 16 programs funded in fiscal year 2018, we assessed the reliability of key data elements and methods used to calculate selected performance measures. We determined that data grantees submitted to OJJDP for the programs we selected were not sufficiently reliable for this purpose. Nevertheless, we present rounded numbers of “youth served” for two of the programs we selected (2018 data only), as we found these data to be reliable for the specific purpose of illustrating differences in the relative sizes of the programs. This appendix discusses our approach for selecting programs to review, identifying the timeframe of review, and selecting performance measures to review. It also discusses issues we found with the data we reviewed and steps OJJDP is taking to improve the reliability of data submitted by grantees. Programs Reviewed We selected three of OJJDP’s 16 programs funded in fiscal year 2018 and listed in appendix I. Specifically, we selected the Title II Formula Grant Program because it is OJJDP’s only formula grant program and because it is a comprehensive nationwide program. We selected the Youth Mentoring and Gang Prevention programs from among the 15 discretionary grant programs as a result of the following process: we ranked programs by total amount awarded during fiscal years 2016 through 2018 and randomly selected one program from the top 50 percent and one from the bottom 50 percent, and we excluded programs that we had recently reviewed in other reports—such as the Tribal Youth and Victims of Child Abuse programs. Timeframe of Review We obtained performance data for the selected programs from OJJDP from October 2015 through December 2018. We chose this timeframe because 2018 was the latest full calendar year for which data were available, and looking back three years was sufficient to capture variations in the programs’ funding levels. Performance Measures Reviewed We selected seven of OJJDP’s 15 core performance measures for review. Specifically, we selected measures that focus on youth and that objectively measure short-term outputs or outcomes, as follows: number of program youth served number and percent of youth with whom an evidence-based program number and percent of youth completing program requirements number and percent of program youth who offend (short-term) number and percent of program youth who re-offend (short-term) number and percent of program youth who are victimized (short-term); number and percent of program youth who are re-victimized (short- term) Not all of these performance measures are applicable to all programs. Performance measures that assess the number and percent of program youth who offend, re-offend, and are victimized measure both short-term and long-term outcomes, and we focused only on short-term because of the challenge that grantees face in tracking youth after they exit programs, as explained to us by OJJDP officials. Review Process We reviewed relevant performance measures, their definitions, and the methodology for calculating them. We reviewed data cleaning and validation procedures that OJJDP uses to verify data provided by grantees. We also tested grantee-submitted data provided to us by OJJDP for missing data, outliers, and inconsistencies. For example, we tested for illogical values, such as different numbers of youth served within the same time period, or numerators that were higher than the denominators (i.e. more youth successfully exiting the program than the total number exiting). Finally, we interviewed knowledgeable OJJDP officials and contractors on several occasions. Results of Review We determined that the data were not sufficiently reliable for the purpose of providing examples of program performance results due to several limitations (bulleted below). Although not every limitation applies to every program we reviewed, the number and significance of the collective issues identified led us not to use the data as examples of performance results in this report. Performance measure results for discretionary programs are often double-counted when presenting data by year. OJJDP collects data from discretionary grantees every 6-months; however, some performance measures are not designed to be aggregated across time periods. OJJDP acknowledges that aggregating data from two, 6-month time periods often results in double-counting when presenting annualized data. Some grantees report inconsistent numbers of youth served. For instance, in the Gang Prevention program, the total number of youth served from January through June 2018 was reported as 387 and 267 for different performance measures in the same data set (within one reporting period). We also identified instances where grantees reported inconsistent numbers of youth across reporting periods. Some grantees do not respond to requests to verify questionable data. According to OJJDP officials grantees may lack resources or staff capacity to collect data and track youth’s outcomes. As a result, some grantees may submit incorrect data or submit data after the reporting deadline. Beginning October 2017 (for formula grant programs) and July 2018 (for discretionary grant programs), OJJDP implemented a process whereby contractors flag inconsistent or otherwise questionable data and reach out to grantees for clarification. However, according to officials, grantees do not face any consequences if they do not respond to requests from OJJDP for data verification. For instance, for Youth Mentoring program data submitted by grantees for July through December 2018, OJJDP contractors flagged 122 of 630 grantees as having potentially inaccurate data, but subsequently received responses from only 74 of the 122 grantees. Inconsistent performance measure definitions. We found inconsistencies in the definitions for the following performance measures: (1) number and percent of program youth who offend (short-term), and (2) number and percent of program youth who re- offend (short-term). For both measures, one documented definition states that they apply only to youth who offend or re-offend during the reporting period, and another documented definition states that it also applies to youth who exited the program 0-6 months ago. Reporting on a subset of youth not representative of all program youth. OJJDP uses the number of youth “tracked” as the denominator for several of its performance measures, including the number of youth who offend and reoffend. According to OJJDP, the number of youth tracked for the offend and reoffend measures should ideally be the same number as total youth served by a program. However, the number of youth tracked is usually lower than the number of youth served. As a result, the measure often reflects a subset of youth that, according to one official, may have characteristics that are not well understood—such as youth or families who are more willing to be tracked because they have not offended recently—and thus may skew the results. OJJDP Steps to Improve Reliability of Grantee- Submitted Data OJJDP acknowledges there are concerns with the quality of the grantee- reported performance measures data. According to OJJDP officials, the limitations are the result of several challenges which they are addressing: Replacing outdated data collection tool. Officials said their current data collection tool is outdated and can be unwieldy and confusing for grantees. Along these lines, the tool only includes automated error checks for a limited amount of data fields and does not include an auto-populate feature, which would prevent grantees from entering illogical or inconsistent data. However, according to Office of Justice Program (OJP) officials, as of October 2019, OJP is designing a new data collection tool for all OJP components—including OJJDP—that will include automated error checks and an auto-populate feature, and they plan to implement this tool beginning in October 2020. Updating performance measures. Officials said that some performance measures are also outdated, such as those that result in duplication when reported annually. Officials also said that some performance measures are confusing to grantees. Nevertheless, as part of an ongoing OJP-wide review of performance measures, OJJDP is in the process of reviewing and updating all OJJDP performance measures and plans to provide updated definitions and instructions to grantees. Increasing grantee response rate for data verification. To increase grantee response rates to data verification requests, OJJDP reported that it is exploring possible consequences for grantees if they do not respond, such as increased scrutiny by OJJDP staff who oversee awards or temporary withholding of funds until verifications are submitted. According to officials, whatever approach (or approaches) they decide on, they will implement them by March 2020. Appendix III: Selected Oversight Requirements Related to the Title II Formula Grant Program The Office of Juvenile Justice and Delinquency Prevention (OJJDP) within the Department of Justice is responsible for administering grant programs under the Juvenile Justice and Delinquency Prevent Act of 1974. One of these programs, the Title II Formula Grant Program, authorizes the award of formula grants to states to develop programs for juveniles and improve their juvenile justice systems. On December 21, 2018, the Juvenile Justice Reform Act of 2018 (“Reform Act”) enacted amendments to the Title II Formula Grant Program, including new accountability and oversight requirements for grantees and OJJDP. The amendments were not effective until the fiscal year 2020 grant award cycle and did not apply to the period of performance we evaluated for this report, which was through fiscal year 2018. Table 4 summarizes the accountability and oversight requirements now in effect for the Title II Formula Grant Program and the status of OJJDP’s efforts to implement them. The Reform Act also requires several evaluations and assessments to help strengthen OJJDP’s internal controls and identify fraud, waste or abuse in its programs. Table 5 summarizes selected oversight requirements related to the Title II Formula Grant Program. Appendix IV: Compliance Auditing Applicable to the Title II Formula Grant Program Prior to Fiscal Year 2020 The Title II Formula Grant Program Prior to Fiscal Year 2020 The Title II Formula Grant Program—so called because it was authorized by Title II of the Juvenile Justice and Delinquency Prevention Act of 1974 (JJDPA)—is a state formula grant program, administered by the Office of Juvenile Justice and Delinquency Prevention (OJJDP). The program has been amended several times since 1974—most recently, by the Juvenile Justice Reform Act of 2018 (“Reform Act”), which also called for this evaluation of OJJDP’s performance. The performance data we reviewed (which covers Title II Formula Grants from October 2015 through September 2018) corresponds with statutory requirements in effect at that time, not the current requirements, as amended by the Reform Act, which apply to grant awards made in fiscal year 2020 and subsequent fiscal years. To be consistent with the data we reviewed, this appendix presents information on program requirements that applied prior to fiscal year 2020. Because these requirements are no longer current, we will differentiate them from those that are by citing the superseded edition of the U.S. Code in which they appear—(2012 & Supp. V 2018)—in comparison to the 2018 Main Edition (2018), which contains the provisions now in force. Statutory Basis for Compliance Auditing The term “compliance auditing” refers to OJJDP’s process for (1) auditing the compliance monitoring systems used by states, and (2) evaluating states’ compliance with four core requirements specified in law. During the grant application process, the four core requirements are among several (previously 28, now 33) that a state’s 3-year plan must satisfy for the state to be eligible for award. However, unlike the other eligibility requirements, the four core requirements can trigger a reduction to a state’s grant allocation unless the state maintains compliance during performance. States must provide adequate systems of monitoring their compliance with three of the four core requirements—i.e. those related to when and where juveniles may be detained in detention or correctional facilities—and OJJDP must audit the adequacy of states’ compliance monitoring systems. OJJDP must also determine whether states maintained compliance with each of the four core requirements and, if not, OJJDP must reduce the state’s allocation the following fiscal year by at least 20 percent for each core requirement that the state failed to meet. During the period covered in our review (i.e., prior to fiscal year 2020), the four core requirements subject to compliance auditing were: 1. Deinstitutionalization of status offenders—which prohibits states from using secure detention or correctional facilities to hold juveniles charged with status offenses (except for a listed few). This requirement also applies to juveniles not charged with an offense but who enter the justice system as aliens or as dependent, neglected or abused youths. 2. Separation of juveniles from adult inmates—which prohibits a state from detaining or confining juveniles protected by the deinstitutionalization of status offenders requirement (see above), or juveniles who are alleged or found to be delinquent, in any institution where they have contact with adult inmates. 3. Removal of juveniles from adult jails and lockups—which prohibits a state from detaining or confining juveniles in adult jails or lockups, except in limited circumstances and for specified periods of time, and only if the juvenile has no contact with adult inmates. 4. Addressing disproportionate minority contact—which requires a state to address the disproportionate number of minority youth who come into contact with the juvenile justice system. OJJDP’s Compliance Auditing Process OJJDP’s compliance auditing process during the majority of the period of our review is set forth in a 2017 OJJDP policy document. According to this policy, OJJDP conducts a comprehensive assessment and makes a determination whether the state is in compliance with each of the four core requirements. The comprehensive assessment includes verification of the data submitted, an analysis of the data submitted by the state to evaluate compliance with each of the four core requirements, and a review to assess the adequacy of internal controls over the state’s compliance monitoring process for collecting and reporting compliance monitoring data. According to this policy, the OJJDP Administrator issues correspondence annually regarding final compliance determinations. These determinations include, as necessary, specific details regarding why a state was determined to be out of compliance with any of the four core requirements or the required compliance monitoring system. Per the policy, a state’s formula grant funding will be reduced by 20 percent for each core requirement with which OJJDP has determined a state to be out of compliance. Additionally, if OJJDP determines that the state has an inadequate system of monitoring, the state may have receipt of its formula grant funding withheld or may be deemed ineligible for a formula grant award. Finally, according to this policy, OJJDP conducts field audits on a rotating schedule. The purpose of the field audits is to confirm state compliance monitoring activity and practices through direct onsite observation and file review, and to identify needed areas for technical assistance. OJJDP anticipates, with available funding and resources, that every state will receive a field audit every three years. Appendix V: Comments from the Office of Justice Programs Appendix VI: GAO Contact and Staff Acknowledgements GAO Contacts Staff Acknowledgements In addition to the contacts named above, Tonnyé Conner-White (Assistant Director), Jonathan Oldmixon (Assistant Director), Jeff Jensen (Analyst-in-Charge), James Ashley, Dominick Dale, Christine Davis, Caroline DeCelles, Elizabeth Dretsch, Eric Hauswirth, Elizabeth Kowalewski, Ben Licht, Jan Montgomery, Heidi Nielson, and Abby Volk, made key contributions to this report.