Background GAO’s Questions for Assessing Reform Efforts In developing our June 2018 report to assist the Congress, OMB, and agencies in assessing agency reform plans, we reviewed our prior work and leading practices on organizational transformations; collaboration; government streamlining and efficiency; fragmentation, overlap, and duplication; and high-risk and other long-standing agency management challenges. The resulting June 2018 report includes 58 key questions to aid in assessing reform efforts. These questions are organized into four broad categories and 12 subcategories. We determined that the questions most relevant to the current implementation stage of State’s reform efforts are found in two subcategories: (1) Leadership Focus and Attention and (2) Managing and Monitoring. Table 1 lists the key questions in these subcategories. State’s 17 Reform Projects In response to the March 2017 Executive Order 13781 and the ensuing OMB memo, State launched a “listening tour” intended to gather ideas and feedback from State and USAID employees. As a key component of this outreach effort, State hired a contractor to design and administer a confidential online survey, which was sent to all State and USAID employees in May 2017. According to the contractor’s report, the survey had a 43 percent response rate, with 27,837 State employees and 6,142 USAID employees responding to the survey. The contractor also conducted in-person interviews with a randomly selected cross section of personnel, which included 175 employees from State and 94 from USAID. The contractor’s report on the results of the survey and the interviews highlighted five areas for State reforms. In July 2017, the Deputy Secretary of State created five planning teams to develop multiple projects in those five areas. The Deputy Secretary also established an Executive Steering Committee composed of senior State and USAID officials to guide the five planning teams and provide direction during the reform process. Led jointly by State and USAID, each planning team comprised participants from a cross section of overseas and domestic workforces. The planning teams were tasked with gathering information and conducting analysis as described below: Foreign Assistance Programs: Analyze current foreign assistance programs at State and USAID to develop a future vision, ensuring alignment with national priorities. Overseas Alignment and Approach: Assess key diplomatic activities and identify required platforms, including the balance of work between headquarters and the field. Human Capital Planning: Identify ways to promote an agile and empowered workforce as part of an overarching talent map. Management Support: Identify opportunities to streamline administrative support functions at the bureau and agency levels to ensure front line effectiveness. Information Technology (IT) Platform Planning: Focus on improving the employee experience through increased use of cutting- edge technology and streamlining duplicative systems and processes. Figure 1 shows a timeline of key events in State’s initial reform efforts. The planning teams developed specific reform projects, listed below in table 2, which State described in the fiscal year 2019 budget justification it submitted to Congress in February 2018. According to implementing officials, all these projects predated the Executive Order and OMB memo issued in the spring of 2017. They also noted, however, that the administration’s reform-related directives helped advance State’s preexisting efforts by focusing management attention and agency resources on these projects. As of April 2019, State Had One Completed and 13 Continuing Reform Projects; Two Other Projects Had Stalled and One Project Was Discontinued As of April 2019, according to State officials and status reports, State had completed one of its 17 reform projects; 13 projects were continuing; two projects were stalled pending future decisions or actions; and one project was discontinued. Table 3 provides additional details on each project and a summary of the results of our analysis. Loss of Leadership Focus Contributed to Staff Uncertainty about Some Reform Efforts, Although Bureaus and Offices Have Taken Steps to Manage and Monitor Continuing Projects Leadership Focus and Attention As State shifted into the implementation phase of its reform efforts in early 2018, multiple transitions within the agency contributed to a loss of leadership focus on the efforts, resulting in uncertainty about leadership’s support for some reform projects. In February 2018, State reported to Congress in its fiscal year 2019 budget justification that it was pursuing the reform projects we described above. In March 2018, the first transition affecting the implementation of those projects occurred when the President removed the then Secretary of State and nominated the then CIA director to replace him; in April 2018, the Senate confirmed the current Secretary. According to senior State officials, when the new Secretary took office, his top priority was ending the hiring freeze and restarting a concerted recruitment effort because vacancies in key positions and a general staffing shortfall would otherwise have led to what one senior official described as a “cataclysmic failure” at State. These senior officials noted that the new Secretary decided some of the existing reform projects were not well designed and that he wanted greater emphasis on cybersecurity and data analytics. They said he also wanted to pursue other initiatives, including a new proposal to create a Global Public Affairs Bureau by merging two existing bureaus. The senior officials told us that the Secretary authorized responsible bureaus and offices to determine whether to continue, revise, or terminate existing reform efforts or launch new initiatives. However, State did not formally communicate other changes in its reform priorities to Congress, such as its plan to no longer combine State and USAID’s real property offices. State initiated another transition in leadership of the reform efforts in April 2018 when it disbanded the dedicated planning teams overseeing the reform efforts and delegated responsibility for implementing the reform projects to relevant bureaus and offices. As the planning teams finished working on their particular reform efforts and prepared to transfer these projects to the bureaus, some planning teams provided memos and reports on the status of their efforts and offered recommendations for the bureaus to consider when determining next steps in implementing the projects. Some implementing officials, however, reported that they received little or no direction regarding their projects or any other indication of continued interest in their project from department or bureau leadership aside from the initial notification that the project had been assigned to them. For example, in separate discussions with implementing officials responsible for three different projects, the officials reported that they had not received any direction or other guidance related to their assigned project since it was delegated to them in April 2018. In one case, this lack of communication continued for nearly a year. In addition, although implementing officials said that they have managed to incorporate reform-related work into their daily responsibilities, they noted that there were multiple benefits from having had dedicated planning teams to lead earlier phases of State’s reform efforts. For example, they said that the dedicated teams included senior officials and the regular involvement of high-level leadership facilitated by these teams had helped advance the reform efforts. These dedicated teams also required staff to set aside time to focus on reform initiatives, which allowed them to develop holistic solutions to reform-related challenges. Conversely, implementing officials reported negative implications of not having dedicated teams. For example, one implementing official described how positive work initiated under the leadership of these dedicated teams—including efforts to eliminate redundancies and identify opportunities for consolidation—ended when the teams were disbanded because the staff and resources needed to continue these efforts were no longer available. Various State officials noted that the prolonged absence of Senate- confirmed leadership in key positions posed additional challenges. We have previously testified that it is more difficult to obtain buy-in on long- term plans and efforts that are underway when an agency has leaders in acting positions because federal employees are historically skeptical of whether the latest efforts to make improvements are going to be sustained over a period of time. For example, State did not have a Senate-confirmed Under Secretary for Management from January 2017 to May 2019. In November 2018, the Deputy Secretary of State told us that the lack of a confirmed Under Secretary for Management was hindering State’s ability to conduct business and implement reforms. The bureaus and offices responsible for 12 of State’s 13 continuing reform projects reported directly to an Acting Undersecretary for Management from January 2017 through May 2019. Moreover, State officials told us that both projects that we determined to be stalled were, among other things, awaiting the confirmation of an Under Secretary for Management to make key decisions. Furthermore, some implementing officials told us that the lack of confirmed officials in leadership positions within the bureaus responsible for implementing the projects added to a lack of leadership focus on implementing some of State’s reform projects. According to State officials, as of April 2019, although 13 of the reform projects described in the fiscal year 2019 Congressional Budget Justification were considered by State to be continuing, some had been scaled back, slowed down, or both as a result of senior leadership’s shifting priorities and attention. For example, one of State’s initial reform projects was related to better management of real property. However, State ultimately scaled back this project, effectively splitting it into two projects: One project focused on real property process improvements is continuing, but State has discontinued the other project to consolidate its and USAID’s real property function. Implementing officials told us in November 2018 that they were still pursuing the internal real property process improvements. They said then that they expected this reform project would likely progress at a slower pace without the dedicated team that previously had provided direct access and frequent interaction with senior department leadership. However, these officials recently informed us that the pace of progress on this project actually increased under the leadership of the bureau’s Senate-confirmed Director. The bureau was led by acting directors from January 2017 through September 2018. We have identified leadership focus and attention as practices vital to successfully implementing reform efforts. These practices include communicating clear and compelling reasons for the reforms, having a dedicated implementation team to manage the transformation process, and designating leaders responsible for implementing reforms and holding them accountable. Dedicating a strong and stable implementation team responsible for a transformation’s day-to-day management is important to ensuring that reforms receive the focused, full-time attention needed to be sustained and successful. One of the key responsibilities of a dedicated team is communication, particularly answering questions about the reform process from employees and other stakeholders. An implementation team is also important to ensuring that reform efforts are implemented in a coherent and integrated way. Because an agency’s transformation process is a large undertaking, we have found that an implementation team must have direct access to and be accountable to top leadership. In turn, top leadership must vest the team with the necessary authority and resources to set priorities, make timely decisions, and move quickly to implement top leadership’s decisions regarding the transformation. In addition, we previously reported that the single most important element of successful improvement initiatives is the demonstrated commitment of top leaders. This commitment is most prominently demonstrated through top leaders’ personal involvement in developing and directing reform efforts. Federal standards for internal control in the federal government also emphasize the importance of maintaining leadership continuity in order to achieve agency objectives. As a result, in other reports, we have recognized that agency reform efforts can take years to implement and that the time frame required for change typically takes longer than the tenures of political leaders. Similarly, the time it takes to nominate and confirm officials for senior management positions can also hamper efforts to initiate reforms or sustain momentum needed to successfully implement reform initiatives. For these reasons, and others, we have highlighted the need to ensure that top leadership drives the transformation and establishes dedicated teams to manage the transformation process. Taken together, the leadership transitions at State had two significant effects on State’s reform efforts. First, the transition of departmental leadership and lack of direction and communication about subsequent changes in leadership’s priorities contributed to uncertainty among implementing officials about the future of individual reform projects. Second, according to implementing officials, the transition of project responsibility from dedicated teams to bureau-level implementing officials resulted in fewer resources and a lack of senior leadership involvement and attention for some projects. Absent leadership decisions, implementing officials will continue to struggle with understanding leadership priorities with regard to State’s reform efforts. Similarly, for any projects that are determined to be leadership priorities, day-to-day implementation activities will continue to be hampered by the lack of a dedicated team to guide and manage the agency’s overall reform effort. Managing and Monitoring Although uncertainty exists about the leadership priorities regarding reform efforts, the bureaus and offices responsible for implementing State’s reform projects have taken steps to manage and monitor their reform projects. Our previous work has identified monitoring as another important practice when implementing reform efforts, including, among other things, developing implementation plans and ensuring transparency by publicly reporting on progress toward milestones. These practices are also incorporated into State’s Foreign Affairs Manual and other department policies. We found that the relevant bureaus and offices responsible for implementing reform projects had developed implementation plans and that these plans identified milestones and deliverables for the projects. For example the Human Resources Services Delivery project had an implementation plan with milestones and deliverables, such as identifying programs and functions for consolidation in 2019 and reducing human resource delivery costs by 14 percent by 2022. Similarly, we found that the implementation plan for the IT Modernization project incorporated milestones that including, among other things, implementing a comprehensive enterprise IT risk management program by fiscal year 2020; reducing average deployment time for new IT capabilities by 10 percent annually from fiscal year 2019 through fiscal year 2021; and increasing workforce access to cloud-based email and business data from 10 percent to 100 percent by September 30, 2019. With regard to monitoring, while there is no centralized mechanism for reporting progress on all projects, we found that each of the ongoing projects currently has some form of progress reporting. For example, State reports progress on projects with IT components—such as Real- Time Collaboration and Work Anytime, Anywhere and Improve Enterprise-Wide Data Accessibility—as part of its quarterly reporting on IT Modernization under the Government Performance and Results Modernization Act of 2010. As a result, these projects have continued within a formal monitoring structure that involves regular web-based status updates and progress reporting. Other reform efforts—such as human capital and real property projects—are monitored against milestones established in State’s Joint Strategic Plan and progress is reported in State’s Annual Performance Reports. Progress for certain projects is also monitored and reported in other reports, such as State’s joint strategic plan, IT strategy, or human capital plan. Finally, other reform projects, such as State’s acquisition reform efforts, are reported at the government-wide level as part of the Cross-Agency Priority Goals outlined in the President’s Management Agenda. State collects data and evidence in order to measure progress in achieving outcome-oriented goals it sets for these projects. State reports these goals and relevant performance data in its annual performance plans and reports. For example, State uses the U.S. General Services Administration’s Customer Satisfaction Survey to measure and report the performance of its Human Capital Delivery Services reform efforts. State also uses data collected through the Office of Personnel Management’s Federal Employee Viewpoint Survey to measure employee satisfaction, which State established as a performance indicator for this project. Conclusions Effectively implementing major reforms can span several years and must be closely managed. In 2017, State began a reform effort that led to 17 reform projects, most of which are unimplemented but still continuing. State notified both OMB and the Congress of these projects. Nevertheless, State leadership has not provided the focus necessary to support the officials responsible for implementing all these reform projects. When a new Secretary of State took charge in March 2018, he transferred responsibility for implementing the reform efforts from dedicated teams led by senior department leadership to bureaus and offices. In addition, key political appointee positions remained filled by officials in an acting capacity until only recently. These transitions at State have had an effect on its reform efforts. Without explicit direction from senior leadership, some implementing officials involved in the reform efforts remain unclear about whether their projects are an agency priority. Further, for the reform efforts that remain an agency priority, a dedicated team to oversee implementation could help accelerate State’s efforts to improve the efficiency and effectiveness of its operations. Recommendations for Executive Action We are making the following two recommendations to State: The Secretary of State should determine which of the unimplemented reform projects included in its fiscal year 2019 Congressional Budget Justification, if any, should be implemented and communicate this determination to Congress and appropriate State personnel. (Recommendation 1) The Secretary of State should establish a single dedicated team to manage the implementation of all reform efforts that the Secretary decides to pursue. (Recommendation 2) Agency Comments We provided a draft of this report to State, USAID, and OMB for review and comment. We received comments from State and USAID, which are reprinted in appendixes II and III, respectively. In response to our recommendation that State determine which reform projects should be implemented and communicate that information to Congress and appropriate State personnel, State indicated that it concurred but suggested it should inform OMB instead of Congress. While we agree that it is important for State to share information regarding its reform efforts with OMB, we remain concerned about State’s lack of communication with Congress regarding the status of the projects State initially reported in its fiscal year 2019 Congressional Budget Justification. Congress is a key stakeholder in State’s reform efforts and should be informed of changes in State’s priorities and the status of these projects to help ensure successful implementation. In response to our recommendation that State establish a dedicated team to manage the implementation of all reform projects, State suggested that leadership of its reform projects should be decided on a case-by-case basis with the latitude to determine whether projects will be assigned to a higher level or within individual bureaus. We stand by our recommendation that State should establish a single dedicated team to manage the implementation of all its reform efforts. This is a key practice for implementing agency reforms identified in previous GAO reports, as well as in State’s Foreign Affairs Manual (1 FAM 014.2), which calls for State to “dedicate an implementation team to manage the transformation process” for major reorganizations of bureaus or offices. Because reform efforts can span several years, dedicating a strong and stable team is important to ensure that the transformation receives the needed attention to be sustained and successful. In its comments, USAID expressed several concerns about the leadership of State’s reform efforts and State’s coordination with USAID. OMB did not provide written comments on the report. We also received technical comments from State and USAID, which we incorporated throughout our report as appropriate. We are sending copies of this report to the appropriate congressional committees, the Secretary of State, the Administrator of USAID, and other interested parties. In addition, the report is available at no charge on the GAO website at http://www.gao.gov. If you or your staffs have any questions about this report, please contact me at (202) 512-6881 or BairJ@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are listed in appendix IV. Appendix I: Objectives, Scope, and Methodology We prepared this report under the authority of the Comptroller General to conduct work to assist Congress with its oversight responsibilities. This report examines (1) the status of the reform efforts that the Department of State (State) reported to Congress in its fiscal year 2019 Congressional Budget Justification and (2) the extent to which State addressed key practices we previously identified as critical to the successful implementation of agency reform efforts. For the purposes of this review, we use the term “reform efforts” to refer to all reform-related projects, proposals, plans, activities, and documents related to the 16 projects identified in State’s fiscal year 2019 Congressional Budget Justification. The term “projects” refers specifically to the 16 reform projects identified in State’s fiscal year 2019 Congressional Budget Justification. State subsequently split one of these 16 projects into two separate projects; thus, we refer to 17 reform projects throughout the report. For both objectives, we reviewed State’s reform plans, proposals, and related documents. We also interviewed four senior officials—generally at or above the assistant secretary level—that had responsibility for the reform efforts as a whole, as well as all implementing officials responsible for each of the continuing reform projects. To determine the status of State’s reform efforts, we reviewed documents and reports related to each of the reform projects described in State’s fiscal year 2019 Congressional Budget Justification. To determine the extent to which State addressed key practices for implementing agency reforms, we assessed State’s reform efforts against key questions identified in the implementation category of our June 2018 report. Specifically, we assessed State’s implementation efforts against key questions from the two implementation-related subcategories of our 2018 report: (1) Leadership Focus and Attention and (2) Managing and Monitoring. We considered the nature of each of State’s reform projects and the efforts taken to implement them, reviewed project-specific reports and other relevant State documents, interviewed State officials responsible for implementing each project, and then made qualitative determinations about the extent to which State’s overall reform efforts addressed these criteria. A second analyst then independently reviewed and validated each determination. Subsequently, other GAO staff reviewed and concurred with these determinations. We only applied criteria from our June 2018 report that we determined were relevant to the scope of our review, which was limited to the implementation phase of State’s reform efforts—from April 2018 to the present—to avoid duplicating the reviews of earlier phases of State’s reform efforts conducted by State’s and the U.S. Agency for International Development’s Offices of Inspector General (OIG). Because State’s OIG was also reviewing State’s reform efforts, we coordinated regularly with State’s OIG to avoid duplication. We did not consider criteria from the first two categories of our June 2018 report—(1) Goals and Outcomes and (2) Process for Developing Reforms—because these applied to the initial phases of State’s reform efforts, which were outside the scope of our work and central to the broader historical review that State’s OIG was conducting at the time of our review. We also did not apply criteria from the final category of our June 2018 report—Strategically Managing the Federal Workforce—to avoid duplicating work State’s OIG recently conducted on State’s workforce management. For the two sub- categories that we selected, we considered the key questions in the report in light of their relevance to State reforms efforts, and also employed other relevant criteria, where appropriate, most notably criteria for leadership from federal internal control standards. We conducted this performance audit from October 2018 to August 2019 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives. Appendix II: Comments from the Department of State Appendix III: Comments from the U.S. Agency for International Development Appendix IV: GAO Contact and Staff Acknowledgments GAO Contact Jason Bair, (202) 512-6881, or bairj@gao.gov. Staff Acknowledgments In addition to the contact named above, Thomas Costa (Assistant Director), Joshua Akery (Analyst in Charge), Peter Beck, David Dayton, Martin de Alteriis, Emily Gupta, Patrick Hickey, Chris Keblitis, Sarah Veale, and Alex Welsh made key contributions to this report.


Background GDUFA and GDUFA II both provided supplemental resources to FDA by giving it the authority to collect user fees from the generic drug industry, in addition to its regular appropriations, in order to make improvements to the generic drug application review process. GDUFA was enacted in July 2012, in part, to provide funding for more generic drug application reviewers at FDA in order to handle an increase in the volume of application submissions and speed up reviews. GDUFA II was enacted in August 2017 to reauthorize the generic drug user fee program from fiscal year 2018 through fiscal year 2022. In return, FDA provided Commitment Letters to Congress that detailed its plans to implement program enhancements and meet certain performance measures related to the review of generic drug applications. For example, in its Commitment Letter for GDUFA II, FDA stated that, for applications in the first review cycle, it would review and act on at least 90 percent of them within specified timeframes—8 months for certain priority applications and 10 months for all other applications. FDA stated that it met this performance measure for fiscal year 2018. The Commitment Letter also indicated that one of the goals of GDUFA II is to minimize the number of review cycles for applications to attain approval. Generic Drug Application Review Process FDA’s generic drug application review process includes a number of steps. The process begins when a generic drug application is submitted to FDA for review by the Office of Generic Drugs and the Office of Pharmaceutical Quality within FDA’s Center for Drug Evaluation and Research. The Office of Generic Drugs is responsible for providing regulatory oversight and strategic direction for FDA’s generic drug program to make safe, effective, and high-quality generic drugs available to the public. The Office of Generic Drugs’ Division of Filing Review determines whether the generic drug application is acceptable for review, meaning that the application is sufficiently complete for FDA to review the application, such as information about the amount of active ingredients present in the drug. If the generic drug application is not acceptable for review, FDA issues a Refuse to Receive letter to the applicant explaining what additional information is required before the application can be accepted for review. In response, the applicant can resubmit the generic drug application with additional information and officials within the Division of Filing Review will assess the information and determine if the resubmitted application is acceptable for review. Once the Division of Filing Review determines that a generic drug application is acceptable for review, the first review cycle begins, and FDA officials review the application across the following three review disciplines: Bioequivalence. Officials within the Office of Bioequivalence are responsible for examining whether the generic drug application is bioequivalent to the brand-name drug, meaning that the drug delivers the same amount of active ingredient in the same amount of time as the brand-name drug. Labeling. Officials within the Division of Labeling Review are responsible for ensuring that the proposed labeling language in the generic drug application, including the drug’s prescribing information, matches the language found in the labeling of the corresponding brand-name drug. Pharmaceutical quality. Officials within the Office of Pharmaceutical Quality have responsibility for ensuring the quality of drug products and assessing all drug manufacturing facilities, including both domestic and foreign. Officials within this office assess the risk of toxic substances or bacterial content in the drug, among other things. All generic drug applications are reviewed by primary reviewers and secondary reviewers. Primary reviewers assess the application to ensure that the documentation meets regulatory requirements in their specific disciplines. For example, primary reviewers within the Office of Pharmaceutical Quality evaluate documentation related to the proposed drug manufacturing process to ensure that the process produces quality drugs consistently. Secondary reviewers ensure the quality and consistency of primary reviewers’ assessments and the clarity of communication to applicants. During the first review cycle, FDA communicates with applicants when issues arise during its review that may prevent the agency from approving the generic drug application. This communication is typically made through the issuance of information requests or discipline review letters: Information requests. Information requests are letters sent to applicants to request clarification or additional information that is needed or would be helpful for FDA to complete its review. These letters may be sent by any review discipline and can be sent at any point after the Office of Generic Drugs accepts the generic drug application for review. Discipline review letters. Discipline review letters are letters issued by each review discipline at about the mid-point of the review cycle to identify possible deficiencies. FDA officials said they aim to issue these letters no later than 6 months into the first review cycle. In response to these letters, applicants can submit additional information for FDA to consider before the end of the review cycle. After FDA’s three review disciplines complete their review of an application and any additional information the applicant has submitted in response to information requests and discipline review letters, the agency issues an action letter that informs the applicant of whether the application is approved, marking the end of the first review cycle and the end of FDA’s review if the application is approved. There are three types of action letters: Approval letter. Issued when the agency has concluded its review of a generic drug application and the applicant is authorized to commercially market the drug. Tentative approval letter. Issued when the agency has completed its review of an application and has concluded that the generic drug application is sufficient, but patents or other exclusivities prevent approval. A tentative approval letter does not allow the applicant to market the generic drug. Complete response letter. Issued at the completion of the review of an application where deficiencies remain at the end of the review cycle. The complete response letter describes any deficiencies that must be corrected in order for the application to be approved. For a generic drug application that receives a complete response letter, the applicant can amend the application and seek another full review, which begins the second or subsequent review cycles. During these cycles, FDA officials review changes made to generic drug applications in response to deficiencies that FDA identified. FDA Approved 12 Percent of Generic Drug Applications in the First Review Cycle and Several Factors May Have Contributed to Whether Applications Were Approved Our analysis of FDA data shows that 12 percent of the 2,030 generic drug applications that FDA reviewed in fiscal years 2015 through 2017 received approval in the first review cycle. See figure 1. We identified several factors, including certain characteristics of generic drug applications, that may have contributed to whether an application received approval in the first review cycle, including the sufficiency of the application, deficiencies in drug quality, the type of drug reviewed, and the application’s priority status. Sufficiency of the application. We found that the sufficiency of the generic drug application, including the completeness of the application and the degree to which the applicants understood and fulfilled application requirements, affected its likelihood of receiving an approval in the first review cycle. According to FDA, one indication of the sufficiency of the generic drug application is whether FDA had previously refused to receive the application for review because it was not substantially complete upon its first submission. Our analysis of FDA data found that applications that had previously been refused were slightly less likely to be approved in the first review cycle compared with applications that had not previously been refused, and rates of approvals decreased for applications with two previously refused attempts. See table 1. According to stakeholders we interviewed, the sufficiency of a generic drug application may partially reflect the level of experience the applicant has in submitting applications, and we found some evidence to support this explanation. FDA managers and reviewers said that, in general, less experienced applicants are more likely to produce lower-quality generic drug applications compared to applicants with relatively more experience. Our analysis of FDA data found that applicants that submitted just one application during fiscal years 2015 through 2017 had a slightly lower rate of approval in the first review cycle (10 percent) compared to the rate across all applicants (12 percent). Additionally, in our review of 35 selected generic drug applications, we identified three applications that were from applicants that had no previously approved generic drug application submissions, an indication that they may have little or no experience with these applications. None of the three applications were approved in the first review cycle. One of these applications elicited reviewer comments that outlined basic application requirements, potentially reflecting the lack of experience of the applicant. Drug quality deficiencies. Our review of documentation from the first review cycle for 35 generic drug applications included 26 that were not approved in that cycle. Among those 26 applications, the most common deficiencies that remained at the end of the first cycle were related to the quality of the drug—356 out of 435 deficiencies. These deficiencies included issues related to the drug manufacturing facilities, which can affect the quality of the drug produced and the stability of the drug over time, among others. Officials from one large applicant told us that most of the comments they received from FDA reviewers are related to the quality of the drug. Three out of five applicants we interviewed also noted that the results from inspections of drug manufacturing facilities—which FDA includes as a component of its review of the quality of the drug—are a factor that may cause an application not to be approved in the first review cycle. Among the 26 applications we reviewed that were not approved, eight had an outstanding deficiency related to the manufacturing facility. Type of drug. We also found that the rate of approval in the first review cycle differed based on certain characteristics of the type of drug reviewed, including the route of administration, which may indicate the complexity of the drug. Complexity can also be influenced by other factors including, for example, the drug’s active ingredient or formulation. FDA officials noted that some complex drugs—including those that combine drug products with drug delivery devices, such as asthma inhalers—are less likely to be approved in the first review cycle. Officials we interviewed from one large applicant—which we identified based on the number of approved generic drug applications it had in fiscal year 2018—reported that their company had never submitted a generic drug application for a complex drug product that received approval in the first review cycle despite having significant experience with producing complex drugs. Officials we interviewed from another applicant said that very few of its dermatological products, which are considered complex, had received approval in the first review cycle. In our review of FDA data, we also found that applications for drugs with certain routes of administration—the method by which the drug is taken, such as oral, topical, or intravenous—had different rates of approval in the first review cycle. In particular, from fiscal years 2015 through 2017, FDA reviewed generic drug applications for 41 ophthalmic and 20 transdermal drugs—types of drugs that FDA considers complex—and none of these applications received approval in the first review cycle. In contrast, generic drug applications for topical drugs, which FDA also identifies as complex, had higher approval rates. Specifically, our analysis found that the rate of approvals in the first review cycle for generic drug applications for topical drugs was 25 percent—more than double the rate for all applications included in our analysis. FDA officials stated that in recent years FDA released several product-specific guidances for topical drugs—technical guidance intended to help applicants identify the most appropriate methodology for developing certain drugs and generating the evidence needed to gain approval. FDA officials told us these guidances may have contributed to the higher rates of approval in the first review cycle for topical drugs. See table 2. Generic drug application priority review designation. In addition, we found that a generic drug application’s priority review status may affect the rate of approval in the first review cycle. FDA may grant priority review status to applications under several circumstances, including for the first generics of brand-name drugs and other designations, such as for drugs that could help address public health emergencies. Our analysis of FDA data found that the rates of approval in the first cycle were lower for applications for first generics than for applications with no priority designation—6 percent and 14 percent, respectively. One potential explanation for the relatively low rate of approval is that for a first generic, applicants have a financial incentive to be the first to submit an application to FDA. Officials from one trade association stated that applications for first generics may be of lower quality because the applicants are rushing to submit their applications. In other cases, priority designations were associated with higher first-cycle approval rates. First-cycle approval rates for applications with other types of priority designations were higher than for applications with no priority designation—18 percent for applications that were marked as priority for other reasons, such as drug shortages or public health emergencies. See table 3. FDA Made Changes That Could Increase the Rates of Approval for Generic Drugs in the First Review Cycle, but Opportunities Exist to Enhance Its Efforts FDA Has Taken Steps to Enhance Communication with Applicants and Improve Reviewer Consistency to Increase the Rate of Generic Drug Approvals in the First Review Cycle Our review of FDA guidance and regulations found that FDA has taken steps to enhance communication with applicants to increase the rate of generic drug application approvals in the first review cycle. Specifically, since the beginning of fiscal year 2013 in response to GDUFA, FDA increased communication with applicants prior to and during a generic drug application’s first review cycle consistent with its goal of helping applicants prepare approvable applications. These changes have included the following: Additional product-specific guidance. FDA has continued to release new and revised product-specific guidance to support a generic drug application’s approval within the first review cycle. Since GDUFA’s implementation and at the time of our review, FDA told us that it issued 993 new and revised product-specific guidance documents that describe acceptable methodologies for developing generic drugs and generating evidence needed to support a generic drug application’s approval. FDA officials indicated that product- specific guidance helps streamline both application development and review. Additional regulatory guidance. FDA has also issued regulatory guidance to communicate the agency’s expectations for the content and format of generic drug applications, which can facilitate approval in the first review cycle. In addition, in 2018, FDA issued draft guidance that described common application deficiencies and sought to promote approval during the first review cycle by providing recommendations on avoiding these recurring deficiencies. Presentations to industry. In presentations to applicants and others, FDA officials have presented information about generic drug application reviews and deficiencies frequently identified in generic drug applications. FDA has posted some of these presentation materials, including several video recordings, publicly on its website to share the information with industry. Communication during the review cycle. FDA has changed its review process to encourage reviewers to communicate with applicants at about the mid-point of the review cycle. FDA reviewers now aim to issue discipline review letters at about the mid-point of the first review cycle, rather than waiting until the end of the review cycle to communicate deficiencies. According to FDA, these earlier communications are intended to provide applicants with an opportunity to address issues before the end of the first review cycle and facilitate more approvals during that cycle. Assistance with applications for complex drugs. FDA has taken steps to assist applicants developing generics of complex drugs, such as drugs with complex active ingredients, formulations, or routes of administration. Beginning in fiscal year 2018 when GDUFA II was implemented, applicants developing complex drugs may request meetings with FDA prior to submitting a generic drug application and at the mid-point of the review cycle. In addition, FDA officials told us that since fiscal year 2013 when GDUFA was implemented, the agency has released 378 product-specific guidance documents focused on complex drugs. While all five applicants we interviewed generally described FDA’s efforts to increase communication as helpful, they offered opportunities for improvement. For example, four out of the five applicants said that product-specific guidance helps them understand exactly what the requirements are for product development, which can facilitate approval in the first review cycle. However, two of these applicants also said that FDA does not obtain sufficient input from the generic drug industry when developing product-specific guidance documents. To increase transparency, stakeholders said that FDA could solicit industry input to avoid proposing unrealistic guidance and one stakeholder suggested that FDA could create a workgroup to prevent unintended consequences following the implementation of new and revised guidance. However, FDA officials told us that draft guidance documents have a public comment period for stakeholders to provide comments on guidances before they are finalized. In December 2017, we similarly found that stakeholders indicated they would benefit from greater transparency in FDA’s process for developing guidance. We recommended that FDA publicly announce the agency’s plans for issuing new and revised product-specific guidance for nonbiological complex drugs within the next year. FDA agreed with this recommendation and published a website in April 2019 that provides information about upcoming product-specific guidance documents for complex generic drugs. In addition, all five applicants we interviewed said that FDA has improved communication with them, such as by increasing the frequency and timing of communications, and two applicants indicated that discipline review letters add predictability to the review process. (See app. I for more information on FDA’s changes to the generic drug application review process to improve communication with applicants.) While all five applicants we interviewed generally agreed that the increased communications would help increase rates of first-cycle approval, they also suggested that additional flexibility to communicate with FDA informally mid-cycle, such as by phone, could further facilitate the review process by helping applicants respond to questions or get clarity on questions included in the information requests or discipline review letters. FDA officials told us that applicants currently can request teleconferences with FDA, such as after receiving a complete response letter; however, they also noted that applicants generally prefer email communications and such teleconferences are not frequently utilized. FDA has also taken steps to improve consistency among reviewers. These steps could facilitate more approvals in the first review cycle because receiving consistent comments from FDA reviewers typically makes it easier for applicants to respond more quickly, which—according to some stakeholders—can result in approval in the first review cycle. These changes include the following: Creating review templates. Officials from FDA’s review disciplines have developed templates to guide reviewers through the generic drug application review process. According to FDA’s Manual of Policies and Procedures, these templates are intended to increase reviewers’ efficiency and improve assessment consistency. Developing common phrases. FDA also issued internal guidance on common phrases that reviewers may use to communicate generic drug application deficiencies in their comments to applicants during the first review cycle. For example, officials from the Division of Labeling Review said they maintain a database of common phrases and train reviewers on how to explain deficiencies to applicants. Officials explained that the Division of Labeling Review is also working toward pre-populating some parts of the review template to increase efficiency and consistency in the review process. Understanding the generic drug industry. FDA has taken steps to increase reviewers’ and applicants’ common understanding of industry practices and FDA review standards, such as through visits to manufacturing facilities, to improve the quality and consistency of reviewers’ comments in the first review cycle. However, all five applicants we interviewed noticed inconsistency among reviewers. FDA officials and two applicants suggested that this may be because FDA reviewers have different professional backgrounds. One applicant noted that some reviewers benefit from visiting manufacturing facilities if they do not have prior experience in the generic drug industry. Officials from a trade association said that such steps improve applicants’ understanding of what FDA reviewers are looking for in generic drug applications and may enhance their ability to submit applications that are approvable in the first review cycle. Opportunities Exist to Enhance FDA’s Efforts to Increase the Rates of Approval for Generic Drugs in the First Review Cycle While stakeholders stated that the changes FDA made to improve reviewer consistency were positive, they noted that inconsistency among reviewers still remained, and we also found inconsistencies among reviewers. In addition, while stakeholders we interviewed raised concerns that the timing of brand-name labeling changes could affect whether applications were approved in the first review cycle, FDA has not taken steps to assess the validity of these concerns. Inconsistency among FDA reviewers. While FDA has taken steps to improve consistency among generic drug application reviewers, stakeholders noted that inconsistencies persist, and these inconsistencies may influence whether an application is approved during the first review cycle. For example, most stakeholders we interviewed (three out of five trade associations and four out of five applicants) indicated that they were aware of examples when different FDA reviewers within the same review discipline provided substantively different assessments of similar generic drug applications, specifically by requesting additional information from applicants for some applications and not others. For example, one applicant cited an example of two similar topical drugs whose applications relied on the same data set. The reviewer for one application required additional data, while the reviewer for the other application did not. To improve consistency, four applicants we interviewed suggested that FDA improve its reviewer training and one suggested that FDA create a workgroup to examine and address inconsistencies among reviewers. Four of the five applicants we interviewed also reported variation in the consistency of reviewers’ comments, including a lack of clarity in the information required for the applicant to address the comments. For example, one applicant said that they have received comments where reviewers did not specify what further information was required, and added that comments that suggest specific resolutions are extremely helpful, which would be consistent with FDA’s Manual of Policies and Procedures. This manual describes a standard process for FDA reviewers to use when assessing the completeness of generic drug applications, including clearly communicating with applicants about deficiencies that must be corrected for their applications to be approved in order to reduce the number of review cycles. According to the manual, primary reviewers are responsible for assessing whether applications meet the regulatory requirements for approval, while secondary reviewers are responsible for ensuring consistency among assessments and quality of communications to the applicant. Further, the manual advises primary and secondary reviewers to ensure that comments to applicants about deficiencies include similar content. The manual also indicates that comments should include the following four elements: (1) refer to a specific location within the generic drug application; (2) identify the omitted information or explain the problem with the information submitted; (3) explain the actions necessary to resolve the deficiency; and (4) explain why the information or revision is needed. Finally, FDA provides reviewers with plain language writing guidelines and other writing resources to support the development of clear messages for external communications. In our review of documentation from the first review cycle for 35 generic drug applications, we found variation in the clarity and specificity of some reviewer comments that may have influenced the outcome of the first review cycle. For example, some discipline review letters included clear descriptions of potential remedies for some deficiencies, while others did not clearly describe the deficiency or FDA’s expectations for an approvable generic drug application. Of the 35 generic drug applications, we conducted a more detailed review of four applications from fiscal year 2018—two that received approval in the first review cycle and two that did not. In the two applications that were not approved in the first review cycle, we identified 32 instances in which the comments did not fully meet FDA’s criteria. Two of the four applications we reviewed in more detail had similar numbers and types of deficiencies identified in the discipline review letters sent to the applicants mid-way through the first review cycle, such as quality deficiencies related to the drug substance and drug product, but the clarity and content of the comments included in the discipline review letters varied considerably. For one of the applications, the comments were written clearly and, consistent with FDA’s Manual of Policies and Procedures, identified options for addressing the deficiencies. For the other application, the comments were less clear and did not clearly identify ways to address the deficiencies. The applicant of the generic drug application with clearly written comments resolved all of the deficiencies raised in their discipline review letter and the generic drug application was approved in the first review cycle. In contrast, the applicant of the other application did not resolve all deficiencies raised in their discipline review letter and was not approved in the first cycle. See Table 4 for more detail on these two examples. Inconsistency among reviewers could affect the rate of approvals in the first review cycle if comments provided to applicants differ in content or are not clearly communicated. For example, if some reviewers provide unclear comments, it could be more difficult for the applicant to address deficiencies in a timely manner, while applicants that received clear comments could potentially address deficiencies within the first review cycle. This could delay some generic drugs from entering the market if applicants require more time, including potentially additional FDA review cycles, to understand and respond to unclear comments. This has potential impacts on patient access to generic drugs and on applicants’ abilities to effectively manage their expectations for when their generic drug applications will be approved. FDA officials explained that although secondary reviewers are experienced, they do not consistently receive additional training to ensure clarity and consistency among primary reviewers. They noted that FDA offers training in clear writing for FDA employees but it is not required for reviewers. FDA managers noted that some inconsistency among reviewers may persist due to various factors such as tenure with the agency and different professional backgrounds or interpretations of the generic drug application information. They also said that standardizing reviewers’ writing is challenging since each reviewer might have his or her own writing style and scientific expertise. Two applicants and one trade association we interviewed also said that the length of reviewers’ tenure with the agency could impact the substance of their comments in information requests and discipline review letters and the likelihood of the applicant attaining approval in the first review cycle. For example, one trade association we interviewed said that inexperienced reviewers typically request information from applicants that a more experienced reviewer would already know, such as information about drug manufacturing facilities. Unknown effects of labeling changes. Three applicants we interviewed noted that they believe FDA could improve the rate of approvals during the first review cycle if they took steps to mitigate delays that stakeholders said result from brand-name drug labeling changes that occur mid-cycle. Because generic drug labels generally must match brand-name labels, most applicants we interviewed said that changes made by brand-name drug companies to the labeling of drugs during the review process can delay or prevent approval of generic drugs in the first review cycle because the applicant of the generic drug would likely need to update the label before it is approved. In addition, five of the 10 stakeholders we interviewed said they believe such labeling changes negatively impact the rate of first-cycle approvals, and three said they believe that brand-name companies may strategically time updates to a brand-name drug’s labeling to occur right before the approval of a generic competitor in order to delay generic drug approvals. Although our review of 35 selected applications did not identify examples where labeling changes made during the first review cycle were the only factor that prevented approval, we identified two instances where labeling changes were among multiple factors that prevented approval. Specifically, we identified two applications for which the complete response letters noted recent changes in the brand-name drug’s labeling as one of multiple factors that contributed to the generic drug application’s failure to receive approval in the first review cycle. One of these applications had successfully addressed several labeling deficiencies during the first review cycle, but received a complete response letter that included new labeling deficiencies because of recent changes in the brand-name drug’s labeling. Three applicants and one trade association identified labeling changes as a concern during our interviews. Two of these applicants and the trade association suspected updates are strategically timed to delay generic drug approvals. However, FDA does not know whether there is validity to these concerns because it has not conducted analysis that would enable it to assess their validity. FDA officials noted that they were aware of these types of concerns, but thought it would be difficult for brand-name drug companies to successfully time changes in their drugs’ labeling to affect applications under review, and that labeling changes for brand- name drugs must be justified, for example, to note an adverse reaction identified after approval. However, FDA officials also acknowledged that there may be an incentive for brand-name drug companies to change the label on a drug frequently to make it more difficult for a generic drug application to be approved. FDA officials stated that the Office of Generic Drugs does not assess how often brand-name companies change their labeling or track how often such labeling changes occur because such changes are reviewed by the Office of New Drugs. Further, while FDA officials noted that the two offices have coordinated on the review of some labeling changes, they stated that they do not coordinate on the timing of approval of brand- name drug label changes. This is inconsistent with federal internal control standards, which state that agencies should identify risks that affect their defined objectives and use quality information to achieve these objectives, including by identifying the information required to achieve the objectives and address related risks. In addition, FDA can approve a generic drug application even though changes have been made in the brand-name drug labeling that the applicant has not incorporated into its proposed labeling, provided the applicant meets certain criteria; however, FDA officials told us that applications rarely meet the required criteria. Conducting an assessment of the extent to which the timing of such changes affect the approval of generic drugs in the first cycle of review would provide FDA with the necessary information to respond to stakeholder concerns and take action, as appropriate, such as by coordinating with the Office of New Drugs on this issue. To the extent that brand-name companies’ labeling changes are creating unnecessary delays in generic drug approval, such delays may impede generic drug entry into the market, which would be inconsistent with FDA’s stated goals of speeding up generic drug reviews. Conclusions The timely approval of safe generic drugs in FDA’s first review cycle can provide substantial cost savings to patients and third-party payers. Since the enactment of GDUFA, FDA has taken steps to help applicants submit stronger generic drug applications and correct deficiencies within the first review cycle. However, according to FDA its most recent analysis found that the average generic drug application required three cycles of review before approval. Opportunities exist to enhance FDA’s efforts to increase the rates of approval for generic drugs in the first review cycle, including improving the consistency and clarity of reviewer comments and assessing the effects of the timing of brand-name companies’ changes to labeling. Taking such steps could help FDA meet the agency’s goals of minimizing the number of review cycles necessary for generic drug application approval and increasing the overall rate of approval, including within the first review cycle. Increasing the rate of approval in the first review cycle, while maintaining the efficacy and safety of generic drugs, can expand consumer access to relatively lower cost medications and has the potential to save patients and third-party payers billions of dollars. Recommendations for Executive Action We are making the following two recommendations to FDA. The Commissioner of FDA should take additional steps to address inconsistency in its written comments to generic drug applicants— including the clarity of writing and the content of comments—among reviewers, such as requiring additional training for reviewers. (Recommendation 1) The Commissioner of FDA should assess the extent to which the timing of brand-name drug companies’ drug labeling changes affect the approval of generic drug applications in the first review cycle, and take steps, as appropriate, to limit the effect of brand-name drug labeling changes on pending generic drug applications. (Recommendation 2) Agency Comments We provided a draft of this report to HHS for review and comment. In its written comments, which are reproduced in appendix II, HHS concurred with our recommendations. HHS stated that it will take steps to improve the clarity and content of primary reviewers’ comments by, for example, providing training on written communication. Additionally, HHS stated that it will take steps to assess examples in which a brand-name drug company labeling change impacted the timeline of a generic drug approval and assess what actions could address this issue. In addition, HHS provided technical comments, which we incorporated as appropriate. We are sending copies of this report to the congressional addresses, the Secretary of Health and Human Services, and other interested parties. In addition, the report is available at no charge on the GAO website at http://www.gao.gov. If you or your staff have any questions about this report, please contact me at 202-512-7114 or dickenj@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are listed in appendix III. Appendix I: Changes to the Generic Drug Review Process to Improve Communications with Applicants The Food and Drug Administration (FDA) made several changes to its review process for generic drug applications in its implementation of the Generic Drug User Fee Amendments of 2017 (GDUFA II) that are intended to improve communications with applicants, such as drug companies, and included the following: Additional communication with applicants. In its GDUFA II Commitment Letter to Congress, 1. FDA committed to notify applicants of potential deficiencies in an application that could prevent approval in the first review cycle through information requests or discipline review letters by about the mid-point of the review cycle; and 2. FDA committed to continue to issue additional information requests or discipline review letters late in the review cycle as needed, and, in certain circumstances, to work beyond the review timeframe to issue an approval. Pre-submission facility correspondence. In its GDUFA II Commitment Letter to Congress, FDA committed to communicating with applicants of priority generic drug applications before the application is submitted. For priority generic drug applications, FDA accepts information about facilities associated with an application, such as manufacturing facilities, at least 2 months before the application is submitted. If FDA finds the pre-submission facility correspondence includes complete and accurate information, the application may receive a review timeline of 8 months, rather than 10. Figure 2 provides an overview of the timeline for the first review cycle for generic drug applications under the Generic Drug User Fee Amendments of 2012 (GDUFA) and the revised review process under GDUFA II. Appendix II: Comments from the Department of Health and Human Services Appendix III: GAO Contact and Staff Acknowledgments GAO Contact Staff Acknowledgments In addition to the contact named above, Gerardine Brennan (Assistant Director), Rebecca Rust Williamson (Analyst-in-Charge), Caroline Hale, and Elizabeth Leibinger made key contributions to this report. Also contributing were Kaitlin Farquharson, Cathy Hamann, Dan Lee, Laurie Pachter, and Vikki Porter.