Background FAR Part 15 describes negotiated contracting, which includes the use of several competitive source selection processes. The processes are associated with the best value continuum, which includes the LPTA process on one end and the trade-off process on the other (see figure 1). Federal agencies may elect to use the LPTA process where the requirement is clearly defined and the risk of unsuccessful contract performance is minimal. In such cases, agencies can determine that cost or price should play a dominant role in the source selection. When using the LPTA process, the agency specifies the evaluation factors that establish the requirements of acceptability in the solicitation. Firms submit their proposals and the agency determines which of the proposals meet those requirements. No trade-offs between cost or price and non-cost factors (for example, technical capabilities or past performance) are permitted. Non-cost factors are rated on an acceptable or unacceptable basis. The award is made based on the lowest priced, technically acceptable proposal submitted to the government. In contrast, agencies may elect to use the trade-off process in acquisitions where the requirement is less definitive, more development work is required, or the acquisition has a greater performance risk. In these instances, non-cost factors may play a dominant role in the source selection process. Trade-offs between price and non-cost factors allow agencies to accept other than the lowest priced proposal. The FAR requires the solicitation to state whether all evaluation factors other than cost or price, when combined, are significantly more important than, approximately equal to, or significantly less important than cost or price. Contracting officials have broad discretion in the selection of the evaluation criteria that will be used in an acquisition. When one is required, a written acquisition plan generally should include a description of the acquisition’s source selection process and the relationship of the evaluation factors to the acquisition objectives. The FAR does not explicitly require contracting officials to document the reasons why the specific source selection process was chosen. The defense and civilian provisions required the DFARS and FAR, respectively, be revised to require that the LPTA process only be used if certain criteria are met, as described in table 1. The defense and civilian provisions also required that the use of the LPTA process be avoided, to the maximum extent practicable, in procurements that are predominantly for the products and services identified in table 2. The FAR and DFARS Rulemaking Process The process for revising the FAR and DFARS is governed by statute, which generally requires agencies to issue a proposed rule in the Federal Register. Agencies are also required to provide at least a 30-day public comment period following publication of the proposed rule. Figure 2 illustrates the basic process that is generally used to revise the FAR and the DFARS. Recent Reports on DOD’s Use of the LPTA Process We have issued two reports in response to the defense provisions requiring us to review DOD’s use of the LPTA process. In November 2017, we found that the Army, Navy, and Air Force used the LPTA process for information technology and other services in 9 out of 133 instances when awarding contracts valued at $10 million or more in the first half of fiscal year 2017. Contracting officials stated that the LPTA process was used in these instances, in part, because the requirements were well-defined, noncomplex, or recurring. We also found that contracting officials’ use of the LPTA process was generally consistent with the criteria listed in the defense provisions. In November 2018, we estimated that about 26 percent of DOD’s contracts and orders valued at $5 million or more in fiscal year 2017 were competitively awarded using the LPTA process. We found that DOD used the LPTA process to buy equipment, fuel, information technology services, and construction services, among other things. We also found that contracting officials used the LPTA process for reasons consistent with the criteria in the defense provisions. Specifically, contracting officials associated with the 14 contracts and orders we selected used the LPTA process, in part, because they determined there was no trade-off available or determined that DOD would not derive any benefit from paying a premium for offers that exceeded the minimum capabilities. Finally, we found that some contracting officials were confused about how to apply two of the criteria included in the defense provisions. Specifically, contracting officials were confused regarding how to assess life cycle costs associated with their procurements (shown as criterion 6 in table 1) or whether the products and services they were acquiring would be considered expendable in nature (criterion 8). Absent clarification on how to consider these two criteria, we found there was potential for increased risk that DOD contracting officials would not consistently apply the criteria of the defense provisions. Accordingly, we recommended that DOD address how contracting officials should apply these two criteria when using the LPTA process. DOD concurred with our recommendations, and plans to address them by issuing guidance concurrent with publication of the final rule at the end of fiscal year 2019. Status of Revisions to Regulations Addressing Use of the LPTA Process Defense and Civilian Agencies’ Revisions to the DFARS and the FAR In December 2018, DOD issued a proposed DFARS rule for public comment to address the defense provisions for using the LPTA process. The December 2018 proposed rule reflected the criteria and limitations for using the LPTA process set forth in the defense provisions, and provided further clarification that these provisions were applicable to both contracts and orders. The public comment period ended on February 4, 2019, during which time the Defense Acquisition Regulations Council received 15 comments. In commenting on the proposed rule, industry representatives generally indicated their support for the proposed rule. On June 19, 2019, the Council agreed to move forward with the process for issuing a final rule revising the DFARS. Defense Pricing and Contracting officials stated that DOD expects to finalize the rule by the end of fiscal year 2019. The time required to develop and finalize the revisions to the DFARS has been longer than provided for under the NDAA for fiscal year 2017, which required the DFARS be revised within 120 days after enactment, which would have been in April 2017. In July 2019, we found that it can take a year or longer to issue a final DFARS rule. For this DFARS case, a Defense Pricing and Contracting official cited several reasons why the revisions have been delayed, including the need to address LPTA-related provisions in two separate NDAAs and the need to resolve a backlog of DFARS changes. In addition to ongoing efforts to update DFARS regulations, DOD officials plan to update the DFARS Procedures, Guidance and Information to provide defense contracting officers with supplemental guidance on applying the new criteria for using the LPTA process. A Defense Pricing and Contracting official stated that this update would be finalized by the end of fiscal year 2019 to coincide with the issuance of the final DFARS rule. The FAR Council has also initiated efforts to incorporate the civilian provisions for using the LPTA process into the FAR. The NDAA for Fiscal Year 2019 required that the FAR be revised to incorporate the civilian provisions within 120 days after enactment, which would have been in December 2018. Officials from the Office of Federal Procurement Policy told us, however, that it generally takes much longer than 120 days to revise the FAR. According to an analysis provided by DOD, it takes 483 days on average to issue a FAR rule. The FAR case to implement the civilian LPTA provisions was initiated in August 2018—the same month the NDAA for Fiscal Year 2019 was enacted. Office of Federal Procurement Policy officials stated that a proposed FAR rule is scheduled to be published in the Federal Register in September 2019. The public comment period for the proposed rule is scheduled to end in November 2019. Figure 3 shows when the defense and civilian provisions were enacted, when the rules were required to be implemented, and some of the efforts associated with revising both the DFARS and the FAR. Current Agency Guidance for Using LPTA Of the six agencies we reviewed, we found that DOD and DHS had existing source selection guidance that already reflected some of the criteria for using the LPTA process identified in the defense and civilian provisions. The other four civilian agencies did not have source selection guidance specific to using the LPTA process. Table 3 shows the status of selected agencies’ existing guidance related to the LPTA process. We found the following: DOD’s March 2016 Source Selection Procedures generally includes five of the eight criteria for using the LPTA process. A Defense Pricing and Contracting official stated that this guidance could be updated after the DFARS rule is implemented and the Procedures, Guidance, and Information resource is updated. The DHS September 2013 Source Selection Guide generally includes the first four of the six criteria for using the LPTA process. DHS officials stated that they plan to update their guidance after the FAR is amended to reflect the criteria and limitations for using LPTA. Acquisition policy officials from VA, GSA, USDA, and HHS stated that they do not have agency-specific guidance for using the LPTA process beyond what is currently provided for under the FAR. These officials stated that they were waiting for regulations to be finalized before determining if there is a need to develop any new guidance. DOD Used the LPTA Process More Frequently Than Selected Civilian Agencies in Fiscal Year 2018 Based on the results of our generalizable samples, we estimate that the selected DOD components used the LPTA process for about 25 percent of competitive contracts and orders valued at $5 million or more in fiscal year 2018, compared to about 7 percent of such contracts and orders at selected civilian agencies, as shown in Table 4. Our findings regarding how often DOD uses the LPTA process are consistent with what we found in our prior work. In November 2018, for example, we reported that Army, Navy, Air Force, and DLA awarded about 26 percent of contracts and orders using the LPTA process in fiscal year 2017. In November 2017, we reported that officials told us the LPTA process was used in instances where the requirements were well- defined, noncomplex, or recurring. This is the first year we were required to evaluate civilian agencies’ use of the LPTA process. Civilian agency officials we interviewed provided various perspectives on the extent to which their agency used the LPTA process. HHS officials told us that their acquisitions are generally complex, so the LPTA process is not often deemed the appropriate mechanism for determining best value. USDA officials told us that they have few acquisitions valued at more than $5 million, and that those acquisitions are likely to have more complex requirements. In such cases, the officials told us, technical and performance considerations generally would be more important than price factors. In analyzing FPDS-NG data, we found that 1 percent of USDA’s fiscal year 2018 contracts and orders were valued at more than $5 million. GSA officials told us their agency often procures services where it is beneficial for industry to propose solutions to a stated need, rather than GSA dictating the solution, such as professional services or information technology systems for a secure network solution. In these cases, officials said they would not have the technical specifications that an LPTA process would require. Officials from DHS and VA stated that they do not centrally track the source selection method used and they do not have sufficient information to say why their agencies use LPTA less frequently than other source selection methods. Within the sample of contracts we reviewed, we found DOD and the five selected civilian agencies bought a variety of products and services using the LPTA process in fiscal year 2018 (see table 5). We found that four of these DOD contracts and orders and one civilian agency order were for services that could be considered within the categories for which the defense and civilian provisions place limitations on, but do not prohibit, use of the LPTA process. In November 2018, we found that DOD contracting officers generally justified the use of the LPTA process for products and services in these categories. As described earlier in this report, the DFARS and FAR are in the process of being revised and do not currently address the limitations on the use of LPTA for these products and services. Agency Comments We provided a draft of this report to OFPP, DOD, VA, HHS, GSA, DHS, and USDA for review and comment. OFPP, DOD, GSA, DHS and HHS provided technical comments, which we incorporated as appropriate. VA and USDA told us that they had no comments on the draft report. We are sending copies of this report to the appropriate congressional committees, the Director of the Office of Management and Budget, the Secretary of Defense, the Administrator of General Services, the Secretary of Veterans Affairs, the Secretary of Homeland Security, the Secretary of Agriculture, and the Secretary of Health and Human Services. In addition, the report is available at no charge on the GAO website at http://www.gao.gov. If you or your staff have any questions about this report, please contact me at (202) 512-4841 or dinapolit@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are listed in appendix II. Appendix I: Objectives, Scope, and Methodology Section 813 of the National Defense Authorization Act (NDAA) for Fiscal Year 2017, as amended, included a provision that we report on the number of instances where Department of Defense (DOD) used the lowest price technically acceptable (LPTA) process for contracts exceeding $5 million, as well as provide an explanation of how acquisition officials considered the new criteria in making a determination to use the LPTA process. We have previously issued two reports in response to this provision. Subsequently, Section 880 of the NDAA for Fiscal Year 2019 included a provision that we report on the number of instances where civilian agencies used the LPTA process for contracts exceeding $5 million, as well as provide an explanation of how acquisition officials considered the six criteria in making a determination to use the LPTA process. This report, which addresses both provisions, describes (1) the status of regulatory changes required by the defense and civilian provisions for using the LPTA process and (2) the extent to which DOD and selected civilian agencies used the LPTA process to competitively award contracts and orders valued at $5 million or more in fiscal year 2018, and what they bought using this process. To address both objectives and select the DOD components and civilian agencies included in our scope, we used data from the Federal Procurement Data System-Next Generation (FPDS-NG) to identify the population of DOD and civilian agency contracts and orders that were reported as competitively awarded and valued at $5 million or more in fiscal year 2018. For DOD, we focused our review on the top four DOD components—Army, Navy, Air Force, and Defense Logistics Agency (DLA)—because they accounted for about 5,400—or about 88 percent— of all DOD contracts and orders valued at $5 million or more that were reported as competitively awarded in fiscal year 2018. Similarly, we focused our analysis on the top five civilian agencies—the Departments of Veterans Affairs (VA), Health and Human Services (HHS), Homeland Security (DHS), and Agriculture (USDA) and the General Services Administration (GSA)—which accounted for about 3,000—or about 66 percent—of all civilian agency contracts and orders valued at $5 million or more that were reported as competitively awarded in fiscal year 2018. To describe the status of regulatory changes governing the use of the LPTA process, we obtained information on agency officials’ efforts to amend the Defense Federal Acquisition Regulation Supplement (DFARS) and the Federal Acquisition Regulation (FAR). To do this, we met with DOD and Office of Federal Procurement Policy officials responsible for overseeing the regulatory changes. We also reviewed DOD’s December 2018 proposed rule to revise the DFARS and the 15 public comments DOD received on the proposed rule. Because revisions to the FAR and DFARS have not been finalized, regulations do not yet require or provide guidance to acquisition officials on how to consider the new criteria. Therefore, we also analyzed agency guidance and interviewed acquisition and contracting policy officials at DOD and each of the selected civilian agencies to determine whether agencies had existing guidance that addressed the defense and civilian provisions, in whole or in part. Specifically, we reviewed agency-specific source selection guidance from DOD, DHS, and VA. GSA, USDA, and HHS do not have source selection guidance that specifically addresses the LPTA process. According to officials, DOD and the selected civilian agencies do not maintain centralized data on whether the LPTA process is used to award contracts and orders. Consequently, to describe the extent to which DOD and civilian agencies used the LPTA process in competitively awarded contracts and orders valued at $5 million or more in fiscal year 2018, we used FPDS-NG to select two generalizable random samples of contracts and orders to estimate the use of LPTA by the DOD components and the civilian agencies within our scope. This resulted in samples of 102 contracts and orders for the four selected DOD components and 100 for the five selected civilian agencies. We removed five contracts and orders from our DOD sample: two contracts and one order because they were incorrectly reported by the agency in FPDS-NG as having been competitively awarded, and two contracts because they were classified. We removed three contracts and orders from our civilian agency sample: two orders because they were incorrectly reported by the agency in FPDS-NG as having been competitively awarded, and one contract because it was incorrectly reported as having an estimated value of more than $5 million. After removing these contracts and orders, our generalizable sample consisted of 97 DOD contracts and orders and 97 civilian agency contracts and orders. For each contract and order in our sample, we requested that the selected agencies identify whether the LPTA process was used. We independently verified agency responses by reviewing the solicitations for each of the contracts and orders within our two samples. We also verified relevant FPDS-NG data on estimated value and competition using agency-provided documentation for the contracts and orders we reviewed. Based on this, we determined these data were sufficiently reliable for us to estimate the percentage of contracts and orders valued at $5 million or more that the four components within DOD and the five selected civilian agencies competitively awarded in fiscal year 2018 using the LPTA process. We also used FPDS-NG product and service codes to identify whether the LPTA contracts and orders in our sample could be considered to be within one of the categories that the defense and civilian provisions direct agencies to avoid use of the LPTA process to the maximum extent practicable. The regulatory changes required by the defense and civilian provisions are not yet in place, and the defense and civilian provisions do not explicitly prohibit use of the LPTA process to acquire these categories of products and services. Therefore, we did not evaluate the reasons why an agency may have used the LPTA process in these instances. The findings based on our review of the product and services codes for the LPTA contracts and orders in our sample are not generalizable. We conducted this performance audit from February 2019 to September 2019 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives. Appendix II: GAO Contact and Staff Acknowledgments GAO Contact Staff Acknowledgments In addition to the contact named above, Justin Jaynes (Assistant Director), Heather B. Miller (Analyst-in-Charge), Sarah Cantatore, Matthew T. Crosby, Lorraine Ettaro, Lori Fields, Stephanie Gustafson, Julia Kennon, Sarah Martin, Alyssa Weir, and Khristi Wilkins made key contributions to this report.


Background FBI’s Use of Face Recognition Technology For decades, fingerprint analysis was the most widely used biometric technology for positively identifying arrestees and linking them with any previous criminal record. However, beginning in 2010, the FBI began incrementally replacing the Integrated Automated Fingerprint Identification System (IAFIS) with Next Generation Identification (NGI). NGI was not only to include fingerprint data from IAFIS and biographic data, but also to provide new functionality and improve existing capabilities by incorporating advancements in biometrics, such as face recognition technology. As part of the fourth of six NGI increments, the FBI updated the Interstate Photo System (IPS) to provide a face recognition service that allows law enforcement agencies to search a database of criminal photos that accompanied fingerprint submissions using a photo of an unknown person—called a probe photo. The FBI began a pilot of NGI-IPS in December 2011, and NGI-IPS became fully operational in April 2015. NGI-IPS users include the FBI and selected state and local law enforcement agencies, which can submit search requests to help identify an unknown person using, for example, a photo from a surveillance camera. When a state or local agency submits such a photo, NGI-IPS uses an automated process to return a list of candidate photos from the database. The number of photos returned ranges from 2 to 50 possible candidate photos from the database, depending on the user’s specification. According to the FBI, in fiscal year 2018, NGI-IPS returned about 50,000 face recognition search results to law enforcement agency users, a decrease from about 90,000 search results in fiscal year 2017. Figure 1 describes the process for a search requested by state or local law enforcement. In addition to the NGI-IPS, the FBI has an internal unit called Facial Analysis, Comparison and Evaluation (FACE) Services that provides face recognition capabilities, among other things, to support active FBI investigations. FACE Services has access to NGI-IPS, and can also search or request to search databases owned by the departments of State and Defense and 21 states, which use their own face recognition systems. Figure 2 shows which states partnered with the FBI for FACE Services requests, as of May 2019, according to the FBI. Unlike NGI-IPS, which primarily contains photos obtained from criminal justice sources, these external systems primarily contain photos from state and federal government databases, such as driver’s license photos and visa applicant photos. According to the FBI, the total number of face photos available in all searchable repositories for FACE Services is over 641 million. Biometric images specialists for FACE Services manually review any photos received from their external partners before returning a photo as an investigative lead to the requesting FBI agents. No more than two photos are returned as a lead after the specialist for FACE Services completes the review. However, according to FACE Services officials we met with during our May 2016 review, if biometric images specialists determine that none of the databases returned a likely match, they do not return any photos to the agents. According to the FBI, from August 2011 (when searches began) through April 2019, FACE Services received 153,636 photos of unknown persons (often called probe photos) from FBI headquarters, field offices, and overseas offices, which resulted in 390,186 searches of various databases in attempt to find photo matches of known individuals in these databases. Privacy Laws and Responsibilities at DOJ Federal agency collection and use of personal information, including face images, is governed primarily by two laws: the Privacy Act of 1974 and the privacy provisions of the E-Government Act of 2002. The Privacy Act places limitations on agencies’ collection, disclosure, and use of personal information maintained in systems of records. The Privacy Act requires that when agencies establish or make changes to a system of records, they must notify the public through a system of records notice (SORN) in the Federal Register. According to Office of Management and Budget (OMB) guidance, the purposes of the notice are to inform the public of the existence of systems of records; the kinds of information maintained; the kinds of individuals on whom information is maintained; the purposes for which they are used; and how individuals can exercise their rights under the Privacy Act. The E-Government Act of 2002 requires that agencies conduct Privacy Impact Assessments (PIAs) before developing or procuring information technology (or initiating a new collection of information) that collects, maintains, or disseminates personal information. The assessment helps agencies examine the risks and effects on individual privacy and evaluate protections and alternative processes for handling information to mitigate potential privacy risks. OMB guidance also requires agencies to perform and update PIAs as necessary where a system change creates new privacy risks, for example, when the adoption or alteration of business processes results in personal information in government databases being merged, centralized, matched with other databases or otherwise significantly manipulated. Within DOJ, preserving civil liberties and protecting privacy is a responsibility shared by departments and component agencies. As such, DOJ and the FBI have established oversight structures to help protect privacy and oversee compliance with statutory and policy requirements. For example, the FBI drafts privacy documentation for its face recognition capabilities, and DOJ offices review and approve key documents developed by the FBI—such as PIAs and SORNs. DOJ and FBI Have Taken Steps Since May 2016 to Better Ensure Privacy but Work Remains to Fully Address Prior Recommendations DOJ Has Taken Steps to More Quickly Publish Privacy Impact Assessments but Has Not Fully Implemented Its Revised Process We reported in May 2016 that the FBI did not (1) update the NGI-IPS PIA in a timely manner when the system underwent significant changes, or (2) develop and publish a PIA for FACE Services before that unit began supporting FBI agents. However, DOJ and the FBI have since taken steps to review and publish PIAs more quickly. As discussed in our 2016 report, consistent with the E-Government Act and OMB guidance, DOJ developed guidance that requires initial PIAs to be completed at the beginning of development of information systems and any time there is a significant change to the information system in order to determine whether there are any resulting privacy issues. In accordance with this guidance, FBI published a PIA at the beginning of the development of NGI-IPS in 2008, as required. However, the FBI did not publish a new PIA or update the 2008 PIA before beginning to pilot NGI-IPS in December 2011 or as significant changes were made to the system through September 2015. During the pilot, the FBI used NGI- IPS to conduct over 20,000 searches to assist in investigations. Similarly, DOJ did not approve a PIA for FACE Services when it began supporting investigations in August 2011. As a new use of information technology involving the handling of personal information, it too required a PIA, according to the E-Government Act, as well as OMB and DOJ guidance. Figure 3 provides key dates in the implementation of these face recognition capabilities and the associated PIAs. DOJ approved the NGI-IPS PIA in September 2015 and the FACE Services PIA in May 2015—over 3 years after the NGI-IPS pilot began and FACE Services began supporting FBI agents with face recognition services. Among other factors, implementation of the NGI-IPS pilot constituted a significant change in the FBI’s use of the technology that, consistent with the E-Government Act and OMB guidance required DOJ/FBI to update the PIA. Similarly, DOJ/FBI acknowledged that FACE Services began supporting FBI investigations in 2011, which involved storing photos in a new work log and also performing automated searches instead of manual searches. As a new use of information technology involving the handling of personal information, it too required a PIA. While DOJ and the FBI updated the internal drafts of these PIAs, the public remained unaware of the department’s consideration for how the FBI uses personal information in the face recognition search process. Given the issues we identified, we recommended that DOJ assess the PIA development process to determine why PIAs were not published prior to using or updating face recognition capabilities. Although DOJ officials did not concur with this recommendation, they did agree that all DOJ processes may be reviewed for improvements and efficiencies. In November 2018, DOJ officials told us that they had reviewed the PIA development process and determined that one reason the FBI’s face recognition PIAs were not completed more quickly was because the FBI and DOJ engaged in an extensive PIA revision process. As a result, DOJ reported that it implemented a pilot in 2018 to expedite the PIA approval process, which included developing a PIA approval template, conducting DOJ’s review earlier in the process, and focusing the review solely on legal sufficiency instead of a more comprehensive review that included less significant editorial changes. According to DOJ, this new process has significantly reduced the time required between the completion of the PIA process by the FBI and the review by DOJ. Further, DOJ reported that it has applied the same process to other DOJ components since December 2018, and that the pilot is evolving into an operational process. We will continue to monitor DOJ’s implementation of its review process changes. DOJ Did Not Complete a SORN Addressing FBI’s Face Recognition Capabilities in a Timely Manner and Has Not Implemented Corrective Actions We reported in May 2016 that DOJ did not publish a SORN, as required by the Privacy Act, that addresses the collection and maintenance of photos accessed and used through the FBI’s face recognition capabilities, in a timely manner. The DOJ published the SORN on May 5, 2016—after completion of our review—even though those capabilities were in place since 2011. According to OMB guidance then in effect, the SORN “must appear in the Federal Register before the agency begins to operate the system, e.g., collect and use the information.” However, from 2011 through May 2016, the agency collected and maintained personal information for these capabilities without the required explanation of what information it was collecting or how it was used. For example, at the time of our review, the existing version of the SORN that covered FBI’s face recognition capabilities was dated September 1999. According to DOJ officials, it did not address the collection and maintenance of photos accessed and used through NGI for the FBI’s face recognition capabilities but rather discussed fingerprint searches. Given that DOJ did not publish the SORN in a timely manner, we recommended DOJ develop a process to determine why a SORN was not published for the FBI’s face recognition capabilities prior to using NGI-IPS, and implement corrective actions to ensure SORNs are published before systems become operational. DOJ agreed, in part, with our recommendation and submitted the SORN for publication after we provided our draft report for comment. According to DOJ, it continues to review and update its pre-existing SORNs on an ongoing basis and is continually improving the scope and efficiency of its privacy processes. However, as of May 2019, DOJ had not taken actions to address our recommendation. Further, in April 2019, DOJ stated that with respect to transparency, a published PIA will provide much the same information that would be contained in a SORN and may provide it in a timelier manner. However, according to OMB guidance, the purpose of the SORN is to inform the public of the existence of systems of records; the kinds of information maintained; the kinds of individuals on whom information is maintained; the purposes for which they are used; and how individuals can exercise their rights under the Privacy Act. Further, PIAs and SORNs both contain information key to providing the public with information about the collection of their personal information, among other things. We continue to believe that by assessing the SORN development process and taking corrective actions to ensure timely development of future SORNs, DOJ would be better positioned to provide the public with a better understanding of how personal information is being used and protected by DOJ components. FBI Has Conducted Audits to Oversee the Use of NGI-IPS and FACE Services The Criminal Justice Information Services Division (CJIS), which operates FBI’s face recognition capabilities, has an audit program to evaluate compliance with restrictions on access to CJIS systems and information by its users, such as the use of fingerprint records. However, at the time of our May 2016 review, it had not completed audits of the use of NGI-IPS or FACE Services searches of external databases. We reported that state and local users had been accessing NGI-IPS since December 2011 and had generated IPS transaction records since then that would enable CJIS to assess user compliance. In addition, we found that the FACE Services Unit had used external databases that included primarily civil photos to support FBI investigations since August 2011, but the FBI had not audited its use of those databases. Standards for Internal Control in the Federal Government calls for federal agencies to design and implement control activities to enforce management’s directives and to monitor the effectiveness of those controls. In May 2016, we recommended that the FBI conduct audits to determine the extent to which users of NGI-IPS and biometric images specialists in FACE Services are conducting face image searches in accordance with CJIS policy requirements. DOJ partially concurred with our recommendation. Specifically, DOJ concurred with the portion of our recommendation related to the use of NGI-IPS. In March 2017, DOJ reported that the FBI began assessing NGI-IPS requirements in participating states in conjunction with its triennial National Identity Services audit, and by February 2018 had conducted eight NGI-IPS audits, which found no significant findings of noncompliance. In February 2018, DOJ provided us with copies of the final audit results for one state and its NGI-IPS audit reference guide. The FBI reported that it conducted an audit of FACE Services in September 2018. According to FBI documentation, the purpose of the audit was to determine the extent to which specialists in FACE Services conducted face image searches in accordance with FBI privacy laws and policies. The scope of the audit focused on determining adherence to policies which govern the appropriate use of NGI-IPS, including those for policy development as well as authorized requests and responses. The FBI reported that it finalized the audit report in April 2019, which concluded that the Face Services Unit is operating in accordance with privacy laws and policies. Further, the FBI stated in May 2019 that audits of FACE Services will continue on a triennial basis and that it conducts triennial audits of states that use NGI-IPS. As a result, DOJ has fully implemented our recommendation. FBI Has Taken Limited Actions to Address Our Recommendations for Ensuring the Accuracy of Its Face Recognition Capabilities FBI Has Conducted Limited Assessments of the Accuracy of NGI-IPS Face Recognition Searches In May 2016, we reported that prior to accepting and deploying NGI-IPS, the FBI conducted testing to evaluate how accurately face recognition searches returned matches to persons in the database. However, we found that the tests were limited because they did not include all possible candidate list sizes and did not specify how often incorrect matches were returned. According to the National Science and Technology Council and the National Institute of Standards and Technology at the time, the detection rate (how often the technology generates a match when the person is in the database) and the false positive rate (how often the technology incorrectly generates a match to a person in the database) are both necessary to assess the accuracy of a face recognition system. The FBI’s detection rate requirement for face recognition searches at the time stated that when the person exists in the database, NGI-IPS shall return a match of this person at least 85 percent of the time. However, we found that the FBI only tested this requirement with a candidate list of 50 potential matches. In these tests, 86 percent of the time, a match to a person in the database was correctly returned. The FBI had not assessed accuracy when users requested a list of 2 to 49 matches. According to FBI, a smaller list would likely lower the accuracy of the searches as the smaller list may not contain the likely match that would be present in the larger list. Further, FBI officials stated during our May 2016 review that they had not assessed how often NGI-IPS face recognition searches erroneously match a person to the database (the false positive rate). If false positives are returned at a higher than acceptable rate, law enforcement users may waste time and resources pursuing unnecessary investigative leads. In addition, we concluded that by conducting this assessment the FBI would help ensure that it is sufficiently protecting the privacy and civil liberties of U.S. citizens enrolled in the database. Therefore, we recommended that the FBI conduct tests of NGI-IPS to verify that the system is sufficiently accurate for all allowable candidate list sizes and ensure that both the detection rate and the false positive rate are identified for such tests. In comments on our draft report in 2016, and reiterated during recommendation follow-up in May 2019, DOJ did not concur with this recommendation. DOJ officials stated that the FBI has performed accuracy testing to validate that the system meets the requirements for the detection rate, which fully satisfies requirements for the investigative lead service provided by NGI-IPS. As of May 2019, DOJ has not taken action to address the recommendation. We continue to believe that the recommended action is needed. Such action would allow the FBI to have more reasonable assurance that NGI- IPS provides leads that help enhance, rather than hinder, criminal investigations and that helps protect the privacy of citizens. As noted above, a key focus of our recommendation is the need to ensure that NGI-IPS is sufficiently accurate for all allowable candidate list sizes. As we reported, although the FBI tested the detection rate for a candidate list of 50 photos, they did not do such tests when NGI-IPS users request smaller candidate lists—specifically between 2 and 50 photos. Further, according to the FBI Information Technology Life Cycle Management Directive, testing needs to confirm the system meets all user requirements. Because the accuracy of NGI-IPS’s face recognition searches when returning fewer than 50 photos in a candidate list is unknown, the FBI is limited in understanding whether the results are accurate enough to meet NGI-IPS users’ needs. In comments on our May 2016 report, DOJ officials also stated that searches of NGI-IPS produce a gallery of likely candidates to be used as investigative leads, not for positive identification. As a result, according to DOJ officials, NGI-IPS cannot produce false positives and there is no false positive rate for the system. We disagree with DOJ. According to the National Institute of Standards and Technology, the detection rate and the false positive rate are both necessary to assess the accuracy of a face recognition system. Generally, face recognition systems can be configured to allow for a greater or lesser number of matches. A greater number of matches would generally increase the detection rate, but would also increase the false positive rate. Similarly, a lesser number of matches would decrease the false positive rate, but would also decrease the detection rate. Reporting a detection rate of 86 percent without reporting the accompanying false positive rate presents an incomplete view of the system’s accuracy. FBI Agreed to Conduct Annual Operational Reviews of NGI-IPS but Implementation Is Incomplete We reported in May 2016 that FBI, DOJ, and OMB guidance all required annual reviews of operational information technology systems to assess their abilities to continue to meet cost and performance goals. For example, the FBI’s Information Technology Life Cycle Management Directive required an annual operational review to ensure that the fielded system is continuing to support its intended mission, among other things. In May 2016, we reported that the FBI had not assessed the accuracy of face recognition searches of NGI-IPS in its operational setting—the setting in which enrolled photos, rather than a test database of photos are used to conduct a search for investigative leads. According to FBI officials, at the time of our May 2016 review, the database of photos used in its tests was representative of the photos in NGI-IPS, and ongoing testing in a simulated environment was adequate. However, according to the National Institute of Standards and Technology, as the size of a photo database increases, the accuracy of face recognition searches performed on that database can decrease due to lookalike faces. At the time of our review, FBI’s test database contained 926,000 photos while NGI-IPS contained about 30 million photos. We concluded that by conducting an operational review of these systems, FBI officials would obtain information regarding what factors affect the accuracy of the face recognition searches, such as the quality of the photos in the database, and if NGI-IPS is meeting federal, state, and local law enforcement needs. As a result, we recommended the FBI conduct an operational review of NGI-IPS, at least annually, that includes an assessment of the accuracy of face recognition searches and take actions, as necessary, to improve the system. In May 2016, DOJ concurred with this recommendation and has taken steps to seek input from its users. For example, the FBI submitted a staff paper through the fall 2016 Advisory Policy Board Process to solicit feedback from its users. Specifically, officials said the paper requested feedback on whether the face recognition searches of the NGI-IPS are meeting their needs, and input regarding search accuracy. According to FBI officials, no users expressed concern with any aspect of the NGI-IPS meeting their needs, including accuracy. DOJ reported that it repeated this process in the fall of 2017. Although FBI’s action of providing working groups with a paper presenting our recommendation is a positive step, FBI’s actions do not fully meet the recommendation. FBI’s paper was presented as informational, and did not result in any formal responses from users. We disagree with FBI’s conclusion that receiving no responses on the informational paper fulfills the operational review recommendation, which includes determining that NGI-IPS is meeting user’s needs. In addition, in May 2019, the FBI stated that it will be working with the National Institute of Standards and Technology on annual operational testing and that such testing meets the intention of this recommendation. However, the proposed testing, while promising, will not occur in an operational environment. As such, we continue to believe the FBI should conduct an operational review of NGI- IPS at least annually, as we recommended. FBI Has Not Assessed the Accuracy of External Partners’ Face Recognition Systems Used by FACE Services In May 2016 we reported that FBI officials had not assessed the accuracy of face recognition systems operated by external partners. Specifically, before agreeing to conduct searches on, or receive search results from, these systems, the FBI did not ensure the accuracy of these systems was sufficient for use by FACE Services. Standards for Internal Control in the Federal Government calls for agencies to design and implement components of operations to ensure they meet the agencies mission, goals, and objectives, which, in this case, is to identify missing persons, wanted persons, suspects, or criminals for active FBI investigations. As a result, we recommended the FBI take steps to determine whether each external face recognition system used by FACE Services is sufficiently accurate for the FBI’s use and whether results from those systems should be used to support FBI investigations. In comments on our draft report in 2016, and reiterated during subsequent recommendation follow-up, DOJ officials did not concur with this recommendation. DOJ officials stated that the FBI has no authority to set or enforce accuracy standards of face recognition technology operated by external agencies. In addition, DOJ officials stated that the FBI has implemented multiple layers of manual review that mitigate risks associated with the use of automated face recognition technology. Further, DOJ officials stated there is value in searching all available external databases, regardless of their level of accuracy. We acknowledge that the FBI cannot and should not set accuracy standards for the face recognition systems used by external partners. We also agree that the use of external face recognition systems by the FACE Services Unit could add value to FBI investigations. However, we disagree with DOJ and continue to believe that the FBI should assess the quality of the data it is using from state and federal partners. We also disagree with the DOJ assertion that manual review of automated search results is sufficient. Even with a manual review process, the FBI could miss investigative leads if a partner does not have a sufficiently accurate system. The FBI has entered into agreements with state and federal partners to conduct face recognition searches using hundreds of millions of photos. Without assessments of the results from its state and federal partners, the FBI is making decisions to enter into agreements based on assumptions that the search results may provide valuable investigative leads. For example, the FBI’s accuracy requirements for criminal investigative purposes may be different than a state’s accuracy requirements for preventing driver’s license fraud. By relying on its external partners’ face recognition systems, the FBI is using these systems as a component of its routine operations and is therefore responsible for ensuring the systems will help meet the FBI’s mission, goals and objectives. Until FBI officials can assure themselves that the data they receive from external partners are reasonably accurate and reliable, it is unclear whether such agreements are beneficial to the FBI, whether the investment of public resources is justified, and whether photos of innocent people are unnecessarily included as investigative leads. Chairman Cummings, Ranking Member Jordan, and Members of the Committee, this concludes my prepared statement. I would be happy to respond to any questions you may have at this time. GAO Contact and Staff Acknowledgments For questions about this statement, please contact Gretta Goodwin at (202) 512-8777 or GoodwinG@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this statement. Individuals making key contributions to this statement include Dawn Locke (Assistant Director), Jason Jackson (Analyst-In- Charge), Jennifer Beddor, Ann Halbert-Brooks, Eric Hauswirth, Paul Hobart, Richard Hung, Susanna Kuebler, Kay Kuhlman, Tom Lombardi, and Dina Shorafa. Key contributors for the previous work that this testimony is based on are listed in the previously issued product. This is a work of the U.S. government and is not subject to copyright protection in the United States. The published product may be reproduced and distributed in its entirety without further permission from GAO. However, because this work may contain copyrighted images or other material, permission from the copyright holder may be necessary if you wish to reproduce this material separately.


Background FDA approves reference listed drugs and generic drugs that meet safety and efficacy standards for marketing in the United States. Generic drug companies must show that their drug is (1) the same as the reference listed drug with respect to the active ingredient(s), conditions of use, route of administration, dosage form, strength, and labeling (with certain permissible differences, as approved by FDA); and (2) bioequivalent to the reference listed drug, meaning it generally delivers the same amount of active ingredient(s) in the same amount of time as the reference listed drug. When the reference listed drug is available, it is also designated as the reference standard drug, which is the product generic drug companies must use to conduct bioequivalence testing. When the reference listed drug is not available, FDA will select an approved generic of the reference listed drug to serve as the reference standard drug. All drugs pose some level of safety risk to patients. According to FDA, for most drugs, routine, risk-minimization measures, such as FDA-approved professional labeling, are sufficient to protect the public from the drug’s risks. However, in some cases, FDA may require a drug company to take additional actions to ensure that the benefits of the drug outweigh its risks and to help mitigate or prevent serious risks of adverse side effects. Specifically, FDA may require the drug company to establish a REMS that includes one or more risk-mitigation strategies beyond the drug’s professional labeling. According to FDA, most REMS are designed to reinforce patients’ and health care providers’ behaviors and actions that support the safe use of the particular drug they cover. For example, FDA may require drug companies to give patients and health care providers additional information to reinforce certain safe use conditions or specific risks described in the approved labeling of a certain drug. FDA may require a REMS either before a drug is approved or after approval if FDA becomes aware of new safety information. When determining whether a REMS is necessary, FDA considers several factors, including, for example, the estimated size of the population likely to use the drug and the seriousness of the disease or condition being treated. FDA may require a REMS to include one or more components. For example, FDA may require drug companies to provide patients with certain information in the form of medication guides. Generally, medication guides include information on serious side effects, including those that might require emergency medical care or involve life- threatening conditions. Similarly, FDA may require drug companies to develop communication plans for how the drug company will disseminate information to health care providers. Communication plans can include, for example, information on any serious risks of the drug and any safety protocols to ensure its safe use. Thus, for one REMS, FDA could require a drug company to provide a medication guide. For a second REMS, FDA could require a drug company to provide both a medication guide and a communication plan. Table 1 below includes a list of selected REMS components. Additionally, if a reference listed drug is subject to a REMS, an approved generic drug is also subject to some of the same REMS requirements. FDA can also require drug companies to implement another REMS component, called “elements to assure safe use” (ETASU), if a drug has been shown to be effective, but is associated with a specific serious risk. Depending on the risk, FDA may require any or all of the following ETASU measures: Prescribers have specific training or special certifications; Pharmacies or health care settings where the drug is dispensed have Drugs are dispensed only in certain health care settings, such as hospitals; Drugs are dispensed with evidence of safe-use by the patient, such as requiring a patient’s acknowledgement that she has been counseled on a drug’s risks and understands and accepts these risks; Patients are monitored, for example, while taking the drug for specific adverse events or outcomes; or Patients are enrolled in a registry for collection of certain information, such as patient outcomes and adverse reactions associated with the drug. According to FDA, these measures are for drugs that can be marketed only if there are requirements in place to mitigate a specific serious risk listed in the drug’s labeling. If FDA requires certain ETASU measures, it may also require a drug company to develop an implementation system to enable the drug company to monitor and evaluate implementation of the ETASU measures by health care providers, pharmacists, and other responsible parties. Also, if a reference listed drug is subject to a REMS with ETASU and a generic version is being developed, the reference drug company and the generic drug company are required to develop a shared system—a system that is used by participating companies to coordinate their REMS activities and information about a drug’s risks. Under a required shared system, the generic drug company and the reference drug company use the same REMS documentation and other materials on the drug’s risks and generally share in the implementation and maintenance of any database and infrastructure (e.g., call center). According to FDA, shared systems can be beneficial in reducing the burden for patients and health care providers, such as prescribers and pharmacies, when accessing REMS informational materials or completing administrative requirements, including any required training or certifications for providers. Generic drug companies must submit REMS documentation and materials as part of their generic drug application. Generally, before FDA can approve generic drugs that are subject to REMS with ETASU, reference drug companies and generic companies must reach agreement on a required shared system. According to FDA, generic drug companies that are developing a required shared system should submit their proposed REMS materials to FDA by the midpoint of the application review process or another time as specified by the agency. Any delays in the development of a required shared system can affect FDA’s ability to approve a generic drug application. A generic company may request a waiver from FDA, which if granted, would allow the generic company to develop a separate system that includes the same ETASU measures required for the reference listed drug. For example, if the reference listed drug’s ETASU measures require prescriber certification and the dispensing of the drug in certain health care settings, then the generic drug company’s separate system must also include the same ETASU measures. Practices Identified by FDA and FTC that May Hinder Generic Drug Development and Marketing In recent years, FDA and FTC have identified two practices that can hinder competition by preventing or delaying the development and marketing of generic drugs. The first practice the agencies identified involves limiting access to samples of reference standard drugs, which generic companies generally need to conduct bioequivalence testing. This practice can apply both to reference standard drugs subject to REMS, specifically those subject to certain ETASU measures, and those not subject to REMS. For example, some drug companies might limit access to samples of reference standard drugs subject to REMS, citing ETASU measures that limit distribution, such as the measure that limits distribution of drugs subject to REMS to only certain health care settings. Additionally, drug companies may limit access to samples of reference standard drugs that are not subject to REMS. Typically, generic companies obtain samples through normal distribution channels such as wholesale distributors. However, drug companies could, for example, limit the sale of their reference standard drugs to certain pharmacies, such as specialty pharmacies. FDA and FTC have testified before Congress that these distribution limits—for reference standard drugs with and without ETASU-related distribution measures—can hinder generic companies’ ability to develop generic drugs and to submit a generic drug application to FDA for review. The second practice involves circumstances when a reference drug company delays its negotiations with generic drug companies on a required shared system. The negotiations to develop a required shared system can be complex because all parties must agree on the implementation of the REMS as well as issues related to cost-sharing, confidentiality, and product liability concerns. As part of their generic drug application, generic companies must include an adequate REMS program in order to be approved. Therefore, delays in the development of a required shared system can affect FDA’s ability to approve a generic drug application. Drugs Subject to REMS Vary in the Risks They Pose, Treat a Variety of Conditions, and About Half of Approved REMS Place Limits on Distribution Drugs Subject to REMS Vary in Risks, Treat a Variety of Conditions, and Accounted for at Least $11 Billion in Federal Spending Our analysis of FDA data shows that as of March 18, 2019, there were 74 approved active REMS that apply to 523 drugs. These drugs pose a variety of risks to users, treat a variety of conditions, and some are generics. A REMS can apply to one drug, more than one drug, or to a large number of drugs. Specifically, the approved REMS apply to: 136 drugs because they pose a high risk of serious medical side effects, 384 drugs because they pose a high risk of serious medical side effects from misuse and abuse, and Three drugs because they have the risk of medical side effects from both the use of the drug and from misuse and abuse. These drugs also treat at least 15 different types of medical conditions such as cancer, cardiovascular, and respiratory conditions. Twenty-two are orphan drugs, which are drugs intended to treat rare diseases. One hundred forty-three of these drugs are reference standard drugs, and 64 of these reference standard drugs have one or more approved generics that are also subject to REMS. (See Table 2) For example, FDA approved a generic of the drug Clozaril, which is used to treat mental and mood disorders. Both Clozaril and its generic, Clozapine, are subject to a REMS to prevent adverse medical side effects. Medicare and Medicaid paid at least $11.8 billion in 2017 for reference standard drugs subject to REMS. Specifically, in 2017 Medicare paid at least $8.5 billion for 83 of 139 reference standard drugs subject to REMS. This amount accounted for at least 8 percent of all Medicare drug spending in 2017. In the same year, Medicaid paid at least $3.3 billion—or at least 15 percent of all Medicaid drug spending—for 83 of the 139 reference standard drugs subject to REMS. Appendix I provides information from available data on Medicare and Medicaid spending on reference standard drugs subject to REMS. Almost Half of FDA’s 74 Active REMS Include Limits on How Drugs Are Distributed, and 10 Established a Shared System Of the 74 active REMS in our analysis, 51 have at least one required ETASU measure, and 35 specifically limit how drugs are distributed. Thirty-one of the 74 active REMS also require medication guides explaining the risks of the drug to be given to patients, and 12 require a communication plan for how the company will disseminate information to health care providers. Similar to how the 74 active REMS can have more than one REMS component, the 51 active REMS with ETASU measures can have more than one required measure. For example, 19 active REMS have an ETASU measure requiring patients to be enrolled in a registry and an ETASU measure requiring drug companies to provide training to prescribers of the drugs. Over half of the 51 active REMS with ETASU include measures that may limit how drugs are distributed. Specifically, 35 active REMS with ETASU measures include a requirement for drug companies to ensure drug dispensing settings are specially certified before they distribute the drugs. The certification process may require dispensing pharmacies to enroll in education programs provided by the drug companies. For example, to mitigate the risk of accidental overdoses from the misuse and abuse of fentanyl products, dispensing pharmacies are required to complete an education program that addresses—among other things— the risks of fentanyl products, patient selection, drug dosage, and patient counseling. In addition, for 10 of the 74 active REMS, companies have entered into a shared system. In three of the 10 shared systems, generic companies received a waiver from the shared system requirement after they were not successful in negotiating a shared system with the reference drug companies. In these three cases, the generic drug companies entered into shared systems that are separate from the reference drug company systems. For example, after developing generics of Lotronex, which is subject to REMS with ETASU and intended to treat gastrointestinal conditions, the generic companies were required to enter into a shared system with the reference drug company. When these companies were not successful in negotiating a required shared system, FDA determined the burden of developing a required shared system with the reference drug company outweighed the benefits of having one and waived the requirement. Once FDA granted the waiver, multiple generic companies were allowed to share REMS materials and administrative requirements with heath care providers via one shared system that is separate from Lotronex’s REMS system. FDA and FTC Have Taken Actions to Address Practices They Identified; Drug Companies and Stakeholders Disagreed on the Usefulness of the Actions FDA and FTC Have Taken Actions to Facilitate Access to Samples of Reference Standard Drugs and Required Shared Systems Negotiations FDA and FTC have taken four actions to address circumstances when generic drug companies cannot access samples of reference standard drugs or experience delays in negotiating required shared systems. According to FDA and FTC, both circumstances can hinder generic drug companies’ ability to develop and market generic drugs. Three of the actions focus on making samples of reference standard drugs accessible and the fourth focuses on facilitating the development of a required shared system. While all four of the actions pertain to drugs subject to REMS, only two of the actions pertain to drugs both subject to REMS and not subject to REMS. According to FDA officials, the agency is even more limited in what actions it can take when drugs not subject to REMS are involved. Drug Companies’ Perspectives on Limited Access to Samples of Reference Standard Drugs with Elements to Assure Safe Use (ETASU) that Limit Distribution Officials from all four of the generic drug companies we interviewed told us that their inability to access samples of reference standard drugs with ETASU measures that limit distribution either delayed or discouraged them from developing generic drugs. Officials from two of the five reference drug companies we interviewed told us they were unaware of specific instances when generic companies had difficulty obtaining samples or that generic companies had requested samples of reference standard drugs with ETASU measures that limit distribution. Also, officials from two reference drug companies cited safety concerns as the reason for limiting the distribution of their drug. FDA issued draft guidance on how to obtain a safety determination letter. One of FDA’s actions focused on facilitating generic drug companies’ access to reference standard drugs with ETASU-related limited distribution measures. (See sidebar for drug companies’ perspectives on this practice.) In 2014, FDA issued draft guidance describing how a generic drug company could ask the agency to send what is known as a safety determination letter to the reference drug company on the generic drug company’s behalf. The draft guidance explains how FDA could send a letter stating that the agency had reviewed the generic company’s plans for its bioequivalence testing and determined that these plans included safety measures that were comparable to those in the ETASU measures for the reference standard drug. For example, if the reference standard drug’s ETASU required protections to prevent fetal exposure to the drug, the generic company’s plans should include the same protections. The safety letter would also note that FDA would not consider it a REMS violation to provide reference standard drug samples to the generic company requesting the safety determination letter. According to FDA, some reference drug companies were concerned that providing samples to the generic drug company would violate REMS requirements. From 2016 to 2018, FDA issued 12 safety determination letters to reference drug companies on behalf of generic companies, according to agency data. However, FDA did not issue a safety determination letter for all of the requests it received. According to FDA officials, there are various reasons why they might not issue a safety determination letter to the reference drug company. For example, a generic company must sign a disclosure form in order for FDA to send the letter to the reference drug company, but the generic company does not always choose to do this. Additionally, the generic company might have withdrawn its request for a safety determination letter, or FDA might be waiting for additional information from the generic company in order to complete its review. According to FDA officials, there is no need for a safety determination letter (which assures the reference drug company that providing samples to the generic drug company will not be considered a violation of their REMS) when there is no REMS for the product in the first place. Drug Companies’ Perspectives on Limited Access to Samples of Reference Standard Drugs Not Subject to Risk Evaluation and Mitigation Strategies (REMS) Officials from three of the four generic companies in our review told us they had experience with drug companies’ imposed distribution limits on reference standard drugs not subject to REMS. Of the four generic companies in our review, officials from one company said they were not able to obtain the samples they needed and chose not to pursue developing a particular drug. Officials from one of the five reference drug companies said they have limited distribution for reference standard drugs not subject to REMS. They said their companies do so to ensure that their products are efficiently distributed, in part by using certain pharmacies. FDA published a web page with information about inquiries that included drugs both subject to and not subject REMS. In February 2019, FDA published a web page with information on inquiries made to FDA by generic companies seeking to obtain samples of reference standard drugs in order to develop generic drugs. (See sidebar for drug companies’ perspectives on this practice.) FDA officials said they published this list to increase transparency about continuing issues related to accessing samples and to raise awareness about the potential effect these issues might have on reducing competition in the drug market. This list included drugs subject to and not subject to REMS, the names of reference drug companies, and the number of inquiries made. According to the web page as of February 2019, inquiries were made for 54 reference standard drugs, including 25 drugs with ETASU-related limited distribution measures and 29 drugs without such measures. According to FDA data, the number of inquiries had been generally decreasing in the years prior to when the list was published. FTC officials reviewed inquiries the agency received from FDA and generic companies and filed two briefs. FTC told us it reviewed inquiries the agency had received from generic companies and FDA, including those related to information on FDA’s published web page. However, FTC officials said, to date, they have not brought a case charging a reference drug company with violating federal antitrust law for refusing to provide samples to a generic drug company. In order to take enforcement action, FTC needs to find sufficient evidence of activity that violates the Federal Trade Commission Act or the Sherman Act. For example, FTC would need to find that a reference drug company’s practice constituted monopolization in violation of the Sherman Act. According to FTC officials, they have not brought any antitrust cases to the courts, but have filed two amicus briefs related to cases involving drugs subject to REMS. In both of these briefs, FTC noted that the generic companies’ respective allegations, if true, established an antitrust violation and that the generic companies’ lawsuits should be allowed to continue. Drug Companies’ Perspectives on Negotiating Required Shared Systems Officials from one generic company said that the respective reference drug companies would not meet with them to negotiate the development of a required shared system REMS. Officials from another generic company said the negotiation process with the reference drug company lasted almost 2 years. Officials from four of the five reference drug companies we interviewed had experience negotiating required shared systems. Officials from three of these four companies told us that developing a shared system is a difficult, challenging, and complex process. Officials from one reference drug company said that the level of complexity can increase based on the number of companies and the different people involved. FDA issued waivers that allowed generic drug companies to develop a separate system from the REMS of the reference listed drug and issued draft guidance on how to obtain such a waiver. According to FDA, since 2007, the agency has received 13 requests for a waiver from the shared system requirement, and at the time of our data collection and analysis, FDA had approved three, the first in 2013. (See sidebar for drug companies’ perspective on required shared systems.) These waivers allowed the generic drug company to develop a separate system that includes the same ETASU measures required for the reference listed drug. According to officials, FDA was unable to grant the remaining waivers for different reasons. For example, the agency may still be reviewing the generic drug application submitted by the company that requested a waiver. Officials explained that the waiver request is part of the overall generic drug application and the agency cannot approve a waiver without approving the application as well. To further facilitate the process, in 2018, FDA issued draft guidance describing what factors the agency considers when granting waivers. The statute authorizes FDA to grant a waiver (1) if the burden of creating a required shared system outweighs the benefit of having it, taking into account the impact on the health care providers, patients, and drug companies involved or (2) if an aspect of the ETASU is covered by an unexpired patent or entitled to trade secret protection, and the generic company was unsuccessful in obtaining a license for use. FDA’s guidance describes examples of the potential benefits of having a shared system and the burdens of forming a shared system on health care providers, patients and drug companies that FDA will consider. For example, having a shared system could benefit drug companies by making a REMS for multiple products more efficient. In contrast, the drug companies negotiating a required shared system could be market competitors and involved in patent litigation related to the drug product. Selected Drug Companies Had Differing Views on the Usefulness of FDA’s and FTC’s Efforts In general, the four generic drug companies and five reference drug companies we interviewed disagreed on the usefulness of FDA’s and FTC’s efforts to address the practices that may affect the development of generic drugs. FDA’s safety determination letters. Officials from three of the generic companies in our review said that the safety determination letters were not useful because they were not enforceable and did not require a reference drug company to provide a generic company with samples of a reference standard drug. In its comments on FDA’s draft guidance on obtaining a safety determination letter, one stakeholder representing generic companies expressed concern that reference drug companies now use safety determination letters as another requirement to obtain samples. In contrast, officials from three reference drug companies we interviewed told us that FDA’s safety determination letters addressed their safety concerns regarding sharing samples of reference standard drugs with generic companies. Further, officials from two of these three reference drug companies said they request these letters from generic companies that request samples of reference standard drugs. Officials from the remaining two reference drug companies we interviewed said they were not aware of FDA’s safety determination letters or did not have concerns or a position on the issue. FDA’s publication of its web page. Officials from one of the four generic companies we interviewed told us they thought the inquiries web page published by FDA was helpful. However, this same company said it had not noticed a significant effect in being able to access samples of reference standard drugs because of the web page. Officials from another generic company said it was too early to tell about the usefulness of FDA’s web page. Of the remaining two generic companies, officials from one company were unaware of the web page and officials from the second company noted that they were uncertain why a generic company would be included in the list of companies on FDA’s web page. Officials from two of the five reference drug companies we interviewed, and whose companies appeared on the web page, said they were unaware of any inquiries made to their companies requesting samples of reference standard drugs. Additionally, officials from one company told us they did not know why they were on FDA’s published web page because the company had sold the reference standard drug to another company and had informed FDA that this had occurred. According to FDA, the web page reflects the owner of the reference standard drug at the time the agency received an inquiry, regardless of whether the drug was later sold. Additionally, some generic companies might contact FDA directly without contacting the reference drug company because they anticipate having difficulties accessing samples of the reference standard drug. FDA notes on its web page that the agency did not independently investigate or confirm the access limitations described in the inquiries it received. FTC’s filing of amicus briefs. Officials from two of the generic companies in our review said FTC’s filing of amicus briefs was generally a positive step. Officials from two companies said the amicus briefs helped negotiations with reference drug companies. A third generic company said the amicus briefs helped raise awareness about issues generic companies are having. Officials from a fourth generic company said FTC’s actions could impact the company’s efforts to develop generic versions of reference listed drugs in the future. Officials from the five reference drug companies we interviewed did not have any comments on FTC’s specific amicus briefs. Waivers for a required shared system. Officials from three of the four generic companies we spoke with had experience with waivers. Officials from one of these three companies said the waiver guidance was helpful. However, officials from this generic company and a second company said it took FDA almost a year to grant their waivers. According to officials from a third company, they obtained their waiver within a month, in part, because negotiations had been ongoing for more than a year. According to FDA officials, the review of a waiver request is part of the generic drug company’s drug application. FDA will not grant a waiver unless the generic drug company meets the waiver requirements and its generic drug application is approved. Reference drug companies and other stakeholders expressed concerns about these waivers. Officials from three of the five reference drug companies we spoke with said the burden on health care providers or patients should be considered when granting waivers. Officials from one company specifically expressed concerns that as FDA grants additional waivers, it could place an additional burden on the health care system. For example, health care providers could be required to use multiple systems to access REMS information on the drug’s risks or to complete administrative requirements, such as required certification. The remaining reference drug companies did not have comments on the topic. In comments we reviewed on FDA’s draft guidance on these waivers, stakeholders noted concerns similar to those raised by the reference drug companies. For example, two groups representing pharmacists and pharmacies said that if FDA grants additional waivers, it could place a burden on the health care system. Historically, FDA has attempted to limit the number of required shared systems created under waivers. If a generic drug company is granted a waiver, it is allowed to create a separate system that includes the same ETASU measures required of the reference listed drug. However, to date, FDA has only granted waivers to generic drug companies that agree to share their systems with other drug companies that concurrently or subsequently develop generic or brand versions of the same reference listed drugs. Agency Comments We provided a draft of this product to FDA and FTC for their review and comment. Both agencies provided technical comments, which we incorporated as appropriate. As agreed with your office, unless you publicly announce the contents of this report earlier, we plan no further distribution until 30 days from the report date. At that time, we will send copies to the appropriate congressional committees, the Secretary of HHS, and other interested parties. In addition, the report is available at no charge on the GAO website at http://www.gao.gov. If you or your staff have any questions about this report, please contact me at (202) 512-7114 or dickenj@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are listed in appendix II. Appendix I: 2017 Medicare and Medicaid Spending on Reference Standard Drugs Subject to REMS In order to estimate Medicare and Medicaid spending on reference standard drugs subject to risk evaluation and mitigation strategies (REMS), we compared data from the Food and Drug Administration (FDA) on drugs subject to REMS as of March 18, 2019, to publicly available 2017 data on the Medicare Part D Drug Spending Dashboard and the Medicaid Drug Spending Dashboard, maintained by the Centers for Medicare & Medicaid Services (CMS). These data covered drug spending and utilization for both of these programs for calendar year 2017, the most current data available. However, not all spending and utilization data for reference standard drugs subject to REMS were available. Since we analyzed data as of March 18, 2019, we were able to identify 139 reference standard drugs subject to REMS with corresponding cost data. To assess the reliability of these data, we interviewed knowledgeable agency officials. We determined that the data were sufficiently reliable for the purposes of our report. Medicare and Medicaid paid at least $11.8 billion in 2017 for reference standard drugs subject to REMS, according to cost data available from the CMS’s drug pricing dashboard. Specifically, Medicare Part D paid at least $8.5 billion for reference standard drugs subject to REMS. This amount—which includes Medicare Part D plan sponsors and beneficiaries Part D payments such as copays, but not price concessions, such as manufacturers’ rebates—accounted for at least 8 percent of all Medicare drug spending in 2017. Similarly, Medicaid paid at least $3.3 billion for reference standard drugs subject to REMS, or at least 15 percent of all Medicaid drug spending in 2017. Of the 139 reference standard drugs in our analysis, the greatest share of these programs’ spending, across medical conditions, was on reference standard drugs subject to REMS for cancer, based on our analysis of available data. Specifically, Medicare and Medicaid spent at least $4.6 billion on 8 reference standard drugs that treat cancer. See table 3 below for Medicare and Medicaid spending for reference standard drugs subject to REMS by medical condition treated. Further, our analysis of available data showed that Medicare and Medicaid spent the most on Revlimid, a drug used to treat cancer, totaling $3.6 billion with Medicare accounting for $3.3 billion of this total. More than 37,000 Medicare beneficiaries used this drug, at an average cost per dosage unit of $626.94. (See table 4.) In contrast, Medicaid spent the most on Suboxone, a drug used to treat opioid dependence, totaling $0.7 billion, based on available data. More than 3 million Medicaid claims were filed for this drug in 2017, at an average cost per dosage unit of $7.89. Vivitrol was the third most utilized drug under Medicaid. (See table 5.) Based on our analysis of available data, the selected examples of reference standard drugs subject to REMS had higher average cost per dosage unit compared to the generic. For example, Medicare spent an average cost per unit of $12.20 for Clozaril, a drug used to treat mental health conditions, compared to $0.99 for clozapine, a generic version of Clozaril. Table 6 below shows selected examples comparing Medicare spending for reference standard drugs to Medicare spending for a generic version, based on our analysis of available data. Our analysis of available Medicaid data showed similar results to our analysis of Medicare data. For example, Medicaid spent an average cost of $11.71 for Clozaril, a drug used to treat mental health conditions, compared to $0.97 for clozapine, a generic version Clozaril. Table 7 below shows selected examples comparing Medicaid spending for reference standard drugs to Medicaid spending for a generic version, based on our analysis of available data. Appendix II: GAO Contact and Staff Acknowledgments GAO Contact Staff Acknowledgments In addition to the contact named above, Geri Redican-Bigott and Tom Conahan, Assistant Directors; Carolyn Garvey, Analyst-in-Charge; Zhi Boon, Gay Hee Lee, and McKenna Storey made key contributions to this report. Also contributing were Sam Amrhein, Kaitlin Farquharson, Cathy Hamann, and Diona Martyn.