Actions Needed to Address Weaknesses in TSA’s Pipeline Security Program Management In our December 2018 report, we found that TSA provides pipeline operators with voluntary security guidelines that operators can implement to enhance the security of their pipeline facilities. TSA also evaluates the vulnerability of pipeline systems through security assessments. Pipeline operators and industry association representatives who we interviewed also reported exchanging risk-related security information and coordinating with federal and nonfederal entities, including TSA. However, we also identified weaknesses in several areas of TSA’s pipeline security program management, including: (1) updating and clarifying pipeline security guidelines; (2) planning for workforce needs; (3) assessing pipeline risks; and (4) monitoring program performance. Exchanging Security Information and Coordinating with Federal and Nonfederal Entities We found in our December 2018 report that all of the pipeline operators and industry association representatives that we interviewed reported receiving security information from federal and nonfederal entities. For example, DHS components including TSA’s Intelligence and Analysis and NCCIC share security-related information on physical and cyber threats and incidents. Nonfederal entities included Information Sharing and Analysis Centers, fusion centers, industry associations, and subsector coordinating councils. Pipeline operators also reported that they share security-related information with TSA and the NCCIC. For example, TSA’s Pipeline Security Guidelines requests that pipeline operators report physical security incidents to the Transportation Security Operations Center (TSOC) and any actual or suspected cyberattacks to the NCCIC. According to TSA officials, TSOC staff analyzes incident information for national trends and common threats, and then shares their observations with pipeline operators during monthly and quarterly conference calls. Updating Pipeline Security Guidelines In our December 2018 report, we found that the pipeline operators we interviewed reported using a range of guidelines and standards to address their physical and cybersecurity risks. For example, all 10 of the pipeline operators we interviewed stated they had implemented the voluntary 2011 TSA Pipeline Security Guidelines that the operators determined to be applicable to their operations. Five of the 10 pipeline operators characterized the guidelines as generally or somewhat effective in helping to secure their operations, 1 was neutral on their effectiveness, and 4 did not provide an assessment of the guidelines’ effectiveness. Pipeline operators and industry association representatives reported that their members also use the Interstate Natural Gas Association of America’s Control Systems Cyber Security Guidelines for the Natural Gas Pipeline Industry, the American Petroleum Institute’s Pipeline SCADA Security standard, and the National Institute of Standards and Technology’s (NIST) Cybersecurity Framework as sources of cybersecurity standards, guidelines, and practices that may be scaled and applied to address a pipeline operator’s cybersecurity risks. We found that TSA’s Pipeline Security Branch had issued revised Pipeline Security Guidelines in March 2018, but TSA had not established a documented process to ensure that revisions occur and fully capture updates to supporting standards and guidance. The guidelines were revised to, among other things, reflect the dynamic threat environment and to incorporate cybersecurity principles and practices from the NIST Cybersecurity Framework, which was initially issued in February 2014. However, because NIST released version 1.1 of the Cybersecurity Framework in April 2018, the guidelines that TSA released in March 2018 did not incorporate cybersecurity elements that NIST added to the latest Cybersecurity Framework, such as the Supply Chain Risk Management category. Without a documented process defining how frequently TSA is to review and, if deemed necessary, revise its guidelines, TSA cannot ensure that the guidelines reflect the latest known standards and best practices of physical security and cybersecurity. We recommended that TSA implement a documented process for reviewing, and if deemed necessary, revising TSA’s Pipeline Security Guidelines at regular defined intervals. DHS agreed and estimated that this effort would be completed by April 30, 2019. In April 2019, TSA provided us with documentation outlining procedures for reviewing these guidelines. We are currently assessing this information to determine if it sufficiently addresses this recommendation. We also found that TSA’s Pipeline Security Guidelines lacked clarity in the definition of key terms used to determine critical facilities. TSA initially identifies the 100 highest risk pipeline systems based on the amount of material transported through the system. Subsequently, pipeline operators are to use criteria in the Guidelines to self-identify the critical facilities within those higher risk systems and report them to TSA. TSA’s Pipeline Security Branch then conducts CFSRs at the critical facilities identified by pipeline operators. However, our analysis of TSA’s data found that at least 34 of the top 100 critical pipeline systems TSA deemed highest risk indicated that they had no critical facilities. Three of the 10 operators we interviewed stated that some companies that reported to TSA that they had no critical facilities may possibly be taking advantage of the guidelines’ lack of clarity. For example, one of TSA’s criteria for determining pipeline facility criticality states that if a facility or combination of facilities were damaged or destroyed, it would have the potential to “cause mass casualties or significant health effects.” Two operators told us that individual operators may interpret TSA’s criterion, “cause mass casualties or significant health effect,” differently. For example, one of the operators that we interviewed stated that this criterion could be interpreted either as a specific number of people affected or a sufficient volume to overwhelm a local health department, which could vary depending on the locality. Without clearly defined criteria for determining pipeline facilities’ criticality, TSA cannot ensure that pipeline operators are applying guidance uniformly, that all of the critical facilities across the pipeline sector have been identified, or that their vulnerabilities have been identified and addressed. We recommended that TSA’s Security Policy and Industry Engagement’s Surface Division clarify TSA’s Pipeline Security Guidelines by defining key terms within its criteria for determining critical facilities. DHS agreed and estimated that this effort would be completed by June 30, 2019. Planning for Workforce Needs TSA conducts pipeline security reviews—CSRs and CFSRs—to assess pipeline vulnerabilities and industry implementation of TSA’s Pipeline Security Guidelines. However, the number of reviews conducted has varied widely from fiscal years 2014 through 2018. These reviews are intended to develop TSA’s knowledge of security planning and execution at critical pipeline systems and lead to recommendations for pipeline operators to help them enhance pipeline security. For an overview of the CSR and CFSR processes, see Figure 1 below. We found that the number of CSRs and CFSRs completed by TSA has varied during the last five fiscal years, ranging from zero CSRs conducted in fiscal year 2014 to 23 CSRs conducted in fiscal year 2018, as of July 31, 2018 (see Figure 2 below). TSA officials reported that staffing limitations had prevented TSA from conducting more reviews. TSA Pipeline Security Branch staffing levels (excluding contractor support) also varied significantly over the past 9 years ranging from 14 full-time equivalents in fiscal years 2012 and 2013 to one in fiscal year 2014 (see Table 1 below). TSA officials stated that, while contractor support has assisted with conducting CFSRs, there were no contractor personnel providing CSR support from fiscal years 2010 through 2017, but that contractors increased to two personnel in fiscal year 2018. TSA officials stated that they expected to complete 20 CSRs and 60 CFSRs per fiscal year with Pipeline Security Branch employees and contract support, and had completed 23 CSRs through July 2018 for fiscal year 2018. In addition, pipeline operators that we interviewed emphasized the importance of cybersecurity skills among TSA staff. Specifically, 6 of the 10 pipeline operators and 3 of the 5 industry representatives we interviewed reported that the level of cybersecurity expertise among TSA staff and contractors may challenge the Pipeline Security Branch’s ability to fully assess the cybersecurity portions of its security reviews. We found that TSA had not established a workforce plan for its Security Policy and Industry Engagement or its Pipeline Security Branch that identified staffing needs and skill sets such as the required level of cybersecurity expertise among TSA staff and contractors. We therefore recommended that TSA develop a strategic workforce plan for its Security Policy and Industry Engagement Surface Division, which could include determining the number of personnel necessary to meet the goals set for its Pipeline Security Branch, as well as the knowledge, skills, and abilities, including cybersecurity, that are needed to effectively conduct CSRs and CFSRs. DHS agreed and estimated that this effort would be completed by July 31, 2019. Pipeline Risk Assessments The Pipeline Security Branch has developed a risk assessment model that combines all three elements of risk—threat, vulnerability, and consequence—to generate a risk score for pipeline systems. The Pipeline Security Branch developed the Pipeline Relative Risk Ranking Tool in 2007 for use in assessing various security risks to the top 100 critical pipeline systems based on volume of material transported through the system (throughput). The risk ranking tool calculates threat, vulnerability, and consequence for each pipeline system on variables such as the amount of throughput in the pipeline system and the number of critical facilities using data collected from pipeline operators, as well as other federal agencies such as the Departments of Transportation and Defense. The ranking tool then generates a risk score for each of the 100 most critical pipeline systems and ranks them according to risk, which was information used by TSA to prioritize pipeline security assessments. However, in our December 2018 report we found that the last time the Pipeline Security Branch calculated relative risk among the top 100 critical pipeline systems using the ranking tool was in 2014. Since the risk assessment had not changed since 2014, information on threat may be outdated and may limit the usefulness of the ranking tool in allowing the Pipeline Security Branch to effectively prioritize reviews of pipeline systems. We recommended that the Security Policy and Industry Engagement’s Surface Division update the Pipeline Relative Risk Ranking Tool to include up-to-date data to ensure it reflects industry conditions, including throughput and threat data. DHS agreed and in March 2019 TSA officials reported taking steps to update the data in the Pipeline Risk Ranking Tool to reflect current pipeline industry data. We are currently reviewing those actions to determine if they sufficiently address our recommendation. We also found that some of the sources of data and vulnerability assessment inputs to the ranking tool were not fully documented. For example, threats to cybersecurity were not specifically accounted for in the description of the risk assessment methodology, making it unclear if cybersecurity threats were part of the assessment’s threat factor. We recommended that the Security Policy and Industry Engagement’s Surface Division fully document the data sources, underlying assumptions, and judgments that form the basis of the Pipeline Relative Risk Ranking Tool, including sources of uncertainty and any implications for interpreting the results from the assessment. In March 2019, TSA officials stated that they had taken steps to document this information. We are currently reviewing those steps to determine if they sufficiently address our recommendation. Monitoring Program Performance In our December 2018 report, we also found that TSA developed three databases to track CSR and CFSR recommendations and their implementation status by pipeline facility, system, operator, and product type. TSA officials stated that the primary means for assessing the effectiveness of the agency’s efforts to reduce pipeline security risks was through conducting pipeline security reviews—CSRs and CFSRs. However, while TSA does track CFSR recommendations, we found that TSA had not tracked the status of CSR recommendations for security improvements in over 5 years—information necessary for TSA to effectively monitor pipeline operators’ progress in improving their security posture. We recommended that TSA take steps to enter information on CSR recommendations and monitor and record their status. DHS agreed and estimated that this effort would be completed by November 30, 2019. Chairman Rush, Ranking Member Upton, and Members of the Subcommittee, this completes my prepared statement. I would be pleased to respond to any questions that you may have at this time. GAO Contact and Staff Acknowledgments If you or your staff members have any questions about this testimony, please contact me at (202) 512-8777 or russellw@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this statement. Other individuals making key contributions to this work include Ben Atwater, Assistant Director; Steve Komadina, Analyst-in-Charge; Nick Marinos, Michael Gilmore, Tom Lombardi, Chuck Bausell and Susan Hsu. This is a work of the U.S. government and is not subject to copyright protection in the United States. The published product may be reproduced and distributed in its entirety without further permission from GAO. However, because this work may contain copyrighted images or other material, permission from the copyright holder may be necessary if you wish to reproduce this material separately.


Background DOD’s guidance states that its FFRDCs are created to (1) provide strategic value through independent, intellectually rigorous, relevant, and timely products and services; and (2) support the department’s goals of long-term improvement in operations and enhanced national security. They are managed by various military departments or divisions within the department, called primary sponsors. FFRDCs are operated by universities, other not-for-profit or nonprofit organizations, or private firms—called parent organizations—under long-term contracts. They provide special research and development services that generally cannot be readily satisfied by government personnel or private contractors. For example, the Lincoln Laboratory develops key radar and electronic warfare technologies for integrated air and missile defense systems. The Software Engineering Institute provides cybersecurity solutions for defense entities. DOD’s FFRDCs are grouped into three categories: research and development laboratories, study and analysis centers, and systems engineering and integration centers. DOD oversees 10 FFRDCs (see table 1). According to the Director of Laboratories and Personnel within the OUSD for Research and Engineering, he took over responsibility for managing FFRDCs in July 2018, following a reorganization of OUSD for Acquisition, Technology and Logistics. DOD and each FFRDC have a sponsoring agreement, which is a stand- alone, bilateral, written agreement between the primary sponsor and the parent organization. It must be approved by the Under Secretary of Defense for Research and Engineering prior to award of an FFRDC contract and is incorporated into the contract. According to DOD’s guidance, the sponsoring agreement defines the FFRDC’s purpose and mission, establishes the conditions under which DOD may award an FFRDC contract, and describes the overarching requirements for operation of the FFRDC. For example, the primary sponsor must include provisions in the sponsoring agreement to prevent real or perceived organizational and personal conflicts of interest. As part of that, sponsors are to require FFRDC parent organizations to establish and maintain policies and procedures to protect information, such as sensitive data, from disclosure and provide training that covers ethics and conflicts of interest. We reported in December 2019 that representatives from the five study and analysis center FFRDCs said they provide annual training covering ethics and conflicts of interest for all personnel. DOD may use FFRDCs to perform work that is closely associated with the performance of inherently governmental functions or that is critical to maintaining control of the department’s missions and operations. Work could include activities such as support for financial analyses, policy development, acquisition planning, source selection, and contract management. In the course of performing work, FFRDCs may need access to acquisition data collected from DOD’s prime contractors and program offices. FFRDCs may obtain these data through DOD personnel, government databases, or directly from prime contractors. Government- held data may be stored and managed in department-wide databases or by individual program offices. For example, the Cost Assessment Data Enterprise is a web application that allows users access to various reports that include information such as major defense acquisition programs’ cost, software, and technical data. On December 21, 2017, DOD issued implementing guidance that marked the launch of its 3-year pilot program. According to DOD officials, prior to the start of the pilot program, FFRDC researchers needed to obtain permission from each data owner (e.g., DOD prime contractor or supplier), typically by signing a nondisclosure agreement. According to DOD officials that requested the authority to allow FFRDCs to have increased access, one of the purposes of the pilot was to allow for a streamlined nondisclosure agreement process. Under the pilot program, FFRDC researchers no longer have to obtain nondisclosure agreements with each data owner. To participate in the pilot, the FFRDC and DOD sponsor must first take steps to ensure certain protections are in place to protect against unauthorized disclosure or use of the data being accessed. For example, according to the statute, in order to be eligible, participating FFRDCs and its personnel (FFRDC researchers) had to agree to be subject to and comply with appropriate ethics standards and requirements applicable to government personnel, including the Ethics in Government Act of 1978, the Trade Secrets Act, and the Procurement Integrity Act. After the protections are in place, the FFRDC and DOD sponsor can enroll individual projects in the pilot program. Per the implementing guidance, the FFRDCs and DOD sponsors agree to collect and provide information about the enrolled projects. For example, DOD sponsors must provide the Laboratories and Personnel Office quarterly updates on a project’s progress obtaining data and, once the project is complete, information on the results of its access to sensitive data under the pilot program. Over Half of DOD’s FFRDCs Used the Pilot for A Small Percentage of Projects and Reported Benefits of Participating As of September 2019, Six of 10 DOD FFRDCs Participated in the Pilot and Enrolled 33 Projects Six of 10 DOD FFRDCs elected to participate in the pilot program during its first 21 months (figure 1). According to FFRDC representatives, the decision regarding whether an FFRDC would participate in the pilot program primarily depended on two factors: (1) the data needs of the FFRDC’s projects and (2) the ability of FFRDCs to access necessary data without the pilot program. Representatives from the six participating FFRDCs told us they elected to participate because they required access to sensitive data and, in some cases, lacked viable options for obtaining that data. For five of these FFRDCs, representatives said their researchers had identified specific projects for which they were interested in using the pilot to gain access to data sources with sensitive data from numerous contractors. Representatives from the four nonparticipating FFRDCs said that the existing processes the FFRDCs have in place provide the access they need for their projects. For example, Lincoln Laboratory representatives said their researchers are often working with an individual program or working to advance a specific technology; therefore, their work is generally with a limited number of contractors. In cases where they have needed access to sensitive data to do this work, they have executed a blanket nondisclosure agreement with their primary sponsor and, in some cases, have executed more tailored nondisclosure agreements with companies when obtaining information directly from a defense contractor. Aerospace has a blanket nondisclosure agreement with the Air Force Space and Missile Systems Center and the center included a provision in its contracts that requires its prime contractors to directly share information with the FFRDC. A Center for Communications and Computing representative said existing processes already provide the access they need for their projects. According to the FFRDC’s sponsor, its work focuses more on technological development rather than acquisitions analysis. Participating FFRDCs reported 33 projects enrolled in the pilot program from January 2018 through September 2019. Pilot projects represented about 1.5 percent of these FFRDCs’ total number of projects as of June 2019. Of the projects enrolled in the pilot program, 11 were complete and 10 were ongoing as of the end of September 2019. In addition, 11 projects enrolled in the pilot initially, only to realize they did not require access to the requested data and thus were removed. One project was put on hiatus pending a decision about whether it will continue. Table 2 summarizes the status of the projects in the pilot program as of September 2019. According to DOD officials, the fiscal year 2018 reorganization of OUSD for Acquisitions, Technology and Logistics into two offices, coupled with changes in leadership, shifted attention away from the pilot program design and implementation. For example, an official from the Acquisition and Sustainment office also told us it missed an opportunity to conduct outreach with its FFRDC because the office did not hold its biannual meetings in 2017 or 2018 due to the reorganization. In these biannual meetings, he explained, they would have discussed the department’s future research priorities and how the pilot program may have helped. This official—who was involved in the pilot’s implementation—also noted that the shift in attention meant they did not engage with the offices that maintain the data repositories as fully as they would have liked. We found it took sponsors and FFRDCs from a few weeks to 7 months to resolve questions about pilot program requirements and update the FFRDCs’ sponsoring agreements to incorporate the pilot protections. During that time, FFRDCs were unable to move forward with certain analyses for their proposed projects. DOD Officials and FFRDC Representatives Reported Benefits from Using the Pilot Of the six projects we selected for further review (shown in table 3), four have been completed and FFRDCs reported benefits from their pilot program participation. The two remaining projects are on hiatus or removed. DOD officials and representatives from the four completed projects shared with us the following benefits: Systems and Analyses Center assessment of the U.S. munitions defense industrial base capacity. Portions of the research required access to sensitive data about the availability and production levels of manufacturing parts for a large number of contractors and suppliers. The FFRDC researchers used these data in their microlevel assessments of the manufacturing capacity and supply chain resiliency of the U.S. defense munitions industrial base. They said they were able to provide DOD’s Industrial Policy office with a more complete picture by combining these microlevel analyses with broader analyses of employment trends and economic outputs. A DOD industrial policy official who requested the work also said that the analysis enabled her office to meet an executive branch reporting requirement, which DOD did not have the manpower to conduct. National Defense Research Institute support for analysis of munitions industrial base. FFRDC researchers worked in collaboration with government officials to perform analyses on the adequacy of the munitions and missiles industrial base using government-held data from prime contractors and subcontractors. For example, the researchers supported working groups examining propulsion and chemicals in munitions and provided analysis for a report to Congress on solid rocket motors. The DOD official that requested the work and National Defense Research Institute representatives said that, without the pilot, the FFRDC would not have been able to access the data used to support DOD in these efforts. The official also noted that in this case the FFRDC helped fill a gap in DOD’s workforce to meet a congressional reporting requirement. National Security Engineering Center and Software Engineering Institute analysis of software acquisitions practices. FFRDCs supported a Defense Innovation Board study that aimed to identify correlations between software complexity, cost, and schedule evolution. FFRDC researchers’ access to and use of the data provided important insights about the quality and reliability of the department’s data. Specifically, DOD gained further insight into the kinds of software data the department holds and the significant gaps that would need to be addressed to improve overall DOD-held data quality. The Defense Innovation Board’s report included findings related to the quality of the software data accessed and analyzed by these two FFRDCs. Project AIR FORCE assessment of contractor risk. According to FFRDC representatives and an Air Force official involved in the work, the pilot program facilitated the FFRDC’s access to sensitive data held by the Defense Contract Management Agency that researchers used to identify early indicators of contractor performance risks. In response to the results of this work, the Air Force has funded a follow- on project to further research the potential of data analytics to provide early indicators of challenges in contract execution, according to an Air Force official involved with project. In addition to the benefits at the project level, several DOD sponsor officials and FFRDC representatives also noted the benefits of using the streamlined nondisclosure agreement process to accomplish their work. According to several DOD sponsor officials and FFRDC researchers we spoke to, completing the requested analysis without the pilot program would have required individual nondisclosure agreements with hundreds of individual contractors and suppliers. Systems and Analyses Center representatives said this would have been essentially impossible, and would have prevented researchers from completing important parts of the analyses. In another case, a Software Engineering Institute representative told us that, before the pilot, their team could not access software data when attempting to complete a 2017 project involving DOD software costs and production time frames. For that project, the DOD organization responsible for the data repository had recommended researchers send out a data request letter to each of the contractors with data in the system. Researchers sent out roughly one hundred requests to contractors for permission, but received no responses. They pointed out this was in part because contractors have no incentive to respond to an FFRDC’s request for access to their data. As a result, Software Engineering Institute was unable to use updated data for the 2017 report. While several sponsoring agency officials noted benefits of using the data for analyses to inform key program decisions, they also noted that a causal relationship between the pilot program activities and acquisition process improvements would be hard to establish, in part due to the length of time needed for projects to effect change. DOD officials responsible for two completed projects examining the munitions industrial base said they expect the analyses performed will lead to improved acquisition processes but that it would take many years to see the benefits. Specifically, they said the FFRDCs’ work helped identify areas for improvement in the department’s budget and acquisition strategy to better signal future demand to its lower tier munitions industrial base suppliers. In addition to noting these expected improvements, several DOD officials also acknowledged that expanding access of sensitive data to more people increases the potential for unauthorized use or disclosure but said that the pilot program put in place important protections to help mitigate these risks. DOD’s Pilot Established Protections for Accessing Sensitive Data but Did Not Establish Procedures to Verify Compliance DOD’s guidance to implement the pilot program outlined protections the FFRDCs must agree to, in order to guard against unauthorized disclosure or use of sensitive data, and required that these protections be incorporated into the sponsoring agreements between the FFRDCs and the DOD sponsor. However, we found some instances where details of the required protections were not incorporated into the agreements. We also found that the Laboratories and Personnel Office, which is responsible for managing the pilot program, does not have a procedure to verify whether protections were implemented, in part because it has not developed a process for doing so. Table 4 explains the protections. Some of these protections were already part of the FFRDCs’ business operations, while others are new. For example, the prohibition on their use of sensitive data to compete against a third party was already a fundamental aspect of FFRDCs’ role in supporting DOD. Similarly, participating FFRDC representatives told us that certain protections, such as implementing nondisclosure agreements and training, required only small adjustments to their existing procedures. However, the pilot’s financial disclosure program, annual certifications by parent organizations, and instructions for researchers to notify contracting officers of employment offers when supporting source selection decisions were new and specific to the pilot, according to a DOD official involved in the pilot’s implementation. We found that not all the details of protections were incorporated into the sponsoring agreements we reviewed. According to DOD’s implementing guidance, to participate in the pilot program, FFRDCs must agree to and follow these protections, which are to be incorporated into FFRDCs’ sponsoring agreements. We found that all six participating FFRDCs’ sponsoring agreements were updated and that most of the protections were incorporated. However, none included the instructions for FFRDC personnel involved in source selections to notify contracting officers if they are contacted about employment by an entity whose proposal is being evaluated and recuse themselves. We also found that the sponsoring agreements omitted one of the three officials that should be notified in the event of a Trade Secrets Act violation. These details were also not included in the templates DOD provided sponsors to use when updating FFRDC sponsoring agreements. When we raised these gaps to the attention of the DOD office responsible for managing the pilot program, the officials we spoke with were unaware of these omissions. In addition, the Laboratories and Personnel Office has not taken steps to ensure that another protection—the certification of the annual review of financial disclosure forms—has occurred, even though it was incorporated into the sponsoring agreements. The implementing guidance states that FFRDCs’ parent organizations must certify the annual review of financial disclosure forms and archive these forms for 6 years. However, only two FFRDC parent organizations provided us with this certification. According to representatives from parent organizations of the other four FFRDCs, the review of financial disclosures is generally performed as part of their conflict of interest programs. They review the disclosures on an annual or rolling basis when researchers are assigned to new projects but had not certified, as the sponsoring agreements require, that they have taken this step for the pilot program. We found that the Laboratories and Personnel Office had not taken steps to verify FFRDC parent organization compliance with this protection, such as collecting or reviewing the certification. When we raised this gap to the attention of the DOD office responsible for managing the pilot program, the officials were unaware of the missing annual certifications. By not ensuring the annual review is occurring, DOD has limited information about FFRDCs’ adherence to this pilot program protection. The pilot’s implementing guidance also states that, before government personnel provide access to sensitive data, the FFRDCs and researchers must have addressed these protections. However, the Laboratories and Personnel Office has not taken steps to ensure it is done. In our review of the six specific projects, we found that different people were checking that some of the protections were in place. For example, For two of the six projects, a primary sponsor official had a copy of the FFRDC addendum, and collected and reviewed the nondisclosure agreements and certifications of financial disclosure for individual researchers on each project. For a third project, a DOD official in the office that requested the project told us she confirmed that FFRDC researchers working on the project were part of the pilot program and told the official from the data repository that he could share information with the researchers. For the remaining three projects, representatives for a data repository that provided researchers with data access told us they confirmed that the addendum was incorporated into the sponsoring agreement and that researchers had the individual protections, such as a nondisclosure agreement, in place before providing access to the data. Standards for internal controls in the federal government state that responsibilities for control activities, such as sponsors ensuring the protections are incorporated into the agreements and that FFRDCs are following these protections, should be documented through policy and procedures. Without a process that includes clearly defined roles and responsibilities to ensure the protections are followed, DOD cannot ensure that its goal to safeguard sensitive data is achieved. DOD Is Collecting Some Pilot Information but Lacks a Plan for Evaluation DOD Collects Information from FFRDCs Quarterly but Does Not Ensure Comprehensive Reporting DOD established what information sponsors must collect about the projects enrolled in the pilot in its implementing guidance to sponsors and notified FFRDCs about these responsibilities. The requirements include: pre-action information to be collected when the project is enrolled in the pilot, which includes basic details about the project, the data required, and planned analysis; quarterly status updates, which include progress obtaining access to sensitive data and any challenges or barriers to access; and post-action information regarding the results of pilot access when the project’s analyses are completed, which includes a summary of how the pilot supported FFRDC research, and any benefits accrued to DOD from pilot participation. The implementing guidance also instructs sponsors to collect information about the project’s results again 6 months after the project is completed. The Laboratories and Personnel Office, which is responsible for managing the pilot program, sends an email quarterly requesting that sponsors submit information. However, we found that the Laboratories and Personnel Office did not receive pre-action information from 11 of the 33 projects in the pilot program. For example: Systems and Analyses Center has not submitted pre-action information for six of its eight projects. An official from the Systems and Analyses Center’s primary sponsor office told us that in his view, pre-action information can be obtained by other means, and he had not requested it. Project AIR FORCE and Arroyo Center have submitted project pre- action information for five projects to the FFRDCs’ primary sponsors: Air Force and Army, respectively. However, the primary sponsors have not provided this information to the Laboratories and Personnel Office. An Air Force official explained that he gets a request from the Laboratories and Personnel Office for the quarterly reports, but not the pre-action information, and thus had not provided it. In addition, the office did not collect a quarterly report for three projects in the pilot, and, as of September 2019, two completed projects had passed the 6-month post-completion time frame and only one had submitted post-action information. These gaps in reporting have occurred because the Laboratories and Personnel Office is not monitoring the project information it receives to ensure sponsors are submitting all required reporting. DOD’s implementing guidance states that primary sponsors will collect and submit this information for each project enrolled in the pilot program. Further, GAO’s leading practices for pilot design state, among other things, that a well-designed pilot program should have a clear approach to gathering information for the purpose of supporting the future evaluation of the pilot and tracking the pilot program’s implementation and performance. Consistency in collecting pre-action, quarterly, and post- action reports is important because each contains different information, which could be useful for the department to track the pilot program’s progress and in an evaluation of the pilot program. For example, without the pre-action information from Systems and Analyses Center, Arroyo Center, and Project AIR FORCE, the Laboratories and Personnel Office will not have general descriptions of their pilot projects or information about the kinds of data these FFRDCs initially planned to access. Without complete information, DOD will not be able to effectively evaluate the pilot program and inform future decisions about the program’s status. DOD Established Pilot Goals but Does Not Have a Plan to Evaluate Its Results We found that DOD followed some but not all of the leading practices for evaluating its pilot program. According to GAO’s leading practices for pilot design, a well-developed and documented pilot program can help ensure that agency assessments produce information needed to make effective program and policy decisions. Such a process enhances the quality, credibility, and usefulness of evaluations, in addition to helping to ensure that time and resources are used effectively. Five leading practices form a framework for effective pilot design and evaluation. (See figure 2.) We found DOD generally addressed the first of the leading practices of pilot design—establish objectives—by establishing goals for the pilot program, summarized below. Make sensitive data previously restricted or unavailable available for analysis. Use sensitive data in accordance with the FFRDC contract. Safeguard sensitive data. Document results of pilot program. Document risks or costs of FFRDC access to sensitive data. Gain analytic value from FFRDC access to sensitive data. Demonstrate benefits to government from sharing sensitive data. Inform future actions for making FFRDC access to sensitive data permanently available. However, DOD has not fully addressed the other leading practices. Specifically, we found that the Laboratories and Personnel Office does not have: a plan that (1) includes an assessment methodology to ensure DOD is collecting the correct information to evaluate whether the pilot has met the department’s goals and (2) defines how DOD will use the information collected to evaluate the implementation and performance of the pilot program, when the evaluation will take place, and by whom; a plan for identifying or documenting lessons learned; and a plan for gathering input from stakeholders, such as DOD sponsors, FFRDCs, and officials from DOD’s data repositories, for the pilot program’s evaluation. According to officials involved in its implementation, DOD did not consider creating such plans when developing the pilot program. The pilot program guidance, however, stated that information collected would be used for the department to assess the ongoing efficacy of the pilot program and GAO’s evaluation. These officials explained that, when the pilot’s guidance was formulated, the department was in the process of reorganizing the former Acquisition, Technology, and Logistics Office and they pointed out that GAO was to do the assessment of the pilot program. They said the reason the program collected information—such as quarterly reports—was to inform our review. Thus, they had no plan to assess the information collected and no plans to talk to stakeholders or to collect and share lessons learned. While our review occurred during pilot implementation, an evaluation of the pilot conducted by DOD after more projects are completed would provide an opportunity to identify lessons learned and gather valuable input from stakeholders—such as the offices that manage the data repositories and the sponsors requesting the projects. We found cases where FFRDC researchers had problems accessing data and where gathering this input from stakeholders involved with pilot projects would have been useful for DOD. For example, some FFRDC researchers described barriers when trying to gain access to certain government- and department-wide databases. In one case, Project AIR FORCE researchers reported not being able to access information in the Electronic Document Access database and various databases containing contractor performance information because researchers lacked military or government email addresses. Further, some of the databases that FFRDC researchers and a DOD official said would be useful are not owned by DOD. We found that guidance for one such database explicitly prohibits disclosure of contractor evaluation data to any contractor or non- government entity. In addition, the researchers were able to gain only partial access to DOD’s Acquisition Information Repository (a database containing acquisition documents for DOD’s major weapons programs) and, as a result, were unable to access individual documents, such as program assessment reports. They told us the repository is set up such that the researchers must request access to individual documents directly from document owners, who set permissions when uploading documents, rather than from a central source that can grant access across the repository. Without further evaluation of the pilot, DOD is missing an opportunity to benefit from gathering input from its stakeholders and identifying lessons learned, such as learning and understanding more about these barriers to accessing certain databases. There is still time for DOD to develop an evaluation plan with elements described in our leading practices. The pilot program ends December 21, 2020. Our review comes at a time when 11 of the overall 33 projects have been completed; therefore, information exists to report on outcomes. Officials and representatives from the Laboratories and Personnel Office and participating sponsors and FFRDCs expressed a continued need for access to the sensitive data. Without an evaluation plan, DOD will have difficulty determining the effectiveness of the pilot to meet its goal of accruing more analytic value for the department while also safeguarding sensitive data. Conclusions The FFRDC pilot program has already provided DOD with some benefits, as a few FFRDCs have reported success in completing analysis that would not have been possible without it. However, in implementing the pilot, DOD has room for improvement. A key control of this pilot that provides access to sensitive data is ensuring protections are in place to prevent improper disclosure. Another control is to establish a process to ensure these protections are followed, yet the responsible office within DOD has not done so. Further, despite the fact that the pilot is past the midpoint of implementation, this office still has an opportunity to develop a plan on how to evaluate it. But to do this, it must develop a mechanism to ensure it is collecting complete information on the pilot activities. Ensuring comprehensive reporting and implementing a well-developed evaluation plan will help DOD understand and articulate the benefits the department has accrued because of FFRDC’s access to sensitive data. Further, through identifying lessons learned and obtaining stakeholder input, the Laboratories and Personnel Office has an opportunity to better understand the challenges FFRDCs and the department face when attempting to access and use sensitive data included in government- and department-wide databases. Such an evaluation could help inform Congress’ decision whether to extend, make FFRDC access permanent, or end the pilot. Recommendations for Executive Action We are making the following six recommendations to the Department of Defense: The Under Secretary of Defense for Research and Engineering should direct the Laboratories and Personnel Office to take steps to ensure that the details of the pilot program’s data protections are incorporated into the existing agreements. (Recommendation 1) The Under Secretary of Defense for Research and Engineering should direct the Laboratories and Personnel Office to take steps to ensure that the FFRDCs and sponsors are implementing the pilot program’s protections for sensitive data. (Recommendation 2) The Under Secretary of Defense for Research and Engineering should direct the Laboratories and Personnel Office to establish a monitoring and oversight mechanism to ensure that primary sponsors submit complete information on pilot projects, as required by DOD’s guidance for the pilot program. (Recommendation 3) The Under Secretary of Defense for Research and Engineering should direct the Laboratories and Personnel Office to develop a plan that outlines the methodology by which DOD will assess the pilot and how and when information collected will be analyzed to evaluate the pilot program. (Recommendation 4) The Under Secretary of Defense for Research and Engineering should direct the Laboratories and Personnel Office to develop a plan to identify and evaluate lessons learned from the pilot program. (Recommendation 5) The Under Secretary of Defense for Research and Engineering should direct the Laboratories and Personnel Office to develop a plan for obtaining input from stakeholders on the pilot program. (Recommendation 6) Agency Comments We provided a draft of this product to DOD for comment. DOD provided a letter response, reproduced in Appendix I. DOD agreed with our recommendations and described actions that it intends to take in response. We also provided excerpts of this product to FFRDCs for comment, of which three provided technical comments that we incorporated as appropriate. We are sending copies of this report to the appropriate congressional committees and the Secretary of Defense. In addition, the report is available at no charge on the GAO website at http://www.gao.gov. If you or your staff have any questions about this report, please contact me at (202) 512-4841 or makm@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are listed in Appendix II. Appendix I: Comments from the Department of Defense Appendix II: GAO Contact and Staff Acknowledgments GAO Contact Staff Acknowledgments In addition to the contact named above, Tatiana Winger (Assistant Director), Leslie Ashton (Analyst-in-Charge), Evan Nemoff, Tanya Waller, Jenny Chanley, Laura Greifner, Christine Pecora, and Roxanna Sun made key contributions to this report.


Background DOD’s policy is to ensure that eligible personnel and their families have access to affordable, quality housing facilities and services consistent with grade and dependent status, and that the housing generally reflects contemporary community living standards. From the inception of MHPI, the military departments were provided with various authorities to obtain private-sector financing and management to repair, re novate, construct, and operate military housing in the United States and its territories. Through these authorities, the military departments have entered into a series of agreements with private partners to provide housing to servicemembers and their families. The military departments have flexibility in how they structure their privatized housing projects, but typically the military departments lease land to private developers for 50-year terms and convey existing housing located on the leased land to the developer for the duration of the lease. The developer then becomes responsible for renovating and constructing new housing and for the daily management of these housing units. At the end of fiscal year 2017, 14 private partners were responsible for 79 privatized military family housing projects—34 for the Army, 32 for the Air Force, and 13 for the Navy and the Marine Corps—in the United States, each of which includes housing at one or more military installation. Each privatized housing project is a separate and distinct entity governed by a series of legal agreements that are specific to that project. However, there are some common elements in how projects invest and use funds. Every project takes in revenue, which consists mostly of rent payments. Projects then pay for operating expenses, including administrative costs, day-to-day maintenance, and utilities, among other things. After that, projects generally allocate funds for taxes and insurance, followed by debt payments. In the typical funding structure for a privatized housing project, once debt payments are made, funds are allocated to accounts that fund scheduled maintenance, such as repair and replacement of items like roofs, heating and cooling systems, and infrastructure. After that, funds are then allocated to a series of management incentive fees, such as the property management fee. Finally, the project divides these remaining funds according to a fixed percentage between accounts that (1) fund major renovations and rebuilds and (2) are provided to the developer. The Deputy Assistant Secretary of Defense for Facilities Management, under the authority, direction, and control of the Assistant Secretary of Defense for Sustainment, is responsible for all matters related to MHPI and is the program manager for all DOD housing, whether DOD-owned, DOD-leased, or privatized. In this capacity, the Deputy Assistant Secretary is to provide both guidance and general procedures related to military housing privatization, as well as required annual reports to Congress on the status of privatized military housing projects. However, it is the responsibility of the military departments to execute and manage privatized housing projects, including conducting financial management and monitoring their portfolio of projects. Each military department has issued guidance that outlines its responsibilities for privatized housing, such as which offices are responsible for overseeing privatized housing projects. DOD Conducts Some Oversight of the Condition of Privatized Housing, but Efforts Are Limited in Key Areas In our draft report, currently with DOD for review and comment, we found that each military department conducts a range of oversight activities— some more extensive than others—for its privatized housing projects. For example, among other things, military departments review sample work order requests and inspect housing during the change–of-occupancy process. DOD guidance states that because privatization creates a long- term governmental interest in privatized housing, it is essential that projects be attentively monitored. Through its guidance, DOD delegates oversight responsibility of the individual privatized housing projects to each of the military departments. In our draft report, we noted that OSD and the military departments’ oversight efforts have been limited in the following key areas. Specifically, we found that (1) the scope of oversight of the physical condition of privatized housing has been limited; (2) performance metrics focused on quality of maintenance and resident satisfaction do not accurately reflect private partner performance related to the condition of privatized housing; (3) there is a lack of reliable or consistent data on the condition of privatized housing; and (4) past DOD reports to Congress on resident satisfaction are unreliable due to the inconsistent handling and calculation of the data and therefore may be misleading. Military Departments Conduct Some Oversight of the Physical Condition of Privatized Housing, but the Scope of Efforts Is Limited DOD delegates oversight responsibilities of the individual privatized housing projects to each of the military departments, and each military department has subsequently issued guidance outlining oversight roles and responsibilities. Military department oversight activities generally fall into two categories—(1) daily oversight of management and operations and (2) periodic reviews of compliance with each project’s business agreements. Daily oversight of management and operations. Daily oversight of a project’s management and operations is to be conducted by each installation’s military housing office. Military housing officials told us that activities to monitor the physical condition of housing units generally include reviewing sample work order requests, following up with a sample of residents to check on their experience with recently completed work, and inspecting housing units during the change-of-occupancy process. As we noted in our draft report, the implementation and scope of these activities varies and can be limited. For example, during our site visits conducted from June through August 2019, we observed that the rate of inspections of homes following change-of-occupancy maintenance varied at the installations we visited. Military housing office officials at one Air Force installation told us that they inspect 100-percent of homes that have completed change-of-occupancy maintenance, while officials from a different Air Force installation stated they inspect 10 to 20 percent of these homes. Military department officials told us that in spring 2019, each department conducted a “100-percent” review of privatized housing by directing installation commanders to contact all residents of privatized housing and offering a visual inspection of their privatized housing unit. In addition, in March 2019 the Army issued an order directing military housing office officials to inspect 100-percent of homes where change-of-occupancy maintenance has been completed. Officials from Army installations we visited noted that this was an increase from previous practices, and for one installation was a change in practice from only conducting inspections during the move-out process, which occurs prior to change- of-occupancy maintenance. Similarly, in November 2019, Air Force officials told us they were moving to a 100 percent inspection policy. Periodic reviews of compliance with each project’s business agreements. Periodic reviews of compliance with a project’s business agreements are a joint effort between the local military housing office, the private partners, military department installation commands, and other echelons of command. These reviews can include neighborhood tours to view project amenities such as community centers, playgrounds, and pools, all of which are owned, maintained, and operated by the private partner companies, as well as exteriors of housing units. However, our draft report showed these annual reviews have been narrow in the scope of their assessment of the physical condition of the housing units, as interior walk-throughs were, at times, focused on just a few homes at each installation. According to military department officials, each department has completed initiatives and is undertaking initiatives to revise guidance and standardize daily oversight activities in an effort to provide consistent oversight across projects and installations, and to increase the focus on the physical condition of housing. In addition, the military departments have initiatives to increase staffing levels, improve training for military housing office officials, and ensure that military department housing officials have independent access to work order data, to strengthen their oversight activities. However, each military department is working to implement service-specific initiatives with only limited guidance from OSD on the level of oversight expected of the services as it relates to the condition of the housing. Specifically, OSD guidance is focused on the oversight of the implementation of projects, the construction of new housing units, and project financial monitoring. The guidance stipulates that after privatized housing projects are awarded, monitoring should include descriptions of deal structure and strategies for project monitoring. In contrast, OSD guidance for military-owned housing provides clearly defined objectives to the military departments for oversight, including the physical condition of the homes. Unless OSD updates its guidance on the oversight of privatized housing with objectives for overseeing the physical condition of housing units, it cannot be assured that the military departments’ oversight activities will be sustained over time or be sufficiently consistent across projects, raising the risk that private partners may not provide adequate quality housing. DOD Uses Several Metrics to Monitor Private Partner Performance, but the Indicators Underlying Those Metrics Do Not Provide Meaningful Information on the Condition of Privatized Housing The military departments each use a range of project-specific performance metrics to monitor private partner performance, but as we note in our draft report, the metrics designed to focus on resident satisfaction and on the quality of the maintenance conducted on housing units do not provide meaningful information or reflect the actual condition of the housing units. Most, but not all, of the private partners are eligible to receive performance incentive fees based on generally meeting the performance metrics established in each individual project’s business agreement. Private partner performance is commonly measured through four key metrics—resident satisfaction, maintenance management, project safety, and financial management. To determine how well the private partners are performing under the metrics, military housing office officials told us they rely on a range of specific indicators established in the project business agreements. However, the indicators themselves do not provide meaningful information on the private partner’s performance in maintaining quality housing units. For example, we identified the following in our draft report: Maintenance management. One indicator of performance of maintenance management that is regularly included in project business agreements measures how often the property manager’s response time to work orders meets required timeframes established in the project’s business agreements. While this indicator measures the timeliness of the private partner’s response, it does not measure or take into account the quality of the work that was conducted or whether the resident’s issue was fully addressed. As such, a property manager may fully meet the metric for maintenance management even if a given repair has not been adequately completed. Residents in 13 of our 15 focus groups noted that they typically have had to submit multiple work order requests before an individual maintenance issue has been fully addressed. Some projects include indicators that aim to more directly measure quality, such as the number of work orders placed during the first 5 business days of residency, which may indicate the extent to which all of the change-of-occupancy maintenance was completed. Resident satisfaction. One example of an indicator of resident satisfaction is whether a project has met target occupancy rates established in the project’s business agreements. An OSD official and private partner representatives told us they use occupancy as an indicator of satisfaction based on the assumption that residents would move if they were dissatisfied with their home’s condition. However, based on our focus groups, this may not be a reliable assumption. Although most residents are not required to live in military housing, residents in each of our 15 focus groups indicated a variety of reasons for choosing to live in privatized housing, many of which did not have to do with their satisfaction with the quality or condition of their homes. For example, residents in our focus groups cited other factors influencing their decision to live in privatized housing, such as living in close proximity to military medical or educational services for children or other family members that are part of the military’s Exceptional Family Member Program, a lack of safe and affordable housing in the surrounding community, and access to quality schools. OSD and military department officials have recognized that the current indicators for measuring performance do not consistently focus on or prioritize the private partners’ performance with maintaining housing units and ensuring resident satisfaction. For example, Army officials told us they are no longer using occupancy rates as an indicator of resident satisfaction and have taken steps to standardize performance indicators across all Army projects, while still allowing for flexibility at the installation level to modify the weight of indicators to provide incentives reflective of the specific needs of the installation. Limitations to the current indicators may hinder the military departments’ ability to accurately determine private partner performance. However, OSD and military department officials told us they have not yet reevaluated the specific indicators used to determine whether a private partner has met a specific metric because doing so will require negotiation with each of the private partners for each project. Nonetheless, without reviewing the specific indicators used to award performance incentives, OSD and the military departments do not have assurance that the information the military departments are using to award these incentives reflects the actual condition of the housing. DOD and Private Partners Collect Maintenance Data on Privatized Housing, but These Data Are Not Captured Reliably or Consistently for Use in Ongoing Monitoring of Housing Units The housing projects’ business agreements typically include a requirement for the private partner to maintain a records management system to record, among other things, maintenance work requested and conducted on each housing unit. According to private partner officials, each company uses commercial property management software platforms for activities such as initiating maintenance work orders and dispatching maintenance technicians. Some private partner representatives stated that while data from the work order tracking systems are primarily used to prioritize and triage maintenance work, the data were never intended to monitor the overall condition of privatized housing units. While data from these work order tracking systems may be useful for point-in-time assessments of work order volume at a given installation, military department officials told us that efforts are underway to monitor work order data to increase the military departments’ oversight and the accountability of the private partners for providing quality housing. However, as we noted in our draft report, we found that these data are not captured reliably or consistently for use in the ongoing monitoring of the condition of privatized housing units. We received and reviewed data from each of the 14 private partners’ work order tracking systems covering each of the 79 privatized family housing projects. Based on our review of these data and discussions with private partner representatives for our draft report, we found two primary factors that would limit the reliability or consistency of using these data for ongoing monitoring of the condition of privatized housing units over time—(1) inconsistent use of terminology in work order records and (2) differing practices for opening and closing work orders: Inconsistent use of terminology. Based on our review of the data provided by the private partners and discussions with private partner officials, we noted cases where work orders were inconsistently entered into the work order tracking systems with respect to two primary factors—(1) how the request is described by the resident or interpreted by the official entering the data, which can differ for each work order, and (2) the existing range of pre-established service category options in the private partner’s work order tracking system, which differ among the partners. Differing practices for opening and closing work orders. At some installations we visited, private partners noted changes in practices for opening and closing work orders, limiting the usefulness of the data in monitoring the status of work orders over time and thus the condition of privatized housing. In addition, we identified other anomalies in work order data from each of the 14 partners. For example, we identified instances of, among other things, duplicate work orders, work orders with completion dates prior to the dates that a resident had submitted the work order, and work orders still listed as in-progress for more than 18 months. According to military department officials, efforts to review data from the private partners’ work order tracking systems have increased, and military department officials told us they have found similar limitations. However, neither OSD nor the military departments have identified minimum data requirements, established consistent terminology or practices for data collection, or developed processes for the military departments to validate the work order data collected by the private partners. Without direction from OSD to establish minimum data requirements and consistent terminology or practices for data collection, as well as a requirement for the military departments to validate data, the military departments’ ability to use data from the private partners’ work order tracking systems to monitor the condition of privatized homes over time will remain limited and may vary across projects. DOD Provides Reports to Congress on Resident Satisfaction with Privatized Housing, but Data in These Reports Are Unreliable, Leading to Misleading Results DOD is statutorily required to provide reports to Congress that include, among other things, information about military housing privatization projects’ financial health and performance and backlog, if any, of maintenance and repairs. These reports have included information on resident satisfaction with privatized housing based on the results of the annual military department satisfaction surveys. As we state in our draft report, we determined that information on resident satisfaction in these reports to Congress on privatized housing have been unreliable and are misleading due to (1) variances in the data the military departments collect and provide to OSD and (2) OSD’s calculation and presentation of the data. In May 2019, OSD issued its report for fiscal year 2017, which stated that overall resident satisfaction for calendar year 2017 was 87 percent. For OSD’s fiscal year 2017 report, the military departments provided data on resident satisfaction based on information from the annual resident satisfaction surveys. Specifically, OSD’s instructions to the military departments required the military departments to report satisfaction based on resident responses to the question that asks: “Would you recommend privatized housing,” with results indicating how many tenants responded “yes,” “no,” or “don’t know.” However, the military departments’ approaches for collecting data in their annual resident satisfaction surveys vary, which limits their ability to assess whether residents would recommend privatized housing. Instead of asking whether residents would recommend privatized housing, the military departments’ annual resident satisfaction survey asks residents the following: “How much do you agree or disagree with the following statement, ‘I would recommend this community to others.’” A resident’s satisfaction with his or her community and inclination to recommend it to others may not be reflective of satisfaction with either the privatized housing unit or privatized housing in general. Residents are then provided the following response categories on a scale of five to zero: (5) strongly agree, (4) agree, (3) neither agree nor disagree, (2) disagree, (1) strongly disagree, and (0) not applicable, no opinion, don’t know, or no answer. Through our analysis, we have identified variances in the methods that each of the military departments use to translate the residents’ responses into the “yes,” “no,” or “don’t know” categories. The variances in how the military departments calculate “yes,” “no,” or “don’t know” resulted in inconsistencies in how resident satisfaction is ultimately reported to Congress. For example, for the fiscal year 2017 report, Navy and Army officials told us they counted responses reported in category 3 (neither agree nor disagree) as “don’t know.” For the same time period, however, Air Force officials told us they counted responses in category 3 (neither agree nor disagree) as “yes.” If the Air Force had not counted category 3 as “yes,” reported resident satisfaction rates would have been lower. For example, for one Air Force installation, if officials had not counted responses in category 3 as “yes,” the resident satisfaction rate for newly constructed units would have been more than 20 percent lower than what was reported. In our draft report, we also identified instances of errors and inaccuracies in how OSD calculates these data and reports on resident satisfaction to Congress. Specifically, we found missing data points and incorrect formulas, among other errors, in OSD’s calculation of the data submitted by the military departments for OSD’s fiscal year 2017 report to Congress. For example: The formula used by OSD to calculate overall resident satisfaction for the fiscal year 2017 report did not include data for several projects, including for four Army projects that, as of September 30, 2017, accounted for over 18 percent of the Army’s total housing inventory. For one Air Force project, OSD reported identical resident satisfaction data for the fiscal year 2015, 2016, and 2017 reports, despite the fact that Air Force officials had noted in their submissions to OSD that the resident satisfaction data were from the annual resident satisfaction survey conducted in December 2013. In our draft report, we also found that presentation of data in OSD’s report to Congress may be misleading because OSD did not explain the methodology it used to calculate the overall resident satisfaction percentage or include caveats to explain limitations to the data presented. Specifically, OSD did not include information on overall response rates to the annual satisfaction survey for each military department, nor did it include response rates by project. Low response rates can create the potential for bias in survey results. For example, in the report for fiscal year 2017, OSD reported that 25 percent of residents living in renovated housing units for one privatized housing project were satisfied with their housing, but we found that only four residents had provided responses to this question. Thus, only one resident reported being satisfied. In addition, we found that OSD did not provide an explanation in the report for why five projects were listed as “not applicable.” According to OSD officials, this error was a quality control issue that they plan to address. According to OSD officials, there are no plans for quality control in development at this time. The National Defense Authorization Act for Fiscal Year 2020 (fiscal year 2020 NDAA) includes a provision requiring each military installation to use the same satisfaction survey for tenants of military housing— including privatized military housing—the results of which are not to be shared with private partners until reviewed by DOD. Until OSD makes changes to the data collection and calculation efforts that make up the department’s report to Congress and provides explanations of the data in the reports, OSD will not be able to provide Congress with an accurate picture of resident satisfaction with privatized housing. Military Housing Offices Have Not Effectively Communicated Their Role as a Resource for Servicemembers Experiencing Challenges with Privatized Housing Military housing office officials, located at each installation, are available to provide resources to servicemembers experiencing challenges with their privatized housing, among other services. However, as we stated in our draft report, we found that these offices have not always clearly and systematically communicated this role to residents of privatized housing. The military housing office is to provide new residents with information on their local housing options, to include referral services for housing options. According to some military housing office officials, the military housing office then works with the private partner to identify the eligibility and type of home the servicemember qualifies for, if the resident chooses to live in privatized housing. According to some residents we spoke with in one of our focus groups, beyond this initial interaction, military housing office officials generally do not interact with residents on a regular basis. Additionally, residents who participated in our focus groups noted they were sometimes confused about the military housing offices’ roles and responsibilities with regard to the maintenance of their home; there was a perception that the military housing office was not working independently of the partner in the residents’ best interest; or they did not know the military housing office existed. The military department oversight agencies have also found that the military departments have not clearly and systematically communicated their roles to residents, and resident confusion and a lack of awareness regarding the role of the military housing offices is an issue. In April 2019 the Air Force Inspector General reported that less than half of the residents interviewed used their military housing office to resolve complaints, and at some installations officials visited, many residents did not know the military housing office had an oversight role. Similarly, in May 2019, the Army Inspector General reported to the Secretary of the Army that at 82 percent of Army installations with privatized housing, residents did not know how to escalate issues to either the private partner or the Army housing office. Additionally, the Army Inspector General reported that installation command teams and staff cited multiple circumstances where military housing offices and tenant advocacy roles and responsibilities were unclear. Further, some military housing office officials with whom we spoke during our site visits acknowledged the gap in resident awareness regarding the existence and purpose of the military housing office. Some military housing officials also noted that some residents are unaware of the difference between the military housing office and the private partner office, due in part to their physical co- location and unclear building signage. Each military department has issued information that establishes that its housing offices can assist in the resident dispute resolution process. Specifically, if servicemembers are experiencing a dispute with a private partner, military department guidance establishes varying roles for their respective military housing office officials. For example, Army policy states that each installation should have an official tasked with supporting servicemembers regarding resident issues that cannot be resolved by the private property manager. This individual is also responsible for resolving every resident complaint and the military housing office, if required, can request mediation by the garrison commander. OSD has recognized that the military departments’ communication with residents about their role as a resource for them has been limited. In February 2019, the Assistant Secretary of Defense for Sustainment testified before Congress that a way forward in addressing resident concerns would require focus in three key areas: communication, engagement, and responsiveness. Some military housing office officials told us they have taken steps to increase resident awareness, such as increasing the advertising of the military housing office’s role and contact information, conducting town hall meetings, and rebranding their military housing offices to differentiate them from the private partners. For example, a Marine Corps housing office official stated that the housing office established a document, which is distributed to residents by the private partner, informing residents of housing office contact information and the service’s three-step dispute resolution process, but efforts have not been standardized across all projects. Moving forward, having plans in place to clearly and systematically communicate the difference between the military housing office and the private partners—including the military departments’ roles, responsibilities, and military housing office locations and contact information—will better position the military departments to achieve the intended objectives of their initiatives aimed at improving residents’ experience. DOD and Private Partners Are Implementing Initiatives to Improve Privatized Housing, but May Face Challenges OSD, the military departments, and the private partners have identified and begun collaborating on a series of initiatives aimed at improving residents’ experiences with privatized housing, but as we state in our draft report currently with DOD for review and comment, these efforts face challenges. In addition, in the fiscal year 2020 NDAA, Congress established several requirements regarding privatized military housing reform. Several of the statutory requirements provide specific provisions that DOD will need to incorporate into its development and implementation of existing MHPI initiatives, as well as additional requirements aimed at improving oversight of privatized housing. In our draft report, we discuss several of these key initiatives, including the following. Development of a resident bill of rights. DOD has been working to develop a resident bill of rights intended to provide clarity to residents on their rights and responsibilities while living in privatized military housing. The fiscal year 2020 NDAA includes specific requirements to be included in the bill of rights, for example, ensuring residents have the right to have their basic allowance housing payments segregated and held in escrow, with approval of a designated commander, and not used by the property owner, property manager, or landlord pending completion of the dispute resolution process. In January 2020, DOD officials told us that they were in the process of updating their existing resident bill of rights to include these provisions. In February 2020, the Secretary of Defense signed the resident bill of rights, noting that the rights would be available to residents on May 1, 2020. Implementation of a common (enterprise) dispute adjudication process that will apply to all projects. The military departments and private partners have been working to develop a common dispute resolution process that would apply to all privatized housing projects. The fiscal year 2020 NDAA includes requirements reinforcing this initiative, specifically stating that the military department Secretary concerned shall implement a standardized formal dispute resolution process to ensure the prompt and fair resolution of disputes between landlords providing housing units and tenants residing in housing units concerning maintenance and repairs, damage claims, rental payments, move-out charges, and such other issues relating to housing units as the Secretary determines appropriate. Additionally, the statute requires that each military department Secretary designate the installation or regional commander in charge of oversight of housing as the deciding authority under the dispute resolution process. Reviewing MHPI resident satisfaction data collection process and the process by which DOD measures and reports on resident satisfaction data. According to OSD officials, the department is reviewing the process by which it measures and reports resident satisfaction data, and has plans to review the survey questions used to measure resident satisfaction. In line with these planned efforts, the fiscal year 2020 NDAA further requires that DOD’s reports to Congress include additional information, such as the results of residence surveys and other factors related to the condition of privatized housing. Standardizing Performance Incentive Fee Ranges. In October 2019, OSD issued new guidance standardizing the performance incentive fee ranges across the military departments. The fiscal year 2020 NDAA requires that DOD publically report information regarding the use of performance incentive fees. The statute also requires that DOD take into consideration any decision a commander renders in favor of the tenant in the formal dispute resolution process in determining whether to pay or withhold all or part of any incentive fees for which a private partner may otherwise be eligible under the contract. In addition to requirements impacting current DOD initiatives, the fiscal year 2020 NDAA included requirements for increased oversight of the physical condition of privatized housing. For example, the statute requires the Secretary of Defense to designate a Chief Housing Officer to oversee housing units, including the creation and standardization of policies and processes regarding housing units. The statute also requires the Secretary of Defense to establish a uniform code of basic standards for privatized military housing, as well as plans to conduct inspections and assessments of the condition of privatized homes. However, both DOD and private partner representatives have cited several challenges that could affect their ability to implement initiatives aimed at improving MHPI. Specifically: Timeliness of implementation due to the need to collaborate with and obtain input and agreement from the large number of stakeholders involved in privatized housing. According to DOD officials and private partner representatives, many of the initiatives designed to improve privatized housing not only require agreement between DOD and the private housing partners, but also discussion with and, in some cases, approval by the project bond holders. Because DOD does not have the ability to unilaterally make changes to existing business agreements, this need for stakeholder agreement limits DOD’s control over the implementation timeline of any initiative that requires changes to a project’s business agreement. The need for more military department staff with targeted expertise. The military departments reduced their involvement in daily privatized military housing operations as part of the overall privatization effort, to include reducing staffing levels at the installations. Military housing office officials at over half of the installations we visited stated that reduced staffing levels impacted their ability to carry out oversight duties, such as work order data analysis and housing inspections. Each of the military departments has plans to increase the military housing office staffing at each installation to allow for enhanced oversight. In particular, according to military department officials, these positions will focus on quality control and quality assurance of the maintenance of privatized homes. The potential for unintended negative financial impacts on the projects that could outweigh the intended benefits of the initiatives. OSD officials and private partner representatives have expressed concern that some initiatives could result in unintended financial consequences for the housing projects. For example, increased frequency of change-of-occupancy inspections could result in homes remaining vacant longer than planned and therefore not collecting rent. This could unintentionally impact a project’s cash flow. Some of the private partners noted that the financial impact of unfunded requirements to projects that are already experiencing financial distress could result in even fewer funds available to reinvest in improvements to the current and future physical condition of the homes. Without assessing risks to the financial viability of the MHPI projects associated with the implementation of these initiatives aimed at improving privatized housing, DOD’s efforts to improve the privatized housing program over the long term could be compromised. In summary, as we state in our draft report, we found that while DOD and the private partners have taken steps to address concerns raised about their ability to adequately maintain and oversee the condition of these housing units and provide quality housing for servicemembers, the extent to which the efforts will be sustained and result in improvements remains unclear. Our draft report includes several recommendations to OSD to strengthen oversight of MHPI, such as updating oversight guidance and assessing the risks to the financial viability of housing projects. Our draft report also includes recommendations to the military departments to enhance monitoring of privatized housing projects, such as improving processes used for data collection; reviewing private partner performance; collecting and reporting resident satisfaction data; and communicating with residents. Chairwoman Wasserman Schultz, Ranking Member Carter, and Members of the Subcommittee, this concludes my prepared statement. I would be pleased to respond to any questions you may have at this time. GAO Contact and Staff Acknowledgments If you or your staff members have any questions about this testimony, please contact Elizabeth A. Field, Director, Defense Capabilities and Management, at (202) 512-2775 or FieldE1@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this statement. Individuals who made key contributions to this testimony include Kristy Williams (Assistant Director), Tida Reveley (Analyst in Charge), Austin Barvin, Ronnie Bergman, William Carpluk, and Jordan Tibbetts. In addition, key support was provided by Vincent Buquicchio, Juliee Conde- Medina, Mae Jones, Kelly Rubin, Monica Savoy, John Van Schaik, Madeline Welter, and Kelsey Wilson. This is a work of the U.S. government and is not subject to copyright protection in the United States. The published product may be reproduced and distributed in its entirety without further permission from GAO. However, because this work may contain copyrighted images or other material, permission from the copyright holder may be necessary if you wish to reproduce this material separately.