Background HUD created REAC in 1997 to obtain consistent information on, among other things, the physical condition of its public and multifamily properties. REAC generally inspects properties every 1 to 3 years, using a risk-based schedule (discussed in detail below). REAC developed a standardized protocol to inspect properties, referred to as the Uniform Physical Condition Standards. As part of the protocol, REAC also inspects properties to identify health and safety deficiencies, including exigent health and safety deficiencies, which are life-threatening and require immediate action or remedy (such as exposed electrical wires or blocked access to windows or doors in case of a fire). REAC’s data system automatically generates an overall inspection score for the property from 0 to 100 based on the information an inspector records. At the end of each day of an inspection, an inspector is required to inform a property manager or other representative if the inspection identified exigent health and safety issues. Before releasing the inspection score, REAC reviews the inspection through a quality assurance process to ensure it is accurate. Following verification of the inspection score, REAC releases an inspection report to the property owner or PHA and the relevant HUD program office. The inspection report contains the overall inspection score, as well as more detailed information on physical deficiencies identified during the inspection. REAC primarily uses contractors—who are trained and certified in REAC’s Uniform Physical Condition Standards protocol—to conduct inspections of multifamily and public housing properties. In addition to these contract inspectors, REAC uses quality assurance inspectors, who are HUD employees, to oversee and monitor contract inspectors, as well as to ensure that REAC provides accurate and reliable inspections. Both contract and quality assurance inspectors complete several phases of training on the inspection protocol, including online, classroom, and field- based training. To procure inspections of HUD-assisted properties, REAC primarily uses an auction process to award contracts either to eligible contract inspectors or to companies that employ contract inspectors. This process, called a reverse auction program, occurs at least once a quarter. Contract inspectors or companies bid to inspect properties across the United States and its territories in a web-based auction. At the close of the auction, REAC awards the inspection to whoever bids the lowest price and is eligible to conduct inspections. The contract inspector then schedules and performs the property inspections in accordance with Uniform Physical Condition Standards protocol. According to REAC officials, this process is designed to increase cost savings and small business participation. REAC Roles and Responsibilities REAC is situated within PIH. Several departments within REAC are involved in facilitating the physical inspection process: Physical Assessment Subsystem (PASS): PASS has three primary divisions that are responsible for different aspects of the inspection process. The PASS Physical Inspection Operations division coordinates the procurement of inspections. The PASS Quality Assurance division evaluates and monitors REAC’s inspection program to ensure reliable, replicable, and reasonable inspections; trains contract and quality assurance inspectors; and provides technical assistance to HUD-assisted properties and other relevant stakeholders. The PASS Inspector Administration division monitors the performance of inspectors and takes administrative actions, such as decertifying inspectors who do not meet REAC’s standards for inspectors. Research and Development: REAC’s Research and Development division produces data analysis and statistical reports on REAC’s information products (e.g., physical inspection reports and Public Housing Assessment System scores) and assesses these products to ensure they are accurate and valid. REAC is also responsible for evaluating additional conditions, beyond physical conditions, of multifamily and public housing properties. Specifically, REAC evaluates the financial conditions of multifamily properties and assesses the financial and management performance of public housing properties. This performance assessment is conducted through the Public Housing Assessment System. REAC uses several data systems to collect, score, and report on the financial and management conditions of public housing properties, along with evaluating the utilization of property modernization and development funds (capital funds). We describe this process in more detail later in the report. HUD Offices Involved in Monitoring and Enforcement of Physical Condition Standards HUD’s PIH, Multifamily Housing, and Departmental Enforcement Center are responsible for ensuring that the owners of REAC-inspected properties (including PHAs) correct the identified physical deficiencies. PIH: This office helps low-income families by providing rental assistance through three programs; our review focuses on physical inspections of the public housing program. In 2018, HUD’s public housing program provided low-rent housing units to over 1 million eligible households. Public housing consists of reduced-rent developments owned and operated by local PHAs and subsidized by the federal government. About 3,300 PHAs own and manage public housing properties. These properties can include high-rise and low-rise buildings and scattered single-family properties, or they can be part of mixed-income housing developments, and they can range in size from fewer than 100 units to more than 30,000 units. PHAs typically have an executive director to manage their operations, as well as a governing board—called a Board of Commissioners—to approve policy, clarify goals, and ensure compliance with federal regulations. PHAs have contracts, called Annual Contributions Contracts, with the federal government. Under the terms of their contracts, PHAs agree to administer their properties according to federal regulations, in exchange for federal funding in the form of operating and capital grants. PIH is organized into six geographic networks, each with several field offices. Multifamily Housing: This office manages HUD’s portfolio of multifamily properties and provides rental assistance through several programs, including Section 8 project-based rental assistance, in which HUD contracts with private property owners to rent housing units to eligible low-income tenants for an income-based rent. Multifamily Housing also oversees the Federal Housing Administration’s multifamily mortgage insurance on loan originations and administers supportive housing for the elderly and programs for persons with disabilities. Collectively, the properties that Multifamily Housing oversees provided affordable rental housing to more than 1.2 million low-income households in 2017. Property owners or management agents of multifamily properties sign business agreements with HUD. Under these agreements, owners or agents agree to administer their properties according to federal rules and regulations, and in exchange, among other benefits, they receive federal assistance through mortgage insurance or housing assistance payments. Multifamily Housing has 12 field offices across five geographic regions. Departmental Enforcement Center: The Departmental Enforcement Center is located within HUD’s Office of General Counsel and works with several of HUD’s program offices, including PIH and Multifamily Housing, to ensure that program funds are used according to federal regulations. These program offices make referrals for the Departmental Enforcement Center to review the financial and other conditions of properties receiving rental assistance from HUD. Based on these reviews, the Departmental Enforcement Center can take various enforcement actions, such as imposing administrative sanctions to bar individuals from participating in HUD programs or civil money penalties for violations. Inspection Frequency REAC conducts inspections on multifamily and public housing properties using a risk-based schedule defined in federal regulations. According to our analysis of REAC inspection data, REAC conducted 44,486 inspections of multifamily properties and 15,156 inspections of public housing developments from fiscal years 2013 through 2017. For multifamily properties, REAC inspects properties every 1 to 3 years. Generally, properties that receive an inspection score below 80 are inspected within 1 year of the previous inspection; between 80 to 89 within 2 years; and 90 to 100 every 3 years. The inspection frequency for public housing developments varies depending on the overall size of the PHA (that is, the number of units and properties that they manage), an individual housing development’s inspection score, and the PHA’s overall performance on the Public Housing Assessment System. For PHAs with 250 housing units or more, REAC inspects developments every 1 to 3 years, using the same risk- based thresholds as Multifamily Housing. For small PHAs with fewer than 250 units, their score on the Public Housing Assessment System determines the inspection frequency, with higher scores associated with less frequent inspections. However, all developments—regardless of the number of units—that receive an overall performance assessment score (as part of the Public Housing Assessment System) of less than 60 out of 100 are designated to have a physical inspection every year. REAC’s Inspection Process Has Some Weaknesses That May Hinder Its Ability to Identify Physical Deficiencies REAC Has a Standardized Inspection Process for Identifying Physical Deficiencies at HUD- Assisted Properties REAC’s Uniform Physical Condition Standards inspection protocol is designed to help provide assurance that physical deficiencies will be identified at HUD-assisted properties. Under the protocol, contract inspectors inspect five areas of a property using a handheld data collection device to help identify and record deficiencies (see fig. 1). The devices have embedded software that provides step-by-step instructions on conducting the inspection. The software helps to ensure consistency between inspectors and consistency with the protocol, according to REAC staff. The software includes a decision-tree model to guide the inspectors on recording and classifying the severity of deficiencies they identify. For example, if an inspector identifies a deficiency with a door in a dwelling unit, the software will ask the inspector to identify which door has the deficiency and the nature of the deficiency (e.g., door lock does not work). The software then assigns a severity level to the deficiency and, if it is severe enough, requires the inspector to take a photo (see fig. 2). REAC has a number of quality assurance processes intended to ensure that contract inspectors identify deficiencies and conduct quality inspections: Collaborative quality assurance (CQA) review. In CQA reviews, REAC quality assurance inspectors observe contract inspectors to help ensure their inspections are accurate and consistent with protocol. REAC uses CQA reviews to coach contract inspectors to help improve their performance. Post-inspection review process. Completed inspections receive two levels of review by REAC quality assurance staff, who use software that compares certain aspects of the current and previous inspections—such as inspection scores, property profiles (for example, number of units), site measurements, and time taken to complete the inspection—and highlights large variances. Quality control inspection (QCI). If REAC reviewers find large variances in current and previous inspection scores and other aspects, they may reject the inspection and schedule a QCI. The QCI is a review of a previously inspected property to evaluate an inspector’s performance and identify potential weaknesses in the quality of the inspection. This review process requires a REAC quality assurance inspector to conduct a second inspection of the same property, including selecting the same sample of buildings and units of the original inspection. Once the QCI is completed, REAC’s reviewers, in collaboration with REAC’s Research and Development division, identify any deficiencies missed and determine whether the contract inspector was complying with REAC’s physical inspection standards. Property owners may appeal deficiencies REAC has identified during the physical inspection. For example, an owner might appeal a deficiency resulting from a window air conditioner blocking egress by providing evidence that this is permitted by local building code. If the appeal is successful, REAC removes the deficiency and the inspection software updates the score. Contract inspectors, REAC quality assurance inspectors, and representatives of property owner associations with whom we spoke had mixed views on REAC’s inspection process. Participants in three of the five discussion groups we held with contract and quality assurance inspectors said that the inspection process provides a comprehensive review of a property and that the inspection software helps promote consistency in inspections. Likewise, representatives from one property owner association we met with said that the inspection process was more standardized and less subjective than in the past. Representatives from another association said that the inspection process effectively identified deficiencies. However, participants in three of the same five discussion groups with contract and quality assurance inspectors noted inconsistent application of protocols and standards, noting that some cases were unclear and required judgment in identifying deficiencies. REAC’s inspection process has features similar to those of home inspection organizations such as the American Society of Home Inspectors (ASHI) and the International Association of Certified Home Inspectors (InterNACHI). For example, ASHI and InterNACHI have developed standards of practice that guide their inspectors on conducting inspections, similar to the role of REAC’s Uniform Physical Condition Standards inspection protocol. In addition, ASHI and InterNACHI require their inspectors to inspect the same five areas of a property that REAC does. Finally, ASHI and InterNACHI have codes of conduct that specify what constitutes ethical conduct for their inspectors; similarly, REAC has developed business rules that define ethical conduct for contract inspectors. REAC has made two major changes to the inspection process over the past 6 years. First, in 2012, REAC updated its inspection software to include the decision-tree model previously discussed and established a point-loss cap to limit the amount by which a single deficiency in an inspectable area could reduce the overall property score. For example, if an inspector found numerous tripping hazards within the same inspectable area, the inspector would record all instances of this hazard, but the software would only deduct from the inspection score once rather than multiple times, according to REAC staff. Second, in 2017, REAC updated its compilation bulletin to address concerns that property owners were making cheap, non-industry-standard repairs to disguise deficiencies during a REAC physical inspection. REAC now requires its inspectors to determine if deficiencies have been corrected consistent with industry standards. For example, property owners cannot use materials such as asphalt, caulking, spray foam, or screws to cover or fill a crack or opening in an electrical panel because that repair would not be consistent with industry standards (see fig. 3). As shown in table 1, from fiscal years 2013 through 2017, the median inspection scores for multifamily and public housing properties were in the mid- to high-80s, with scores trending downward toward the end of that time frame. (See apps. II and III for additional data on REAC scores.) However, a small percentage of multifamily properties scored below 60, which for multifamily properties is defined as a failure and triggers enforcement actions that Multifamily Housing or the Departmental Enforcement Center can take to require the correction of physical deficiencies. Of 27,486 multifamily properties that were inspected during fiscal years 2013 through 2017, 1,760 properties (6 percent) failed at least one inspection, and 272 properties (1 percent of the total) failed two or more inspections. Staff in Multifamily Housing field offices said multiple failed inspections are a sign of serious owner noncompliance, such as an owner who plans to sell and thus lacks motivation to make needed repairs. Multifamily Housing staff said that they take enforcement action in these cases. A higher percentage of public housing properties scored below 60 during this same period. Of the 7,699 public housing properties that were inspected during this period, 887 (11 percent) scored below 60 for at least one inspection, and 291 (4 percent) scored below 60 for two or more inspections. REAC Has Not Conducted a Comprehensive Review of Its Inspection Process since 2001 REAC has not conducted a comprehensive review of its inspection process since 2001, even though new risks to its process have emerged since then. A concern of REAC staff is that some property owners have taken advantage of the scoring system and others have misrepresented the conditions of their properties. Specifically, because more points are deducted for deficiencies on the property site than for deficiencies in a dwelling unit, some property owners prioritize site repairs over unit repairs. Additionally, some property owners attempt to cover up, rather than address, deficiencies—such as by using mulch on a building exterior to hide erosion. REAC staff have also raised concerns about property owners employing current or former REAC contract inspectors to help prepare for an inspection, sometimes by guiding owners to repair just enough to pass inspection rather than comprehensively addressing deficiencies. REAC also continues to find that some contract inspectors are conducting inspections that do not meet REAC’s quality standards (discussed later in the report). Property owner associations we met with also raised concerns about the fairness of the inspection process. Specifically, representatives of two property owner associations said that REAC’s inspection process penalizes properties for items that do not affect the livability of a unit (e.g., property receives severe deficiency for chips on exterior bricks even though the dwelling units are in good condition). Representatives from one property owner association said that some properties’ scores have fluctuated even though the condition of the property has not changed. HUD’s Office of Inspector General (OIG) also identified some weaknesses in the inspection process. Specifically, the OIG found that REAC did not verify the accuracy of sampled units for public housing agencies. Further, REAC fundamentally changed the entities that conduct inspections. In 1998, REAC employed a few large inspection companies to conduct the inspections. However, in 2005, REAC introduced the reverse auction program and opened up the inspection process to a larger number of small businesses, which resulted in a change in the composition of inspectors conducting the inspections. One of the subgoals of REAC’s strategic plan for 2011–2015 was for REAC to produce inspections of HUD-assisted properties that are reliable, replicable, and reasonable. To meet this subgoal, the plan states that REAC should assess its inspection process and apply lessons learned over the last 10 years in order to improve the process. The plan also states that REAC should conduct independent, internal audits and reviews of the inspection process to identify strengths and weaknesses and develop recommendations for improvement. Further, federal internal control standards state that management should implement control activities through policies, such as by periodically reviewing policies, procedures, and related control activities for continued relevance and effectiveness in achieving the entity’s objectives or addressing related risks. REAC officials stated that they understand the importance of conducting a comprehensive review of the inspection process similar to what they did in 2001 but that they have focused their staff and resources on other priorities—for example, upgrading their technology and quality assurance processes, hiring and training quality assurance inspectors, and conducting targeted assessments of their inspection process in reaction to specific events or risks. For example, REAC staff worked on an intra- agency team to develop recommendations to address weaknesses in the inspection process that were identified as part of the assessment of Eureka Gardens. (We describe this effort later in the report.) REAC staff also said that they updated the compilation bulletin in reaction to property owners who were making cheap, non-industry-standard repairs to disguise deficiencies during a REAC physical inspection. In addition, REAC staff noted that they meet biweekly to address certain parts of the inspection process, such as the appeals and quality assurance processes. However, these efforts help identify weaknesses in the inspection process related to specific risks and were not comprehensive enough to identify and address broader risks. For example, REAC has not assessed how changes to one part of its inspection process (for example, changing how many points are deducted for a particular inspectable area) can affect other parts of the process or result in unintended consequences. Without a comprehensive review to assess its inspection process, REAC cannot determine if it is meeting the goal of producing inspections that are reliable, replicable, and reasonable. REAC May Not Be Identifying All Properties in Need of More Frequent Inspections or Enforcement Actions REAC may not be identifying all properties that need more frequent inspections or enforcement actions because it does not consider sampling errors of the inspection scores. REAC’s inspection process does not require the inspection of all units and buildings within large properties due to REAC’s limited inspection resources. For these properties, the inspection process provides for inspecting statistical samples of units and buildings. The results for the sample are then used to estimate a score that represents the condition of the entire property. Sampling introduces a degree of uncertainty, called sampling error, which statisticians commonly express as a range associated with numerical results. For example, for a property that scored 62 on its physical inspection, REAC would consider this a passing score that requires an annual inspection and no enforcement action. However, due to sampling error, the range associated with this score could be between 56 on the lower bound and 68 on the upper bound. HUD takes enforcement action for multifamily properties with a score below 60. Federal internal control standards state that management should use quality information to achieve the entity’s objectives. In particular, internal control standards note the importance of using the entity’s objectives and related risks to identify the information requirements needed to achieve the objectives and address the risks. REAC’s property inspection scores are currently presented as numerical results without any information on the range associated with the score. REAC’s prior version of its scoring software automatically calculated the sampling errors in the inspection scores, and this information was available for inspection scores from fiscal years 2002 through 2013. However, according to REAC staff, the current version of its scoring software does not automatically calculate the sampling errors, in part because of a lack of resources and also because they believe there is no need to calculate them. Yet, in a review we conducted of REAC in 2000, officials told us that they planned to adjust the score downward and take appropriate actions for inspection scores with a lower bound that fell under an administrative cutoff, such as 60 points. During our current review, REAC staff told us that they did not implement this plan because they would need to coordinate with other HUD offices, such as the Office of Housing, and issue a notice in the Federal Register for public comment. Based on our analysis of REAC inspection data, HUD potentially could have taken enforcement actions against more properties if REAC had taken sampling errors in inspection scores into account. For example, from fiscal years 2002 through 2013, about 4.3 percent of inspections of multifamily and public housing properties had an inspection score of 60 or slightly above 60 but had a lower bound score under 60. In addition, some multifamily and public housing properties might have been inspected more frequently if the sampling errors were taken into account. For example, federal regulations require inspections of multifamily properties scoring 90 or greater once every 3 years; scoring 80 to 89 once every 2 years; and scoring less than 80 every year. Taking sampling errors into account, about 7.1 percent of multifamily properties inspected from fiscal years 2002 through 2013 might have been inspected 1 year after the most recent inspection rather than 2 years. Likewise, about 7.2 percent of inspections of multifamily properties might have occurred 2 years after the most recent inspection, rather than 3 years. Without reporting on sampling errors and considering the results, REAC will not identify some properties which could require more frequent inspections or enforcement actions. REAC Lacks Comprehensive or Organized Documentation of Sampling Methodology REAC lacks comprehensive or organized documentation of the sampling methodology it uses to make generalizable estimates about the condition of properties with its scoring system. REAC’s documentation supporting its sampling methodology is contained in five documents, none of which provides a comprehensive description of the methodology with all changes to the methodology incorporated. The main document that describes the sampling methodology is a paper presented to the American Statistical Association in 2002. This document provides a very short summary of the sampling methodology, but some key assumptions, calculations, and details are not included. For example, this document does not show how REAC derived one of the variables used to calculate the number of units to sample. When we asked REAC staff to provide us with documentation on how they derived this variable, they could only provide us with an email from 2005 from a former REAC statistician that discussed some of the statistical considerations that went into the derivation of the sample-size formula. The other four documents related to the sampling methodology are dated prior to 2002 and include the initial methodology developed and subsequent changes, but these also do not provide complete information on why key assumptions were used, or the documents lack certain formulas. Further, REAC has not updated any of its documents related to the sampling methodology since 2002 to reflect current practices. Federal internal control standards state that management should establish an organizational structure, assign responsibility, and delegate authority to achieve the entity’s objectives. In particular, the standards note the importance of developing and maintaining documentation of the internal control system. This documentation provides a means to retain organizational knowledge and mitigate the risk of having that knowledge limited to a few personnel, as well as a means to communicate that knowledge as needed to external parties, such as external auditors. Further, this documentation of controls, including changes to controls, is evidence that controls are identified, capable of being communicated to those responsible for their performance, and capable of being monitored and evaluated by the entity. However, REAC does not have a process to ensure comprehensive and organized documentation of the sampling methodology of its inspection process. Instead, REAC relies on the institutional knowledge of individual staff members. For example, when we requested documentation of its sampling methodology, REAC relied on a statistician who had been with the organization for many years to locate and provide us with the documents we requested. In addition, we had to interview this individual to better understand the methodology because key pieces of information were missing from these documents. REAC staff told us that since the inspection process has remained relatively consistent over time, they have not seen the need to ensure that documentation of the sampling methodology is comprehensive and organized. By interviewing multiple individuals, reviewing multiple documents, and conducting our own calculations, we were able to determine that REAC’s sampling methodology is suitable for making generalizable estimates about the condition of a property with the scoring system. However, the lack of comprehensive and organized documentation could affect REAC’s ability to preserve institutional knowledge and make changes or improvements to its inspection process if key staff leave the agency. REAC Does Not Always Meet Its Schedule for Inspecting Multifamily Properties or Track Progress toward Meeting Scheduling Requirements REAC schedules inspections of multifamily properties based on the prior REAC inspection score, but it did not meet its schedule for about 20 percent of inspections from calendar years 2013 through 2017. As discussed earlier, federal regulations require inspections of multifamily properties scoring 90 or greater once every 3 years, those scoring 80 to 89 once every 2 years, and those scoring less than 80 every year. Our analysis of REAC inspection data showed that about 20 percent of the properties were not inspected within 3 months before or after what HUD has identified as the ideal date to conduct the inspection, called an ideal future date. On average, REAC conducted inspections for these properties about 6 months past the ideal future date. REAC staff told us that there may be legitimate reasons for not conducting an inspection according to the ideal future date. For example, Multifamily Housing can delay an inspection because of natural disasters or major rehabilitations to the property, among other reasons. However, REAC maintains limited data on the reasons why inspections have been rescheduled or cancelled. In addition, these data are not readily available to understand retrospectively why an inspection did not occur on schedule. REAC also does not track its progress toward meeting its requirement for inspecting multifamily properties within prescribed time frames. Federal internal control standards state that management should use quality information to achieve the entity’s objectives. In particular, the standards note the importance of designing a process that uses the entity’s objectives and related risks to identify the information requirements needed to achieve the objectives and address the risks. Further, management should obtain relevant data from reliable internal and external sources in a timely manner and process these data into quality information that supports the internal control system. Multifamily Housing depends on REAC inspections to provide assessments of the physical condition of properties under its jurisdiction (discussed in greater detail later in this report). REAC’s inability to adhere to the inspection schedule for multifamily properties could hinder Multifamily Housing’s ability to monitor the physical condition of properties on a timely basis and take enforcement actions when warranted. The lack of a mechanism to track REAC’s progress toward meeting its requirement for inspecting multifamily properties also hinders REAC’s ability to determine what factors are contributing to delays in conducting the inspections. As a result, REAC lacks the information needed to determine the scope of the problem and what actions it can take to ensure multifamily properties are inspected on a timely basis. REAC Is Piloting a Process for Hard-to-Staff Inspections but Lacks Plans to Evaluate Pilot Results REAC has started a pilot program to staff inspections that contractors typically do not bid on, but it has not developed a formal plan to evaluate the results of this pilot. Since 2005, REAC has used the reverse auction program to save money on inspections and increase small business participation. However, under the reverse auction program, REAC has faced challenges in obtaining bids for inspections in some urban areas, such as Chicago, and some remote areas. To address this challenge, for a select number of properties, REAC has implemented a pilot program as an alternative to the reverse auction program. Under this alternative process, REAC has awarded multiple Indefinite Delivery/Indefinite Quantity (IDIQ) contracts to four companies to conduct these inspections. The IDIQ contracts are intended to ensure that REAC obtains physical inspections of all HUD-assisted properties on one task order rather than allowing contractors to selectively choose properties under the current program. The “all or none” approach, a key feature of these IDIQ contracts, eliminates the need to re-auction the same properties multiple times at higher prices to incentivize contractors to bid on the property. The pilot differs from REAC’s current physical inspection process in a number of ways. The pilot requires the companies that have been awarded the IDIQ contract to inspect all properties in a geographic region rather than to select which individual properties they want to bid on. Another difference is that the companies conduct quality assurance functions normally conducted by REAC staff, such as ensuring that inspectors are certified and identifying and addressing any gaps in inspectors’ performance. As of November 2018, REAC had focused its efforts on implementing the pilot but had not developed a formal plan to evaluate its results. GAO’s guide for designing evaluations states that a program evaluation is a systematic study using research methods to collect and analyze data to assess how well a program is working and why. Some key attributes of effective program evaluation design include the following: identification of data sources and collection procedures to obtain relevant, credible information; clear criteria for making comparisons that will lead to strong, defensible evaluation conclusions; and an established evaluation scope that will ensure the evaluation is tied to research questions. Federal internal control standards also state that management should use quality information to achieve the entity’s objectives. In particular, the standards note the importance of management designing a process that uses the entity’s objectives and related risks to identify information requirements needed to achieve the objectives and address the risks. Further, the standards stress the importance of management obtaining relevant data from reliable internal and external sources in a timely manner based on the identified information requirements. REAC staff told us that they plan to measure the success of the pilot program by determining whether companies are completing quality inspections in a timely manner. However, REAC staff did not provide details about how the results of the pilot would be compared to the existing process and how the quality of inspections and the performance of inspectors would be measured and assessed. Absent a formal process that incorporates key attributes for effectively evaluating the results of the pilot program, REAC may lack the information needed to determine if the pilot is a success or whether changes are needed before moving from a pilot to a permanent process. HUD Has Made Limited Progress in Implementing Recommendations from an Internal Review of REAC HUD has made limited progress in implementing recommendations from an internal review of REAC that was conducted in 2016. HUD created the Rapid Response and Resolution team—which consisted of staff from REAC and other units within HUD, including Multifamily Housing—in response to, among other things, problems associated with Eureka Gardens, a multifamily property in Jacksonville, Florida. In 2015, REAC conducted a physical inspection of Eureka Gardens, and the contract inspector gave the property a score of 85. However, REAC later declared that the inspection was out of standard when it learned that the contract inspector had only inspected one of the two properties associated with Eureka Gardens (the better of the two properties). REAC officials told us that property management engaged in some deceptive practices (such as making quick, cheap repairs) in an attempt to influence the inspection score. According to these officials, the inspector did not conduct the inspection consistent with REAC’s standards and was subsequently decertified. REAC then reinspected the entire Eureka Gardens property with its own quality assurance staff and found numerous deficiencies, which resulted in the property receiving an inspection score of 62. The Rapid Response and Resolution team developed 31 recommendations, 8 of which were specific to REAC, in January 2017. As of December 2018, nearly 2 years after the recommendations were developed and 3 years after the initial inspection of Eureka Gardens, REAC had reached concurrence with Multifamily Housing on 3 of these recommendations and asked for Multifamily Housing’s consideration of the funding and rulemaking requirements for the remaining 5. HUD had also not yet implemented the 3 recommendations on which it reached concurrence. Some of these recommendations address REAC’s management of the inspection process. They include the following: Weighting of dwelling units in inspection score. The review team recommended that REAC consider increasing the weight of dwelling- unit deficiencies in the physical condition score. This recommendation attempts to address the issue, discussed earlier, of property owners who focus their repairs on common areas of the property over dwelling units. Notice provided to property owners of impending inspection. This recommendation reduces the time that REAC can take to notify property owners of an upcoming inspection from 15 days to 3 days for properties that have failed their previous REAC inspection. REAC staff said that this change would provide a more accurate picture of the condition of properties since property owners generally address the maintenance of the property just before an inspection. In addition, this recommendation could address the concern discussed earlier of property owners hiring current or former REAC contract inspectors to help them prepare for an inspection. This recommendation should also encourage property owners to maintain properties in good condition at all times, according to REAC staff. Exigent health and safety risks. Another recommendation was that REAC work with Multifamily Housing and PIH to implement a risk- based exigent health and safety abatement verification policy. According to REAC staff, some properties certify that they have corrected exigent health and safety deficiencies when they have not done so. We found that many inspections conducted from fiscal years 2013 through 2017 had at least one exigent health and safety deficiency, and the percentage has been higher in recent years (see table 2). Field office staff from PIH and Multifamily Housing may check to ensure that these repairs have been made when they are onsite. However, neither of these offices has a formal program to ensure that property owners are actually addressing the exigent health and safety issues. As a result, property owners may choose to correct only those deficiencies that they believe will be checked by HUD field office staff, according to REAC staff. Federal internal control standards state that management should identify, analyze, and respond to risks related to achieving the defined objectives. By establishing the Rapid Response and Resolution team, HUD took the steps of identifying the risks to its inspection process and designing responses to these risks. However, the standards also call for remediating identified internal control deficiencies on a timely basis. HUD officials we met with attributed the delay in implementing the recommendations to prior vacancies in some senior leadership positions, including positions in Multifamily Housing. HUD’s delay in implementing most of the recommendations from the Rapid Response and Resolution team affects REAC’s ability to respond to weaknesses it has identified in the inspection process in a timely manner. REAC’s Processes for Selecting, Training, and Developing Inspectors Have Weaknesses REAC Sets but Does Not Verify Qualification Requirements for Contract Inspector Candidates Contract inspector candidates certify through an application that they meet REAC’s qualification requirements, but REAC does not currently verify that candidates have met these requirements before REAC selects them for training and determines them to be eligible to inspect HUD- assisted properties. Before inviting candidates to participate in inspector training, REAC requires them to certify that they meet three main qualifications: Inspections. Candidates must have conducted a minimum of 250 residential or commercial inspections. Building trades knowledge. Candidates must have building trades knowledge, such as knowledge of construction methods or electrical systems. Computer literacy. Candidates must be able to use email, the internet, and Microsoft Windows. However, REAC does not require documentation from contract inspector candidates demonstrating that they successfully conducted 250 inspections. REAC officials told us that they intend to verify a sample of the 250 inspections for each inspector, but as of November 2018 they had not yet developed a process for doing so, such as by developing a methodology for sampling and a timeline for contacting references. In contrast, one of the home inspection associations we met with, ASHI, requires certified inspector candidates to submit a list of 250 fee-paid home inspections that meet or exceed the ASHI standards and to provide a notarized affidavit validating those inspections. In addition, REAC staff told us that some contract inspector candidates have inspection experience based on inspections that are not as rigorous as those conducted using the Uniform Physical Condition Standards protocol. Participants in three of the four discussion groups we held with REAC quality assurance inspectors and supervisors told us that they had trained candidates who had included information on their applications about previous inspection experience that was not well matched to REAC’s inspection process. For instance, some inspector candidates submitted Federal Emergency Management Agency inspections and U.S. Army Office of Housing inspections as evidence of having completed 250 inspections, but REAC officials said these inspections are not as comprehensive as REAC inspections because they do not assess building systems, such as electrical or heating, ventilation, and air conditioning systems. Federal internal control standards call for management to recruit competent individuals so that they are able to accomplish their assigned responsibilities. In addition, key principles for workforce planning state that agencies need to determine the critical skills and competencies necessary to achieve their goals. REAC officials told us that the inspector training program should weed out inspector candidates that may not have the appropriate qualifications. However, although REAC officials told us that inspector candidates have been removed from training for not having the requisite skills, the officials were not able to determine how many candidates had misrepresented their qualifications on their application or had failed training for other reasons. REAC does not verify the inspections submitted by inspector candidates—relying instead on training to screen out unqualified candidates—and does not determine the type of inspection that may count as a qualifying inspection. As a result, REAC may be allowing candidates with insufficient experience to proceed in the training process, which may waste resources by training candidates who are unlikely to become successful inspectors. Training for Contract Inspectors Is Not Consistent with Key Attributes of Effective Training and Development Programs Contract inspector candidates must complete several phases of REAC training—online, in-class, and field—and pass associated examinations, as well as a background check. Online training. Inspector candidates first complete a 6-week online training that includes web-based modules on the Uniform Physical Condition Standards protocol and the use of the software system for the handheld data collection device. Candidates must pass a pre- certification examination to progress to the next phase of training. In-class training. After passing a background check, inspector candidates then begin in-class training. This phase consists of 3 to 4 days of in-class training led by REAC quality assurance inspectors and covers the compilation bulletin, Uniform Physical Condition Standards protocol, best practices, simulations of the inspection software, and hands-on practice exercises using the software. To proceed to field training, inspector candidates must pass a certification examination with a minimum score of 75 percent that covers material from both the compilation bulletin and the Uniform Physical Condition Standards protocol. Field training. The last phase is a 5-day field training course that culminates in a field examination. REAC quality assurance inspectors lead and provide instruction for the first 4 days of field training. Inspector candidates independently conduct a mock inspection using the Uniform Physical Condition Standards protocol on the fifth day, and a quality assurance inspector evaluates the candidate’s performance. REAC has made changes to training in recent years. For example, REAC began using actual HUD-assisted properties, rather than simulated properties, for the mock inspection. Some quality assurance staff and property owner associations told us they regarded the changes made in recent years to be beneficial. Participants in three of the four discussion groups we held with quality assurance supervisors and inspectors, as well as two representatives of property owner advocacy organizations, said that, in addition to classroom training, field training on a physical property helped to assess the competency of inspector candidates. In addition, stakeholders—including property managers, contract inspectors, and REAC staff—told us the mock inspections have been effective at providing training to new inspectors, and that the professionalism of contract inspectors has improved. REAC contracts with a private vendor to provide the contract inspector online training, and the vendor provides data and reports that REAC staff use to track inspector candidates’ progress through the online training modules. REAC officials told us they use this information to identify areas of the training where candidates struggle and to help revise the training material. In addition, REAC solicits feedback from contract inspector candidates on the online training. REAC also uses key performance indicators to track the number of inspector candidates who enroll and whether they pass or fail training. However, REAC does not currently have formal metrics or use data to track the effectiveness of its three phases of training. For instance, REAC does not track key measures of performance that could provide management with information to improve the training process, such as how individuals score in each section of the in-class training examination and their field examinations. REAC also does not track the resources spent on training, either in terms of funds spent or number of quality assurance inspectors who participate. According to key practices we have identified for training and development, agencies should have processes to systematically track the cost and delivery of training and measure the effectiveness of those efforts. REAC officials said that they would like to have such mechanisms and have developed a proposal to consolidate training functions and better align training to REAC’s strategic goals. However, the proposal does not include performance measures for evaluating the effectiveness and efficiency of training. Use of cost-tracking and performance measures tied to its strategic goal of improving the inspection process could improve REAC’s ability to manage scarce resources, evaluate the effectiveness of its training program, and plan for future training. Quality Assurance Inspector Training Requirements May Not Cover All Job Duties and Are Not Documented REAC’s quality assurance inspectors—who train and oversee contract inspectors—must be able to conduct physical inspections of properties as well as assess contract inspectors’ performance. To assess contract inspector performance, quality assurance inspectors oversee and mentor contract inspectors during CQA reviews and provide them feedback in a collaborative manner, an approach REAC management implemented in 2017. Some senior quality assurance inspectors are also responsible for leading classroom and field training for contract inspectors. According to REAC officials, REAC’s quality assurance inspectors receive the same training as contract inspectors on the Uniform Physical Condition Standards inspection protocol. However, REAC’s training for quality assurance inspectors does not include formal instruction on how to coach or provide feedback during CQA reviews. Instead, new quality assurance inspectors are provided with on-the-job training, and they can only conduct CQA reviews independently when quality assurance supervisors are satisfied that they are sufficiently competent. Beyond the on-the-job training, quality assurance inspectors are encouraged to undergo additional online training on coaching, but there are no specific training requirements related to conducting CQA reviews. Participants in two of our three discussion groups with quality assurance inspectors told us they were not sure how to provide the collaborative coaching and mentorship REAC officials said they wanted. REAC also does not specifically train quality assurance inspectors in how to provide classroom and field training to contract inspectors, and participants in all three discussion groups with quality assurance inspectors told us that instructors do not seem to take a consistent approach to classroom and field training. In addition, REAC’s training requirements for quality assurance inspectors are not documented in the quality assurance standard operating procedures or other documents we reviewed. For example, REAC’s new, more collaborative approach to CQA reviews was communicated to quality assurance inspectors during staff meetings, but REAC staff have not documented the approach or developed any specific training. Some contract inspectors told us that quality assurance inspectors often have less experience conducting inspections than they do. They suggested that this gap may affect quality assurance inspectors’ ability to competently oversee CQA reviews and conduct QCIs. REAC officials told us they are considering changing quality assurance inspector training requirements to be more rigorous than contract inspector training. For example, staff from REAC’s Quality Control group, created in 2017, told us that they are considering expanding training on the five inspectable areas and assessing quality assurance inspectors to see if they need additional support in any of these areas. They would also like to require quality assurance inspectors to pass the training examinations with minimum scores of 90 percent, instead of the score of 75 percent that currently applies to both contract and quality assurance inspectors. However, the Quality Control group has not implemented these changes, officials said, because its staff resources are limited, and staff have been reallocated to support other projects within REAC. In comparison, one of the home inspector associations we met with, InterNACHI, has specific requirements for its instructors. According to the association, its instructors are certified master inspectors, have completed a minimum of 1,000 paid inspections or hours of education or some combination thereof, and have conducted inspections for a minimum of 3 years. The instructors also assist in developing the educational material for training courses. Federal internal control standards state that management should demonstrate a commitment to recruit, develop, and retain competent individuals. In particular, the standards note that agency personnel need to possess and maintain a level of competence that allows them to accomplish their assigned responsibilities. The standards also note the importance of management enabling individuals to develop competencies appropriate for key roles and tailoring training based on the needs of the role. Without assessing whether training for quality assurance inspectors is sufficient and requiring additional training as needed, REAC may not have reasonable assurance that these inspectors have the skills required to oversee contract inspectors. Federal internal control standards also state that management should design control activities to achieve objectives and respond to risks, and they note the importance of documenting internal control—for example, in management directives, administrative policies, or operating manuals. These standards also state that management should establish an organizational structure, assign responsibility, and delegate authority to achieve the entity’s objectives. For example, the standards note that effective documentation assists in management’s design of internal controls by establishing and communicating the who, what, when, where, and why of internal control execution to personnel. Without documenting training requirements that encompass all job responsibilities, REAC may not have reasonable assurance that the required skills and competencies are clearly communicated to and understood by quality assurance inspectors and aligned with job duties. REAC Does Not Require Continuing Education for Contract and Quality Assurance Inspectors REAC has ongoing requirements for contract inspectors to maintain their eligibility, but these do not include continuing education requirements. Contract inspectors must conduct at least 25 successful inspections per year—that is, inspections found to be within REAC’s inspection standards—and pass a background check every 5 years to remain certified. REAC also offers optional training through online refresher modules. However, REAC does not know how many contract inspectors use these resources or how effective they are. In comparison, ASHI and InterNACHI have continuing education requirements for their certified inspectors. ASHI requires inspectors to earn 20 continuing education credits annually. The qualifying training courses must be ASHI-approved, the inspector must submit a signed affidavit attesting to having attended the training, and ASHI spot checks to monitor compliance. InterNACHI requires inspectors to earn 24 continuing education credits annually and to pass the InterNACHI Online Inspector Examination with a score of 80 percent or better every 3 years. REAC encourages quality assurance inspectors to take additional training but does not require continuing education. REAC offers optional “dine- and-learn events” to update both contract and quality assurance inspectors on policy and procedure changes and point out errors they commonly observe. In 2017 REAC also began offering limited coaching to contract inspectors by quality assurance inspection reviewers, a process separate from CQA reviews. The reviewers compare physical defects identified by contract inspectors to those that were identified by quality assurance inspectors during QCIs of the same property. Reviewers then provide one-on-one feedback to contract inspectors to address any discrepancies in inspection scores. REAC officials said that for continuing education they prefer self-paced learning to formal instruction because it more appropriately matches the varying education needs of inspectors. REAC’s strategic plan proposes developing REAC-wide policies for staff training and skill development, but it does not include any requirements for continuing education. Key practices we have previously identified for training and development suggest that agencies should encourage employees to take an active role in their professional development, which can include requiring employees to complete a specific level of continuing education. Ongoing training requirements for contract and quality assurance inspectors could help REAC ensure that inspectors are up-to- date on REAC policies and industry standards. Such continuing education could also refresh existing knowledge, helping contract and quality assurance inspectors conduct high-quality inspections consistently. Continuing education could also help quality assurance inspectors to develop their mentoring and coaching skills, which would better enable them to develop and oversee contract inspectors. REAC’s Processes for Monitoring and Evaluating Contract and Quality Assurance Inspectors Have Weaknesses REAC Uses Several Mechanisms to Monitor and Evaluate Contract Inspectors Collaborative Quality Assurance Reviews REAC’s mechanisms to monitor and evaluate its contract inspectors include collaborative quality assurance reviews, quality control inspections, and various other monitoring tools. REAC uses CQA reviews to monitor, evaluate, coach, and provide feedback to contract inspectors. REAC has documented its processes for conducting and reporting on CQA reviews in field training guidance, standard operating procedures, and the compilation bulletin. To determine which inspections will receive a CQA review, REAC combines a risk-based approach—targeting low-performing inspectors—with scheduling needs, based on the timing and geographic location of the inspection. REAC conducted almost 3,000 CQA reviews from fiscal years 2013 through 2017. As shown in table 3, the percentage of inspections each year that received a CQA review ranged from about 3 to 8 percent. Our analysis of CQA review data shows that some contract inspectors are not conducting inspections in accordance with REAC standards. From fiscal years 2013 through 2017, an average of 17 percent of CQA reviews resulted in contract inspectors receiving a rating that was outside of REAC’s physical inspection standards, referred to as an outside standard rating. This rating is based on the contract inspector committing at least 1 of 18 types of performance- or scheduling-related REAC protocol violations, with 8 of the performance violations resulting in an automatic outside standard rating. The percentage of outside standard ratings was significantly higher in fiscal years 2015 through 2017, as compared to 2013 and 2014 (see fig. 4). According to REAC officials, this increase was likely due, in part, to an increase in the number of less experienced contract inspectors. Specifically, from September 2013 through December 2015, REAC attempted to expand the pool of contract inspector candidates by lowering the required number of inspections from 250 to 50. REAC officials confirmed that these inspectors were less experienced and more likely to violate protocols. As previously discussed, REAC has taken steps to make its CQA reviews more collaborative, but some stakeholders said that challenges remain. REAC’s 2015 standard operating procedures stated that the goal for CQA reviews should not be to designate a contract inspector as outside standard, but rather to ensure inspection accuracy and to improve the knowledge of the contract inspector. However, it was not until more recently that REAC management emphasized in several staff meetings the need for quality assurance inspectors to take a collaborative approach, according to REAC staff. Despite this new emphasis, participants in a discussion group we held with contract inspectors told us that they believe a punitive approach persists with some quality assurance inspectors, that repeated pairings of contract and quality assurance inspectors could lead to bias, and that contract inspectors who receive a high number of CQA reviews may feel like REAC is targeting them. Quality Control Inspections REAC uses QCIs to further ensure the accuracy of inspections conducted by contract inspectors. As previously noted, a combination of factors can lead REAC to reject an inspection and trigger a QCI, including significant differences between the current inspection and previous inspections and other contextual factors, such as the inspector’s past CQA review performance. In a QCI, the quality assurance inspector reviews an inspection report and then conducts a new inspection to identify potential weaknesses and evaluate the inspector’s performance. The QCI results in a new inspection score. REAC has standard operating procedures that document requirements for scheduling, conducting, and reporting QCIs. REAC completed 226 QCIs from March 2017 through June 2018. Our review found that more than 50 percent of QCIs resulted in an outside standard rating. On average, contract inspectors gave properties a score that was 16 points higher than the score given subsequently by quality assurance inspectors, indicating that those contract inspectors missed deficiencies. Of these inspections, about 15 percent that had initially received a passing score from the contract inspector failed the subsequent QCI. Other Monitoring Tools for Contract Inspectors REAC also uses ratings and reports to oversee contract inspectors. REAC assigns each contract inspector a rating based on a combination of factors, including CQA results and percentage of inspections rejected, and these ratings help target which inspectors should receive CQA reviews and CQIs. REAC also produces two reports for all contract inspectors to assist in its oversight: Defect Comparison Reports compare the specific deficiencies reported by an inspector to the frequency with which other contract inspectors reported that same deficiency across properties. REAC primarily uses the results to target areas to coach contract inspectors who have been flagged for a QCI. Defect Delta Reports compare deficiencies in a contract inspector report to deficiencies a quality assurance inspector reported in a follow-up inspection (usually a QCI). REAC primarily uses this information to identify the types of deficiencies the contract inspector is missing. REAC Has Not Met Management Targets for Reviews of Contract Inspectors REAC did not meet management targets for the number of CQA reviews to be conducted in any quarter from fiscal years 2013 through 2017 (see fig. 5). REAC officials told us their management target is to conduct three CQA reviews for each high-risk contract inspector and two CQA reviews for lower-risk contract inspectors each quarter. REAC officials told us that in developing the targets, they attempted to balance risks to the quality of inspections and resources available. In addition, as of June 2018, REAC had not met management targets for timeliness of QCIs in any quarter. REAC has no documented timeliness goals for QCIs, but REAC officials told us that QCIs are supposed to take place within 30 days of the original inspection date because the condition of the property can change over time (see fig. 6). REAC officials told us they did not meet these management targets for CQAs and QCIs because they did not have enough quality assurance inspectors. In addition, when quality assurance inspectors are pulled onto other projects, such as supporting HUD’s efforts related to natural disasters, REAC’s ability to conduct CQAs and QCIs is reduced. For example, in fiscal year 2018, 28 quality assurance inspectors were pulled offline to assist HUD in areas affected by hurricanes. REAC officials told us these temporary reassignments have affected their ability to implement the quality assurance process. REAC staff told us that the quality of inspections may have deteriorated because some contract inspectors were aware that quality assurance inspectors would not be conducting CQA reviews or QCIs during those post-disaster periods. REAC has recently hired more quality assurance inspectors to help address staffing shortages. In addition, REAC officials told us that they intend to take into account the likely effects of natural disasters on their ability to conduct quality assurance reviews when planning for these reviews in the future, but REAC has not yet developed a plan to meet its targets that includes, for example, mechanisms to mitigate resource constraints and unforeseen demands on staff. According to REAC’s strategic plan, to produce physical inspections that are reliable, replicable, and reasonable, REAC is to look for patterns and trends in inspection results, such as inconsistencies between inspectors, regional and area differences, and patterns in different inspection criteria. In addition, the strategic plan calls for REAC to assess and improve the quality of contract inspectors. However, if REAC is unable to meet its management targets for CQA reviews, it may not be able to consistently produce high-quality inspections because it is not providing routine opportunities for contract inspectors to receive coaching from quality assurance inspectors, which could include addressing deficiencies that the contract inspectors did not initially identify and record. In addition, if QCIs are not conducted shortly after the original inspections, REAC may not be able to verify the quality of the inspection because the condition of the property could change over time. REAC’s inability to meet management targets for CQA reviews and QCIs could also affect its ability to monitor patterns in inspection results because, for example, the quality of QCI data would be less reliable due to the lapse in time. As a result, REAC may not be using quality assurance inspector resources as effectively as possible. REAC Takes Administrative Actions against Contract Inspectors Who Do Not Meet REAC Requirements REAC’s Inspector Administration division administers a variety of administrative actions and disciplinary sanctions against contract inspectors in response to complaints or CQA reviews and QCIs. Inspector Administration officials said that they acknowledge and follow up on all complaints received from property representatives and residents, among others. Complaints about a contract inspector can relate to conduct, inspection protocol violations, scheduling, and conflicts of interest, among other issues. Inspector Administration uses the Code of Conduct, Uniform Physical Condition Standards inspection protocol, and compilation bulletin as standards for evaluating contract inspector conduct. In order of increasing severity, Inspector Administration can issue a letter of warning, issue a performance deficiency, suspend or decertify the inspector, or refer the inspector to HUD’s OIG, among others. While Inspector Administration can use professional judgement in adjudicating complaints, some actions are automatic. For example, decertification is automatic for inspectors with three or more performance deficiencies, inspectors found to have engaged in egregious misconduct, or inspectors who conduct fewer than 25 inspections annually. Inspector Administration took more than 700 administrative enforcement actions against contract inspectors from fiscal years 2013 through 2017 (see fig. 7). As part of its effort to reform its contract inspector pool, REAC decertified 127 inspectors from fiscal years 2013 through 2017 due to inactivity, conduct, or performance issues. For example, REAC decertified the contract inspector who gave Eureka Gardens a passing inspection score even though it was in poor physical condition. Two advocacy organizations told us they noticed that REAC was decertifying more inspectors than in the past, and one said that the quality of contract inspectors had improved as a result. In response to concerns from contract inspectors, Inspector Administration is proposing changes to, among other things, provide contract inspectors who are subject to potential enforcement actions with more opportunities to present their perspective. For example, Inspector Administration would allow contract inspectors to appeal performance deficiencies earlier in the process. Other proposed changes would make performance deficiencies based on outside standard ratings discretionary instead of automatic and would remove a performance deficiency from a contract inspector’s record after 25 consecutive inspections without a new performance deficiency instead of 30. Inspector Administration officials said the new rules would also adjust decertification sanction periods, which specify the amount of time a decertified contact inspector must wait before reapplying to REAC, to account more appropriately for the reason the contact inspector left REAC (e.g., resignation, performance, or conduct). REAC’s Quality Control Group Has Not Yet Implemented Procedures for Inspector Oversight REAC created the Quality Control group to standardize quality assurance inspector reviews by conducting more frequent oversight and looking for trends across all quality assurance inspectors, according to a Quality Control official. This official said that one type of oversight involves a quality control staff member conducting an identical inspection 1 day after that of a quality assurance inspector to determine how well the inspector recorded deficiencies. Inspections are then rated as either “acceptable” or “unacceptable” based on whether the inspector followed established protocols and observed and accurately recorded 90 percent or greater of the existing deficiencies. According to the official, inspection reviews are expected to be shared with quality assurance management and individual supervisors to support quality assurance inspector development. This official also told us Quality Control plans to conduct reviews of quality assurance inspectors at least once a year, or more frequently as needed. In November 2018, Quality Control developed a mission statement which says that the primary goal of the group is to improve the consistency of inspections. Also in November 2018, Quality Control developed procedures for reviewing quality assurance inspectors, which include processes for conducting field reviews of completed inspections, criteria for acceptable inspections, and processes for providing feedback. An official from Quality Control said that the group worked with other divisions within REAC, such as PASS Quality Assurance and Research and Development, to develop the procedures and criteria for evaluating quality assurance inspectors. The official told us both its mission statement and procedures have not been implemented, in part because Quality Control staff have been repeatedly pulled onto other special projects. The official told us that these documents have been approved by REAC management and that Quality Control intends to implement the procedures in 2019. According to federal internal control standards, management should implement control activities through policies. For example, the standards call for documenting in policies each unit’s responsibility for an operational process’s objectives and related risks. Without finalizing and implementing its policies and procedures for reviewing quality assurance inspectors, Quality Control may not be able to provide consistent reviews of quality assurance inspectors, which could affect the quality of inspections as well as the feedback and coaching quality assurance inspectors provide to contract inspectors. Prioritizing the implementation of Quality Control’s review procedures could help ensure that Quality Control achieves its objectives and provides consistent reviews of quality assurance inspectors. Performance Standards for Quality Assurance Inspectors Do Not Fully Align With Job Duties The standards REAC uses to measure quality assurance inspectors’ performance do not fully align with their job duties. Quality assurance supervisors are primarily responsible for evaluating quality assurance inspector performance using five performance elements, which REAC’s performance appraisal system describes as follows: Collaboration: Provide customer service communication both verbally and in writing to internal and external HUD stakeholders, customers, or anyone who comes in contact with quality assurance services. Individual training: Develop competencies and perform individual training associated with job duties. Personal investment: Improve processes, such as through special projects or self-initiated projects that improve the quality assurance division’s standard operating procedures, the Uniform Physical Condition Standards inspection protocol, the compilation bulletin, or others. Risk management: Maximize scarce resources and be cost efficient to the government in all aspects of job duties and assignment. Meeting the need for quality affordable rental housing: Perform in accordance with all protocols and standard operating procedures, and complete CQA reviews and Uniform Physical Condition Standards inspections. REAC’s performance appraisal system includes descriptions of the standards for each of the five performance elements, as well as supporting behaviors. For example, to be rated fully successful for “meeting the need for quality affordable rental housing,” quality assurance inspectors should independently complete Uniform Physical Condition Standards inspections with no more than two inspections being rejected by REAC within the rating period. Based on the standards, quality assurance inspectors are also expected to conduct CQA reviews and field trainings for contract inspector candidates. However, the performance appraisal system for quality assurance inspectors does not include performance elements with competencies that relate to all of their job duties. For example, the performance appraisal system does not define expectations for performing CQA reviews or QCIs. In addition, it does not include criteria for evaluating the training, coaching, and mentoring that quality assurance inspectors are expected to provide to contract inspectors. Quality assurance supervisors can incorporate information from reviews of quality assurance inspectors by Quality Control in their performance evaluations. However, REAC officials told us that Quality Control does not evaluate inspectors based on the performance elements and standards. Instead, Quality Control’s reviews only evaluate an inspector’s performance as it relates to the QCI being reviewed, such as following established protocols and observing and accurately recording 90 percent or greater of the existing deficiencies. In addition, Quality Control’s reviews do not include evaluations of a quality assurance inspector’s performance for other key job duties, such as training and mentoring contract inspectors. According to key practices we have identified for effective performance management, agencies should use competencies to define the skills and supporting behaviors that individuals need to effectively contribute to organizational results. REAC staff told us they do not know when the performance elements and standards for quality assurance inspectors were last revisited, and new job duties such as conducting QCIs have been added that are not part of the performance elements. Better alignment between the performance competencies and the job responsibilities of quality assurance inspectors would help ensure that inspectors are assessed on all their key duties—including training and mentoring contract inspectors—which could improve the quality of inspections and reviews. HUD’s Key Rental Programs Rely on REAC Physical Inspection Scores as Part of Their Monitoring and Enforcement Processes PIH and Multifamily Housing each have separate processes to monitor the conditions of HUD-assisted properties, including physical conditions, and take enforcement actions if properties are not decent, safe, sanitary, and in good repair. PIH assesses the performance of PHAs on key indicators through a federal regulatory process—the Public Housing Assessment System. PIH also monitors PHAs through a Risk Assessment Protocol, which incorporates qualitative data and determines actions to address identified risks. The Risk Assessment Protocol is intended to be a proactive approach to address risk at PHAs and use resources efficiently. Separately, Multifamily Housing monitors properties that score below 60 on the REAC physical inspection. To account for properties scoring 60 or above on the REAC inspection, as well as to monitor property characteristics other than physical conditions, Multifamily Housing assesses properties through its risk rating system. REAC Scores Factor into PIH’s Assessment of Public Housing Agencies’ Performance and Help Determine Actions to Address Deficiencies Public Housing Assessment System Process The Public Housing Assessment System uses the REAC physical inspection score for each public housing development to determine the physical performance of the PHA. The physical performance of PHAs is one of four indicators within the Public Housing Assessment System, which assesses the performance of PHAs and determines a performance designation. The four indicators are (1) the physical condition of the PHA’s housing developments, (2) the financial condition of the agency, (3) the management operations of the agency, and (4) utilization of property modernization and development funds (capital fund). REAC inspection scores are adjusted to reflect the size of each housing development, and their weighted average is the physical performance indicator for a PHA. The physical indicator score, worth a maximum of 40 points (out of 100 points total) toward the overall Public Housing Assessment System score, has the highest value of the four indicators. To determine the financial, management, and capital fund indicators, PHAs upload information electronically to REAC, and REAC’s data systems generate a score for each indicator and overall. Figure 8 shows the maximum value for each indicator and overall score. Table 4 explains how the indicator and overall assessment scores lead to a performance designation. PHAs are assessed and receive a performance designation every 1 to 3 years, according to their size and prior performance designation. PHAs with at least 250 units receive an assessment annually. PHAs with fewer than 250 units receive an assessment every 3 years if designated as a high performer, 2 years if designated as a standard or substandard performer, and annually if designated as a troubled or capital fund troubled performer. In years that smaller PHAs do not receive an assessment, they must provide financial data to REAC but do not receive a published assessment score or designation. Following REAC’s release of the performance designations, PHAs and PIH each have a role in ensuring that physical deficiencies are corrected. REAC is not responsible for ensuring that PHAs correct physical deficiencies. According to federal regulations, PHAs must take certain actions depending on their performance designation. PHAs designated as troubled must enter into a recovery agreement with PIH to improve their performance within 2 years. PHAs designated as standard or substandard performers must correct deficiencies identified in the assessment within 90 days, or they may develop a plan to correct the deficiency within a specified time frame. PIH officials told us they monitor whether PHAs designated as standard or substandard performers have developed a plan or if field offices are assisting the PHA. PHAs designated as high performers are not required to correct deficiencies. Table 5 shows the number of PHAs in each designation for fiscal years 2013 through 2017, including some PHAs exempt from receiving a performance designation (e.g., small PHA deregulation). If PHAs do not correct deficiencies or improve their performance, PIH officials told us they can initiate a series of actions. First, PIH field offices are to remind PHAs of their obligation to provide housing that is decent, safe, sanitary, and in good condition. If those conversations are not effective, PIH can take administrative or enforcement actions. For example, PIH can refer PHAs to the Departmental Enforcement Center, which can exclude PHA leadership from participating in HUD programs. However, we previously found that PIH refers PHAs to the Departmental Enforcement Center infrequently, making 12 referrals in 2017 and 25 referrals in 2016. In rare instances, PIH also can place the PHA into administrative receivership and take control of the PHA’s operations. These actions also could be part of a recovery agreement for troubled performer PHAs. PIH officials told us they initiate actions specified in the recovery agreement for troubled performer PHAs that do not improve their performance within 2 years. PIH’s Risk Assessment of Public Housing Agencies Incorporates REAC Scores and Determines Actions to Address Physical and Other Risks Risk Assessment Protocol To inform its monitoring efforts, PIH uses the Risk Assessment Protocol to assess PHAs in four risk categories: physical, governance, financial, and management. PIH collects quantitative data from various HUD data systems and qualitative data from a survey administered by PIH field offices. The physical risk category uses a PHA’s Public Housing Assessment System physical indicator score—which is determined using the REAC inspection score—as one factor in determining physical risk. PIH also assesses physical risk using the qualitative survey and location of the PHA. Additionally, the performance designation from the Public Housing Assessment System—which incorporates the REAC inspection score—is included as part of assessing governance risk. The financial and management categories do not incorporate the REAC physical inspection score. For each risk category, PIH assigns points and designates a risk level for each PHA, as shown in figure 9. A higher number of points is associated with higher risk. For example, PIH assigns 25 points to PHAs with a physical indicator score of 25 or below and zero points to PHAs with a physical indicator score of 28 or higher. After assigning points, PIH designates a risk level for each risk category, as well as overall, based on the average PHA score for each category and overall. These risk designations are very high, high, moderate, and low. PHAs furthest from and above the average score are designated as very high risk, and PHAs closest to the average score are designated as low risk. PIH designates a risk level to PHAs every quarter, although some information used to determine the risk level is not updated every quarter. For example, the qualitative survey is updated every other quarter. PIH determines actions—called risk treatments—to address each risk category based on a PHA’s risk level. PIH determines actions each quarter for every PHA newly designated as very high or high risk, and it determines actions every other quarter for all other very high, high, or moderate risk PHAs. To address physical risks, field office staff may provide training or technical assistance to PHAs. For example, field office staff told us they provided technical assistance by explaining the physical inspection standards and policies related to using operating funds to make physical repairs. Risk treatments have a completion date, and PIH field office staff are to monitor whether the treatment is effective. If the risk treatments do not result in improvements, PIH officials told us they can seek technical assistance from subject matter experts within PIH or can elevate the risk treatment, among other actions. For example, PIH can provide on-site assistance rather than remote assistance. Multifamily Housing’s Process for Directing Property Owners to Correct Physical Deficiencies Is Based on REAC Scores Process for Correcting Deficiencies Multifamily Housing is required to direct property owners to correct physical deficiencies based on the REAC inspection score. For properties that score below 60 on the REAC physical inspection, Multifamily Housing issues property owners a notice to take the following actions: (1) provide a copy of the notice to residents; (2) survey 100 percent of the property to identify all physical deficiencies; (3) correct all deficiencies identified during the survey and the REAC inspection; (4) certify that they have corrected all deficiencies; and (5) submit a 100- percent survey of the property and certification of corrected deficiencies to HUD. Property owners should complete these actions within 60 days of receiving the notice but may request an extension if correcting the deficiencies will take longer than 60 days. For example, Multifamily Housing may issue extensions for notices received in winter months because seasonal conditions may make certain repair work, such as pouring concrete, more difficult to complete within 60 days. Multifamily Housing schedules a follow-up inspection depending on whether property owners submit a certification, as well as if the property scores 30 or below on the inspection. For property owners who certify that deficiencies have been corrected, Multifamily Housing schedules the property’s next inspection to take place within 1 year after the date of the last inspection. For property owners who do not submit the certification or for properties that score 30 or below on the REAC inspection, Multifamily Housing or the Departmental Enforcement Center schedules a follow-up inspection as soon as possible. Multifamily Housing uses the REAC score from that next inspection to determine whether the owner corrected deficiencies. Actions for Low-Scoring Properties After issuing a notice, Multifamily Housing can take various actions when properties’ scores on the REAC inspection remain below 60, if owners do not certify or correct physical deficiencies. Table 6 summarizes the actions Multifamily Housing took in fiscal years 2016 and 2017. For example, Multifamily Housing officials initially can place a flag in a data system to indicate that an owner has not met requirements for properties to be decent, safe, sanitary, and in good repair. This flag may prevent the owner from further participation in HUD programs. Another action Multifamily Housing officials can take is to change the property’s management agent. Multifamily Housing officials told us this action has been successful in improving the physical conditions of properties when properties do not require significant repair work. Multifamily Housing also can take more significant actions, such as terminating a rental assistance contract or foreclosing on a loan and relocating tenants from these properties. In addition to taking these actions, REAC and Multifamily Housing refer properties to the Departmental Enforcement Center when they score below a determined threshold. Upon publishing the inspection score, REAC refers properties that score 30 or below on a REAC inspection to the Departmental Enforcement Center automatically. Multifamily Housing officials told us they coordinate with relevant stakeholders to discuss these properties. Multifamily Housing also can refer properties electively to the Departmental Enforcement Center when they score between 31 and 59 on the REAC inspection. Further, Multifamily Housing can recommend specific actions for the Departmental Enforcement Center to take regardless of a property’s inspection score. The Departmental Enforcement Center can impose civil money penalties to encourage compliance with HUD’s regulations or limit a property owner from participating in HUD programs. Our analysis of referral data for physical conditions from fiscal years 2012 through 2017 shows that for 12 referrals, the Departmental Enforcement Center imposed money penalties through a settlement, and that no referrals resulted in a suspension or debarment. However, according to our previous work on the Departmental Enforcement Center, most referrals result from financial reviews rather than physical inspections. HUD’s Threshold for Issuing Notices for Property Owners Is Inconsistent with Requirements of Appropriations Legislation The Office of Multifamily Housing’s current practice of issuing notices to property owners when the REAC score is 59 or below is inconsistent with the legal requirement. As previously discussed, for properties that score 59 or below on the REAC inspection, HUD issues notices for property owners to certify that deficiencies have been identified and corrected within 60 days. However, the 2017 and 2018 Consolidated Appropriations Acts state that HUD must provide a notice to owners of properties that score 60 or below on the REAC physical inspection. Multifamily Housing officials told us that they believe language in the appropriations acts is not clear regarding the threshold to issue notices to property owners. Specifically, the appropriations acts state that HUD should issue a notice for properties that score 60 or below, and also that HUD may withdraw the notice to property owners when they successfully appeal their inspection score to 60 or above. Additionally, Multifamily Housing officials told us that HUD’s long-standing and current practice is to issue notices when a property receives a score of 59 or below. According to our analysis of inspection data, 30 properties received a score of 60 from May 2017 to December 2017 and would not have received a notice to correct physical deficiencies under HUD’s approach. Unless Congress changes the threshold identified in appropriations acts from 60 to 59 or HUD changes its practice to issue notices to properties that score 60 or below, HUD’s actions will continue to be inconsistent with the legal requirement. Other Multifamily Housing Monitoring Processes Also Incorporate REAC Scores Multifamily Housing also uses other processes to monitor the physical condition of properties, including properties that score 60 or above on the REAC inspection. These other processes incorporate additional aspects of properties beyond physical conditions. Risk Rating System Multifamily Housing’s risk rating system uses information on properties’ physical, financial, and management conditions to assign one of three risk ratings—troubled, potentially troubled, or not troubled—to each property. The REAC inspection score, along with actions taken to correct deficiencies, is one factor that determines the risk rating. Specifically, properties that score between 30 and 70 on the REAC inspection are rated as potentially troubled if the property owner is addressing physical deficiencies. Properties that score between 30 and 59 are rated as troubled if the owner has not certified that deficiencies have been corrected. Properties that score below 30 are rated as troubled and maintain that rating until the next REAC inspection. Multifamily Housing field office and headquarters staff told us they provide greater monitoring and oversight to properties rated as troubled and potentially troubled. Properties rated as troubled are required to develop an action plan to identify and document steps to address their risk, including physical risk. For example, a plan to improve the physical condition of a property may direct property owners to rehabilitate units. Properties rated as potentially troubled may develop such a plan but are not required to do so. Additionally, Multifamily Housing headquarters staff conduct a monthly call with field office staff to discuss properties rated as troubled. Multifamily Housing officials told us they review properties every 3 to 12 months based on the risk rating and can take actions if properties are not correcting issues. If property owners do not correct issues outlined in their plan, Multifamily Housing can take many of the actions listed previously, such as changing the management agent. Other Monitoring Processes Multifamily Housing can also monitor properties through other processes, such as site visits or other reviews. According to Multifamily Housing officials, field office staff conduct site visits of properties if they receive multiple complaints from tenants or notice a particular concern, or if the property receives media attention. Multifamily Housing also can conduct site visits of properties through a Management and Occupancy Review. Multifamily Housing officials told us they are moving toward a risk-based approach, using results from prior reviews and a property’s risk rating to determine how often to conduct these Management and Occupancy Reviews. However, Multifamily Housing officials told us that budget and staffing constraints continue to limit the number of reviews completed annually, with less than half of project-based rental assistance properties reviewed in 2017. To complete Management and Occupancy Reviews, HUD staff or contractors review documentation to monitor whether properties are adhering to requirements for receiving HUD funding and to target potential issues. This review gathers information on seven factors of property management, including management of a property’s physical condition. As part of gathering information, HUD staff or contract administrators interview the property owner or agent and may visit a sample of housing units to verify that deficiencies identified in the REAC inspection have been corrected. The Management and Occupancy Review specifies—in a summary report for owners and agents— corrective actions to take within targeted completion dates, not to exceed 30 days, based on the documentation review and on-site visit. Properties that perform poorly on the review also must provide proof of taking these actions. If properties do not provide proof of taking these corrective actions, Multifamily Housing can take some of the previously listed actions, such as changing the agent of a property. Conclusions REAC’s inspection process annually identifies properties that are in poor physical condition and contain life threatening health and safety issues. With over 2 million moderate- and low-income households living in public housing or multifamily properties assisted or insured by HUD, it is imperative that these properties are decent, safe, sanitary, and in good repair. Our review of REAC found areas for improvement in its inspection process: Review of inspection process. A comprehensive review of the inspection process could help REAC identify risks and ensure it is meeting the goal specified in its strategic plan that inspections be reliable, replicable, and reasonable. Sampling errors in inspection scores. If REAC were to resume reporting on sampling errors and develop a process to address properties that fall below certain cutoff scores when the sampling error is taken into account, it would have the information it needs to identify properties that may require more frequent inspections or enforcement actions. Sampling methodology documentation. Comprehensive and organized documentation of the sampling methodology could help REAC preserve the institutional knowledge of important features of its inspection process, particularly when key staff leave the agency. Timing of housing inspections. Improvements in REAC’s on-time performance of multifamily property inspections could provide HUD with more timely information on the physical condition of these properties and the information it needs to take any enforcement actions. Further, by developing mechanisms to track its progress on meeting the schedule for inspections and improving its collection of data on why inspections are delayed, REAC could better determine what factors are contributing to delays in conducting inspections. Staffing inspections. A formal evaluation plan could help REAC determine if its pilot program for staffing inspections in difficult geographic areas is a success or whether changes are needed before moving from a pilot to a permanent process. Implementation of open recommendations. Taking timely actions on internal-review recommendations could help HUD to improve REAC’s inspection process and the safety of HUD-assisted properties. We also found areas for improvement in REAC’s processes for selecting, training, and overseeing contract and quality assurance inspectors: Inspector candidates’ qualifications. A more robust process for verifying contract inspectors’ qualifications could reduce the number of candidates with insufficient experience who participate in REAC’s training program, which could help REAC to expend fewer resources on training candidates who are unlikely to become successful inspectors. Contract inspector training. Evaluating the effectiveness of its training program for contract inspectors could help REAC better assess the quality of the program and plan for future training. Quality assurance inspector training. By developing and documenting training for quality assurance inspectors that encompasses all of their job responsibilities, REAC can better ensure that inspectors have the skills required to oversee contract inspectors. Continuing education requirements. Continuing education requirements for contract and quality assurance inspectors could help REAC ensure that inspectors are up-to-date on REAC policies and industry standards. Targets for reviews of contract inspectors. Improving its ability to meet management targets for CQA reviews and QCIs could help REAC better ensure that contract inspectors are receiving the feedback needed to improve their performance, thereby improving the quality of inspections. Formal policies for Quality Control group. By implementing policies and procedures for the Quality Control group, REAC can help ensure that the group achieves its objective of providing consistent reviews of quality assurance inspectors that will enable these inspectors to improve their oversight roles. Performance standards for quality assurance inspectors. Reviewing and updating REAC’s performance standards for quality assurance inspectors so that they align with their job duties can help REAC ensure that staff understand how their duties are prioritized within REAC’s mission and improve the quality of performance reviews. Finally, Multifamily Housing’s current practice of taking actions against property owners when the REAC score is 59 or below is inconsistent with the legal requirement to take action when the score is 60 or below. While in practice this affects very few properties, without either Congress changing the threshold identified in appropriations acts from 60 to 59 or HUD changing its practice to issue notices to properties that score 60 or below, HUD’s actions will continue to be inconsistent with the legal requirement. Recommendations for Executive Action We are making the following 14 recommendations to HUD: The Deputy Assistant Secretary for the Real Estate Assessment Center should conduct a comprehensive review of the physical inspection process. (Recommendation 1) The Deputy Assistant Secretary for the Real Estate Assessment Center should resume calculating the sampling error associated with the physical inspection score for each property, identify what changes may be needed for HUD to use sampling error results, and consider those results when determining whether more frequent inspections or enforcement actions are needed. (Recommendation 2) The Deputy Assistant Secretary for the Real Estate Assessment Center should develop comprehensive and organized documentation of REAC’s sampling methodology and develop a process to ensure that documentation is maintained going forward. (Recommendation 3) The Deputy Assistant Secretary for the Real Estate Assessment Center should track on a routine basis whether REAC is conducting inspections of multifamily housing properties in accordance with federal guidelines for scheduling and coordinate with the Deputy Assistant Secretary for Multifamily Housing to minimize the number of properties that can cancel or reschedule their physical inspections. (Recommendation 4) The Deputy Assistant Secretary for the Real Estate Assessment Center should design and implement an evaluation plan to assess the effectiveness of the Indefinite Delivery/Indefinite Quantity pilot in ensuring timely and quality inspections for properties in hard-to-staff geographic areas. (Recommendation 5) The Deputy Assistant Secretary for Multifamily Housing and the Deputy Assistant Secretary for the Real Estate Assessment Center should expedite implementation of the recommendations from the Rapid Response and Resolution Team. (Recommendation 6) The Deputy Assistant Secretary for the Real Estate Assessment Center should follow through on REAC’s plan to create a process to verify candidate qualifications for contract inspectors—for example, by calling references and requesting documentation from candidates that supports their completion of 250 residential or commercial inspections. The plan should also consider whether certain types of inspections—such as Federal Emergency Management Agency inspections and U.S. Army Office of Housing inspections—satisfy REAC’s requirements. (Recommendation 7) The Deputy Assistant Secretary for the Real Estate Assessment Center should develop a process to evaluate the effectiveness of REAC’s training program—for example, by reviewing the results of tests or soliciting participant feedback. (Recommendation 8) The Deputy Assistant Secretary for the Real Estate Assessment Center should revise training for quality assurance inspectors to better reflect their job duties. Revised training should be documented, include expanded subject matter training, and address skills that may not be included in training for contract inspectors—for example, instructing contract inspector candidate trainings and coaching and providing feedback. (Recommendation 9) The Deputy Assistant Secretary for the Real Estate Assessment Center should develop continuing education requirements for contract and quality assurance inspectors. (Recommendation 10) The Deputy Assistant Secretary for the Real Estate Assessment Center should develop and implement a plan for meeting REAC’s management targets for the timeliness and frequency of CQA reviews and QCIs. The plan should include consideration of resources of and demands on quality assurance inspectors, including the effect of natural disasters and other special assignments. (Recommendation 11) The Deputy Assistant Secretary for the Real Estate Assessment Center should ensure that Quality Control’s policies and procedures for overseeing quality assurance inspectors are implemented. (Recommendation 12) The Deputy Assistant Secretary for the Real Estate Assessment Center should review quality assurance inspector performance standards and revise them to better reflect the skills and supporting behaviors that quality assurance inspectors need to effectively contribute to REAC’s mission. (Recommendation 13) The Deputy Assistant Secretary for Multifamily Housing should report to Congress on why the agency has not complied with the 2017 and 2018 Consolidated Appropriations Acts requirement to issue notices to properties when the REAC score is 60 or below, including seeking any statutory flexibilities or exceptions believed appropriate. (Recommendation 14) Agency Comments and Our Evaluation We provided a draft of this report to HUD for review and comment. In written comments, reproduced in appendix V, HUD agreed with 11 recommendations, partially agreed with 2, and neither agreed nor disagreed with 1. In its written comments, HUD noted that it largely agreed with the findings and has been examining how it can develop, pilot, and evaluate an alternate approach to its inspection model that will address the issues raised in our report. Consistent with our report, HUD recognized that after 20 years, its physical inspection process has become susceptible to manipulation. HUD said it plans to pilot a new physical inspection process in one of HUD’s administrative regions later this year. HUD stated that given its limited resources, it will be unable to simultaneously develop the new process and implement all of the recommendations to its current process. We maintain that implementing the recommendations will help REAC to ensure that properties are decent, safe, sanitary, and in good repair. HUD agreed with 11 recommendations and provided specific information about planned steps to implement them. For example, for our first recommendation on conducting a comprehensive review of REAC’s physical inspection process, HUD noted in its written comments that it plans to develop new standards, protocols, scoring approaches, and software to be validated through a demonstration. In addition, if resources are available, HUD plans to contract with an external vendor to assess the accuracy and effectiveness of the new inspection process and the statistical validity of scoring. For our eighth and ninth recommendations on evaluating and revising training for contract and quality assurance inspectors, HUD noted that it would evaluate its internal training program for contract inspectors as it pilots its new inspection process and compare the results with its evaluation of an outsourced training approach. In addition, HUD noted that it would identify the subject matter expertise needed for quality assurance inspectors and provide training to address any skills gaps among these inspectors. HUD partially agreed with our fourth and sixth recommendations and noted some considerations for addressing them. HUD partially agreed with our fourth recommendation regarding tracking its progress on conducting inspections of multifamily properties in accordance with federal guidelines, but did not identify the reason for its partial agreement. In written comments, HUD described actions it plans to take that we consider consistent with the intent of the recommendation. We maintain that this recommendation should be implemented to achieve benefits, including better understanding of the factors that contribute to inspection delays. HUD also partially agreed with our sixth recommendation regarding expedited implementation of recommendations from the Rapid Response and Resolution Team. In written comments, HUD noted that in order to balance resources invested in the current approach with those needed to design future operations, it would consider whether the remaining recommendations from the Rapid Response and Resolution Team fit with the new inspection model that it plans to pilot. Whether in the current inspection model or a future one, we maintain that expediting implementation of the recommendations from the Rapid Response and Resolution Team will support that team’s intention to address conditions at troubled multifamily properties. HUD neither agreed nor disagreed with our second recommendation to resume calculating the sampling error associated with the physical inspection for each property, identify the changes that may be needed for HUD to use sampling error results, and consider those results when determining whether more frequent inspections or enforcement actions are needed. In response to this recommendation, HUD noted in its written comments that it is examining resource implications, regulations and policies that would need to be changed, and the viability and effectiveness of making the changes included in our recommendations. We maintain that implementing this recommendation would improve REAC’s inspection process by identifying properties that may require more frequent inspections or enforcement actions. We are sending copies of this report to the appropriate congressional committees and the Secretary of Housing and Urban Development. In addition, the report is available at no charge on the GAO website at http://www.gao.gov. If you or your staff have any questions about this report, please contact me at (202) 512-8678 or GarciaDiazD@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are listed in appendix VI. Appendix I: Objectives, Scope, and Methodology This report examines (1) the Department of Housing and Urban Development’s (HUD) Real Estate Assessment Center’s (REAC) process for identifying physical deficiencies; (2) REAC’s processes for selecting, training, and developing contract and quality assurance inspectors; (3) REAC’s processes for monitoring contract and quality assurance inspectors; and (4) HUD’s monitoring and enforcement processes for addressing physical deficiencies and how REAC’s information is used to support these processes. To address the first objective, we reviewed regulations and policies and procedures related to REAC’s physical inspection process. Specifically, we reviewed the final notice on REAC’s physical inspection scores and the 2017 update to REAC’s compilation bulletin, which is the guidance document for inspectors conducting physical inspections. We also reviewed REAC’s user guide, which explains how REAC’s inspection software and handheld data collection devices are used to conduct the inspection and record deficiencies. To describe the quality assurance processes for physical inspections, we reviewed REAC’s quality assurance standard operating procedures, which provide instructions to REAC’s quality assurance inspectors on how they are to conduct various monitoring activities over contract inspectors to assess the quality of inspections. We also reviewed REAC’s standard operating procedures for post-inspection reviews. As part of our assessment of the physical inspection process, we reviewed the statistical methodology used by REAC to determine the sample size for dwelling units and buildings. We reviewed REAC’s documentation describing the sample-size calculations for units and buildings and interviewed a REAC statistician to obtain information on the statistical approach and assumptions used in the sample size calculations. With this information, we were able to conduct our own calculations on the sample-size and compare our results to REAC’s. To report on the number of physical inspections conducted from fiscal years 2013 through 2017, as well as other data on inspections over this period, we accessed REAC’s Record and Process Inspection Data database. This database contains information related to physical inspections, such as the types and locations of properties inspected, dates of inspection, and inspection scores. To assess the reliability of the database, we first identified the various tables in the database that held the relevant data we needed for our analysis. We also identified the common identifier in each of these tables to construct records of inspections with the relevant data. We met with REAC’s staff to confirm that our selection of the tables and our construction of records was correct. We then performed our analysis and developed various descriptive statistics, such as the number of inspections per year by property type from fiscal years 2013 through 2017, the number of multifamily properties that failed their REAC inspection (scored below 60) for fiscal years 2013 through 2017, the percentage of multifamily property inspections that occurred on time given their inspection score, and various inspection score ranges by state. We compared our statistics on the number of inspections per year with comparable statistics developed by REAC. In cases where we had differences, we obtained explanations from REAC for these differences and revised our analysis where appropriate. Based on our overall assessment of the REAC data we used, we found them to be sufficiently reliable for analyzing the number and timing of inspections and trends in scoring. To obtain the views of various stakeholders on the inspection process, we held discussion groups with contract inspectors and REAC’s quality assurance inspectors and supervisors. Each discussion group had between 6 and 13 participants and was facilitated by a GAO staff member. We covered a number of topics in these discussion groups, including the inspection and quality assurance processes. We held one discussion group with contract inspectors, three with REAC quality assurance inspectors, and one with REAC quality assurance supervisors: Contract inspectors. For the discussion group with the contract inspectors, we invited all of the contract inspectors who were attending a conference at REAC’s headquarters in Washington, D.C. Thirteen contract inspectors attended the discussion group. Quality assurance inspectors. For the discussion groups with the quality assurance inspectors, we reached out to all quality assurance staff and coordinated with REAC to arrange specific meeting times to maximize the number of participants. We held two separate discussion groups with experienced inspectors. The first of these groups had 11 participants, and the second group had 6. We also held a separate discussion group with 11 newly hired quality assurance inspectors. Quality assurance supervisors. For the last discussion group, we reached out to all quality assurance supervisors and met with 6 of them. We recorded all of the discussion groups to help transcribe the conversations. In order to analyze the discussion group transcripts, we identified phrases that represented key themes across the groups. One GAO analyst reviewed one of the transcripts to identify any additional phrases we should add to our analysis. Once we arrived at our final set of key themes, one GAO analyst reviewed all of the transcripts and matched responses in the transcripts to the key themes. A second GAO analyst then checked the work to determine if he agreed with the coding of the first analyst. If there were any disagreements on the coding, the two analysts met to reach consensus on the appropriate coding. Finally, to obtain the perspectives of property owners on REAC’s inspection process, we met with four organizations representing multifamily or public housing property owners. These organizations were the Council for Large Public Housing Authorities, the National Affordable Housing Management Association, the National Leased Housing Association, and the Public Housing Authorities Directors Association. Also, to understand how private home inspection associations developed their inspection processes, we interviewed staff from the American Society of Home Inspectors and the International Association of Certified Home Inspectors. To address the second and third objectives, we reviewed REAC’s policies and procedures for selecting, training, developing, and monitoring contract and quality assurance inspectors. We reviewed the contract inspector candidate assessment questionnaire and construction analyst job announcement, which describe the requirements to become a contract and quality assurance inspector, respectively. We also reviewed documents describing the online (Phase Ia), classroom (Phase Ib), and field (Phase II) training courses. We also reviewed an assessment that Deloitte, a management consultant firm, conducted of REAC’s training, quality assurance, and inspector oversight processes. We compared REAC’s training processes for inspectors with key attributes of effective training and development programs. In our discussion groups with contract and quality assurance inspectors, we also asked their views on REAC’s selection, training, and monitoring processes. In addition, we interviewed REAC management officials to discuss their processes for the selection, training, monitoring, and oversight of contract and quality assurance inspectors. We spoke with staff from the American Society of Home Inspectors and the International Association of Certified Home Inspectors to understand how their selection and training requirements for inspectors who are members of home inspection associations compared with REAC’s. To examine REAC’s processes for monitoring contract inspector performance, we reviewed REAC’s quality assurance standard operating procedures, REAC’s strategic plan, and various tools REAC has developed to assess how contract and quality assurance inspectors perform relative to their peers. We obtained data on collaborative quality assurance reviews for fiscal years 2013 through 2017 and data on quality control inspections for January 2017 through June 2018. We analyzed the data to determine, for example, how often contract inspectors were conducting inspections in accordance with REAC’s Uniform Physical Conditions Standards protocol and its quality assurance standard operating procedures, and how often REAC was meeting its goals for timeliness and frequency of reviews. We assessed the reliability of the data by interviewing knowledgeable officials and conducting manual testing on relevant data fields for obvious errors. Based on these steps, we found the data to be sufficiently reliable for the purposes of our analyses. To examine REAC’s processes for monitoring and overseeing quality assurance inspector performance, we reviewed the performance standards and performance elements REAC uses to evaluate quality assurance inspectors. We interviewed staff from REAC’s Quality Control department, which conducts inspection reviews on quality assurance inspectors. We compared REAC’s performance management processes to key practices we have identified for effective performance management. We also compared REAC’s policies for oversight and monitoring of quality assurance inspectors to criteria in Standards for Internal Control in the Federal Government. To address the fourth objective, we reviewed documentation related to monitoring and enforcement processes for HUD’s Office of Multifamily Housing (Multifamily Housing), Office of Public and Indian Housing (PIH), and the Departmental Enforcement Center. For example, we reviewed relevant protocols and guidance documents on PIH’s and Multifamily Housing’s processes to address physical risk, among other risks. We also reviewed the relevant legal authorities in the 2014 through 2018 Consolidated Appropriations Acts and federal regulations for these HUD program offices to take enforcement actions for properties with physical deficiencies. We interviewed officials from Multifamily Housing, PIH, and the Departmental Enforcement Center on their processes to monitor the physical condition of properties and take enforcement actions. We further selected two Multifamily Housing and four PIH field offices throughout the United States to understand actions they take to monitor properties and ensure that physical deficiencies are corrected. We developed a two- stage process to select field offices with a higher percentage of inspections with scores 70 and below. We first selected HUD regions based on our score criteria and then selected specific field offices within those regions using similar score criteria. Because we selected a nonprobability sample of field offices, the information we obtained cannot be generalized more broadly to all field offices. However, the information provides important context and insight into how the enforcement process for physical deficiencies works across the country. In addition, we obtained data on performance designations for public housing agencies within PIH, actions taken by Multifamily Housing for properties scoring below 60 on the REAC inspection, and actions taken by the Departmental Enforcement Center for Multifamily Housing properties. We assessed the reliability of the data by reviewing relevant HUD guidance and obtaining written responses from agency officials on how the data were collected, maintained, analyzed, and presented. Based on these steps, we found the data to be sufficiently reliable for the purposes of our analyses. Finally, we reviewed prior reports from GAO and from the HUD Office of Inspector General that discussed efforts to monitor the physical condition of properties, among other conditions. We conducted this performance audit from July 2017 to March 2019 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives. Appendix II: Number of Multifamily Housing Inspections and Percentage of Inspections in Selected Score Ranges, Fiscal Years 2013– 2017 The Real Estate Assessment Center (REAC) conducted 44,486 inspections of Office of Multifamily Housing (Multifamily Housing) properties from fiscal years 2013 through 2017, according to our analysis of REAC’s inspection data. Properties received a score from 0 to 100, with a score below 60 considered as failing. Table 7 shows the percentage of inspections conducted in each state across three score ranges. States varied in the percentage of inspections that fell within the score range considered failing (0 to 59), from a low of 1 percent to a high of 10 percent. REAC inspects properties with lower scores more frequently than properties with higher scores. For example, properties that scored below 80 would have been inspected annually over this period, while properties that scored 90 or above would have been inspected every 3 years. Appendix III: Number of Public Housing Inspections and Percentage and Number of Inspections in Selected Score Ranges, Fiscal Years 2013–2017 The Real Estate Assessment Center (REAC) conducted 15,156 inspections of public housing properties from fiscal years 2013 through 2017, according to our analysis of REAC’s inspection data. Properties received a score from 0 to 100, with a score below 60 considered as failing. Table 8 shows the percentage of inspections conducted in states or U.S. territories across three score ranges. States varied in the percentage of inspections that fell within the score range considered failing (scores 0 to 59), from a low of 1 percent to a high of 34 percent. REAC generally inspects properties with lower scores more frequently than properties with higher scores. According to our analysis, REAC conducted fewer than 100 inspections of public housing properties in 18 states or territories. Table 9 shows the number of inspections conducted within three score ranges for these 18 states or territories. Appendix IV: Recommendations to the Real Estate Assessment Center from the Rapid Response and Resolution Team The Rapid Response and Resolution Team was created by the Department of Housing and Urban Development (HUD) in May 2016 to address troubled multifamily properties by improving HUD’s internal processes for assessing properties and analyzing risk so that properties do not become troubled; improving HUD’s processes for inspecting properties so that troubled ones are identified earlier and more reliably and communicating the results to stakeholders; and improving HUD’s processes for enforcing corrective actions and resolving troubled properties and working with owners so that HUD resources are used only on safe and healthy housing. The team consisted of staff from the Real Estate Assessment Center (REAC) and other units within HUD, including the Office of Multifamily Housing (Multifamily Housing). In January 2017, the team presented 31 recommendations, 8 of which were specific to REAC. As of December 2018, REAC had not yet implemented any of these recommendations. REAC had reached concurrence with Multifamily Housing on 3 of these recommendations and asked for Multifamily Housing’s consideration of the funding and rulemaking requirements for the remaining 5 recommendations. The 8 recommendations that were specific to REAC are as follows: 1. Implement a risk-based exigent health and safety abatement verification policy. 2. Inspect properties that have a REAC physical inspection score of less than 60 after a 3-day notice. 3. Increase the scoring weights of units and reexamine point deduction caps. 4. Expand photo capability in the inspection process to level 1 and level 2 deficiencies and a panoramic photo of the property. 5. Inspect carbon monoxide detectors in the inspection process. 6. Develop health and safety abatement requirements, including focusing on water ponding and missing lead-based paint disclosure forms and inspection reports. 7. Take enforcement action to protect tenants before the 45-day appeal period is over for properties that score under 30 points and that have exigent health and safety deficiencies. 8. Require electronic exigent health and safety certifications and abatements within 24 hours of the inspection. Appendix V: Comments from the Department of Housing and Urban Development Appendix VI: GAO Contact and Staff Acknowledgments GAO Contact Staff Acknowledgments In addition to the contact named above, Andy Pauline (Assistant Director), José R. Peña (Analyst in Charge), Carl Barden, Chloe Brown, Hannah Dodd, Juan Garcia, Jeff Harner, Emily Hutz, Jill Lacey, Jerry Sandau, Jessica Sandler, Jennifer Schwartz, and Jena Sinkfield made key contributions to this report.