Background Schools generally report incidents of restraint and seclusion to their district, and districts are responsible for reporting incidents of restraint and seclusion to Education’s CRDC. Districts are expected to report the number of incidents and the number of students affected for all schools in their district and to use Education’s definitions of restraint and seclusion to determine whether an incident occurred. Education defines two types of restraint: physical and mechanical (see table 1). Education’s definition of a physical restraint makes a distinction between a restraint and a physical escort. Similarly, Education’s definition of seclusion makes a distinction between seclusion and a timeout (see figures 1 and 2). Every two years, OCR administers the CRDC to nearly every public school district in the country. In turn, districts self-report information on a wide variety of topics, including course offerings, discipline, and restraint and seclusion. Education collects these data through an online submission tool. CRDC activities, such as data collection and quality, are managed by a company under contract with Education. The data submission period for the 2017-18 school year ended June 21, 2019. School districts have one year from the end of the submission period to make a request to amend submitted data. As of March 2020, Education had not announced when it will publicly release these data. Education’s primary data quality checks for the CRDC data, including the restraint and seclusion data, are built into the CRDC submission tool. The online CRDC submission tool automatically performs checks that flag data errors or potential errors. These “business rules” occur in real time as districts enter data or after they upload files. The 2015-16 submission tool used three business rules related to restraint and seclusion; for the purpose of this report, we focused on the two rules most relevant to our work. The first business rule pertains to the reporting of zeros for very large districts only—that is, those districts with 100,000 or more enrolled students (see fig. 3). If a very large school district enters zero incidents of restraint and seclusion, it receives a message prompting it to review its enrollment counts and reported incidents, or provide an explanation using a reason code and comment. Importantly, if districts have not collected the data required for the CRDC—or if the data are unavailable for some other reason—districts are to leave relevant data cells blank. A zero in a data cell should represent an actual count— that is, the district restrained or secluded zero students. The second rule applies to schools that both report more than 100 incidents of restraint or seclusion and enter a greater number of students affected by restraint or seclusion than the number of incidents reported (see fig.4). Districts are asked to resolve this error by adjusting their counts so the number of students subjected to restraint or seclusion is less than the number of incidents of restraint or seclusion, or to provide an explanation using a reason code and comment. The last step in the data submission process is the district certification. To complete this step, the district superintendent or an authorized designee must indicate agreement with a statement that acknowledges that they are responsible for verifying the data, the information provided is “true and correct,” and a willfully false statement is punishable by law. The CRDC submission tool is designed so that it should not allow a district to certify its submission unless all required data pass the system validation checks, or all errors are explained. Education also reviews CRDC data quality during other phases of data collection to identify potential data quality issues to improve future collections. For school year 2015-16, Education’s contractor reviewed data quality during and after the collection phase. Education’s contractor contacted school districts about potential errors that Education determined were easily adjustable and asked them to review and correct data or provide an explanation if no corrections were determined to be necessary. Further, some states and school districts have laws, regulations, and/or policies regarding restraint and seclusion. These laws vary from state to state, and sometimes require schools or districts to annually report incidents of restraint and seclusion to either the state or local education agency. In January 2019, Education announced an initiative to address inappropriate use of restraint and seclusion on children with disabilities. As part of this initiative, OCR announced plans to conduct 50 data quality reviews of the 2015-16 restraint and seclusion CRDC data submitted across each of OCR’s 12 enforcement regions. This review had not been completed at the time of our review. In August 2019, Education announced plans for OCR to work collaboratively with Education’s National Center for Education Statistics (NCES). Education stated that the collaboration would help ensure that CRDC data are reliable and authenticated in a manner that provides a more accurate picture of key civil rights issues in education. According to the announcement, under a new agreement, NCES and OCR will work together to improve the quality of the CRDC data by providing school districts with technical assistance, and by reviewing and revising data quality procedures. NCES is the primary federal entity responsible for collecting and analyzing statistical data related to education. Key Quality Control Processes for Restraint and Seclusion Are Ineffective or Do Not Exist We identified four key issues for which Education’s quality control processes for its CRDC restraint and seclusion data are largely ineffective or do not exist (see table 2). CRDC Business Rule Targeting Zero Reporting Is Largely Ineffective Education’s business rule to detect potentially erroneous reporting of zero incidents applied to only 30 of the more than 17,000 school districts nationwide, rendering it largely ineffective for checking the 70 percent of districts that reported zero incidents of restraint or seclusion. This is because the rule only applied to districts with over 100,000 enrolled students. However, in its January 2019 data quality review of 50 districts’ restraint and seclusion data, OCR found erroneous reporting of zeros in districts of all sizes. Of the 50 districts OCR contacted, OCR determined that 40 districts should amend their original 2015-16 submissions. When we analyzed the 2015-16 CRDC restraint and seclusion data, we found that almost three-quarters of small districts reported zeros, while about one-third of large and one-fourth of very large districts reported zero incidents (see table 3). The findings from Education’s data quality review, along with those from our analysis, suggest that misreporting is a problem among districts of all sizes. For the 2017-18 CRDC data collection, Education lowered the threshold for the rule to detect potentially erroneous reporting of zeros to include districts with 50,000 enrolled students, rather than only districts with more than 100,000. However, the new rule counts students with disabilities and students without disabilities separately. To trigger the rule, a district would have to have at least 50,000 students with disabilities and report zero incidents for them, or have at least 50,000 students without disabilities and report zero incidents for them. Of the nation’s more than 17,000 school districts, only 3 (Chicago, Los Angeles, and New York) have at least 50,000 students with disabilities; only 95 have at least 50,000 students without disabilities. Education’s CRDC data show that restraint and seclusion disproportionately affect students with disabilities and its data quality review showed that substantial portions of districts of all sizes inaccurately reported zeros. However, Education could not provide a data-driven basis for the 100,000 or 50,000 student enrollment thresholds or for creating separate thresholds for students with and without disabilities. Rather, Education stated that the thresholds were a management policy decision inherited from previous administrations. Because Education’s business rule targeting districts that report zeros was inadequate, in June 2019, GAO recommended, among other things, that for the 2017-18 data collection Education contact districts that reported all zeros for restraint and seclusion to ask them to ensure that the zeros actually represented zero incidents, and Education did so after the data collection closed. Absent a business rule targeting all districts reporting zeros during data submission, inaccuracies in future data collections will likely be missed, and Education risks expending more time and resources with repeated manual follow up after the fact. Federal standards for internal control state that management should evaluate both internal and external sources of data for reliability. Absent reliable and accurate data, the public’s confidence in the CRDC restraint and seclusion data may be further undermined, and the utility of a dataset intended to assist with federal civil rights monitoring, enforcement, and oversight will remain limited. While it is clear that some school districts have reported inaccurate restraint and seclusion data, Education officials do not fully understand why this is occurring. In technical comments on a draft of this report, Education stated that it is committed to learning more about why this is occurring. While we do not know all of the reasons districts fail to report accurate data, our interviews with over 50 school and district officials provide some insight. School officials in the nine districts we visited cited a variety of reasons districts might not report, including that they were not collecting the data because their state did not require reporting, and that their school district only required them to collect data for students with Individualized Education Programs (IEPs). More fully understanding why districts report inaccurate data is key to correcting the issue. Federal standards for internal control also state that managers should use quality information to achieve the entity’s objectives, assess the risks facing the entity as it seeks to achieve its objectives, and use this assessment to develop appropriate risk responses. By not identifying school districts’ reasons for reporting zero incidents of restraint and seclusion, Education will not know how to best support districts in improving the accuracy of their reported data. Future CRDC data will remain inaccurate, significantly limiting the utility of a key tool on which OCR relies to help it enforce federal civil rights laws. CRDC Lacks Business Rules to Detect Very Low or Very High Rates of Restraint or Seclusion Education has no business rules that flag school districts reporting very low or very high rates of restraint or seclusion, nor has it completed initial efforts to determine a range of rates that might warrant further exploration. Given widespread concerns about misreporting, we devised two possible ways to test for these types of outliers. First, we looked beneath the district level to examine school-level reporting patterns within districts. When we tested the nation’s 30 largest school districts (those with more than 100,000 students), we found patterns that may suggest underreporting in at least 13 of them, in addition to the 10 that reported zeros for the 2015-16 school year. In these 13 districts, we found that all of the incidents of restraint reported occurred in no more than 15 percent of a district’s schools; the rest of the schools in those districts reported zero incidents. (See fig. 5.) For example, the Chicago Public School District—the third largest school district in the country, with nearly 400,000 students enrolled—reported a total of 47 incidents of restraint for school year 2015-16. All of these incidents were reported by just two of its 579 schools. The district’s six incidents of seclusion were clustered in one school. In the Los Angeles Unified school district, the second largest school district in the country, 82 of its 785 schools reported a total of 108 incidents of restraint, with 65 schools reporting exactly one incident each. The district reported no incidents of seclusion. Education has a business rule that targets large districts (for 2017-18, those with over 50,000 students), but only when all schools in a district report zeros. Thus, as long as a large district reports at least one incident of either restraint or seclusion, the business rule would not be triggered. Education’s post-collection data quality reviews for school year 2015-16 did not test below the district level to look for potential underreporting within a district. Second, we tested for outliers by comparing per capita rates of restraint or seclusion in the 30 very largest districts (over 100,000 students enrolled) to average rates in all school districts. In the 30 districts, we found that in addition to the 10 districts that reported zeros, nine districts reported fewer than three incidents of physical restraint per 10,000 students, which is lower than 95 percent of all districts reporting incidents. (See fig. 6.) For example, DeKalb County school district in Georgia reported 0.3 incidents per 10,000 students, and Charlotte Mecklenburg school district in North Carolina reported 0.5 incidents per 10,000 students. We also tested for districts with very high rates of physical restraint. For the 2015-16 school year, we found 52 districts that were outliers, most of which served comparatively large populations of students with disabilities. Forty-nine of these 52 districts had rates of physical restraint per enrolled student higher than 99 percent of all districts that reported incidents of physical restraint. Almost half reported an average of 10 or more incidents per student affected, and almost two-thirds of the districts reported restraining from 25 to100 percent of their students. The Learning Tree preschools in Alabama, which enrolled a combined 135 preschool students ages 3 to 5, reported that it restrained nearly two-thirds of its students in school year 2015-16. Further, Learning Tree reported 5,963 incidents of physical restraint affecting 84 students, or an average of 71 incidents of physical restraint per preschooler. The Morris-Union Jointure Commission School District in New Jersey, where almost all of its 281 students were identified as having a disability, reported restraining over one-third of its students. These students were restrained an average of 20 times in school year 2015-16. (See table 4.) We found a similar pattern in the 2015-16 seclusion data, identifying 36 outlier school districts. For 22 of the 36 districts, the rates of seclusion were higher than 99 percent of districts reporting. (See appendix IV for more information.) For example, CRDC data for the Sangamon Area Special Education District in Illinois, which enrolled 74 students in grades 1 through 12, showed the district secluded one-third of its students an average of 27 times each in school year 2015-16. Similarly, data for the Bi-County Special Education Cooperative, also in Illinois, showed the district secluded over two-thirds of its 48 enrolled students an average of 13 times each in school year 2015-16. Federal standards for internal control state that management is to determine if controls individually and in combination with other controls are capable of achieving an objective and addressing related risks. An internal control design may be deficient when a control necessary to meet an objective is missing. Without business rules or similar analytical processes to flag these outliers, they may continue to go undetected by Education and other stakeholders. Education may be missing an opportunity to identify districts with disproportionately low or high incidents of restraint and seclusion to determine where technical assistance or other intervention may be warranted. Such information is particularly critical given widespread concerns about underreporting and misreporting, and its stated interest in protecting students’ civil rights. CRDC Business Rule Targeting Illogical Data Is Largely Ineffective Education has a business rule that identifies illogical data; that is, when schools report more students affected than incidents of either restraint or seclusion. However, the rule is largely ineffective because it was not designed in a way that would detect logical inconsistencies in the majority of cases, as the rule would have only applied to schools with at least 100 incidents. When we tested Education’s rule on the 462 schools that reported at least 100 incidents in 2015-16, we found no logical inconsistencies in the data. However, when we tested the rule on all schools, we found logical inconsistencies in the data reported by 592 schools with fewer than 100 incidents. For example, a school in Indiana reported that it restrained 156 students, but only reported 80 incidents. (See table 5.) Education could not provide any data-driven basis for its threshold of 100 incidents for this business rule. Education officials said that the threshold was inherited from previous administrations’ business rules. Collecting accurate data is key to the Office for Civil Rights’ (OCR) mission to ensure equal access to education and to promote educational excellence throughout the nation. In addition, federal standards for internal control state that when evaluating the design of internal controls, such as business rules, management should determine if controls are capable of achieving an objective and addressing related risks. An internal control design is deficient if, even though it operates as designed, it does not meet the control objectives. Our analysis shows that the business rule is not effective in its current form, because 592 schools were able to report illogical, and therefore incorrect, data. School and School District Officials Lacked a Common Understanding of Federal Restraint and Seclusion Definitions We talked to more than 50 officials in nine school districts in Kentucky, Washington, and Wisconsin about their interpretations of the CRDC’s definitions of restraint and seclusion. These school districts all reported incidents of restraint and seclusion in 2015-16, but officials we interviewed differed in their interpretations of terms used in the CRDC definitions, such as alone and escort. As a result, districts varied in how they counted incidents of restraint and seclusion. Further, officials we spoke with in the three state educational agencies and all seven stakeholder groups with expertise on the use of restraint and seclusion in public schools also said there was ambiguity regarding terms used in the definitions. For example, an official from one stakeholder group that represents some of the nation’s largest school districts said that its constituents provided feedback that restraint and seclusion terms were ambiguous, open to interpretation, and do not provide enough clarity. Civil Rights Data Collection Definition of Seclusion: Seclusion is the involuntary confinement of a student alone in a room or area from which the student is physically prevented from leaving. It does not include a timeout, which is a behavior management technique that is part of an approved program, involves the monitored separation of the student in a non- locked setting, and is implemented for the purpose of calming. With respect to the definition of seclusion, district and school officials varied in their interpretations of the word alone, and consequently, whether the incident should be counted as seclusion. Officials in three districts said that an incident was not seclusion as long as a teacher was in the room with the student, while officials in several other districts said that even if a teacher was present, it could still be seclusion if the student was prevented from leaving. (See sidebar.) Officials in the nine districts we visited also varied in their interpretation of the word area. Because the CRDC’s definition of seclusion states that seclusion can occur in an area, officials from one stakeholder group representing thousands of school administrators wondered whether it should be considered seclusion if a child is in a classroom with 20 other students and is required to stay alone in the corner of the room. Officials from a district in Wisconsin said that if a student is taken away from peers and placed in one area of the same room, but cannot leave that area, it still might be seclusion, even if the student and peers are in the same room. Officials in another district in Wisconsin said that sending a student to a corner does not count as seclusion. However, they said that the use of mobile partitions to close off an area of a room could constitute seclusion. The phrase physically prevented from leaving also elicited differing interpretations. Officials from the stakeholder group representing administrators said the definition is not clear about what counts as “prevented from leaving”: a barrier, such as a door; the presence of another adult watching the child; or both. School officials we spoke with had differing interpretations of this phrase, which affected how they counted and reported incidents of seclusion. School officials in a district in Kentucky said that the phrase means closing the door and keeping it closed. However, officials in another school in the same district did not specify the use of a door, stating instead that “physically prevented from leaving” means the student cannot walk out of the room. A school official in Washington said that it would count as seclusion if staff put a student in a motorized wheelchair in a room and deactivated the wheelchair’s power. Civil Rights Data Collection Definition of Physical Restraint: Physical restraint is a personal restriction that immobilizes or reduces the ability of a student to freely move their torso, arms, legs, or head. The term does not include a physical escort. Physical escort means a temporary touching or holding of the hand, wrist, arm, shoulder or back for the purpose of inducing a student who is acting out to walk to a safe location. With respect to the definition of physical restraint, school district officials we interviewed varied in their interpretations of the term escort, which the CRDC definition specifies is not a physical restraint. (See sidebar.) While officials in three districts said that an escort meant providing a physical prompt to a student who was not resisting relocation, officials in four districts said that moving a student who was resisting staff still counted as an escort. For example, officials in a school in Wisconsin said that if the student who is resisting is “carried away” from a location, that action would not meet the definition of escort and would count as restraint. Yet a district in Kentucky counted moving students against their will—including by carrying them—as escorts and did not report them as restraints. Further demonstrating differing interpretations of these terms, officials in four districts said they reported all escorts as restraints in the CRDC. Education does not provide schools or school districts with any information that could help clarify its definitions or provide examples on how schools and school districts should apply the definitions of restraint and seclusion to common classroom situations. For example, while Education’s “Restraint and Seclusion: Resource Document” includes the CRDC definitions of restraint and seclusion, it does not include clarifying information or examples about how to apply the definitions. Officials in the schools and districts we visited inconsistently interpreted the definitions for restraint and seclusion; moreover, officials from the seven stakeholder groups we interviewed said the definitions were unclear. These findings raise concern about whether restraint and seclusion data reported by school districts to the CRDC are being reported in a way that is consistent with the CRDC definitions. Federal guidance on data reliability states that data should be well defined enough to yield similar results in similar analyses. In addition, federal standards for internal control state that agency management should use quality information to achieve the entity’s objectives, noting that such data should be reasonably free from error and bias and faithfully represent what they purport to represent. Absent data on restraint and seclusion that is what it purports to be, Education will continue to lack quality information key to fulfilling its mission of ensuring equal access to education nationwide. All Nine School Districts We Visited Used Data to Reduce Incidence of Restraint and Seclusion and Developed Strategies for Improved Reporting All Nine Districts Used their Data on Restraint and Seclusion to Reduce Use of Restraint and Seclusion Officials in all nine school districts we visited said they used their data on restraint and seclusion to help reduce its use. In addition to collecting data for CRDC reporting purposes, these districts also collected and used more current and more detailed data to help reduce the use of restraint and seclusion. Officials in seven of the nine districts said they began collecting the data when their state passed a law requiring reporting. District officials identified several benefits to collecting data and using it to develop strategies to reduce use of restraint and seclusion. Specifically, officials said that the data helped them identify the following: Behavior patterns. Officials in several districts told us that collecting and reviewing data on restraint and seclusion helps them identify patterns in staff and student behavior that may contribute to use of these practices. Specifically, by identifying the circumstances under which a student’s behavior tends to escalate, staff can strategize how to more effectively respond so as to prevent the need to use restraint or seclusion. For example, one official in an elementary school in Wisconsin said that if staff notice more incidents occur on particular weekdays, they can examine those days to understand what may be affecting students’ behavior. Similarly, a teacher of students with autism in a middle school in Washington said that reviewing data helps staff, such as teachers, paraprofessionals, and administrators, determine what triggered a student’s behavior and then determine what to do differently to avoid triggering the student. In all three states, we visited districts that required staff to participate in a debriefing after each incident in an effort to understand what might have triggered the event and to discuss strategies to deescalate future incidents. For example, officials in Washington said that the building administrator and all staff involved discuss every incident. District officials consider this an important step for reducing use of restraint and seclusion, and said holding the discussions was a “game- changer.” Need for training. Officials in several districts said they examine data on restraint and seclusion at the classroom and school level to determine if staff need additional training, including on how to manage student behavior, or appropriately use restraint or seclusion. For example, a behavior coach for a Kentucky school district said that the data on restraint and seclusion helps her determine if certain teachers could benefit from more training on de-escalation techniques. A director of student services in Washington said that he was concerned about the rates of restraint and seclusion in the district, and after implementing more training for teachers, the rates declined. Officials in another Washington district said that after the district began collecting data in response to state law, they discovered that staff were using restraint and seclusion as punishment. As a result, district officials said that they coached teachers on how to manage behavior differently and emphasized that restraint and seclusion should not be an everyday occurrence. Officials at an elementary school in Wisconsin said that de- escalation training helps staff understand that students are trying to communicate with their behaviors. They said that when staff adopt the perspective that students are trying to communicate, staff also see the value of collecting data to improve how they respond to the students’ behaviors. Need for student supports. Officials in five of the nine districts we visited spoke about using restraint and seclusion data to assess when a student required additional support services to be successful in the classroom. For example, officials in one district in Kentucky said their data provides evidence for obtaining additional staff or social emotional learning resources for students. Similarly, officials in one Wisconsin district said the data can be used to allocate funding for school-based services to help address underlying causes of behavior. Officials in another Wisconsin district said that a jump in restraints or seclusions of a particular student could indicate that the student’s individualized education program needs to be adjusted. All Selected Districts Developed Strategies to Encourage Reporting of Incidents Officials in all the districts we visited also shared strategies on how they improved their CRDC data reporting, including communicating with staff about how data are used, training on how to report, and developing processes that encourage reporting. Specifically: Communication and culture. Officials in the majority of school districts said they routinely reviewed their data with school staff and emphasized the value of collecting data on restraint and seclusion. Officials in a district in Wisconsin said that they monitor data on restraint and seclusion on a monthly basis for students with and without disabilities, which increases interest among school staff about what causes the incidents. Officials in three districts we visited said that they explain to staff that documenting incidents of restraint or seclusion ensures that students obtain the support services they need. In a Wisconsin district, officials said they emphasize that reporting helps the students and keeps the school safe by making the district aware that more supports are needed. Officials at a school in Wisconsin said that some staff might worry that the data reflect poorly on them or might fear repercussions, but district officials have worked to shift the culture of reporting to focus on continuous improvement and problem solving. Similarly, officials in another Wisconsin district said that schools might be concerned about the data being used against them; therefore, district officials try to create a culture of curiosity around the data, rather than a culture of punishment. Accountability. To encourage staff to report incidents, officials in some districts developed processes that increased accountability for reporting. For example, school officials at an elementary school in Wisconsin said an administrative assistant in the main office immediately logs calls from classroom teachers requesting help managing a student’s behavior. Officials said this process provides accountability. Two districts said that they used a team approach for restraint or seclusion, which included someone to observe and someone to record details of the intervention, such as the time it began or the events that preceded it. Having multiple people involved increased the likelihood that relevant facts were recorded. Officials in a district in Washington said that keeping teachers and staff honest about reporting requires reiterating the process and procedures, reviewing the forms with staff, and following up with schools that fail to submit reports. Officials in a district in Wisconsin said they have advised staff to write the incident down on paper until staff are able to enter it in the district’s electronic reporting system. Training. Officials from all nine districts said they encourage reporting by provided training on how to report incidents. Generally, this information was incorporated into trainings on when to use restraint and seclusion and how to deescalate a student’s behavior. Officials from five stakeholder groups we interviewed, all of whom have expertise related to the use of restraint and seclusion in public schools, stated that training was necessary to both raise awareness of the requirement to report incidents and to ensure that incidents were reported accurately. For example, in de-escalation training for teachers in a Washington district, the trainer provides examples of restraint and seclusion; presents a variety of scenarios, including ambiguous ones, for discussion; and reviews the appropriate staff response. Conclusions The Civil Rights Data Collection (CRDC) is a longstanding and critical aspect of Education’s Office for Civil Rights’ overall enforcement and monitoring strategy. Collecting accurate data through the CRDC can help Education in its mission to ensure equal access to education, promote educational excellence for all, and enforce various federal civil rights laws prohibiting discrimination on the basis of race, color, national origin, sex, and disability. However, the significant data quality problems that both Education and we identified with the CRDC data on restraint and seclusion, combined with the significant weaknesses we found in Education’s data quality control processes, cast serious doubt on the accuracy of these data. As a result, it is impossible to accurately determine the frequency and prevalence of restraint and seclusion among K-12 public school students. The four recommendations in our June 2019 report urged Education to take immediate steps to address the widespread potential misreporting of zeros for its 2017-18 CRDC. Education took some steps to address the issues we raised, but has not yet fully addressed them. Moreover, those recommendations were intended as stop-gap measures to improve the quality of the 2017-18 data being collected in real time precisely because the CRDC’s business rules related to restraint and seclusion are inadequate. Therefore, addressing our recommendations would not solve the issues that are the subject of this report. Our work makes it clear that an overhaul of the quality control processes is needed to correct fundamental problems with federal restraint and seclusion data collected through the CRDC. Two of the CRDC’s key business rules meant to check data quality and flag potential errors in restraint and seclusion data are poorly designed and the thresholds that trigger these rules have no data-driven basis. Further, Education does not have business rules designed to flag outlier schools and school districts that report relatively low or high rates of restraint and seclusion, nor has it determined a range of rates that might warrant further exploration. Until Education more fully understands why so many school districts are underreporting and misreporting federal restraint and seclusion data, it will likely not be able to help districts improve their reporting, thereby improving the accuracy and utility of the data. There were widely varied interpretations of federal restraint and seclusion definitions among the 50 school and district officials with whom we spoke and officials from the seven key stakeholder groups we interviewed echoed these concerns. As a result, we have concerns that school districts may be inconsistently counting and reporting instances of restraint and seclusion for federal reporting purposes. Clarifying the definitions, for example by explaining to districts how they can be applied to common classroom scenarios, could help produce more consistency in reporting. Ultimately, the issues we found with Education’s restraint and seclusion data have consequences for the students who are restrained or secluded in school and whose restraint or seclusion goes un-reported. When federal data are misreported to the public, it undermines confidence in that data and fails to provide decision makers with reliable information on which to make informed policy decisions to protect students. In addition, Education lacks information that could help it determine whether schools’ use of these practices may be excessive, discriminatory, or both. Recommendations for Executive Action GAO is making six recommendations on restraint and seclusion to the Department of Education’s Office for Civil Rights. The Assistant Secretary for the Office for Civil Rights should revise its CRDC business rule to require that every school district reporting zeros, regardless of district size or numbers of students with disabilities, affirm the zeros are correct during the CRDC data submission process. (Recommendation 1) The Assistant Secretary for the Office for Civil Rights should develop and implement a CRDC business rule that targets schools and school districts that report very low numbers of incidents and set data-driven thresholds to detect such incidents. (Recommendation 2) The Assistant Secretary for the Office for Civil Rights should develop and implement a CRDC business rule that targets schools and schools districts that report very high number of incidents and set data-driven thresholds to detect such incidents. (Recommendation 3) The Assistant Secretary for the Office for Civil Rights should apply the CRDC business rule targeting illogical data at the school level to all schools, regardless of the number of incidents reported. (Recommendation 4) The Assistant Secretary for the Office for Civil Rights should identify the factors that cause underreporting and misreporting of restraint and seclusion and take steps to help school districts overcome these issues. (Recommendation 5) The Assistant Secretary for the Office for Civil Rights should further refine and clarify federal restraint and seclusion definitions and take steps to ensure that this information is conveyed to school districts. This could include providing common classroom scenarios that highlight the differences between a restraint and an escort, and a time out and a seclusion. (Recommendation 6) Agency Comments and Our Evaluation We provided a draft of this report to the Department of Education for review and comment. In its formal comments, which are reproduced in appendix II, Education agreed with all six recommendations. Education also provided technical comments, which we incorporated, as appropriate. In agreeing with GAO’s six recommendations, Education stated that it would determine the best means to implement them. Education also stated that it is fully committed to working with public schools, state educational agencies, and school districts to help ensure accurate reporting of federal restraint and seclusion data, and to improve the quality of the information for all users of CRDC data. We appreciate Education’s willingness to address the serious data quality issues affecting the CRDC restraint and seclusion data. In its response, Education stated that the agency has already made significant improvements to the CRDC in general and has made specific improvements with respect to restraint and seclusion data, especially in response to the four recommendations we made in our June 2019 correspondence. Education asked that we acknowledge the progress it feels it has made in this regard, and we have done so. Importantly, however, our June 2019 recommendations were intended as stop-gap measures to improve the quality of the 2017-18 data that was already being collected in real time precisely because the CRDC’s business rules related to restraint and seclusion were inadequate. Therefore, steps Education has taken toward addressing them do not address the underlying data quality issues that are the subject of this report. In other words, the recommendations in this report urge Education to address data quality problems at the front-end by applying adequate business rules at the time districts submit their data. This could reduce the need for follow-up with districts to correct potentially inaccurate data. More information about our assessment of the steps Education has taken to address the four recommendations from the June 2019 report are available on our website. Education also stated that because our draft report did not mention the methodological improvements OCR made to address the quality of restraint and seclusion data for the 2017-18 CRDC data collection, our draft report overstates the relevance of the data issues from the 2015-16 collection. Education also stated that it provided us with information about the methodological improvements in December 2019, and, in its formal response, requested that we reflect the information in this report. We disagree with this perspective. After we completed our audit work for this engagement, Education provided us an excerpt from its post-collection data quality report for school year 2017-18. At that time and again in its technical comments on this report, Education stated that “information shared with GAO about the results of the 2017-18 data quality review process and what might be addressed is still confidential.” As of March 23, 2020 Education described the 2017-18 data quality control process as “incomplete.” Lastly, the 2017-18 CRDC data, which are the topic of the excerpt Education provided to us, are not yet available. Under our auditing standards, we cannot opine on the quality of data we could not independently assess or on the efficacy of process improvements associated with those data. Education also raised concerns about how we weighted our interviews with school officials, and it questioned the relevance of our discussions about selected school districts’ use of restraint and seclusion data not reported for CRDC purposes. We disagree. Education was concerned about the weight GAO placed on information obtained from 50 officials in 11 school districts across 3 states whom we interviewed during the course of our audit work. As stated in the report, this information cannot be generalized to all districts. However, we believe it provides useful insights into how some districts use their restraint and seclusion data to reduce the incidence of these practices and improve the accuracy of their data. The widespread disagreement among the 50 school officials with whom we spoke also highlights confusion about how to accurately and consistently apply CRDC definitions of restraint and seclusion. This finding is supported by the views of seven nonfederal advocacy organizations that represent parents and families; individuals with disabilities; and other stakeholders, such as representatives of relevant school and special education professional associations. Education questioned the relevance of discussing the benefits that selected school districts said they derive from using restraint and seclusion data not reported for CRDC purposes. Education stated that “attempting to generalize comments about how these nine school districts use restraint and seclusion data” seems inconsistent with Governmental Accounting Standards Board (GASB) statistical principles. We believe that describing selected school districts’ use of their restraint and seclusion data is within the scope of our stated audit objectives. In addition, the explanatory statement from the House Committee on Appropriations accompanying the Consolidated Appropriations Act of 2018 includes a provision for us to provide examples of how schools are adopting effective alternatives to these practices and reducing the incidence of seclusion and restraint, among other things. Further, Education mistakenly asserts that none of the data and analyses that the school districts collected, performed, or used are part of the CRDC and none could be feasibly collected by the CRDC. We have further clarified in the final report that portions of the data these school districts collect are used for CRDC reporting purposes. For example, some of the data elements are the same ones that districts use to calculate aggregate incident counts, which are required by the CRDC. We do not recommend that Education collect such detailed data or perform such analyses. Regarding Education’s concern about “GASB statistical principles” and case selection, all GAO performance audits are subject to Generally Accepted Government Auditing Standards (GAGAS); in contrast, GASB’s Generally Accepted Accounting Principles apply to financial audits of public entities. The applicable methodological guidance we followed -- Selecting a Sample of Nongeneralizable Cases for Review in GAO Engagements -- is designed to ensure that GAO policies on evidence and GAGAS are met, and conforms to the generally accepted principles and practices of the appropriate disciplines. When providing illustrative examples, it is neither necessary nor appropriate to use statistical methods to analyze and interpret evidence. Finally, in its comments, Education stated that it is critical that we emphasize that the CRDC is an aggregate of self-collected and self- reported data from school districts and that the district superintendent or an authorized designee certifies that the data they submit are “true and correct.” We agree, and acknowledged this in several places in both the draft and final reports. At the same time, we believe that self-certified data does not absolve Education of its responsibility to ensure the quality of the data it collects and publicly reports – especially given the CRDC’s longstanding role in Education’s overall enforcement of various federal civil rights laws prohibiting discrimination on the basis of race, color, national origin, sex, and disability. Self-reported data by nature are subject to error, making the need for effective quality control measures before, during, and after collection a necessity. We are sending copies to the appropriate congressional committees, the Secretary of Education, and other interested parties. In addition, the report will be available at no charge on the GAO website at http://www.gao.gov. If you or your staff have any questions about this report, please contact me at (617) 788-0580 or nowickij@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are listed in appendix V. Appendix I: Objectives, Scope, and Methodology This report examines (1) the effectiveness of CRDC data quality control procedures for its restraint and seclusion data, (2) how selected districts interpret the CRDC definitions of restraint and seclusion and (3) how selected districts use data on restraint and seclusion and encourage staff to report incidents. Below are the details of our analysis to determine the extent to which Education ensures the quality of restraint and seclusion data reported by school districts, and of our interviews with officials in selected districts about how they apply Civil Rights Data Collection (CRDC) definitions of restraint and seclusion and use restraint and seclusion data. To inform all of our objectives, we interviewed federal agency officials, representatives from several nonfederal advocacy organizations that represent parents and families, individuals with disabilities, and other stakeholders, such as representatives of professional associations. We also reviewed agency documentation, relevant federal laws, regulations and policies, and selected state laws. Analysis of National Restraint and Seclusion Data To determine the extent to which Education ensures the quality of restraint and seclusion data reported by school districts, we analyzed Education’s Civil Rights Data Collection (CRDC) for school year 2015-16. Specifically, we analyzed the CRDC to determine the extent to which districts reported zero incidents of restraint and seclusion, to identify outliers (districts that reported a high or low incidence of restraint and seclusion), and to identify illogical data. CRDC is a biennial survey that is mandatory for nearly every public school and school district in the United States and is conducted by Education’s Office for Civil Rights. The CRDC collects data on the nation’s public schools (pre-K through 12th grade) that includes the use of restraint and seclusion, student demographics and enrollment numbers, educational and course offerings, and disciplinary actions. In school years 2013-14 and 2015-16, the CRDC collected data from nearly every public school in the nation (approximately 96,000 schools in 17,000 school districts in school year 2015-16). CRDC data are self-reported by districts and schools, and consequently there is potential for misreporting of information. After reviewing their CRDC data, school districts can submit revised data to Education. The public-use data file of the CRDC for school year 2015-16 was the primary source of data for our analyses and the most recent data available at the time. We also used restraint and seclusion data from school year 2013- 14 primarily to analyze how use of restraint and seclusion may have changed between the two time periods. The CRDC collected data on (1) mechanical restraint, (2) physical restraint, and (3) seclusion. Using these data, we performed the following analyses to determine potential inaccuracies or underreporting in the CRDC. Analysis of Extent of Districts Reporting Zeros To examine the extent to which school districts reported zeros, we calculated the percentage of districts and schools reporting zeros for restraint (both mechanical and physical) and for seclusion. We performed this calculation for both districts and schools nationally and by state, district size, and school type (e.g., charter, traditional, and special education schools). Although Education has a business rule that targets very large districts that report zero incidents of restraint or seclusion, we calculated the number of all districts and schools that reported zeros to understand the prevalence of zeros in the reported data. Analysis of Relatively Low Rates of Restraint and Seclusion Incidents To test for potential underreporting, we first limited our analysis to the restraint and seclusion data reported by the 30 largest school districts in the nation (districts with over 100,000 students enrolled). Because of these districts’ size, we reasoned that they would be more likely to have incidents of restraint and seclusion to report. Our analysis found that 20 of the 30 largest school districts reported incidents, and thus we focused our analysis of underreporting on the 20 largest districts that reported incidents. For each of the 20 districts, we calculated the percentage of schools that reported incidents. To compare the 20 largest districts that reported incidents with all 5,252 districts that reported incidents, we calculated the rates of restraint and seclusion per enrolled student and calculated percentile ranges. (See table 6.) We determined that nine of the 20 districts had incidents of physical restraint per enrolled student that were below the 5th percentile of all districts reporting incidents of physical restraint. Analysis of Relatively High Rates of Restraint and Seclusion Incidents To identify school districts with relatively high rates of restraint and seclusion, we examined districts that reported having more incidents than students enrolled. This analysis potentially indicates that some students may have been restrained or secluded multiple times. To illustrate, if a school district reported that it had 24 students enrolled, and also reported that it had 100 incidents of restraint, these reported data would indicate that the reporting was erroneous or that some students were restrained multiple times. Based on this analysis, we then calculated the average number of incidents (of restraint and seclusion) per student affected. Analysis of Extent of Illogical Data To test for illogical data, we analyzed the restricted-use restraint and seclusion data file for schools that reported more students affected than incidents. To illustrate, if a district reported that a school had restrained 80 students, and also reported that the school had 40 incidents of restraint, these reported data are illogical. Education has a business rule to detect illogical data at the school level, but the rule applies only to schools with more than 100 incidents. For our analysis, we looked for all schools with illogical data to determine the prevalence. School District Interviews on Interpreting CRDC Definitions of Restraint and Seclusion To determine how selected school districts interpret the CRDC definitions of restraint and seclusion, we selected 11 schools and nine school districts in three states to serve as illustrative examples. In total, we interviewed about 50 school officials. Information we collected from our 11 selected schools and nine districts cannot be generalized to all districts and schools nationwide. We selected states, districts, and schools to obtain a range of perspectives on federal reporting of restraint and seclusion data. Our selection also accounted for other criteria, such as selecting states that had laws requiring reporting; high or low rates of reporting zeros among districts; relatively high or low rates of restraint or seclusion per capita; grade levels served (e.g., K-6 or 9-12); school type (e.g., traditional or charter); and significant changes—increase or decrease—in incidents across reporting periods. We also selected districts that had reported incidents. As a result, we selected nine school districts to visit: two in Kentucky, three in Washington, three in Wisconsin, and a charter district in Wisconsin (see table 7). To determine how district and school officials, such as assistant superintendents, program managers, department directors, principals, and teachers, were interpreting the CRDC definitions of restraint and seclusion, we made the following statements and asked the following questions in our interviews. 1. We are going to talk to you about the definitions of restraint and seclusion that appear in the CRDC. We have heard that these definitions are not always clear to educators, so we want to get your feedback. Mechanical Restraint: the use of any device or equipment to restrict a student’s freedom of movement. Do you think this definition is clear or does it leave room for ambiguity? Physical Restraint: a personal restriction that immobilizes or reduces the ability of a student to move his or her torso, arms, legs, or head freely. The term does not include a physical escort. Physical escort means a temporary touching or holding of the hand, wrist, arm, shoulder or back for the purpose of inducing a student who is acting out to walk to a safe location Do you think this definition is clear or does it leave room for ambiguity? How do you differentiate between physical escort and physical restraint? Does breaking up a fight constitute a restraint? Seclusion: the involuntary confinement of a student alone in a room or area from which the student is physically prevented from leaving. It does not include a timeout, which is a behavior management technique that is part of an approved program, involves the monitored separation of the student in a non-locked setting, and is implemented for the purpose of calming. Do you think this definition is clear or does it leave room for ambiguity? How do you differentiate between timeout and seclusion? What does physically prevented (from leaving) mean? In what types of physical spaces can seclusion occur? Does your district have dedicated spaces for seclusion rooms? Can you describe where they are generally located, e.g., which types of schools or classrooms? 2. How do staff determine when an incident needs to be recorded as a restraint? 3. How do staff determine when an incident needs to be recorded as a seclusion? We conducted this performance audit from November 2018 to April 2020 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives. Appendix II: Comments from the Department of Education Appendix III: Restraint and Seclusion Data for 20 Largest School Districts Reporting Incidents Appendix IV: School Districts with Relatively High Rates of Reported Seclusion Appendix V: GAO Contact and Staff Acknowledgments GAO Contact Staff Acknowledgments In addition to the contact named above, Sherri Doughty (Assistant Director), Lara Laufer (Analyst-in-Charge), Morgan Jones, and Kristin Petroff, made key contributions to this report. Also contributing were James Bennett, Deborah Bland, Tonnye Conner-White, Holly Dye, Gretta Goodwin, Sheila R. McCoy, Jean McSween, John Mingus, James Rebbe, and Manuel Valverde.