Background Stages in the Life Cycles of NSF’s Major Facilities Projects Each major facilities project has a sponsoring office from within NSF’s seven research directorates. The sponsoring office assesses the scientific merit of a potential project, proposes projects for funding through NSF’s MREFC account, and is responsible for overseeing the project during the following five stages of its life cycle. Development. Initial project ideas emerge, and a broad consensus is built within the relevant scientific community for the potential long-term needs, priorities, and general requirements for research infrastructure that NSF may consider funding. Design. Entrance into this stage occurs when the NSF Director approves the proposed research infrastructure as a national priority and the sponsoring directorate makes an award (either through a cooperative agreement or contract) for developing detailed project cost, scope, and schedule for possible construction. This stage is divided into conceptual, preliminary, and final design phases. According to NSF documentation, the goal of the conceptual design phase is to create a comprehensive design that clearly articulates project elements that NSF will consider, such as a description of research infrastructure and technical requirements, a concept of operations, and an initial risk analysis, among others. The preliminary design phase further develops projects through the formulation of a site-specific scope, an accurate budget estimate, a revised and updated project execution plan, and other deliverables to establish a project baseline. In the final design phase, a candidate project will refine cost and contingency estimates, complete recruitment of key staff needed to undertake construction of the project, and develop the necessary documentation needed to undergo final design review. A candidate project will exit the design stage and enter the construction stage after a successful review by the NSF director and other key stakeholders of its project execution plan and authorization of its not- to-exceed total project cost by the National Science Board, as discussed below. Construction. The construction stage begins when NSF makes awards to external recipients for acquisition or construction of research infrastructure. Such awards generally take the form of cooperative agreements, although NSF occasionally uses contracts, according to agency officials. The policies and procedures in NSF’s Major Facilities Guide apply to research infrastructure projects regardless of the award instrument employed. According to NSF’s Major Facilities Guide, the transition from construction to operations could be a single acceptance event or multiple events depending on the nature of the project, and many projects require an integration and testing phase, followed by a commissioning phase to bring the facility up to the design level of operational readiness. The construction stage ends after final delivery and acceptance of the defined scope of work and facility performance per terms of the award instrument. Operations. The operations stage includes the day-to-day work necessary to operate and maintain the research infrastructure (including refurbishment or upgrade activities) and to perform research. Operations awards, which are separate from construction awards, may be made to the construction award recipients or to a different entity. Depending on the project, initial operations may begin before completion of construction. Integration and testing activities may continue during the operations stage, depending upon the complexity and time needed to reach design specifications. Divestment. Divestment can include the transfer of the research infrastructure to another entity’s operational and financial control or the decommissioning of the research infrastructure, including its complete deconstruction and removal. NSF generally decides to divest when the agency or the scientific community determines that the facility is no longer considered an operational priority with regard to advancing science, according to NSF’s Major Facilities Guide. NSF funding for the development, design, operations, and divestment stages generally comes from the sponsoring directorate. Funding for the construction stage generally comes from the MREFC account. However, if the sponsoring directorate funds construction, the policies and procedures in NSF’s Major Facilities Guide apply if total project costs meet the definition of a major multiuser research facility project under the American Innovation and Competitiveness Act—that is, if the costs exceed $100 million or 10 percent of the responsible directorate’s annual budget, whichever is less. NSF Oversight of Major Facilities Projects NSF has established an oversight structure for major facilities projects that includes offices from across the agency (see fig. 1). This includes the National Science Board, a policy and advisory body that is part of NSF and consists of the NSF Director and 24 members, drawn from industry and universities, who represent a variety of science and engineering disciplines. The NSF Office of the Director and the National Science Board provide high-level, ongoing oversight of major facilities projects, including the approval of new projects to be included in NSF’s annual budget request. Within NSF’s Office of Budget, Finance, and Award Management, the Large Facilities Office (1) develops business-related oversight policies for all life-cycle stages with a focus on the design and construction stages and (2) provides assistance on nonscientific and nontechnical aspects of project planning, budgeting, implementation, and management. To that end, the office maintains the Major Facilities Guide, which contains NSF policies for agency staff and recipients on the planning, management, and oversight of major facilities. Prior to requesting the National Science Board’s authorization to include a proposed project in a future NSF budget request, the Large Facilities Office provides independent assurance—apart from the sponsoring office and external panels—that NSF oversight processes have been followed, project plans are construction ready, and construction and operations budgets are justified. In addition, it prepares a bimonthly status report for NSF leadership on all ongoing major facilities in construction and candidate projects in design. NSF also uses external panels of experts to review projects at several points during their life cycles. An external panel may first review a project proposal during the development stage. Separate panels then review the project at the culmination of each of its design phases. In addition, an external panel periodically reviews each project during both construction and operations; according to NSF officials, those reviews are generally on an annual basis. According to NSF officials and policy documents, the agency selects panelists based on the questions that need to be addressed and on the type of review taking place. For example, for panels charged with reviewing all aspects of a project, NSF will generally select panelists to represent the academic and broader national or international research community, as well as experts in administrative aspects of facilities and project management, according to NSF’s Major Facilities Guide. Furthermore, the responsible directorate and the Large Facilities Office jointly manage the external panel review process and other NSF staff may attend as observers, according to the agency’s Major Facilities Guide. Each panel is to provide NSF with a report summarizing the review’s findings and any recommendations to NSF. Components of Construction Costs and Schedules of Major Facilities Projects Under NSF’s major facilities construction process, the recipients of design awards develop construction cost and schedule estimates for projects and submit them to NSF for review. In particular, after a project’s final design review, the National Science Board authorizes a not-to-exceed award amount and an award duration. According to NSF officials, this finalizes the initial budget request previously submitted to Congress after the project’s preliminary design review. The not-to-exceed award amount that the National Science Board authorizes is the amount against which NSF measures cost increases to implement its no cost overrun policy. NSF’s Major Facilities Guide defines two components that together make up the total project cost and schedule for the construction of major facilities projects. The total project cost awarded in a project’s construction agreement may be less than the not-to-exceed cost but not more. These components of the total project cost and schedule are the following: Performance measurement baseline. During design, the cost, scope, and schedule are refined and eventually become the project baseline. Once the baseline has been authorized and included in a construction award, it is known as the performance measurement baseline. NSF documents the performance measurement baseline in the terms and conditions of the award instrument and requires that any changes to it be made through a formal change control process. The performance measurement baseline does not include the project’s budget or schedule contingency. Contingency. This is an amount of budget or time for covering the cost increases or delays that would result if foreseen project risks were to occur. During development of a total project cost estimate, the timing and impacts of such risks are uncertain. As a project progresses, the impacts of risks that materialize may exceed the cost or schedule in the performance measurement baseline and lead to use of the project’s budget or schedule contingency. According to NSF’s Standard Operating Guidance on budget contingency, it is likely no contingency will be left over by the end of a project because all of it will have been used during normal execution of the project to manage known risks and uncertainties. NSF approval is needed when use of contingency exceeds certain project-specific thresholds, which are described in the project’s execution plan and codified in the award. In this report, we identify total project costs for the construction of major facility projects which were developed during the design phase based on the latest estimates available from NSF officials; those estimates are subject to change before construction awards are made. For projects under construction, we identify total project costs based on the amounts awarded in the cooperative support agreements for construction and the not-to-exceed amounts authorized by the National Science Board. Only at the end of the projects—when construction is complete and the awards have been closed out—will the final total project costs be known. In addition to the performance measurement baseline and budget contingency, a project’s not-to-exceed cost that the National Science Board authorized may include the following: Fee. NSF may provide recipients the opportunity to earn a fee (formerly referred to by NSF as a management fee) for major facilities projects. According to NSF’s Standard Operating Guidance on negotiation, award, and payment of a fee, such a fee can stimulate efficient performance. Management reserve. NSF, not the award recipient, holds management reserve to manage budget uncertainties, unforeseeable events, and risks that the recipient is not able to manage, according to NSF officials. According to agency officials and the Major Facilities Guide, NSF does not hold a management reserve except in rare circumstances. NSF’s No Cost Overrun Policy for Major Facilities Projects Since February 2008, NSF has had a policy to manage cost overruns on major facilities projects. Under this policy, the cost estimate developed at the preliminary design review should have adequate contingency to cover all foreseeable risks. Any cost increases not covered by contingency are generally to be accommodated by reductions in scope. Figure 2 provides a breakdown of the total project cost components in relation to the not-to-exceed award amount. NSF officials said that under this policy, they will only request an increase to the not-to-exceed cost that the National Science Board authorized if the recipient cannot address the increase through use of the project’s budget contingency or acceptable reductions to the project’s scope. Accordingly, at the preliminary design review, projects must have a prioritized, time-phased list of options for reducing scope during construction, known as scope contingency, and the potential cost savings associated with those options is to total at least 10 percent of the project’s baseline. As defined by NSF’s Major Facilities Guide, scope contingency is scope that can be removed without affecting the overall project’s objectives but that may still have undesirable effects on facility performance. NSF Experienced No Recent Cost or Schedule Increases on Ongoing Major Facilities Projects and Completed Construction of One Project As of September 2019, NSF continued construction of three major facilities projects with no changes to their authorized total project costs or scheduled completion dates since our March 2019 report. In addition, NSF approved a fourth project to enter the construction stage, completed construction of one project, and advanced two major facilities projects in the design stage. The four major facilities projects under construction have a combined total cost of approximately $1.6 billion (see table 1). Ongoing construction projects. Three projects—the Daniel K. Inouye Solar Telescope, the Rubin Observatory and the Regional Class Research Vessels—continued construction with no changes to their authorized total project costs or scheduled completion dates since our March 2019 report. Instead, NSF managed cost increases on the projects through the use of budget contingency, as specified under its no cost overrun policy, and managed delays through the use of schedule contingency. For example, the Rubin Observatory utilized $11.9 million in budget contingency and 5 months of schedule contingency to better align testing of the camera within the project schedule due to delays associated with the completion of the dome enclosure and telescope mount assembly, among other delays. The project team for the Rubin Observatory is also evaluating scope reduction options in order to complete the project within its total project cost and by its scheduled completion date of October 2022. New construction project. In February 2019, the National Science Board authorized a not-to-exceed total project cost of $410.4 million for the AIMS project and NSF awarded an initial contract modification for construction. We previously reported that in NSF’s fiscal year 2019 budget request, the estimated total project cost for construction of the AIMS project was $355.0 million. By the project’s final design review in October 2018, the AIMS team determined that it could not execute the project with the desired scope for this amount because of changing market conditions. NSF evaluated scope reduction options for the project but decided to maintain the project’s scope at the higher total project cost of $410.4 million. This change in total project cost did not count as an increase under NSF’s no cost overrun policy because the previous amount had not been authorized by the National Science Board as the project’s not-to-exceed cost. Completed construction project. In May 2019, NSF completed construction of the National Ecological Observatory Network project within the $35.5 million cost increase authorized by the National Science Board and a schedule increase of 2.8 years (57 percent). In 2011, NSF made the original award for construction of this nationwide network of ecological observation sites which was planned for completion in July 2016 at a total project cost of $433.8 million. In 2017, NSF increased the not-to-exceed cost for the project to $469.3 million. In accordance with NSF’s no cost overrun policy, the NEON project implemented scope reductions, such as reducing the number of observation sites from 106 to 81 and eliminating certain scientific instruments at the project’s observation sites. The scope reductions resulted in an estimated cost savings of $62.4 million. According to NSF documentation as of November 2019, NSF obligated a total of $458.9 million from the MREFC account for the construction of NEON, $10 million below the authorized total project cost. As of January 2020, NSF extended the construction stage award for NEON to allow for award close-out activities, which NSF officials expected to be complete in August 2020. Projects in design. In addition, in 2019, NSF advanced the design of two major facilities projects in the design stage, the Large Hadron Collider High Luminosity Upgrade (HL-LHC) and the Leadership- Class Computing Facility (LCCF). Under NSF policy, a major facility project’s cost, scope, and schedule are not finalized until after the final design review, when the National Science Board authorizes a not-to- exceed cost and an award duration. The not-to-exceed cost that the National Science Board authorized is the amount against which NSF measures cost increases to implement its no cost overrun policy. In September 2019, NSF convened two external panel reviews for the final design of the two separate detector upgrades that make up the HL-LHC program. According to NSF officials, the panels recommended to the NSF Director that the detector upgrades proceed to the construction stage. According to NSF documentation dated November 2019, the HL-LHC program had an estimated total project cost of $150 million for both upgrade projects. However, this amount was subject to change since the projects had not yet been authorized by the National Science Board to advance to the construction stage. According to NSF officials, the National Science Board authorized the total program cost at $153 million in early February 2020, setting the not-to-exceed costs for both awards. The LCCF project entered the conceptual design phase in March 2019. As of September 2019, the LCCF project had not developed an initial estimated total project cost because it had so recently entered design. Further details on the two projects in design are located in appendix II. NSF Has Implemented Two Prior Recommendations on Major Facilities and Has Taken Initial Steps to Address Other Recommendations NSF has fully implemented two of the six recommendations we made in June 2018 and March 2019—recommendations on policies for estimating the costs of major facilities projects and revising the Rubin Observatory’s schedule to better meet best practices. NSF has taken steps to address but has not fully implemented the remaining four recommendations concerning the agency’s management of major facilities, specifically our recommendations on policies for developing schedules for major facilities projects, project management competencies of the agency’s major facilities project management expertise of award recipients for major facilities ensuring the sharing of lessons learned or best practices on major facilities projects. NSF Revised Its Cost Estimating Policies and the Rubin Observatory’s Schedule to Better Meet Best Practices Cost estimating policies. In our June 2018 report, we found that procedures documented in NSF’s policies for major facilities projects fully or substantially met many best practices and partially or minimally met others identified in GAO’s guide for developing project cost estimates. Specifically, we found that NSF’s procedures fully or substantially met seven of the 12 best practices in GAO’s cost guide and partially or minimally met the remaining five, such as the best practice for conducting a sensitivity analysis to understand which variables most affect the cost estimate. The American Innovation and Competitiveness Act requires that NSF ensure that its policies for estimating and managing costs and schedules are consistent with the best practices in GAO’s cost guide, and NSF requires the same of its recipients. We recommended that NSF revise the agency’s policies for estimating the costs of major facilities projects, and for reviewing those costs, to better incorporate best practices. In response, NSF revised its Major Facilities Guide and certain internal Standard Operating Guidance policies that documented procedures for estimating costs. In our current assessment of these revised guidance and policy documents, we found that NSF fully met the five cost estimating best practices in GAO’s cost guide that we previously found were minimally or partially met. For example, in our 2018 report, we concluded that NSF’s procedures required a sensitivity analysis but did not describe how one is to be conducted. In our updated assessment, we found that NSF’s procedures describe the best practice and how it should be applied to NSF major facility cost estimates. Specifically, the procedures describe, among other things, (1) identifying key variables—cost drivers, ground rules, and assumptions—for inclusion in the analysis, with examples particular to NSF major projects included as part of the procedures; (2) evaluating the effect of these variables on the cost estimate by varying them one at a time; and (3) developing a strategy to deal with the variables to which the estimate is most sensitive. Table 2 provides an overview of our original and updated assessments of NSF’s cost estimating policies. Between our June 2018 assessment and our current assessment, NSF’s policies substantially or fully met all 12 of the best practices in GAO’s cost guide. Rubin Observatory schedule. In our March 2019 report, we found that the Rubin Observatory’s schedule could not be considered reliable because it did not substantially or fully meet all four characteristics of a reliable schedule from GAO’s schedule guide—comprehensive, controlled, well-constructed, and credible, as described in table 3. While the schedule substantially met the comprehensive and controlled characteristics, it partially met five scheduling best practices associated with the well-constructed and credible characteristics. Specifically, we found certain issues related to the construction of the project’s schedule, including (1) the sequencing of activities, (2) the schedule’s critical path— a chain of dependent activities that drive a project’s earliest completion date, and (3) the amount of float calculated in the schedule—the amount of time by which a project activity can slip before the delay affects the project’s estimated completion date. We recommended that NSF ensure that the project’s schedule meets the well-constructed and credible characteristics of a reliable schedule, as defined in GAO’s schedule guide. Our current assessment found that the revised schedule addressed our recommendation. Specifically, the schedule substantially met four of the five best practices that we previously found had been partially met within the well-constructed and credible characteristics of a reliable schedule and partially met the remaining best practice (ensuring reasonable total float). Between our two assessments, the Rubin Observatory project’s schedule substantially or fully met the four characteristics and nine of the 10 best practices in GAO’s schedule guide. We consider NSF’s actions sufficient to address our recommendation. Table 3 provides our original and current assessments of the Rubin Observatory project’s schedule. NSF Has Taken Initial Steps to Address Four Recommendations Supporting Its Oversight of Major Facilities In addition to implementing two of our recommendations, NSF has taken initial steps to address the other four recommendations from our June 2018 and March 2019 reports, but has not fully implemented them. Once NSF completes the steps discussed below, we will evaluate its actions to determine whether they are sufficient to fully address our recommendations. Policies for developing project schedules. In our June 2018 report, we found that NSF’s procedures for recipients substantially met one of the 10 best practices for developing project schedules—the best practice on conducting a schedule risk analysis. In contrast, NSF’s procedures partially or minimally met six and did not meet three of the remaining best practices. For example, we found that NSF’s procedures did not meet the best practice of establishing the durations of all activities because the NSF documents we reviewed did not include policy or guidance related to this practice, such as guidance on using realistic assumptions in estimating durations. The American Innovation and Competitiveness Act requires that NSF ensure that its policies for estimating and managing costs and schedules are consistent with the best practices in GAO’s schedule guide, and NSF requires the same of its recipients. We recommended that NSF revise its policies for developing schedules for major facilities projects, and for reviewing those schedules, to better incorporate the best practices in GAO’s schedule guide. As of November 2019, NSF had updated its internal guidance on standardized cost analysis to include a new section related to schedule reviews to help address this recommendation. This guidance states that the NSF Large Facilities Office will lead analysis of the schedule for each proposed major facilities project, which will include a technical evaluation by the sponsoring office, and may include input from an independent cost estimate and schedule review, or other reviews. As further steps to implement this recommendation, NSF plans to update two other policy and guidance documents, according to NSF officials. Specifically, NSF plans to: develop a new section of the Major Facilities Guide on schedule development, estimating, and analysis and post the guidance for public comment; and develop new internal guidance to help NSF staff more fully utilize external panels to address elements of schedule—in addition to cost—as part of the panels’ oversight reviews. According to NSF officials, they plan to complete these actions by the end of fiscal year 2020. Once NSF completes these actions, we will re-assess NSF’s procedures against the nine best practices that NSF partially or minimally met or did not meet in the assessment we conducted for our June 2018 report. Project management competencies of NSF’s major facilities oversight workforce. In our March 2019 report, we found that NSF had not (1) assessed potential gaps in how well its key major facilities oversight staff met project management competencies or (2) developed human capital plans for its major facilities oversight staff to address any gaps that may exist. Taking these steps would be consistent with leading principles for strategic workforce planning that we and the Office of Personnel Management have previously identified. Therefore we recommended that NSF assess its major facilities oversight workforce to identify any project management competency gaps, develop a plan to address any gaps and time frames for doing so, and monitor progress in closing them. In September 2019, in response to our recommendation, NSF awarded a contract for a proficiency assessment and workforce gap analysis. NSF expects this analysis to assess the core competencies and necessary proficiency levels of agency staff overseeing the major facilities portfolio and promote long-term workforce development. According to contract documentation, the contractor will take the following actions, among others: conduct a proficiency assessment and gap analysis based on a review of existing workforce materials, such as relevant position descriptions, vacancy announcements, performance plans, and other NSF guidance documents; work with NSF staff to refine competency guidance to better meet needs of the agency; and work with NSF to update training plans as necessary, based on the findings in the gap analysis and a review NSF’s existing training plan. According to contract documentation, NSF anticipates finishing the competency assessment and workforce gap analysis by the second quarter of calendar year 2020 and the implementation of contract tasks by March 2021. According to NSF officials, depending on the results of the assessment and analysis, improvements to address any identified gaps may involve developing standards of performance for the oversight workforce, identifying training opportunities in support or workforce development, and clarifying minimum competency requirements. Project management expertise of award recipients for major facilities projects. In our March 2019 report, we found that NSF had some procedures in place to help ensure that award recipients had project management expertise, but that the agency had not established criteria for the expertise needed by recipients or how they should demonstrate it. We concluded that, as a result, NSF was at risk of making awards to organizations that may not be well qualified to manage construction of major facilities projects. We recommended that NSF establish criteria for the project management expertise of award recipients for major facilities projects and incorporate the criteria in project requirements and external panel reviews. As of November 2019, NSF had drafted new language for the Major Facilities Guide and related supplemental award terms and conditions for major facilities that would require award recipients to document how project management competencies will be met. NSF officials told us they had shared the draft documents with targeted recipient representatives for review and comment in September 2019. NSF officials stated that the supplemental terms and conditions are planned to be published in fiscal year 2020, with an effective date of June 2020. The officials also said that, for existing awards, the agency will work with recipients on a phased implementation of the new guidance and terms and they will automatically be incorporated into future awards. Sharing of lessons learned or best practices on major facilities projects. In our March 2019 report, we found that NSF formalized a process for identifying and sharing lessons learned on major facilities projects. The process, which NSF refers to as its knowledge management program, responded to a 2015 recommendation by the National Academy of Public Administration and to the American Innovation and Competitiveness Act’s requirements that NSF coordinate the sharing of best management practices and lessons learned from major facilities projects. We recommended that NSF ensure, through a requirement or other means, that award recipients for major facilities projects provide information to NSF on any lessons learned or best practices. NSF developed supplemental award terms and conditions for major facilities to require recipients to participate in NSF’s knowledge management program. According to NSF officials, among other things, the requirement can be met by recipients: sending appropriate staff to the annual major facilities workshop that NSF hosts to provide a collaborative forum for continuous learning and information sharing among participants; presenting lessons learned or good practices at the annual workshop; participating in a workshop planning committee; or providing lessons learned or good practices to NSF. According to NSF officials, the draft terms and conditions will be included in the same revision as those related to recipients’ project management expertise, planned for publication in fiscal year 2020. As described above, NSF officials said that for existing awards, the agency will work with recipients on a phased implementation of the new terms and conditions, and they will automatically be incorporated into future awards. NSF Plans to Make Its First Awards for Mid-Scale Research Infrastructure Projects in 2020 and Is Developing Guidance to Manage Projects NSF Plans to Award Its First Set of Mid-Scale Projects in 2020 According to NSF documentation, NSF requested $45 million for fiscal year 2020 within the MREFC account to fund its first set of mid-scale projects with a total project cost between $20 million and $70 million. In response to a solicitation it issued in December 2018, NSF received approximately 50 preliminary proposals for mid-scale projects from research areas spanning all of NSF’s directorates, according to NSF officials. NSF invited 14 of these applicants to submit a full proposal and received full proposals from 11. The solicitation specified a list of information each full proposal should contain, including a project summary and description, a budget, and a project execution plan. NSF is currently reviewing the full proposals and expects to award its first portfolio of mid-scale projects in August 2020, according to NSF documentation. NSF’s solicitation anticipated that $150 million will be available over five years to fund its first batch of mid-scale projects. According to NSF officials, NSF plans to award subsequent sets of mid- scale projects biennially, depending on the availability of funds for future projects. According to NSF’s solicitation, the agency is seeking prospective mid- scale projects that are innovative and potentially transformative, that include a strong component of student training, and that provide unique research capabilities relative to what currently exists in the research community. Based on the definition of mid-scale projects in the American Innovation and Competitiveness Act, the solicitation stated that NSF would consider upgrades to existing major facilities projects currently in operation as candidates for mid-scale projects. The solicitation required full proposals to describe the full life cycle cost and schedule—including development, design, implementation, operations, and divestment. According to agency officials, NSF is only seeking to fund construction and acquisition costs from the MREFC account but needs to understand potential cost impacts on other life cycle stages. According to NSF officials, the mid-scale program is designed to identify potential projects with shorter implementation timelines and high levels of readiness as compared to the multiyear, incremental refinements to cost, scope, and schedule that occur with major facilities projects. NSF officials also stated that, to assess the readiness of the mid-scale projects for which full proposals were received, the agency will use an internal proposal review process similar to the final design review process used for major facilities projects. In addition, NSF policies state that there can be multiple inputs to the proposal review process, such as external panels or ad hoc reviews, which ensure that the mid-scale projects NSF awards will reflect the needs and interests of the scientific community. NSF Has Developed Flexible Guidance to Manage Mid-Scale Projects To provide guidance on oversight for mid-scale projects, NSF has included a chapter in its September 2019 update of the Major Facilities Guide to outline minimum recipient requirements and NSF oversight activities for mid-scale projects. In addition, NSF has created a management plan for NSF personnel that outlines procedures for reviewing proposals, selecting mid-scale projects, and managing the award process. NSF last updated the plan in November 2019, and according to NSF officials, the agency will continue to update the plan as it leads its initial set of projects from award to execution. According to NSF officials, oversight requirements for mid-scale projects will be dependent upon the technical scope and complexity of each individual project. As a result, NSF has tailored its guidance to provide the level of oversight commensurate with each project’s technical scope, type and mix of work, and risk profile. In addition, NSF is incorporating some aspects of its existing guidance for major facilities projects into its guidance for mid-scale projects. However, NSF officials anticipate that mid-scale projects will be less complex than major facilities projects. The following describes aspects where NSF has adapted its guidance for major facilities projects to the lower level of complexity anticipated for mid-scale projects. Performance measurement baselines. Similar to major facilities projects, NSF requires that the scope, cost, and schedule for mid-scale projects be defined at the time of award. In addition, NSF requires budget management, cost controls, and identification of potential risks and mitigation strategies for mid-scale projects, and its guidance states that budgets should be developed in accordance with GAO’s cost estimating best practices. While NSF officials state that NSF will apply substantial rigor in assessing the defined total project cost, mid-scale projects will not be subject to NSF’s no-cost-overrun policy. As a result, unlike for major facilities projects, NSF will not require all mid-scale projects to include budget contingency and scope reduction options, both of which are necessary for implementing the no-cost-overrun policy, although it may choose to include contingency in the budgets for certain mid-scale projects. For those mid-scale projects that have budget contingency, they must follow guidance for budget contingency laid out in the Major Facilities Guide, such as obtaining approval from NSF for using budget contingency. Monitoring and assessment. Like major facilities projects, NSF will monitor the award progress of mid-scale projects through periodic reports that provide quantifiable measurements on technical progress as well as cost and schedule performance. Depending on the complexity of each project, annual site visits or reviews may also be conducted. However, recipients of mid-scale projects may use alternatives to an earned value management system to report progress, such as reporting on milestone events or expenditure reports. According to NSF officials, the burden of establishing an earned value management system for some mid-scale projects may outweigh the benefits of using such a system, depending on the technical nature of the project. Project execution plan. According to the Major Facilities Guide, NSF will require a project execution plan for all mid-scale projects to demonstrate how recipients will manage the projects. A project execution plan serves as the stand-alone document that explains all of a project’s requirements for execution. According to NSF officials, the project execution plan used for major facilities projects would be excessive for mid-scale projects and may discourage potential proposals. Thus, NSF guidance for mid-scale projects requires only nine of the 16 sections normally required in a project execution plan and allows the recipients to tailor the detail and scope of each section to the specifics of each project. In addition, NSF will not require mid-scale projects to include design and development plans or site and environment information, which are required sections for major facilities projects. Since it is only funding the construction of mid- scale projects and seeking to award projects with high levels of readiness, NSF does not consider these sections to be beneficial in assessing how a recipient would manage a mid-scale project. Agency Comments We provided a draft of this report to NSF for review and comment. In its comments, reproduced in appendix III, NSF stated that our report provides the agency with an independent assessment of its oversight of projects in design and construction and its stewardship of the MREFC account. With regard to our recommendations on policies for estimating the costs of and developing schedules for major facilities projects, NSF stated it is proud of the progress it has made in meeting GAO best practices for cost estimating on major facilities projects and that it recognizes the remaining work needed to codify NSF guidance on project schedules. NSF also provided technical comments, which we incorporated as appropriate. We are sending copies of this report to the appropriate congressional committees, the Director of the National Science Foundation, and other interested parties. In addition, the report is available at no charge on the GAO website at https://www.gao.gov. If you or your staff have any questions about this report, please contact me at (202) 512-6888 or neumannj@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are listed in appendix IV. Appendix I: Summaries of the National Science Foundation’s Major Facilities Projects under Construction This appendix provides individual summaries of the National Science Foundation’s (NSF) four major facilities projects under construction: (1) the Daniel K. Inouye Solar Telescope, (2) the Vera C. Rubin Observatory, (3) the Regional Class Research Vessels, and (4) the Antarctic Infrastructure Modernization for Science. Each project’s summary is based on project documents and other information that NSF officials provided and includes the following: An overview of the project and its purpose. A timeline identifying key project dates, including the date of the original construction award, which we report as the start of construction. Project information, such as the project’s estimated completion date for construction (including schedule contingency), the type and latest amounts of the awards for construction, the responsible NSF directorate, project partners, and expected duration of operations. Tables summarizing the project’s current status and its cost, any cost or schedule increases or scope reductions made under NSF’s no cost overrun policy, and changes since our March 2019 report. A summary of the project’s cost and schedule performance history. A chart depicting the latest construction award’s total project cost for construction, including the performance measurement baseline and budget contingency. If applicable, a chart showing the increase in the construction award’s total project cost since the original construction award. Information on remaining project risks and potential for cost or schedule increases, including the amount of remaining contingency and scope reduction options. When completed, the National Science Foundation’s (NSF) Daniel K. Inouye Solar Telescope (DKIST), formerly named the Advanced Technology Solar Telescope, will be the world’s flagship facility for the study of magnetic phenomena in the solar atmosphere. It will help answer fundamental questions in solar physics and enable understanding of solar variability and activity, which can affect Earth through phenomena generally described as space weather. Project Information Location: Maui, Hawaii. Construction of NSF’s DKIST project was 94 percent complete as of September 2019. The project was in its 10th year of construction and in the integration, testing, and commissioning phase. Since our March 2019 report, the project completed installation of all telescope optics. Testing of the optics, originally planned for October 2019, was delayed until January 2020 to allow the project to replace a key piece of equipment that is essential to safely perform the testing. Despite the delay, the estimated completion of construction and beginning of full operations remained unchanged at June 2020, including 1.5 months of schedule contingency. Estimated construction completion date, including schedule contingency: June 2020. Construction award: Cooperative support agreements with the Association of Universities for Research in Astronomy, Inc., consisting of 42 U.S. institutional members and five international affiliates. Responsible NSF directorate: Mathematical and Physical Sciences. Project partners: More than 20 U.S. and international organizations. Kiepenheuer-Institut für Sonnenphysik (Germany) and Queens University Belfast (Northern Ireland) are supplying additional equipment for the project. Expected duration of operations: 50 years. Legend: ▲ = cost or schedule increase; ▼= scope reduction. Latest Construction Award NSF’s DKIST project had no changes to its authorized total project cost, June 2020 completion date or project scope since our March 2019 report, which used data as of September 2018. From April to November 2019, NSF approved the project’s use of about $6.2 million in budget contingency, with the largest usage of about $4.6 million in August 2019. Project delays requiring use of 3 months of schedule contingency— primarily because the project faced challenges with the installation and testing of the mirror systems, as described above—accounted for $4.3 million of the $4.6 million. We previously reported that the DKIST project’s risk of delays had the potential to increase costs for such items as labor, utilities, real estate, and equipment. NSF officials stated that most of the activities at risk of further delays would be achieved during the testing planned for January 2020. In 2013, NSF increased DKIST’s total project cost and the not-to-exceed cost that the National Science Board authorized from $297.9 million to $344.1 million, an increase of $46.2 million (16 percent) since 2010. NSF also delayed the project’s estimated completion date by about 2.5 years (31 percent), from December 2017 to June 2020. Prior to the National Science Board’s authorization to increase the total project cost, the recipient also reduced DKIST’s scope, resulting in estimated cost savings of $5.9 million but generally low expected impacts for the project. According to NSF officials, these cost and schedule increases resulted primarily from unforeseeable legal and administrative challenges to the construction site’s environmental permits. Remaining Project Risks and Potential for Cost or Schedule Increases As of September 2019, the DKIST project had $7.8 million of budget contingency remaining—$0.4 million more than the estimated remaining risk exposure of about $7.4 million when weighted for the risks’ probability. The project also had 1.5 months of schedule contingency remaining to help avoid any potential delays in completing construction. According to the project documentation, the largest remaining risk category is project completion and closeout risks. As of October 2019, 10 risks in this category remained, some of which had been partially realized, according to NSF officials, with about $4.0 million in risk exposure when weighted for probability. The remaining risks included staff retention as the construction project nears completion, and damage to or wear of equipment during integration and commissioning. For example, contingency may be needed to make minor repairs to the dome enclosure in preparation for full operations. In accordance with NSF policy, the project maintains a list of scope reduction options, which as of October 2019 included approximately $56,700 in total possible project de-scopes, such as reductions in travel. However, the ability of these remaining de-scope options to reduce costs will continue to decrease as the project continues to spend down remaining funds as it approaches completion. The National Science Foundation’s (NSF) Vera C. Rubin Observatory (Rubin Observatory), formerly named the Large Synoptic Survey Telescope (LSST), is an 8.4- meter, wide-field optical telescope. It will initially be used to image the entire visible southern sky—every 3 days for a decade—using the world’s largest digital camera (3.2 billion pixels). Built on a mountaintop in Chile to take advantage of the location’s pristine skies, the observatory will collect data and images that will allow for charting billions of galaxies as well as increased knowledge about potentially hazardous asteroids, dark matter, and dark energy. The observatory has the potential to advance every field of astronomical study, from the inner solar system to the large-scale structure of the universe. Project Information Location: Cerro Pachón, Chile. As of September 2019, the Rubin Observatory was 75 percent complete and in its sixth year of construction. NSF made the initial operations award in October 2018, and NSF officials anticipate completion of construction and start of full operations in October 2022, including contingency. Since our March 2019 report, the project has experienced delays related to both the telescope’s dome enclosure and mount assembly, leading NSF to add the project to the Director’s Watch List. Estimated construction completion date, including schedule contingency: October 2022. Construction award: Cooperative support agreement with the Association of Universities for Research in Astronomy, Inc., consisting of 42 U.S. institutional members and five international affiliates. Responsible NSF directorate: Mathematical and Physical Sciences. Project partners: The LSST Corporation, Department of Energy. Expected duration of operations: 50 years. Legend: ▼= scope reduction. Latest Construction Award Since our March 2019 report, NSF’s Rubin Observatory project had no changes to its authorized total project cost and implemented one scope reduction option valued at $1.4 million to increase available budget contingency. In addition, the project utilized $11.9 million in budget contingency and 5 months of schedule contingency to better align the testing of the camera within the project schedule. According to project documentation, the use of schedule contingency was due to delays with completion of the telescope mount assembly and dome enclosure that will house the telescope and other buildings. NSF officials attributed the delays to contractor performance and adverse weather conditions. For example, due to high winds, the project was able to use a crane to complete dome construction for only two days in September 2019. Remaining Project Risks and Potential for Cost or Schedule Increases Project data on the remaining risks and contingencies and the findings of two recent reviews indicate that the final cost of the Rubin Observatory may exceed the not-to-exceed cost authorized by the National Science Board, unless the project implements scope reduction options under NSF’s no-cost-overrun policy. As of September 2019, the project had an estimated remaining risk exposure of $26.4 million, which is equal to the remaining budget contingency of $26.4 million. In addition, the project had 3.5 months of schedule contingency remaining as of September 2019 to help avoid any potential delays in completing construction by October 2022. According to project documentation, the project’s largest remaining risks included delays in the completion of the telescope’s dome enclosure, the installation of the mount assembly, and delivery of the camera from the Department of Energy (DOE). The project team is modifying activity plans to mitigate these delays. For example, the project plans to complete dome enclosure and telescope mount assembly activities in parallel. As part of the Director’s Watch List, NSF plans to closely track updates on the project, including potential execution of scope reduction options. In August 2019, NSF and DOE jointly convened an external committee of experts to review the project’s construction progress. The committee found that the project may face difficulty in completing the baseline scope within the authorized total project cost. Specifically, the committee expressed concerns with the rate at which schedule contingency has been used (5 months of schedule contingency within the past 18 months), delays in completing the dome due to contractor performance issues, and the risks associated with maintaining an aggressive schedule composed of parallel activities in order to minimize further delays. The review committee recommended that NSF direct the project team to develop a proposal for executing scope reductions in fiscal year 2020 to complete the telescope within an acceptable level of risk at the current total project cost, among other recommendations. Remaining Contingency and Scope Reduction Options As of September 2019 with construction 75 percent complete. Budget contingency: $26.4 million (Equal to the probability- weighted risk exposure of $26.4 million). Schedule contingency: 3.5 months (included in the October 2022 estimated completion date). Estimated value of remaining scope reduction options: $24.8 million. as compared to a separate risk exposure analysis from July 2019 that indicated a 50 percent confidence. The panel recommended that the project report risk based on the analysis with the lower confidence level and conduct more frequent risk exposure analyses based on changes that have occurred, such as the realization or retirement of identified risks, to better inform management decisions. According to NSF documentation, the project team has recently acquired enhanced risk management software for analyzing risk exposure, including the effects of mitigating actions within the schedule. In a July 2019 update to its scope management plan, the project team identified 39 scope reduction items with a total value of $25.0 million. Among them is a de-scope option for reducing the amount of final commissioning surveys that may potentially return $4.3 million of budget contingency and 3.5 months of schedule contingency. According to NSF officials, NSF has yet to evaluate the impact of reducing the surveys to the project’s capabilities or operational costs. According to the external panel review convened by NSF and DOE, the project team identified potential scope reductions options valued at $14 million that the project can exercise in fiscal year 2022. However, the panel questioned the feasibility of executing the project’s scope reduction options and recommended that the project prioritize viable options while pursuing a no-cost extension to complete the project without an increase to the total project cost. The U.S. Department of Energy (DOE), a cosponsor of the Rubin Observatory, is responsible for delivering the observatory’s camera at a cost of $168 million. SLAC National Accelerator Laboratory manages a collaboration of DOE national laboratories and universities to develop, fabricate, and deliver the camera. As of September 2019, the project had the camera integration on the telescope scheduled for September 2021. Budget contingency accounts for the risk of a delayed delivery that would impact integration. The LSST Corporation is a not-for-profit organization representing nearly 40 institutional members and 34 international contributors. It acts as the agent for nonfederal funding contributed to the project and has raised more than $50 million for certain long-lead construction items and additional development efforts. The National Science Foundation’s (NSF) Regional Class Research Vessels (RCRV) project will construct three 199-foot vessels to support the nation’s ability to conduct fundamental scientific research in the coastal zone and continental shelf, including from the ocean’s surface through the water column to the sea floor and subsea floor environment. These vessels will provide enhanced capabilities beyond those of the retiring vessels they will replace. The three vessels’ research locations will depend on locations of the greatest science demand, but NSF planned to operate the first vessel along the west coast, the second along the east coast, and the third along the gulf coast of the United States. Project Information Location: Construction site is in Louisiana. As of September 2019, NSF’s RCRV project was 20 percent complete and was in its third year of construction. Since our March 2019 report, the project progressed with construction of the first vessel and began construction of the second vessel in September 2019. NSF also awarded funds for construction of the third vessel, which was scheduled to begin in March 2020, and awarded a cooperative agreement for its future operations to the Gulf-Caribbean Oceanographic Consortium. In February 2019, the RCRV project experienced a partial suspension of work due to the status of necessary production design and modeling deliverables, among other concerns. This resulted in 16 weeks of schedule contingency usage. However, there was no overall increase to the scheduled construction completion date of July 2024. Estimated construction completion date, including schedule contingency: July 2024 for three vessels. Construction award: Cooperative support agreement with Oregon State University, which contracted with Gulf Island Shipyards, LLC. Responsible NSF directorate: Geosciences. Project partners: The U.S. Navy performed initial design for the vessels. Expected duration of operations: 30 years. Construction Status of the Regional Class Research Vessels, as of September 2019 Percentage complete (based on construction of three vessels) aScope changes included are reductions in response to NSF’s policy on cost overruns or as part of a cost increase. Latest Construction Award As of September 2019, the RCRV project had no changes to its authorized total project cost, no changes to its estimated completion date of July 2024 for all three vessels, and no scope reductions. The National Science Board had authorized a not-to-exceed cost of $365.0 million for construction of three vessels. However, the shipyard bid was ultimately lower than expected, reducing the total project cost of building three vessels to $354.0 million. NSF accepted the project’s earned value management system in May 2019, following a surveillance review of the system. The review team found that the project’s system met the intent of NSF requirements and that its data were reliable. (In our March 2019 report, we reported that NSF conditionally accepted the project’s earned value management system in November 2018.) As of September 2019 with construction of three vessels 20 percent complete. Budget contingency: $44.0 million (exceeded the probability- weighted risk exposure of $24.6 million). Schedule contingency: 6 months (included in the July 2024 estimated completion date for three vessels). Estimated value of remaining scope reduction options: $9.8 million. Beginning in February 2019, the RCRV project utilized 16 weeks of schedule contingency and $2.4 million of budget contingency due to a partial suspension of work issued by the construction award recipient, Oregon State University (OSU). OSU was concerned with Gulf Island Shipyards’s (GIS) project management capacity and its ability to manage subcontractors, such as engineering vendors responsible for providing design specifications. During the work suspension, GIS developed a corrective action plan that identified eight areas of improvement, such as a subcontract management plan and updated schedules that better align the development of necessary design specifications with construction activities. OSU’s management team assessed and monitored GIS’s progress on these areas and subsequently lifted the work suspension in May 2019. However, the project continues to face subcontractor management issues. OSU has requested NSF approval for an estimated $6.1 million of budget contingency and 4 months of schedule contingency to compensate for the delays associated with these issues. According to project documentation, this issue may cause the construction completion date of each vessel to slip. Remaining Project Risks and Potential for Cost or Schedule Increases According to project documentation, the project had an estimated risk exposure of $24.6 million and $44.0 million in remaining contingency as of September 2019. With the utilization of 4 months of schedule contingency in 2019, the RCRV project had 6 months of contingency remaining until construction is scheduled to end in 2024. According to project documentation, 12 options for reducing scope were available as of December 2019, with potential savings estimated at $9.8 million. and schedule expertise, which resulted in a decrease in the impact of the risk. In addition, the RCRV project is closely monitoring two risks related to newer technologies and requirements for regional operability of each vessel. First, the project team identified newer technologies for systems such as communications compared to those specified during the design phase. According to project documentation, the project may utilize contingency to integrate such technologies into the vessels. Second, the project may incur additional engineering, labor, and material costs associated with certain potential design changes that NSF and the operating institutions for the three vessels have identified. These design changes are intended to improve quality and performance within the different regions where the three vessels will be operating. The National Science Foundation’s (NSF) Antarctic Infrastructure Modernization for Science (AIMS) project will modernize the core infrastructure of McMurdo Station in Antarctica, the largest of three stations operated by NSF’s United States Antarctic Program and used by multiple agencies. McMurdo Station serves as a logistics hub for remote field sites and for the Amundsen-Scott South Pole Station. The AIMS project is expected to make environmental and safety upgrades to McMurdo Station and redevelop it into a more compact, energy and operationally efficient core facility to support research. The planned core facility will consolidate critical buildings, such as medical facilities and field science support. Project Information Estimated construction completion date, including schedule contingency: 2028. Construction award: February and April 2019 modifications to the existing Antarctic support contract with Leidos Innovations Corporation. Responsible NSF directorate: Geosciences. Project partners: Other federal agencies—such as the National Aeronautics and Space Administration, National Oceanic and Atmospheric Administration, and the Department of Energy—and international programs, such as the Scientific Committee for Antarctic Research. Expected duration of operations: 35 to 50 years. Construction of NSF’s AIMS project was about 6 percent complete as of September 2019. The project was in its first year of construction. In February 2019, the National Science Board approved the project’s not-to- exceed cost of $410.4 million, and NSF awarded an initial contract modification for construction equipment and materials to be delivered to California by December 2019, in time for deployment to McMurdo station through two supply vessels. In April 2019, NSF awarded the second contract modification for construction of the first major components of AIMS: the Vehicle Equipment and Operation Center (VEOC) and a new lodging facility structure and exterior shell. According to NSF, the VEOC will facilitate maintenance and repair of both heavy and light equipment ranging from tractors and cranes to trucks, vans, snowmobiles, and field generators. The lodging facility will include space for 285 beds, which the project’s final design review panel expected to be adequate to support short- and long-term plans for McMurdo station, including construction needs. As of September 2019, the start of initial operations for the VEOC and lodging facility were planned for 2022 and 2023, respectively, and completion of both facilities was planned for 2022, according to NSF officials. Later phases of the AIMS project will include construction of central services, emergency operations, field science support, and industrial trades facilities. In November 2018, the U.S. Army Corps of Engineers completed an independent cost estimate (ICE) report for the AIMS project. According to NSF officials, the ICE was critical for negotiations with the contractor as NSF utilized data within the ICE, such as labor rates and cost of materials, to verify costs. Specifically, the ICE assisted NSF in determining the reasonableness of the contractor’s proposed cost estimate and schedule for the project and associated risks. According to NSF officials, NSF and the contractor resolved all recommendations from the ICE report to NSF’s satisfaction prior to setting the not-to-exceed cost. Cost and Schedule Performance History As of September 2019, NSF’s AIMS project had no changes to its authorized total project cost, changes to its estimated completion date, or scope reductions since the National Science Board authorized the project’s not-to-exceed cost of $410.4 million, which included $67.2 million in budget contingency, in February 2019. Remaining Contingency and Scope Reduction Options As of September 2019 with construction about 6 percent complete. $59.2 million ($7.1 million more than the probability-weighted risk exposure of $52.1 million). 18.4 months (included in the 2028 estimated completion date). By the project’s final design review in October 2018, the AIMS team determined that it could not execute the project with the desired scope for the $355.0 million estimate—as was previously presented in NSF’s fiscal year 2019 budget request—because of changing market conditions. In response, NSF convened a review panel, which evaluated scope reduction options such as relocating and reducing bed space in the lodging facility from 285 to 100 beds, which would also entail keeping the current lodging facility in operation instead of demolishing it to make room for a new facility. While it accepted some of these options, such as a reduction of warehouse space within the VEOC, the panel noted that relocation of the lodging facility and a reduction of bed space would have adverse effects on the project. For example, the panel found that constructing a new 100-bed lodging facility in an alternate location would not support the eventual construction of sky bridges. According to the project’s Final Design Review report, these sky bridges would improve efficiency by avoiding the need for personnel to put on Antarctic gear before moving between buildings, reduce energy use by reducing the need to open exterior doors, and significantly improve the quality of life for personnel. NSF therefore decided to maintain the 285-bed plan and finalized the total project cost at $410.4 million. and four of 11 procurements for the lodging facility to the 2021 vessel, but NSF officials do not expect significant construction delays as a result. The officials explained that the VEOC procurements are not required for 2020 construction and that the deferral of lodging procurements is expected to be accommodated by re-sequencing activities on site. Remaining Project Risks and Potential for Cost or Schedule Increases As of September 2019, the AIMS project had a risk exposure of $52.1 million and $59.2 million in remaining contingency, and all of the project’s 18.4 months of schedule contingency remained available. The project had cumulatively used $7.9 million in budget contingency. Of this, $7.8 million was used during initial award for contract modifications for initial construction, with the remainder used for additional equipment purchases and leases in August and September 2019. As of September 2019, the AIMS project had $14.2 million in high- likelihood risks. The largest remaining risk, with an estimated value of $12.5 million and a 23-day delay, was that subcontractor proposals would exceed planned construction costs. Another such risk was an increase in the estimated base price of key construction materials—such as steel, copper wire, concrete, gypsum, and specialty items—before the materials were procured. NSF’s contractor for the project, Leidos Innovations Corporation, was working with one of its subcontractors to ensure material costs were accurate and consistent with market pricing. In accordance with NSF policy, the project maintains a list of scope reduction options, which as of April 2019 included approximately $34.0 million to $43.1 million in total possible project de-scopes. For example, the largest scope reduction option, with an estimated value of up to $19.1 million, is to remove the new trades shop from the AIMS scope and instead use the current facility. Another option, with an estimated value of up to $4.0 million, is to remove the gymnasium from the emergency operations facility and instead continue to use and maintain the existing gymnasium. Appendix II: Summaries of the National Science Foundation’s Plans for Future Major Facilities Projects in Design This appendix provides individual summaries of the two National Science Foundation (NSF) projects that were in design and planned for construction as major facilities projects: (1) the Large Hadron Collider High Luminosity Upgrade and (2) Leadership Class Computing Facility. As of September 2019, no construction funds had been awarded for these projects and all cost, schedule, scope, and design information for these projects was subject to change. Each project’s summary is based on project documents and other information that NSF officials provided and includes the following: An overview of the project and its purpose. A timeline identifying key project dates. Project information, such as the expected date for completion of construction; the anticipated type of awards for construction; the responsible NSF directorate; project partners; and expected duration of operations. A summary of the project’s current status. A summary of the project’s design and construction costs, if available, and the budget account NSF planned to use for construction of the project. Information on potential project risks. The Large Hadron Collider (LHC) is the world’s most powerful particle accelerator. The facility’s four detectors observe new particles that are produced when high-energy protons are accelerated and collided, providing insight into fundamental forces of nature and the condition of the early universe. Through the National Science Foundation’s (NSF) Large Hadron Collider High Luminosity Upgrade (HL-LHC) program, the agency will fund a portion of a larger international effort to upgrade the facility’s accelerator and detectors. Specifically, NSF plans to fund the design and implementation of certain parts of the upgrades as two separate projects for the facility’s detectors, the A Toroidal LHC Apparatus (ATLAS) and Compact Muon Solenoid (CMS) detectors. The Department of Energy (DOE) is also contributing to upgrades to the LHC’s accelerator and to the ATLAS and CMS detectors. Project Information Location: Geneva, Switzerland. As of September 2019, NSF’s HL-LHC program was approaching its fifth year of design. The program has conducted several required activities to complete the design stage. In September 2019, NSF convened an external panel for the final design review of the program. The panel found that both detector upgrades met the readiness criteria within NSF’s Major Facilities Guide to proceed to construction. NSF also convened the internal Facilities Readiness Panel in November 2019 and conducted life cycle cost reviews for each detector upgrade in October 2019, according to NSF officials. Estimated construction completion date, not including schedule contingency: 2026. Construction awards: If approved, planned for 2020 as cooperative agreements with Columbia University (ATLAS detector) and Cornell University (CMS detector). Responsible NSF directorate: Mathematical and Physical Sciences. Project partners: European Organization for Nuclear Research and the Department of Energy. Expected duration of operations: 12 years. According to NSF officials, NSF planned to request National Science Board authorization in February 2020 to make construction awards. As a prerequisite for making the awards in April 2020, NSF received the independent cost estimates for both projects from the Army Corps of Engineers in January 2020. According to NSF documentation, these results align with the current total project cost reviewed during the final design review. According to the Major Facilities Guide, NSF uses independent cost estimates to validate recipient estimates, negotiate awards, check for compliance with GAO best practices and Uniform Guidance cost principles, and inform NSF’s cost analysis. According to NSF officials, the estimated completion for both upgrade projects is 2026. According to program documentation, NSF had obligated a total of $24.3 million for the design of its detector upgrades as of September 2019. Funding for the design has come from NSF’s Research and Related Activities account, rather than the Major Research Equipment and Facilities Construction account. Planned Contingency and Scope Reduction Options As of November 2019, with finalization of the NSF cost analysis still pending. Budget contingency: $38.9 million as follows $20.0 million for the ATLAS detector. $18.9 million for the CMS detector. Schedule contingency: To be determined. Estimated value of scope reduction options: $15.1 million as follows $8.4 million for the ATLAS detector. $6.7 million for the CMS detector. until authorization by the National Science Board. These figures remained subject to change before completion of the final design phase. According to NSF documentation, the total project cost may increase slightly based on a detailed evaluation of both projects’ contingency budgets following the final design review. NSF plans to fund the upgrades with separate cooperative agreements for each detector and to monitor each agreement in accordance with its distinct terms and conditions, total project cost, and earned value management metrics, according to agency officials. In August 2019, NSF initiated independent cost estimates of both projects (ATLAS and CMS) under the HL-LHC program, as required by the American Innovation and Competitive Act for projects in the design phase. The U.S. Army Corps of Engineers is conducting the estimates under an interagency agreement with NSF, with contractor support. In addition, NSF is conducting a cost analysis that will be informed by the final design review panels, internal assessments by the NSF’s Large Facilities Office and other business units, and the independent cost estimates. DOE’s Contributions to Upgrading the Large Hadron Collider DOE’s High Energy Physics program helped fund the construction of the Large Hadron Collider and continues to support researchers using the facility as well as upgrades to it. According to DOE’s fiscal year 2020 budget request, the department planned to support the upgrades to the ATLAS and CMS detectors at an estimated cost range of $149 million to $181 million for the ATLAS detector and $125 million to $155 million for the CMS detector. The scope of DOE’s work on the detectors was to focus on areas where the expertise and infrastructure of the department’s national labs were needed, whereas the scope of NSF’s work was to focus on areas led by university researchers. In addition, DOE approved upgrades to the accelerator itself with a total project cost of $242.7 million, according to DOE’s fiscal year 2020 budget request. NSF plans to fund the construction of the detector upgrades through its Major Research Equipment and Facilities Construction account. While the upgrades would involve separate cooperative agreements for each detector, NSF considers them one program consisting of two distinct projects, according to agency officials. Project Risks and Potential Scope Reduction Options Under NSF policy, a project’s cost should include enough budget contingency to cover all foreseeable risks. Following the preliminary design review, the amount of budget contingency included in the construction cost for the upgrades was approximately $38.9 million, or 26 percent of the planned total project cost. At the time of this report, the NSF cost analysis following the final design review was still pending and therefore the estimated amount of contingency is subject to change. NSF policy also directs a project’s design to include prioritized, time- phased options for reducing its scope during construction if needed. As of the final design review, the project teams had identified a total of $15.1 million of potential scope reduction options for the projects, which are subject to change throughout the design and construction of a project. According to the projects’ scope management plans we reviewed, the ATLAS detector has nine options to reduce scope totaling $8.4 million, with the options ranging in value from $0.6 million to $1.7 million. The CMS detector has 17 scope reduction options with a total value of $6.7 million. According to the project’s scope management plan, both NSF officials and external panels reviewed and provided input to determine the current scope reduction options. The National Science Foundation’s (NSF) Leadership-Class Computing Facility (LCCF) project is intended to provide advanced computational capabilities to enable transformative research in all areas of science and engineering that would not be possible by theory or experiment alone. According to NSF officials, future research using LCCF might include extremely detailed simulations ranging from biological molecules to supernovae and analyses of very large data streams such as satellite images to create high-resolution Earth maps. Project Information Location: Texas Advanced Computing Center, University of Texas at Austin Project Status As of September 2019, the LCCF project was in its first year of design; consequently, all cost, schedule, scope, and design information for the project was subject to change. In March 2019, the NSF Director approved the project to enter the design stage as a candidate major facilities project. The project represents the final phase of a two-phase deployment of high-performance computing systems. The first phase—known as the Frontera project at the Texas Advanced Computing Center at the University of Texas at Austin—was completed in September 2019. According to NSF, at that time, Frontera was the largest high- performance computing system deployed on a U.S. academic campus. The LCCF project will support the design and construction of an upgrade to the Frontera system as well as to the physical facility that will host it. In project documentation, NSF has described the upgrade as providing a substantial improvement in application performance but has not specified the extent of improvement. Estimated construction completion date, not including schedule contingency: Fiscal Year 2025. Construction award: Planned for 2024. Responsible NSF directorate: Directorate for Computer & Information Science & Engineering. Project partners: None. Expected duration of operations: 10 years. In July 2019, NSF awarded both an overarching cooperative agreement for the LCCF project and a cooperative support agreement for the conceptual design phase to the University of Texas at Austin. As of November 2019, the project was focused on leading and participating in activities with experts within the community for high-performance computing. The purpose of these activities was to document the science, technology, and facilities requirements for LCCF, as well as to shape the design and cost of long-lead items, such as the power and cooling infrastructure to service the facility. NSF plans to conduct the conceptual design review in June 2020. NSF’s Support for High- Performance Computing Systems NSF has supported high-performance computing capabilities for nearly 4 decades. In 2007, NSF awarded $226.6 million for the Blue Waters high- performance computing system through a cooperative agreement with the University of Illinois at Urbana- Champaign. According to NSF, at the time of its deployment in 2013, Blue Waters was one of the most powerful supercomputers in the world and was one of the fastest on a university campus. Scientists and engineers across the country used the computing and data power of Blue Waters to tackle a wide range of problems, including predicting the behavior of complex biological systems and simulating the evolution of the cosmos. Because of the rapid evolution of computer technology, by 2019, NSF no longer considered Blue Waters to be the leadership computing system for fundamental science and engineering research. Anticipating these technological advances, in September 2018, NSF awarded about $63.0 million to the University of Texas at Austin for the follow-on project to Blue Waters. Frontera was intended to provide three to five times the computing capability and twice the storage capacity to support the increased computational requirements for science and engineering research. NSF also anticipated that Frontera would help inform science requirements and reduce risks for LCCF, which is planned to provide substantially more computational capabilities than both Blue Waters and Frontera. obligated $2 million from its Research and Related Activities account for the design of LCCF. According to the project’s cooperative agreement, NSF may provide additional funding to advance the design of LCCF— $3.5 million in fiscal year 2020 and $2.5 million in fiscal year 2022 following successful completion of the conceptual and preliminary design reviews, respectively, subject to availability of appropriations. As of September 2019, NSF had not yet formally identified risks for the LCCF project because the project was early in the design stage. NSF requires recipients to develop and follow formalized risk management during the design and construction stages of major facility projects to identify potential risks, assess the nature of those risks, and identify actions that can be taken to either reduce the probability of those risks occurring or reduce their impact to the project. NSF officials told us that an assessment of risks associated with the LCCF project will be part of the conceptual design review, planned for June 2020. According to NSF officials, one anticipated challenge for the LCCF project is the rapid pace of technological change in the field of high-performance computing. The officials stated that forecasting the technology marketplace in the future can be challenging as technology can change radically because of external market forces. Conversely, the rapid pace of change can also be an opportunity if the LCCF project can incorporate the latest technological advances that result in the most advanced computing capabilities. According to NSF officials, taking advantage of such opportunities as late in the design stage as possible will be important for the success of the project. Appendix III: Comments from the National Science Foundation Office of the Director Appendix IV: GAO Contact and Staff Acknowledgments GAO contact Staff Acknowledgements John Neumann, (202) 512-6888 or neumannj@gao.gov In addition to the contact named above, Joseph Cook (Assistant Director), Sean Manzano (Analyst in Charge), Louise Fickel, Yvette Gutierrez, Patrick Harner, Douglas G. Hunker, Jason T. Lee, Serena Lo, and Anika McMillon made key contributions to this report.