Background Over the next 10 years, the Navy plans to continue developing critical technologies, complete detail design, and begin construction of the lead Columbia class submarine. In December 2017, we found that the schedule to deliver the lead submarine was aggressive, with extensive overlap—or concurrency—between development, design, and construction, as shown in figure 1. Our prior work reviewing shipbuilding programs has shown that the programs with the greatest amount of overlap between shipbuilding phases often have the highest cost and schedule growth, as well as quality and performance issues. The National Defense Authorization Act for Fiscal Year 2018 included reporting requirements for the Columbia class program. As part of these annual reporting requirements, the Navy must submit to Congress matrices that identify (1) key milestones, events, and performance goals for the design and construction of the Columbia class program; and (2) costs associated with the design and construction period of the Columbia class program. The Navy submitted its initial matrices to Congress in February 2018 and an update to the matrices in October 2018. The next matrices update is due in March 2019 and annually, thereafter, until the lead Columbia submarine is delivered. Columbia Class Critical Technologies The Navy is developing a number of new technologies related to submarine propulsion, missile tubes, and survivability that are planned to ensure that the Columbia class will remain operationally relevant throughout its planned 42.5-year service life, as shown in figure 2. In 2015, as part of its technology readiness assessment, the Navy identified two technologies—the advanced carbon dioxide removal unit and the stern area system—as critical technology elements. However, as we found in 2017, several Columbia class technologies that met GAO’s definition of a critical technology element were not identified by the Navy as critical technologies. In addition, several of these were immature, with technology readiness levels (TRL)—used to describe the maturity of critical technologies—of less than 7. See appendix II for a description of TRLs. As part of its matrices to Congress, the Navy is required to report on the TRLs of the integrated power system, nuclear reactor, propulsor, coordinated stern features, stern area system, and common missile compartment—which are the critical technologies we identified in our prior report. Table 1 lists each GAO-identified critical technology and its TRL as of October 2018, as reported by the Navy. Columbia Design and Construction Approach Two shipbuilders—General Dynamics Electric Boat (Electric Boat) and Huntington Ingalls Industries Newport News (Newport News)—design and build nuclear submarines. Electric Boat is the prime contractor for both design and construction of the Columbia class program, with Newport News serving as a subcontractor. Similar to the Virginia class program, each shipbuilder will construct segments of the submarine, but Electric Boat will complete final outfitting and deliver the submarines to the Navy. The Navy awarded a detail design contract in September 2017 to Electric Boat for work including completion of the submarine’s design, component and technology development, and prototyping efforts. The detail design process for the Columbia class program encompasses three activities, which began after the Navy set the technical requirements for the submarine in 2016: Arrangements outline the steel structure and routes distributive systems—such as electrical or piping systems—throughout the submarine. At this time, the shipbuilder generates a three-dimensional computer-aided design model for the area. Disclosures complete the design work for even the lowest-level items of the submarine, including material information. After these are completed, the shipbuilder can begin ordering material and long lead items for the submarine. Work instructions are three-dimensional electronic products that shipyard workers use to construct the submarine. Figure 3 illustrates the design phases for the Columbia class program. The shipbuilder will design and construct Columbia class submarines in six large hull segments, referred to as super modules, a method also used to construct most of the Virginia class submarines. During construction, the modules will largely be outfitted with systems and connections prior to being attached together during final assembly. According to the shipbuilder, this method is more efficient than outfitting the hull after it is constructed because more workspace is available to install equipment. Figure 4 illustrates the super modules within the submarine. Cost Estimating A reliable cost estimate is critical to program success. It provides the basis for informed investment decision making, realistic budget formulation and program funding, meaningful progress measurement, proactive course correction when warranted, and accountability for results. GAO’s Cost Estimating and Assessment Guide states that reliable cost estimates reflect four characteristics, which encompass 19 best practices. These characteristics—comprehensive, well documented, accurate, and credible—are shown in table 2. For Navy shipbuilding programs, including the Columbia class, several different entities are involved in cost estimating: The Naval Sea Systems Command (NAVSEA) Cost Engineering and Industrial Analysis Group develops the program life-cycle cost estimate, which is an estimate accounting for the total cost to the government of acquisition and ownership of a system over its full life. NCCA develops an independent cost assessment for certain Navy programs, such as the Columbia class program, at milestone events in the defense acquisition system. This assessment is not a separate estimate, but rather a review of the NAVSEA program life-cycle cost estimate. A cost review board, comprised of multiple Navy offices, establishes a service cost position based on their review of the program life-cycle cost estimate and the independent cost assessment. The Office of the Secretary of Defense’s CAPE conducts or approves independent cost estimates for major defense acquisition programs. Independent cost estimates are statutorily required for major defense acquisition programs at milestone events. The milestone decision authority, which in the case of the Columbia class program is the Under Secretary of Defense for Acquisition and Sustainment, reviews the service cost position and independent cost estimate and selects the cost estimate to baseline and fund the program. The most recent milestone event for the Columbia program was the Milestone B decision in January 2017, where the program received approval to proceed to the next acquisition phase—engineering and manufacturing development, which includes detail design of the lead submarine. In a memo documenting that decision, the milestone decision authority noted that significant development risks remain for the Columbia program and cost control must remain a priority. To limit program cost growth, the milestone decision authority established an affordability cap: the average submarine procurement cost should not exceed $8.0 billion in constant year 2017 dollars. Figure 5 summarizes the cost estimating process for the Columbia class program’s Milestone B review. Navy Is Managing an Aggressive Build Schedule, but Early Design and Construction Challenges Signal Schedule Risk The Navy is attempting to mitigate an aggressive schedule for lead submarine construction by (1) setting a goal to mature a significant amount of the submarine’s design prior to the start of construction and (2) beginning advance construction of submarine modules prior to October 2020. The shipbuilder is working to improve design performance and would have to maintain this increased pace to achieve its design goal, which is necessary to mitigate schedule risk associated with constructing the lead submarine. This may prove challenging as it must complete an increasingly higher volume and complexity of design products. At the same time, the Navy is continuing to develop several critical technologies and recent manufacturing defects with the integrated power system and missile tubes are among the challenges that the Navy is facing in ensuring timely delivery of critical components to the shipyard. Finally, to achieve Columbia’s aggressive construction schedule, while simultaneously building Virginia class submarines, the shipbuilder is working to ensure that it has sufficient shipyard capacity—including new facilities, additional suppliers, and an increased workforce. Shipbuilder Would Have to Maintain Its Increased Pace to Meet Its Design Maturity Goal and Reduce Schedule Risk The shipbuilder has failed to achieve its planned rates for completing design arrangements and disclosures to meet its design maturity goal in recent months—hampered by implementation of a new design software tool and an insufficient number of designers to meet monthly design completion rates. As we reported in December 2017, the Navy’s priority is to complete a high level of design—specifically, 100 percent of design arrangements and 83 percent of design disclosures—by the start of lead submarine construction in October 2020. By maturing the design before beginning construction on the lead submarine, the Navy is attempting to mitigate the risk of costly rework from design changes and subsequent delays to the Columbia class program’s 84-month construction schedule, which the Navy has acknowledged is aggressive. The Navy established the design maturity goal for Columbia based on lessons learned from the Virginia class program, when the shipbuilder began constructing the lead submarine with only 76 percent of arrangements and 43 percent of disclosures completed and, subsequently, realized 21 percent cost growth. Since the shipbuilder began work on the detail design, it has generally met its overall goal of completing the arrangements on schedule. As detail design continues, however, the shipbuilder is transitioning from relatively simple designs for the hull to the more complex designs for the submarine’s internal systems, increasing the pace needed to complete the remaining designs, as shown in figure 6. Navy officials stated that design disclosures are generally considered the most challenging phase of design work, where the shipbuilder specifies the lowest-level items and defines all aspects of the submarine. The shipbuilder has to maintain this increased pace in order to achieve the design maturity goal by the start of lead submarine construction. However, the shipbuilder’s design progress in completing disclosure products has fallen short of its plan in recent months as the planned pace and complexity of the design has increased. Using data from the program’s cost performance reports, we analyzed the shipbuilder’s monthly design progress according to a schedule performance index that measures the value of the work completed against the work scheduled. For example, if the schedule performance index is less than 1.00, then the shipbuilder has completed less than a dollar’s worth of work for each dollar that was scheduled. As shown in figure 7, since January 2018, schedule performance has consistently fallen below 1.00. Both DOD and Navy officials attributed the shipbuilder’s design delays to challenges adapting to a new design software tool. Beginning with the Columbia class program, the shipbuilder transitioned to a new customized software tool for design and construction because its prior software was no longer supported by the original developer. However, the shipbuilder has experienced problems developing the tool, which has resulted in slower progress to complete both design arrangements and disclosures, as certain aspects of the software’s functionality were delayed. Navy officials stated that, as of June 2018, they believe that design software functionality was performing at a level that no longer impeded design progress. While the designers have gained proficiency with the new design tool to complete arrangements and disclosures, according to Navy officials, the shipbuilder is now facing similar challenges using the tool to generate work instructions. Navy program officials also stated that the shipbuilder has not delivered some of the software functionality needed to produce work instructions as scheduled. Further, Navy officials noted that the process to create work instructions from completed disclosures takes longer with the new design software so the shipbuilder has begun generating work instructions earlier. According to Navy officials and shipbuilder representatives, the shipbuilder hired 150 additional designers in an effort to recover its design schedule and meet future monthly design goals. However, adding designers to recover and maintain the shipbuilder’s design schedule ultimately increases the program’s design costs. Similar to the schedule analysis above, we used data from cost performance reports to analyze the shipbuilder’s monthly design progress according to a cost performance index that measures the budgeted value of the work completed against what it actually costs to complete it. For example, if the cost performance index is less than 1.00, then less than a dollar’s worth of work has been completed for each dollar spent. As shown in figure 8, the shipbuilder’s cost performance has consistently fallen below 1.00 since December 2017. If the shipbuilder cannot address challenges associated with using the software tool to generate work instructions discussed above, it will likely need additional design hours in the future, resulting in higher costs in order to mature the design on schedule. Navy’s Use of Advance Construction to Mitigate Aggressive Schedule Is Not without Risk Navy officials and shipbuilder representatives expect to mitigate risks associated with the Columbia construction schedule by accelerating the building of certain components more than a year in advance of the formal start of construction. They anticipate that this advance construction strategy will allow them to gain 2 months of schedule margin for final assembly and testing prior to delivery of the lead submarine. Starting in December 2018, the shipbuilder will begin constructing modules of the submarine as part of its advance construction effort. In 2017, we reported that the Navy had planned to begin advance construction for four of the submarine’s six super modules, but since our report was issued, it now plans to begin construction on all six super modules including building components like the stabilizers, impulse tanks, and others. Figure 9 shows the start of advance construction for each super module. Navy officials estimate that the current advance construction efforts will require approximately 631,000 labor hours. In addition, advance construction efforts would require that the Navy accelerate delivery of equipment provided to the shipbuilder for installation on the submarine, such as pumps and valves. Shipbuilder representatives stated that a lesson learned from the Virginia class program was that construction of certain complex components should begin as early as possible if capability requirements and designs are stable. However, based on its plan, the shipbuilder will begin advance construction having completed less than 40 percent of the total design disclosures for the Columbia class submarine, as shown in figure 10. The number of disclosures completed at the start of advance construction is less than half of those the shipbuilder plans to complete by the start of lead submarine construction in October 2020. Navy officials stated that they believe the risk associated with beginning construction with a less mature overall design is mitigated because the program selected components for advance construction that are well understood and unlikely to be affected by design changes, like ballast tanks, decking, and hull segments. In addition, Navy officials stated that they will not begin construction on the component or hull unless the arrangements associated with the structure of that area of the submarine are complete. However, based on the shipbuilder’s design plans, the arrangements and disclosures of adjoining areas of the super module may not be complete, which could negatively affect construction. Specifically, the shipbuilder’s design plans indicate that it will have completed 100 percent of disclosures for only one super module at the start of advance construction. As we have found in our prior work, proceeding with construction despite having completed fewer designs than planned increases the likelihood of design changes later that may, in turn, require costly and time-intensive re-work to change components that have already been built. Shipbuilder representatives acknowledged that there is risk in starting construction of some components prior to completing the design for individual super modules or the entire submarine. However, shipbuilder representatives stated that they believe this risk is reduced by only starting construction on components for which the disclosures are complete. Recent Challenges with Critical Technologies Have Reduced Available Schedule Margin While ship design is underway, the Navy is continuing to develop and mature the critical technologies related to the Columbia class program. While these critical technologies are not required at the shipyard for several years, recent challenges have eroded available schedule margin, as illustrated below: Integrated Power System: In 2017, we reported that the Navy experienced manufacturing problems associated with the integrated power system. We found that the Navy continues to experience problems with the electric drive of the integrated power system that could potentially affect construction of the lead submarine. A manufacturing defect that affected the system’s first production- representative propulsion motor required extensive repair that consumed 9 months of schedule margin at the land-based test facility. The Navy now plans to test the motor at the same time it had originally scheduled to make any final design changes before starting production. This could constrain opportunities to implement timely, corrective actions if problems are discovered during testing. Common Missile Compartment: Navy officials stated that, in July 2018, the shipbuilder identified substantial weld defects in missile tubes from one of three tube suppliers and resulted in investigations of the missile tubes from all suppliers. These defects were discovered after seven tubes in various stages of outfitting had already been delivered to the shipyard and five additional tubes under production have been affected. Navy program officials stated defects occurred because inexperienced welders performed the complex work and inspectors at the supplier’s facility subsequently failed to identify the defects. While the Navy and shipbuilder are still determining the cost and schedule impacts of the weld defects, program officials estimated that addressing this issue will consume up to 15 of the 23-month schedule margin for these components. In addition, program officials stated that the Navy likely will be responsible for some of the cost associated with investigating the root cause of the defects and risk mitigation efforts going forward. Given the erosion of available schedule margin, there is less time available to address issues without resulting in schedule delays. For example, the shipbuilder’s construction plans for two super modules do not include schedule margin to accommodate any delays that may occur as the technologies are matured and detail design is completed. One of these, the stern super module contains three technologies that are not fully mature—the integrated power system, stern area system, and advanced propulsor bearing. The integrated power system is not expected to reach full maturity until October 2019 and the remaining two technologies will not be mature until after the shipbuilder begins construction on the lead submarine, not including those components that begin advance construction years earlier. Without schedule margin to accommodate any changes or issues, any delays in delivering equipment to the shipyard on time could disrupt the shipbuilder’s construction sequence for the lead submarine. Shipbuilder Is Facing Oversight and Capacity Challenges in Preparation for Columbia Construction To meet the Navy’s aggressive construction schedule for the lead submarine, the shipbuilder has to ensure that it has the capacity to meet a substantially higher workload and effectively balance Columbia and Virginia class construction. At the same time as construction on Columbia begins in 2020, the shipbuilder will also have begun constructing two modified Virginia class attack submarines per year. To accommodate the construction of both submarine classes, the shipbuilder is planning an extensive expansion of its facilities, including new buildings, a pier, an ocean transport barge, and a floating dry dock. The anticipated increases in workload at the shipyard will also require the shipbuilder to manage a higher volume of build materials and an expansion of its workforce. While construction of new facilities is progressing on schedule, according to shipbuilder representatives, it faces other challenges preparing for Columbia class construction. Ensuring Supplier Oversight Achieving the planned construction schedule will require the Navy and shipbuilder to ensure that materials arrive on time and meet quality expectations, but according to Navy officials, supplier oversight has been a challenge for this shipbuilder in the past. Both Navy officials and shipbuilder representatives stated that they are concerned about the capacity of its suppliers to meet the demand for high-quality components given an industrial base that has diminished significantly since previous major submarine construction efforts in the 1980s. Many of the parts and equipment on Columbia class are common with those used on Virginia class submarines but, in other instances, suppliers are producing components for the first time after a considerable break, such as missile tubes that have not been produced since the early 1990s. Navy program officials and shipbuilder representatives stated that they monitor supplier capacity and quality—among other areas—and they have several methods to intervene if a supplier is not able to perform as needed. The shipbuilder and the Navy have formed a group to assess the three primary areas of supplier performance: Capability: includes the uniqueness of the supplier’s product on the market, challenges in shifting to a different supplier due to intellectual property rights or technical knowledge, and the ability for the supplier to sustain their own supply base. Capacity: includes the supplier’s ability to increase production without decreasing quality, maintain that capacity over the program’s production, their financial dependence on Navy programs for revenue, lead time needed to meet new orders, and the capacity of their own suppliers. Cost: includes the costs of increasing production spread out across demand from Navy programs. In 2017, the shipbuilder assessed its supplier base using these areas, identified the criticality and risk of each supplier based on their potential impact to the program and potential alternate suppliers, and conducted a gap analysis comparing the supplier’s current performance to the program’s desired performance. Based on the results of the analysis, the shipbuilder identified and is monitoring at-risk suppliers in coordination with the Navy to determine if immediate intervention is needed, such as investing in new facilities for the supplier, improving manufacturing workflow, or finding new sources of material from that supplier. Despite these efforts, supplier oversight remains an issue, because—in the instance of the missile tube welds mentioned above—the shipbuilder focused on managing certain anticipated risks, as opposed to actively managing the supplier’s quality and performance with on-site independent inspections, according to Navy officials. In response to the missile tube issues, the shipbuilder has proposed additional supplier oversight by assessing the need for on-site inspection teams depending on the risk each supplier poses to the program. Navy officials stated that they have begun some assessments but, as of March 2019, had yet to determine who will pay for this additional oversight. We plan to more fully assess the Navy and shipbuilder’s oversight of its suppliers for the Columbia class program in future work. Building Workforce Capacity and Capability According to shipbuilder representatives, the start of lead submarine construction for the Columbia class, combined with expanding Virginia class construction, increases the demand for hiring and retaining skilled workers at levels not seen at this shipyard since the 1980s. Navy officials expressed concerns about the risk of adding large numbers of new workers, including an influx of inexperienced welders and inspectors— issues that also contributed to the defects in missile tubes discussed above. To support growing workload from both the Columbia and Virginia submarine programs, the shipbuilder plans to increase workforce at its two facilities over the next decade: by 66 percent at Quonset Point, Rhode Island—where the components and individual submarine modules will be constructed—and 174 percent at Groton, Connecticut—where the super modules will undergo final outfitting and assembly. To meet this increased demand in a skilled workforce, the shipbuilder assessed future demographic trends in the area surrounding its facilities and found that, while sufficient labor will likely be available, more training will be necessary. Consequently, the shipbuilder established internal and external training programs and partnerships with educational institutions in the area to grow the qualified workforce in time to begin lead submarine construction in October 2020. The influx of inexperienced workers can temporarily decrease construction efficiency as compared to a current, more experienced workforce. For example, when the Virginia class program expanded its workforce to build a second submarine each year, the addition of new staff contributed to an 8 percent decrease in cost efficiency for the program. Shipbuilder representatives at one production facility have already reported reduced efficiency following increased hiring of new workers. The shipbuilder’s goal is to maintain an average of 8 years of experience for workers in core trades, such as welding. However, the shipbuilder’s projections show that the new workforce ramp-up at the Groton facility will reduce workers’ average experience from 13.1 years to a low of 5.6 years in 2028—just after the shipbuilder plans to deliver the lead Columbia class submarine. If workforce growth or efficiency assumptions are not met, the shipbuilder may resort to scheduling overtime work or outsourcing some activities to meet the program’s construction schedule, which would have cost impacts for the program. Columbia Class Cost Estimate Is Not Reliable and Does Not Reflect Program Risks The Navy’s procurement cost estimate of $115 billion to construct Columbia class submarines is not reliable because it does not reflect likely program costs and risks. We assessed the Columbia class cost estimate by comparing it with the best practices identified in GAO’s Cost Estimating and Assessment Guide. We found that it substantially met the criteria for the comprehensive characteristic of a reliable cost estimate, and partially met the criteria for the remaining characteristics, including accurate and credible. In particular, we found that the cost estimate does not accurately reflect program costs because it is based on overly optimistic labor hour assumptions, and is not fully credible because while the Navy conducted risk and sensitivity analyses to test the likelihood of achieving its assumptions, it selected a specific cost estimate that informs the program’s budget which does not include any margin in case those assumptions are not achieved. In addition, the cost estimates and assessments conducted by other entities produced a range of results, indicating that there is a high degree of uncertainty regarding program costs. See appendix III for the full results of our assessment of the Navy’s cost estimate. Navy officials stated they plan to update the Columbia class cost estimate in support of DOD’s decision to authorize construction of the lead submarine and this decision is expected to occur in summer 2020. Navy officials also stated that they expect that the cost estimate will be complete by the end of fiscal year 2019, followed by an independent cost assessment to support the authorization decision. However, this timeframe does not provide assurance that both the update and the independent assessment will be complete before the Navy requests funding from Congress for lead submarine construction, as part of its fiscal year 2021 budget request, which could be submitted as early as February 2020. If so, decision makers may be basing their decisions on outdated or incomplete information. Columbia’s Cost Estimate Is Not Accurate Because It Relies on Overly Optimistic Labor Hour Assumptions The Columbia class cost estimate relies on optimistic program assumptions and does not reflect the likely labor hour costs that the Navy will incur to construct the submarines. As part of our assessment of the Columbia program cost estimate, we found that it did not fully meet the best practices for an accurate estimate. A cost estimate is considered accurate when it is based on an assessment of the most likely costs—that is, it is neither overly conservative nor overly optimistic. The Navy estimates that it will need $115 billion to design and construct 12 submarines and NAVSEA cost estimators identified labor costs as a primary source of cost risk. As discussed below, if the program’s optimistic assumptions are not realized, the program may require more funding than originally planned to construct the Columbia class. The Navy anticipates that it will need 12 million labor hours to directly construct the lead submarine—referred to as touch labor. This represents 17 percent fewer labor hours than what was needed for the lead Virginia class submarine, when adjusted for weight differences. To develop this estimate, NAVSEA estimators relied heavily on historical touch labor hour data from the construction of the lead Virginia class submarine and cost data from the Ohio class submarine program for unique ballistic submarine components, such as missiles. NAVSEA estimators took the following steps to develop the Columbia lead submarine estimate: In general, heavier ships cost more to construct, so NAVSEA cost estimators calculated a weight-adjusted estimate based on Virginia class labor hours to account for the heavier weight of the Columbia class. This resulted in an initial estimate of 14.5 million touch labor hours for the lead submarine. NAVSEA cost estimators then made numerous adjustments in the cost estimate that reduced the expected number of labor hours based on multiple assumptions that differences in the design and construction process would lead to more efficient construction of Columbia class submarines than previous submarine classes. These adjustments subsequently decreased the estimate to 12 million touch labor hours for the lead submarine. NAVSEA cost estimators then used the lead Columbia submarine estimate as the basis to calculate labor hours for follow-on submarines, estimating an average of 8.9 million touch labor hours. Figure 11 illustrates NAVSEA’s touch labor hour calculation for the lead submarine. However, the touch labor hour estimate is overly optimistic—with assumptions on construction efficiencies that are either unsubstantiated or unprecedented compared to Virginia class and other shipbuilding historical data. Compared to the Navy’s estimate, Columbia’s estimated touch labor hours, as calculated by other organizations, are more conservative. For example, CBO questioned the Navy’s assumption that ballistic submarines are less expensive to build than attack submarines, after accounting for weight differences and estimated that for the overall class, including the lead and follow-on submarines, the Navy would more likely realize an 8 percent reduction rather than the 19 percent reduction estimated by the Navy. While the shipbuilder will likely realize some efficiencies from initiatives to improve design and construction processes, our analysis of the Navy’s assumptions used to develop the cost estimate indicates that they likely overstate the labor hour reduction the shipbuilder can realistically achieve. These assumptions include that the program (1) achieves its design goals at the start of construction; (2) is constructed more efficiently than Virginia class submarines; and (3) successively reduces the number of hours needed to construct follow-on submarines. If these assumptions are not realized, overall program costs could be higher than the Navy’s procurement estimate of $115 billion. Navy officials stated that they believe that these assumptions are valid and that the cost estimate is achievable. However, our assessment indicates that the assumptions for the cost estimate are overly optimistic, as discussed below. Risk of Unrealized Design Goals The Navy’s cost estimate does not reflect the risk that the shipbuilder may not achieve its planned design completion goals. As we reported above, design performance to date has slowed and the shipbuilder has had to hire additional designers in an effort to mature its design on schedule. NAVSEA cost estimators stated that they recognize that an incomplete design at the start of ship construction was a significant driver of cost growth on other shipbuilding programs. For the Columbia class, NAVSEA cost estimators assumed that achieving the design maturity goal would eliminate 2 million labor hours by reducing costs associated with rework and out of sequence work. In October 2018, NCCA officials stated that they recently reviewed shipbuilder data and the expected design completion at construction start continues to range between 55 and 75 percent complete—the same range that they estimated in their independent assessment. While this lower rate would be an improvement over the Virginia class program, it would still fall short of the 84 percent assumption built into the cost estimate. If the shipbuilder does not complete the design at its planned rate and begins construction with a less mature design, it may need additional labor hours to construct the ship, resulting in increased program costs. Overly Optimistic Estimate of Efficiencies The Navy’s cost estimate includes assumptions that reduce Columbia’s estimated touch labor hours due to efficiencies from constructing Columbia and Virginia class submarines concurrently, an assumption with which the shipbuilder does not agree. NAVSEA cost estimators calculated a 1.1 million-labor hour reduction, attributing the decrease to efficiencies gained from constructing multiple submarines at the same time, basing their assessment on shipbuilder estimates of the Virginia class. However, it is unclear how increased shipyard production would result in fewer labor hours to construct each submarine. Shipbuilder representatives stated that rather than a reduction in touch labor hours, they expect to realize efficiencies from increased production primarily from reduced overhead rates and material costs. Further, the Navy’s independent assessment analyzed labor hour data for Virginia class construction and found that there was no correlation between the number of submarines constructed at a time and the total number of labor hours. However, increasing shipyard production to include both Virginia and Columbia class construction may increase schedule risk for the shipbuilder, which could result in additional costs if the shipbuilder does not achieve planned increases in its workforce and facility upgrades. When the number of Virginia class submarines under construction increased, both shipyards experienced inefficiencies due to poorly planned ramp-up requirements. In addition, DOD officials stated that problems encountered on one program could affect the other as the shipbuilder is relying on the same workforce and vendor base for both programs. The Navy’s cost estimate also assumed construction efficiencies— because the Columbia class submarine will be less dense than the Virginia class submarine—another assumption with which the shipbuilder does not agree. Navy officials stated that less dense submarines are less costly to construct as the additional space within the hull allows for faster and more efficient work. However, the shipbuilder conducted analysis to compare the density of various areas of the Virginia class and Columbia class submarines and found that areas had very similar density. Specifically, there was only a 1 percent and 3 percent difference, between the forward compartments and aft compartments, respectively—some of the more complex areas of the submarine. If the shipbuilder does not realize these construction efficiencies, more total labor hours would be required to construct the submarine, resulting in increased cost. Learning Curve Assumption The Navy’s cost estimate assumes that the costs for follow-on Columbia class submarines will decrease at a rate that may overstate the improvements the shipbuilder can realistically achieve. The Navy expects the number of labor hours to construct Columbia class follow-on submarines to decrease based on an assumed learning curve rate. Learning occurs when construction is consistent and continuous and the shipbuilder learns how to do repetitive tasks more efficiently. The decrease in the number of expected labor hours is expressed as a learning curve rate, where a lower percentage indicates that less labor is required for follow-on units. NAVSEA cost estimators calculated a learning curve of 88.9 percent for Columbia class submarines. A learning curve indicates that as the number of units doubles, unit cost decreases by a constant percentage. In this case, the cost estimate assumed that the fourth submarine would require only 88.9 percent the amount of labor to build the second submarine. NAVSEA’s assumption may overstate the potential learning rate that Columbia can expect to achieve. The first four Virginia class submarines, hull numbers SSN 774 through 777, incorporated modular construction techniques where submarines were built in 10 modules. The next six Virginia class submarines, hull numbers SSN 778 through 783, were constructed using four modules. As a result of the improvements in the modular construction process, construction across the first ten submarines was not consistent, which is a condition that is necessary to determine the learning curve rate. Therefore, there is no way to determine what share of the labor hour reduction on later submarines was due to learning or process improvements. Rather, SSN 778, the first Virginia class submarine to use the four modular construction approach is a better starting point to determine the shipbuilder’s capacity for learning. The Navy’s independent assessment included a separate learning curve analysis for Virginia class submarine hulls SSN 778 through 791 and calculated a potential learning curve rate of 93.9 percent. A learning curve assumption applies to all follow-on submarines and has a cumulative effect on the number of labor hours and, ultimately, the cost of these submarines. In the case of the Columbia program, the rate will apply to the second through twelfth submarines. Figure 12 shows how the difference in the learning curve rate can affect the estimated labor hours for follow-on submarines. Therefore, a small change in the assumed learning curve rate can have a significant effect on the cost estimate for follow-on submarines. For example, the Navy’s independent assessment of the cost estimate calculated that production costs could increase by $3.59 billion in constant year 2010 dollars if a learning curve of 93.9 percent was realized, rather than the 88.9 percent rate estimate. Our previous work on Navy shipbuilding performance has shown that the Navy has consistently underestimated the costs for follow-on ships, with costs for Virginia class submarines underestimated by close to 40 percent. Columbia Cost Estimate Is Not Credible Because It Does Not Adequately Account for Program Risks The Columbia program cost estimate did not fully meet the best practice criteria to be considered credible because, in part, Navy program management did not sufficiently account for program risks when it selected the final estimate. To determine the estimate’s credibility, we examined the extent to which NAVSEA cost estimators tested, among other things, the sensitivity of key cost elements such as labor hours and conducted uncertainty analyses to quantify risks; and an independent cost estimate and assessment were conducted by groups outside the acquiring organization (specifically, CAPE and NCCA) to determine whether other estimating methods produced similar results. We found that while the Navy program management’s $115 billion procurement cost estimate for the Columbia class is overly optimistic in some of its assumptions, the estimate does not reflect any contingency to offset the likely effects of not meeting the assumptions, which is a best practice. In addition, the independent cost estimates and assessments conducted by other organizations had varying results, indicating the high level of uncertainty regarding Columbia program costs. We further address these issues below. Sensitivity and Risk Analysis Indicate Insufficient Cost Risk Coverage Navy leadership’s decision to select $115 billion as the program cost estimate means that there is no margin in the program budget to cover likely program costs if risks are realized. The best practices identified in GAO’s cost estimating guide state that the results of a risk analysis should be used to select a cost estimate that is sufficient to manage program risks. NAVSEA cost estimators conducted a risk analysis to identify and quantify program risks, and determined the effects of changing key cost driver assumptions and factors—important steps in creating a high quality estimate. However, while NAVSEA cost estimators identified 54 risk parameters for construction costs, we found that some of the inputs for these ranges resulted in a cost estimate that understates the potential impact of program cost risks. For example, the risk ranges do not sufficiently account for the issues we identified above, including that increased shipyard construction could result in similar inefficiencies that occurred in the production of the Virginia class, requiring more labor hours than estimated; and shipbuilder workforce ramp-up could result in decreased efficiency and quality due to the influx of new workers even greater than the issues observed on the Virginia class when shipyard construction increased. For other risk parameters, such as cost of material provided by the shipbuilder, the cost estimate documentation was not sufficient for us to analyze whether the risk ranges included in the estimate were reasonable (i.e., not overly optimistic or pessimistic). As a result, we could not determine whether the risk analysis sufficiently captures the risk of program cost growth, or what the probability is of achieving the $115 billion procurement cost estimate. Further, Columbia’s program management and the milestone decision authority selected $115 billion as the program’s procurement cost estimate, without adjusting for the likelihood of cost growth in the design or construction of Columbia class submarines identified in the risk analysis. As we reported in December 2017, the risk analysis developed by NAVSEA indicated that there is only a 45 percent probability that the overall program cost estimate will be sufficient to cover program costs. The cost estimating best practices identified in our cost estimating guide state that a risk-adjusted cost estimate helps ensure that sufficient funding will be available for the expected program costs. Additionally, a risk-adjusted cost estimate is consistent with federal internal control standards, which indicate that risk mitigation efforts should be selected to sufficiently respond to risks. However, Columbia program officials stated that they believe program risks can be managed within the current cost estimate—which they consider to be conservative—as it does not account for all of the program’s potential cost savings. Specifically, the Navy anticipates that the program will realize up to $1.9 billion in additional cost savings from use of authorities associated with the National Sea-Based Deterrence Fund (the Fund), such as the authority to purchase components for multiple submarines—which we discuss later in this report. As a result, the program office estimate represents the program manager’s cost goal for the Columbia program, rather than the risk- adjusted estimate. Even if the Navy were to achieve the full anticipated $1.9 billion savings, these savings represent only 1.5 percent of program costs. Such cost savings are unlikely to cover program cost overruns for a high-risk program, such as Columbia, given that historically shipbuilding programs experience 27 percent cost growth. As the current estimate does not include any reserves for cost overruns, program management is relying on these potential savings to help mitigate likely cost growth. Independent Cost Reviews Indicate Varying Results Several entities have conducted independent reviews of the Columbia program cost estimate, with varying results. CAPE conducted an independent cost estimate and NCCA conducted an independent cost assessment of the program estimate in support of the Columbia class program’s Milestone B review. CAPE’s independent cost estimate was 3 percent lower than the Navy’s service cost position, which it stated was due to CAPE’s use of lower shipyard labor rates. However, NCCA’s assessment did not produce similar results as the program cost estimate and concluded that the program is at risk of up to $6.14 billion in cost growth. The program manager reviewed the recommendations in the independent cost assessment and determined that the program office estimate appropriately weighs program risks. Navy leadership selected the program office estimate to serve as the Navy’s service cost position because program officials stated that they believe program risks can be managed within the program cost estimate. CBO also conducted a cost estimate and projected that procurement of 12 submarines would be 6 percent higher than the program estimated. The results of these cost estimates and NCCA’s assessment are summarized in table 3. As part of the Milestone B review, the milestone decision authority reviewed the service cost position and CAPE’s independent cost estimate. The independent cost assessment was reviewed by Navy leadership as part of the service cost position process and, therefore, was not briefed as part of the milestone review. The milestone decision authority accepted the Navy service cost position and directed the Navy to use this estimate as the basis of its fiscal year 2018 budget request. It also established an $8 billion affordability cap for the average procurement cost of all 12 submarines to control future program costs. Congress May Not Have Up-to-Date Cost Information When Considering Columbia Class Budget Request for Lead Submarine Funding Navy officials stated that they plan to update the cost estimate for the lead submarine in support of a planned Defense Acquisition Board review, in the third quarter of fiscal year 2020. At that point, the Navy will be seeking approval from the milestone decision authority to award the contract for construction of the lead submarine. However, the Navy and DOD’s general timeframes do not provide assurance that the planned update of the cost estimate would be completed prior to the fiscal year 2021 budget request, which will include funding for lead submarine construction, as shown in figure 13 below. The milestone decision authority has directed CAPE, with assistance from NCCA, to assess the lead submarine cost estimate to support the decision to authorize the Navy to award the contract for lead submarine construction. Since this assessment will occur after the Navy has updated the lead submarine cost estimate, it is even less likely that the program budget request will reflect the results from the independent cost assessment. Additionally, the current program cost estimate the Navy developed for the Milestone B review does not reflect the program’s current strategy to use authorities associated with the Fund to achieve cost savings, as discussed further below. The best practices identified in GAO’s cost estimating guide state that cost estimates should be regularly updated and reflect the program acquisition baseline. Updating the cost estimate and risk analysis to include these anticipated savings and current program data would improve its reliability and help ensure that budget requests are sufficient to execute the Columbia program as planned. After we provided our draft report to DOD for comment, Navy officials briefed us on the changes they had made to the program’s estimate to date, stating that they updated the cost risk analysis as part of an internal program review. While the Navy plans to update the lead submarine cost estimate again by the end of fiscal year 2019 to support the Defense Acquisition Board review in the summer of 2020, it has yet to provide specific details on the steps it will take to update this estimate to ensure that it would include likely program costs and risks, such as the cost data it plans to include or the assumptions it may reassess. Further, since the Navy will likely submit its budget request to Congress as early as February 2020, Congress may be asked to authorize and fund lead submarine construction without the benefit of any changes to the estimate that may occur as a result of recommendations stemming from an independent review of the update. Further, although the Navy reports Columbia program cost information to Congress through annual matrices submissions, updates to the program cost estimate will not be reflected in these reports. For example, the Navy plans to report program manager and contractor cost estimates for individual submarines in the matrices once the submarines are under construction. Since these estimates are based on shipbuilder contract performance, they are initially calculated only after construction of each submarine is 15 percent complete, when sufficient data are available to show performance trends. While the Navy plans to award the contract for the lead submarine in October 2020, limited contractor performance data will be available in time for the February 2021 matrix submission. As a result, the earliest opportunity to report on the cost of the lead submarine would be the Navy’s next submission in February 2022, at which point the Navy will have already requested funding for the second and third Columbia submarine. Navy Is Using National Sea-Based Deterrence Fund and Associated Authorities, but Anticipated Savings May Be Overestimated In 2014, Congress created a National Sea-Based Deterrence Fund (the Fund) that provides DOD with greater discretion to fund the design, construction, purchase, alteration, and conversion of the Columbia class. Since then, Congress has provided the Navy with enhanced acquisition authorities to buy and construct submarines and certain key components early, in bulk, and continuously, when using these funds. The Navy anticipates saving over $1.9 billion through use of these authorities, but these savings, which were not included in the Columbia class program’s cost estimate, may be overestimated. Navy Executes Columbia Program through the Fund and Its Associated Authorities Since its inception in 2014, Congress has expanded the special acquisition authorities under the Fund, in part, to allow the Columbia class program to gain economic efficiencies and realize cost savings. The timeline of the establishment of the Fund and legislative changes are shown in figure 14. The following authorities have been included as part of the use of the Fund: Economic order quantity: Permits awarding of contracts that provide a quantity of supplies that will result in a total cost and unit cost most advantageous to the government by achieving economic efficiencies based on production economies. Advance construction: Allows for manufacturing and fabrication efforts prior to ship authorization. Multiyear procurement authority: Permits a single contract for more than one year of critical components. Incremental funding authority: Facilitates the purchase of long lead items through partial funding of a contract with the expectation that full funding will be provided later. Using the Fund’s associated authorities, the Navy is able to purchase significant components and start advance construction prior to receiving Congress’s authorization of and funding to purchase each submarine. In total, the Navy will have requested and received $8.6 billion in funding, including 33 percent of funding for the lead submarine, before it receives authorization and funding to begin construction of the lead submarine in October 2020. At that point, the Navy will also have already requested funding for the propulsor and advance construction for the second submarine. Under law, the Navy is required to deposit all appropriations for the Columbia class construction and design into the Fund. To date, the Navy has made three deposits from the Shipbuilding and Conversion, Navy account into the Fund, totaling over $1.6 billion. The Navy is using initial deposits of $773 million in fiscal year 2017 and $862 million in fiscal year 2018 for detail design and continuous production of missile tube components. The Navy Comptroller initiates all deposits into the Fund, which are approved by the DOD Comptroller as internal reprogramming actions, as shown in figure 15. Anticipated Savings from Use of Fund’s Associated Authorities May Be Overestimated The Navy anticipates achieving over $1.9 billion in savings through the use of the Fund’s associated authorities, but the Navy did not evaluate these savings when it developed the program office cost estimate. Table 4 provides a description of each authority and the Navy’s plans and estimated potential savings resulting from use of the authorities. Overall, while we were unable to fully assess the methodology and assumptions the Navy used to estimate anticipated savings, the information we reviewed indicated that the Navy may have overestimated some of the savings the program can realistically achieve through use of the Fund’s associated authorities. While the Navy provided some documentation of the cost estimate methodologies, we could not fully validate that the estimated savings were realistic because, in general, the documentation provided by the Navy did not include a detailed description of how the estimates were calculated or how historical data were used to develop the estimate—a best practice identified in GAO’s cost estimating guide. In some cases, such as for individual critical components, the total value of the component costs was not documented. For other savings, such as advance construction, the Navy could not provide documentation of the calculations or a rationale for the estimated savings. In addition, the Navy assumes a higher rate for Columbia multiyear procurement savings than what has been typically achieved for other programs. The Navy has generally used multiyear procurement contracts after production has begun and some units have already been purchased. For example, according to the Navy, it did not receive multiyear procurement authority for the DDG 51 Arleigh Burke-class destroyer program—until 1998—more than 10 years after the contract for the lead ship was awarded and 38 ships had been purchased. We have reported that DOD typically overestimates savings from multiyear procurement authority. Further, in a 2017 presentation to Congress, the Navy stated that multiyear procurement savings are historically 10 to 12 percent. When the Navy requested multiyear procurement authority for the DDG 51 program in fiscal year 2013, it estimated achieving a savings of 8.7 percent. Similarly, when planning material purchases for the Virginia class submarine, the shipbuilder estimated that it would achieve 10 to 15 percent savings through the use of multiyear procurement authority. However, the Navy estimates that the Columbia class program will realize savings of 15 to 20 percent using multiyear procurement authority. A realistic estimate of savings is essential because program management is essentially relying on these savings as the only cost reserve to address any issues that arise during design and construction of the submarines. Updating the cost estimate to reflect these savings will provide program management with a more realistic assessment of the margin available and resources needed to achieve their costs. Conclusions The Columbia class program is driven by the continued and pressing need to meet the Navy’s nuclear deterrent requirements as the legacy submarine fleet cannot extend its life any longer. From the outset this has translated into an aggressive and concurrent schedule for lead submarine construction. To counterbalance this schedule risk, the program plans to complete a substantial amount of the design before starting construction, which may prove challenging as the shipbuilder must complete an increasingly higher volume and complexity of disclosures. This, coupled with failures in missile tubes already delivered to the shipyard, highlight the potential for management challenges ahead. This is not to suggest that in a program of this size and complexity that some issues are not to be expected. Rather, the challenge for the Columbia class program is that the Navy has a limited ability to slow the pace of the program given the mission imperatives. At present, the need for additional resources appears likely because the Navy’s margin to mitigate any cost growth from issues that develop during design and construction relies on overestimated savings from use of the Fund’s associated authorities. The steps that the Navy takes between now and the fiscal year 2021 budget request to understand and plan for likely program costs will determine whether sufficient funding is in place to cover potential cost growth. The Navy plans to update the lead submarine cost estimate to reflect its current acquisition strategy and, in doing so, the Navy has the opportunity to incorporate more realistic information into the risk analysis and lead submarine cost estimate. In addition, a realistic and well-documented estimate of savings from use of the Fund’s associated authorities would help ensure that the Navy has allocated the necessary resources to address any issues that emerge during design or construction of the lead submarine. Such steps will likely improve the reliability of the lead submarine cost estimate and would position the Navy to better align its fiscal year 2021 budget request with funding it will likely need to construct the lead submarine—the next key decision point in the Columbia class program. Without an updated cost estimate with more realistic assumptions, Congress will be asked to commit billions of dollars for the lead submarine without knowing the full potential cost of construction and the possible effect on other shipbuilding programs. Recommendations for Executive Action We are making three recommendations to the Secretary of the Navy: The Secretary of the Navy should direct NAVSEA to incorporate current cost and program data and an updated cost risk analysis in its planned update of the Columbia class lead submarine cost estimate. (Recommendation 1) The Secretary of the Navy should direct NAVSEA to develop a realistic and well-documented estimate of savings from use of the authorities associated with the Fund and incorporate the savings associated with the lead submarine into the Columbia lead submarine cost estimate. (Recommendation 2) The Secretary of the Navy should direct the Columbia class program office to update the lead submarine cost estimate and cost risk analysis prior to requesting funds for lead submarine construction. (Recommendation 3) Agency Comments and Our Evaluation We provided a draft of the sensitive report to DOD for comment. DOD’s written comments on the sensitive report are reprinted in appendix IV and summarized below. DOD concurred and described the actions they have taken or plan to take in response to all three of our recommendations. Regarding our recommendations to update its cost estimate update prior to requesting funds for lead submarine construction, the Navy has stated that it incorporated current cost and program data and an updated risk analysis into its cost estimate for the lead submarine in 2018, as part of an annual review. The Navy also stated that it will continue to update the lead submarine cost estimate with current data prior to requesting funding for lead submarine construction in fiscal year 2021. Until the updated estimate is independently validated—an essential cost estimating step— we cannot determine that the updated estimate is credible. Further, in response to our recommendation regarding the development of a realistic and well-documented estimate of savings from use of the Fund’s associated authorities, the Navy stated that it incorporated savings in its updated cost estimate. However, it has not provided any additional evidence to demonstrate that estimated savings from use of the Fund’s associated authorities are realistic and well-documented. Based on documentation that the Navy provided to us, it did not include a detailed description of how the estimates were calculated or how historical data were used to develop the estimate. Until these estimates are independently validated, the Navy cannot be confident that the program will achieve the planned amount of savings. The Navy also provided technical comments, which we incorporated as appropriate. DOD also raised a number of issues related to our assessment of the cost estimate, advance construction, and technology development, which we address in appendix IV. We are sending copies of this report to the appropriate congressional committees, the Acting Secretary of Defense, the Secretary of the Navy, and other interested parties. In addition, the report is available at no charge on the GAO website at http://www.gao.gov. Should you or your staff have questions, please contact me at (202) 512- 4841 or oakleys@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are listed in appendix V. Appendix I: Objectives, Scope, and Methodology This report evaluates the Navy’s Columbia class submarine program. Specifically, we assessed (1) the Navy’s progress and challenges, if any, associated with meeting design goals and preparing for lead submarine construction; (2) the reliability of the Navy’s cost estimate for the Columbia class submarine program; and (3) how the Navy is implementing the National Sea-Based Deterrence Fund (the Fund) and associated authorities to construct Columbia class submarines. This report is a public version of a sensitive report that we issued in March 2019. The Department of Defense (DOD) deemed some of the information in our March report to be sensitive, which must be protected from public disclosure. Therefore, this report omits sensitive information about the Navy’s development of critical technologies for the Columbia class program, including specific details about the technologies. Although the information provided in this report is more limited, the report addresses the same objectives as the sensitive report and uses the same methodology. To assess the Navy’s progress and what challenges, if any, are associated with meeting design goals and preparing for lead submarine construction, we reviewed Navy and shipbuilder documents, including program briefings, schedules, and contract status reports to assess the schedule and performance risks of the Columbia class program. To evaluate the shipbuilder’s progress in maturing the Columbia class design, we reviewed the Navy’s plans for design management and completion, evaluated the shipbuilder’s design schedule, and compared them against design progress reports to identify any delays. To evaluate the Navy’s plans for advance construction, we analyzed metrics reported in Navy and shipbuilder documents, briefing slides, and other documentation including key dates and estimated construction plans. We compared design knowledge on the Columbia class program to our prior work on shipbuilding best practices. We reviewed ongoing development efforts and schedules for the Columbia class program’s critical technologies to determine remaining risks to their development and integration. We also reviewed the matrices submitted by the Navy to Congress in February and October 2018, to determine the status of the program and identify any changes to the Navy’s design and construction goals for the program since our last report in December 2017. We also analyzed available documentation related to the status of the nuclear reactor and integrated power system. We reviewed the shipbuilder’s construction plans for its new facilities and its workforce hiring plans. We also reviewed the shipbuilder’s and Navy’s process for evaluating its suppliers. To corroborate documentary evidence and gather additional information in support of our review, we met with officials from the Navy’s Columbia class submarine program office; Naval Nuclear Propulsion Directorate; Naval Surface Warfare Center Philadelphia; Office of the Chief of Naval Operations; Supervisor of Shipbuilding, Groton; the Office of the Deputy Assistant Secretary of Defense for Systems Engineering; and the Office of Undersecretary of Defense for Acquisition and Sustainment. Additionally, we met with shipbuilding representatives from General Dynamics Electric Boat—the prime contractor—as well as their main subcontractor, Huntington Ingalls Industries Newport News Shipbuilding to understand their role in Columbia class design and construction. To assess the reliability of the Navy’s cost estimate for the Columbia class submarine program, we determined the extent to which the estimate met best practices as identified in GAO’s Cost Estimating and Assessment Guide. We examined cost estimate documentation, such as the Columbia class program life-cycle cost estimate, briefs, memoranda, and other documents that contain cost, schedule, and risk information. We also examined the independent cost estimate conducted by the Office of the Secretary of Defense’s Office of Cost Assessment and Program Evaluation (CAPE), the independent cost assessment conducted by the Naval Center for Cost Analysis (NCCA), and the cost estimate conducted by the Congressional Budget Office, to determine what methodologies and assumptions differed from the program cost estimate. We met with Navy officials who were responsible for developing the cost estimate to understand the processes used by the cost estimators, to clarify information, and to allow the Navy to provide additional documentation on the data and methodologies used in the estimate. We also observed portions of the Columbia class program’s cost model during a presentation and discussion with Navy cost estimators. We also reviewed the matrices submitted by the Navy to Congress to identify any changes to the Navy’s cost goals and reported information. To further corroborate documentary evidence and gather additional information in support of our review, we conducted interviews with relevant DOD and Navy officials responsible for developing, updating, and assessing the Columbia class program cost estimate, including CAPE; NCCA; the Naval Sea Systems Command’s (NAVSEA) Cost Engineering and Industrial Analysis Group; and the Columbia class program office. To evaluate how the Navy is implementing the Fund and associated authorities to construct Columbia class submarines, we reviewed the legislation establishing and modifying the Fund, program budget request documents, and DOD reprogramming approvals. We also reviewed the Navy’s basis of estimate for the savings it plans to achieve from these authorities. To further corroborate documentary evidence and gather additional information in support of our review, we met with officials from the Office of the Assistant Secretary of the Navy for Financial Management and Comptroller; Office of the Under Secretary of Defense (Comptroller); and the Columbia class program office to discuss the Navy’s plans to use and execute the Fund and DOD’s role in approving transfers into the Fund. The performance audit upon which this report is based was conducted from December 2017 to March 2019 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives. We subsequently worked with DOD from February 2019 to April 2019 to prepare this unclassified version of the original sensitive report for public release. This public version was also prepared in accordance with these standards. Appendix II: Technology Readiness Levels Appendix II: Technology Readiness Levels Description Lowest level of technology readiness. Scientific research begins to be translated into applied research and development. Examples might include paper studies of a technology’s basic properties. Invention begins. Once basic principles are observed, practical applications can be invented. The application is speculative and there is no proof or detailed analysis to support the assumption. Examples are still limited to paper studies. Active research and development is initiated. This includes analytical studies and laboratory studies to physically validate analytical predictions of separate elements of the technology. Examples include components that are not yet integrated or representative. Basic technological components are integrated to establish that the pieces will work together. This is relatively “low fidelity” compared to the eventual system. Examples include integration of “ad hoc” hardware in a laboratory. Fidelity of breadboard technology increases significantly. The basic technological components are integrated with reasonably realistic supporting elements so that the technology can be tested in a simulated environment. Examples include “high-fidelity” laboratory integration of components. Representative model or prototype system, which is well beyond the breadboard tested for TRL 5, is tested in a relevant environment. Represents a major step up in a technology’s demonstrated readiness. Examples include testing a prototype in a high-fidelity laboratory environment or in a simulated realistic environment. Prototype near or at planned operational system. Represents a major step up from TRL 6, requiring the demonstration of an actual system prototype in a realistic environment, such as an aircraft, vehicle, or space. Examples include testing the prototype in a test bed aircraft. Technology has been proven to work in its final form and under expected conditions. In almost all cases, this TRL represents the end of the true system development. Examples include developmental test and evaluation of the system in its intended weapon system to determine if it meets design specifications. Actual application of the technology in its final form and under mission conditions, such as those encountered in operational test and evaluations. In almost all cases, this is the end of the last “bug fixing” aspects of true system development. Examples include using the system under operational mission conditions. Appendix III: GAO’s Assessment of the Reliability of the Navy’s Cost Estimate for the Columbia Class Submarine Program To assess the reliability of the Navy’s cost estimate, we determined the extent to which the estimate was consistent with cost estimating best practices as identified in GAO’s Cost Estimating and Assessment Guide. This guide groups the best practices into four general characteristics: well documented, comprehensive, accurate, and credible. We reviewed documentation the Navy submitted for its cost estimate including limited portions of the Navy’s cost model, conducted numerous interviews, and reviewed relevant sources. We determined that the Columbia class cost estimate substantially met one, and partially met three of the four characteristics of a reliable cost estimate, shown in figure 16. We determined the overall assessment rating by assigning each individual rating a number: Not Met = 1, Minimally Met = 2, Partially Met = 3, Substantially Met = 4, and Met = 5. Then, we calculated the average of the individual assessment ratings to determine the overall rating for each of the four characteristics as follows: Not Met = 1.0 to 1.4, Minimally Met = 1.5 to 2.4, Partially Met = 2.5 to 3.4, Substantially Met = 3.5 to 4.4, and Met = 4.5 to 5.0. We consider a cost estimate to be reliable if the overall assessment ratings for each of the four characteristics are substantially or fully met. If any of the characteristics are not met, minimally met, or partially met, then the cost estimate does not fully reflect the characteristics of a high-quality estimate and is not considered reliable. Appendix IV: Comments from the Department of Defense GAO Comments In addition to responding to our recommendations, DOD also provided observations on a number of issues related to our assessment of the cost estimate, advance construction, and technology development. Our response to DOD’s observations is as follows. Assessment of Columbia Class Program’s Cost Estimate In paragraph 4, page 1 of the letter above, the Navy did not agree with our assessment of the accuracy of the cost estimate and stated that the life cycle cost estimate includes accurate calculations, proper inflation tables, and updates to requirements. DOD also stated that GAO or other stakeholders did not identify any errors. This is incorrect. While the Navy allowed us to observe the model, we did not independently check the accuracy of the calculations because Navy officials stated that the cost model, which contains the cost calculations, could not be released. We informed the Navy that this would affect parts of our assessment. After we provided a draft of the report, the Navy provided a briefing summarizing the results of a program office cost checkpoint conducted in September 2018. At the briefing, we received information on updates that the Navy made to the program cost estimate. As a result, we updated our assessment to reflect that the Navy substantially met the best practice to regularly update the cost estimate to reflect significant changes. However, the additional information provided by the Navy did not change our assessment of the accuracy and, therefore, our overall assessment of the Columbia cost estimate remains valid. In paragraph 1, page 2, the Navy did not agree with our assessment of the credibility of the cost estimate and stated that the life cycle cost estimate includes analyses that address sensitivity, risks, and uncertainty within the estimate. As we point out in the report, the estimate is based, in part, on optimistic assumptions regarding the number of labor hours needed to construct Columbia class submarines. The Navy has made updates to the program cost estimate based on a 2018 checkpoint review and stated that the cost risk analysis has been updated and program costs are less than originally estimated. The Navy provided us with a high-level brief of these updates. However, due to the timing of this report, we were not able to fully assess the update to the cost model. Given the size and complexity of the Columbia class program, we continue to believe that the program’s cost estimate does not adequately account for program risks. In paragraph 3, page 1, DOD stated that our findings were largely informed by an assessment conducted by the Naval Center for Cost Analysis (NCCA). However, our process for assessing program cost estimates is based on the extent to which the estimate met best practices outlined in GAO’s Cost Estimating and Assessment Guide. In conducting our assessment, we examined multiple sources of information, including the Columbia class program life cycle cost estimate, NCCA’s independent cost assessment, DOD’s Office for Cost Assessment and Program Evaluation’s (CAPE) independent cost estimate, and the cost estimate conducted by the Congressional Budget Office (CBO), to determine what methodologies and assumptions differed from the program cost estimate. We also relied on prior experience examining and reporting on the cost performance of Navy shipbuilding programs, issuing 26 reports over the past 10 years. We found, for example, that the cost estimate is based on optimistic labor assumptions which, while in agreement with NCCA’s assessment and CBO’s estimate, results from our independent assessment of the evidence we reviewed and on our prior work. Advance Construction In paragraph 2, page 2, the Navy stated that it identified super modules and selected components where acceleration would reduce construction schedule risk. We acknowledge in the report that the design for these components will be complete prior to starting construction. However, we continue to believe that starting construction for components of the lead submarine before the arrangements for the submarine are complete increases design and construction risk. Even if the components included in advance construction are fully designed, risk remains for the adjoining and interfacing components within the module that may have ongoing design work, potentially requiring costly and time-intensive rework. Technology Development In paragraph 4, page 2, the Navy notes that fully maturing all of the key technologies identified in our 2017 report—such as the advanced propulsor bearing—would require substantial investments in money and time. However, we continue to reinforce that a tenet of achieving design maturity is based on demonstrating a prototype in its final form, fit, and function in a realistic environment—which requires a design resembling the final configuration. Integrated Power System Motor Manufacturing Delays In paragraph 6, page 2, the Navy stated that it does not agree with our characterization that the Navy is continuing to experience manufacturing problems with the electric drive of the integrated power system. DOD stated that while the vendor experienced delays in manufacturing the prototype motor, it has taken proactive measures to deliver the motor to the shipyard, as scheduled. However, the Navy’s plan to concurrently test and finalize the design increases risk that any issues identified in testing could delay the delivery of the system to the shipyard. As a result, we continue to identify this as a key risk to the program. Additional details on this system are classified. Appendix V: GAO Contact and Staff Acknowledgments GAO Contact Shelby S. Oakley, (202) 512-4841 or oakleys@gao.gov. Staff Acknowledgments In addition to the contact above, the following staff members made key contributions to this report: Diana Moldafsky, Assistant Director; Laura Jezewski; Jessica Karnis; and Nathaniel Vaught. Other contributions were made by Brian Bothwell; Daniel Glickstein; Kurt Gurka; Stephanie Gustafson; and Robin Wilson.