Background DOD started the F-35 program in 2001 to develop a fifth-generation fighter aircraft intended to replace a range of aging aircraft in the U.S. military services’ inventories and to provide enhanced capabilities to warfighters that capitalized on technological innovations. Among other capabilities, the program designed the F-35 aircraft to be difficult to observe using radar and include sensors that can provide insights into potential targets and other warfighting information. The program is producing and delivering three variants of the F-35 aircraft: the F-35A conventional takeoff and landing variant for the Air Force, the F-35B short takeoff and vertical landing variant for the Marine Corps, and the F-35C carrier-suitable variant for the Marine Corps and the Navy. The characteristics of the services’ variants are similar, but each variant also has unique operating requirements. For example, the Marine Corps requires that the F-35B be capable of operating from aircraft carriers, amphibious ships, and main and austere operating bases alike, requiring the capability to conduct short takeoffs and vertical landings. Figure 1 shows an F-35B exercising this capability. While DOD plans to purchase 2,470 aircraft for the U.S. services, the F- 35 program is acquiring more than just aircraft. The complete F-35 air system has eight elements, including training and maintenance systems. Figure 2 shows the eight elements that make up the entire F-35 Air System and how they each support the aircraft. For example, the program intends for the Automated Logistics Information System (ALIS) to provide the necessary logistics tools to F-35 program participants as they operate and sustain the F-35 aircraft. To do this, ALIS consists of multiple software applications designed to support different squadron activities, such as supply chain management, maintenance, training management, and mission planning. For the F-35 aircraft to have full capability, each element of the air system has to be developed and fielded in sync with the aircraft. However, we found in March 2020 that problems with ALIS still pose significant challenges to day-to-day F-35 operations. According to DOD, it plans to replace ALIS with a new system named the Operational Data Integrated Network (ODIN). Furthermore, DOD reports that it is currently developing a strategy for ODIN, which will include key tasks, milestones and schedule, risks and opportunities, governance structure, and cost estimates. We concluded that, as DOD proceeds with replacing ALIS with ODIN, it will be important for the department to carefully consider and assess the key technical and programmatic uncertainties that we reported in March 2020. These include how much of ALIS will be incorporated in ODIN and the extent to which DOD has access to the data it needs to play a more active role in the management of the system. These issues are complex, and will require significant direction and leadership to resolve. Further, we reported in March 2020 that the F-35 program office was not able to provide us with historic costs showing how much the department had spent on ALIS over the years. Also, because DOD had not answered key questions about the future of the system, such as the extent to which the re-design will incorporate current ALIS software, DOD has not been able to develop accurate cost estimates for the ALIS re-design. We recommended that DOD develop and implement a strategy for the re- design of ALIS. The strategy should be detailed enough to clearly identify and assess the goals, key risks or uncertainties, and costs of re- designing the system. DOD concurred with the recommendation. Status of F-35 Program Development and Costs as of April 2019 DOD began development of the F-35 aircraft in 2001 without adequate knowledge of its critical technologies or a solid design, as we reported in March 2005. DOD’s acquisition strategy also called for high levels of concurrency between development and production—building aircraft while continuing to refine the designs of key components—which runs counter to GAO’s leading practices for major defense acquisition programs. In our prior work, we identified the F-35 program’s lack of adequate knowledge and high levels of concurrency as the major drivers of the program’s eventual significant cost and schedule growth, among other performance shortfalls. Since 2001, the program has been rebaselined with new cost and schedule estimates three times. DOD initiated the most recent restructuring in 2010 when the program’s cost estimates for each aircraft exceeded critical thresholds established by statute—a condition known as a Nunn-McCurdy breach. DOD then established a new acquisition program baseline that increased the program’s cost estimates by $162.7 billion and extended delivery schedules 5-6 years into the future. This last revision is the current program baseline, reflecting the cost and schedule estimates to deliver the aircraft and systems and to meet the original program requirements. From 2018 to 2019, the total cost estimate of the F-35 acquisition program increased by $22 billion, from $406 billion to over $428 billion. This increase was partially due to the addition of the estimated Block 4 modernization costs. Block 4 includes efforts to enhance and add capabilities—beyond the F-35 baseline program—through hardware and software upgrades. In April 2019, the F-35 program estimated that Block 4 development and procurement costs would add $13.9 billion to the program’s total baseline cost. Beyond this Block 4 increase, the F-35 program baseline costs also increased by $8 billion over the program’s 2018 estimate. Table 1 outlines the program’s baseline costs, the Block 4 modernization costs, and the sum total of the baseline and Block 4 cost estimates since 2001. In addition to the acquisition costs above, the program estimates that the sustainment costs to operate and maintain the F-35 fleet for its planned 66-year life cycle are $1.2 trillion, bringing the total cost of the F-35 program to over $1.6 trillion. Status of Testing, Production, and Reliability and Maintainability as of December 2019 The F-35 program office, in coordination with the Director of Operational Test and Evaluation (DOT&E), received approval to conduct some preliminary operational testing in January 2018. This included weapons, cybersecurity, and cold weather testing, among other things. The program’s formal operational testing (conducted by DOT&E) started in December 2018 and was ongoing in 2019. The purpose of operational testing is to assess the effectiveness, suitability, survivability, lethality, and mission capability of the F-35, including the information systems and the air vehicle, in an operationally representative environment. Operational testing includes cybersecurity assessments, some of which the program has conducted. The program plans for the remaining testing to take place through at least September 2020, while the program continues to produce and deliver aircraft. Through 2019, F-35 program test officials had identified over 3,200 deficiencies. Deficiencies represent specific instances where the weapon system either does not meet requirements or where the safety, suitability, or effectiveness of the weapon system could be affected. The test officials categorize deficiencies according to their potential impact on the aircraft’s performance. Category 1 deficiencies are critical and could jeopardize safety, security, or another requirement. Category 2 deficiencies are those that could impede or constrain successful mission accomplishment. In June 2018, we recommended that the program resolve all critical deficiencies before making a full-rate production decision, in part, to reduce the potential for additional concurrency costs stemming from continuing to produce aircraft while testing was ongoing. DOD concurred with our recommendation and stated that it would resolve critical deficiencies before full-rate production, currently planned to occur between September 2020 and March 2021. Production of the aircraft began one year after testing started in 2007, while development was in its early stages. Due to the concurrency of testing and production, according to an F-35 program official, as many as 550 aircraft delivered through 2020 will need retrofits to fix deficiencies and design issues found during testing. The program refers to the cost of these fixes as its concurrency cost, which the program estimates at $1.4 billion; this estimate did not change with the program’s last update in 2019. Until operational testing is complete, there is a risk that the program may identify additional deficiencies. As a result, as we have previously reported, the concurrency costs of retrofitting delivered aircraft could increase. In our June 2018 report, we found that the program was not on track to meet its reliability and maintainability (R&M) performance targets. R&M targets indicate how much time the aircraft will be in maintenance rather than operations. We concluded that the program was missing a prime opportunity to infuse affordability into the aircraft’s future with better R&M performance. As a result, we recommended that the F-35 program office identify what steps it needed to take to ensure the F-35 meets R&M requirements and update the R&M Improvement Program with these steps. DOD concurred with the recommendations, noting that the F-35 program office would update the R&M Improvement Program with the steps needed to ensure continued progress towards its goals. In April 2019, we found that F-35 R&M performance had shown some small improvements but that the program could take more actions to meet the R&M targets. We made additional recommendations to the Secretary of Defense, with which DOD concurred and has taken some actions to implement. Currently, the Office of the Under Secretary of Defense for Acquisition and Sustainment (OUSD (A&S)) is the acquisition decision authority for the F-35 program, and would direct the F-35 program office to take any further actions. In 2019, the program’s R&M performance generally remained unchanged. However, measurable improvements in R&M can take time to manifest. For example, fielded aircraft must be modified and flown for many hours before the program can measure improvements. For details about the R&M performance, see appendix IV. Block 4 Modernization’s Development Approach As we have previously reported, even though operational testing of the baseline program remains ongoing, the F-35 program office has turned its attention to Block 4 modernization activities using a different development approach. DOD refers to this approach as Continuous Capability Development and Delivery (C2D2). This method is loosely based on the Agile software development process. With this approach, the program plans to deliver capabilities to the warfighter faster than it did during the baseline development program. For example, rather than take years to develop and deliver all the required capabilities to the warfighter, the program intends to incrementally develop, test, and deliver small groups of capabilities every 6 months. In January 2018, to transition from the baseline development program to its Block 4 activities, the F-35 program started using the C2D2 approach to develop and test software updates to address deficiencies identified during testing. The planned $13.9 billion Block 4 effort exceeds the statutory and regulatory thresholds for what constitutes a major defense acquisition program, and Block 4 is more expensive than many of the other major weapon acquisitions already in DOD’s portfolio. To provide better oversight into Block 4 activities, in 2016, we recommended that the Secretary of Defense hold a milestone B review—a critical point in an acquisition program leading to the engineering and manufacturing development phase—and manage it as a separate major defense acquisition program. DOD did not concur with our recommendation, and it continues to manage Block 4 within the larger F-35 program. We maintain that DOD should manage the Block 4 activities as a separate program. Operational Testing Delays Provide More Time to Address Deficiencies before Full-Rate Production Decision Completion of Operational Testing Delayed by 9 Months In 2019, the F-35 program conducted a majority of its planned operational testing but added 9 months to the schedule to complete the remaining tests. Specifically, as of February 2020, according to test officials, the program completed 156 flight tests. The program must still conduct four open-air flight tests, the remaining cybersecurity tests of the air vehicle and mission systems, and 64 simulated flight tests. The 9-month delay needed to complete testing, however, also provides additional time for the program to address our June 2018 recommendation that it resolve critical deficiencies before making its full-rate production decision, currently planned to occur between September 2020 and March 2021. Figure 3 shows the test schedule as of 2019, the delay to the schedule into 2020, and the remaining tests events planned. The completion of operational testing hinges on three main tasks: (1) the final four open-air flight tests; (2) cybersecurity testing; and (3) the final development, integration, verification and validation of its simulator and 64 simulated flight tests. First, the program expects to complete the four remaining open-air tests between March and April 2020. To conduct these tests, the program must finish moving the Radar Signal Emulators—test assets that simulate long- range threat radars—from the Nevada Test and Training Range to the Point Mugu Sea Range in California. According to test officials, there is some risk with this move, such as damage to the sensitive test equipment. The test facilities will have to integrate the equipment into the testing infrastructure at Point Mugu. Second, while the program has conducted cybersecurity testing on several aspects of the F-35 aircraft and support systems, three air vehicle subsystems tests and two enterprise-level ALIS tests remain. The program expects to complete these by August 2020.The tests completed to date have identified multiple cybersecurity vulnerabilities. The F-35 program office has taken steps to address some identified vulnerabilities and is working to address the remainder. Test officials stated that some of the delays to cybersecurity testing of the aircraft are due to safety concerns and the risk of losing the use of a test aircraft before testing is complete. According to DOD policy, cybersecurity testing should be conducted as early in the operational test cycle as possible. Leaving this critical testing to the end of operational testing adds risk to the program because the program will not know the extent to which the aircraft may have cybersecurity vulnerabilities until near the expected decision to proceed to full-rate production. If the program cannot finish these tests by September 2020, officials stated that DOT&E could require that the cybersecurity testing be completed in follow-on testing and not hold up the full-rate production decision. Any additional cybersecurity vulnerabilities may require more time to develop and implement plans to address vulnerabilities in aircraft that have already been produced and those slated for production. Lastly, the program has not been able to complete the F-35 Joint Simulation Environment, which we refer to as the aircraft simulator, on time. The simulator runs the F-35’s mission systems software along with other software models (such as other weapons and modern threat systems) to provide complex test scenarios that the program cannot replicate in a real-world environment. We reported in April 2019 that the simulator’s development was behind schedule and was a risk to operational testing. Since then, the program has struggled to develop the complex software and functionality needed to complete the simulator. The difficulties stem, in part, from the program office’s original plan to have the contractor, Lockheed Martin, develop the simulator. However, in August 2017, program office officials decided that the contractor’s proposal was considered to be too expensive. To mitigate concerns over the cost of the proposal, the program decided to have the Navy complete the work. The program originally expected the Navy’s simulator to be ready for testing in 2017, but it is now 3 years behind schedule. According to program office officials, the simulator’s development effort has taken longer than expected to integrate F-35 aircraft and sensor data, in part because the contractor claimed the data as its own intellectual property. These issues were resolved by 2019 when the contractor provided the necessary data. Because of these delays, the program now expects that the simulator will be ready by August 2020, with the planned simulator testing expected to take about 3 weeks. According to test officials, there is increased risk that the completion of the simulator may face additional delays to correct deficiencies and add needed capabilities, but also stated that they can complete the tests by August 2020. Due to these delays to completing operational testing, the program has delayed its full-rate production decision by at least an additional 9 months. Though the program is working toward September 2020, the program has acknowledged this decision could be made as late as March 2021. Any additional delays due to challenges with moving the emulators, completing the simulator, or cybersecurity testing could further delay the end of operational testing and the program’s decision to enter into full-rate production. This delay, however, gives the program more time to complete two key steps consistent with statute and DOD policy. Complete operational testing, which is intended to demonstrate that the aircraft are operationally suitable. Resolve all deficiencies, which should be done prior to full-rate production, and is discussed below. Even with these delays, the program plans to have produced and delivered over 550 aircraft before operational testing is complete, adding to the risk of finding more deficiencies that will require retrofits—at additional cost—for the delivered aircraft. Statute and DOD policy states that the preliminary low-rate production quantities will be set at the development request for proposal decision point. If, at that time, low-rate initial production quantities are determined to be above 10 percent of the total quantity planned, the Secretary of Defense must explain the reasons for the increase in a report to Congress. When a program reaches the planned low-rate initial production quantity, and requires to exceed the quantity, the program may seek approval to produce quantities above that amount. The F-35 program will have delivered more than 10 percent of the total planned production quantities—due to the necessity to prevent a break in production—before operational testing and the full-rate production decision are complete. As noted above, this approach has contributed to the $1.4 billion in concurrency costs already incurred by the program. The program reports that none of the category 1 deficiencies is a safety of flight concern and all of them have operational workarounds. In 2019, the program split the category 1 deficiencies into two groups. Group A are deficiencies that may cause death, severe injury, severe occupational illness, or major loss or damage to equipment and has no workaround. The program has none of these deficiencies currently. Group B are deficiencies that may critically restrict the combat readiness capabilities or may result in adequate performance but not be able to accomplish the primary or alternate missions. All of the 9 category 1 deficiencies are in group B. F-35 Program Made Progress Resolving Deficiencies during 2019, but Many Remain Unresolved In 2019, the F-35 program resolved nearly 300 of the deficiencies it had identified in developmental and operational testing, but discovered even more over the same period. Specifically, 331 new deficiencies were identified in operational testing during 2019. As of December 2019, the F- 35 program had 870 open deficiencies. Of the 870 open deficiencies, the program characterizes nine as category 1 and 861 as category 2. The program reports that none of the category 1 deficiencies is a safety of flight concern and all of them have operational workarounds. In 2019, the program split the category 1 deficiencies into two groups. Group A are deficiencies that may cause death, severe injury, severe occupational illness, or major loss or damage to equipment and has no workaround. The program has none of these deficiencies currently. Group B are deficiencies that may critically restrict the combat readiness capabilities or may result in adequate performance but not be able to accomplish the primary or alternate missions. All of the 9 category 1 deficiencies are in group B. Of the 9 open category 1 deficiencies, the program reports all have operational workarounds—procedures that avoid encountering the deficiency. This represents four fewer open category 1 deficiencies than we reported in April 2019, reflecting the resolution of previously identified deficiencies and the addition of new ones, some of which were resolved. For example, the program fielded a software fix to a category 1 deficiency, which showed that the F-35’s cockpit display could falsely indicate its AIM-9X weapon—an air-to-air missile—selection status as “selected” though the weapon’s status is not selected. Figure 5 shows the F-35 firing an AIM-9X missile. The program office plans to continue to address the open deficiencies, but officials report that some will not be fully resolved for several years. Further, some deficiencies may not be resolved ever and some may be resolved well after the program has completed testing, and after it expects to have made a full-rate production decision. According to DOT&E, there are many significant deficiencies the program should address to ensure the F-35 baseline aircraft configuration is stable prior to adding all of the new capabilities planned in Block 4. As of December 2019, the program office and the contractor have resolved over 2,300 deficiencies and program office officials stated that they have a process in place to address the high priority ones. F-35 Aircraft Prices Decreased but Manufacturing and Supply Chain Risks Remain In 2019, the program reported continuing to negotiate lower unit prices across all F-35 aircraft variants and delivered more aircraft on time. However, officials also reported that the airframe and engine contractors demonstrated some declines in production performance, such as the number of labor hours to produce each aircraft, as production rates increased. We also identified other risk indicators that could affect the contractors’ future production performance. Specifically, the airframe contractor’s manufacturing processes do not meet all manufacturing leading practices that programs should meet before full-rate production. Additionally, parts shortages increased significantly in 2019 and Turkey’s suspension from the program will likely further complicate existing supply chain challenges. Aircraft Prices Decreased and More Aircraft Were Delivered on Time, but Other Measures of Production Performance Declined According to the program office, the negotiated prices for all F-35 variants have generally been decreasing with each production lot and as more aircraft are being procured in each lot. In April 2019, we reported that the program set a goal of reducing the negotiated unit price of an F-35A to less than $80 million by lot 13. According to a program official, in October 2019, the program finalized the contract action for lots 12-14 that met this goal. Specifically, with the most recent contract, the program agreed to purchase 351 F-35As, with unit costs declining to $73 million in lot 14. Figure 6 shows how the negotiated price for an F-35A has decreased since production began, as reported by the program office. According to the program office, it negotiated lower unit prices by working with the airframe contractor to leverage economic order quantity purchases and invest in cost reduction initiatives. Economic order quantities involve the contractor making large purchases of components that it will use across multiple procurement lots of aircraft to reduce production costs by achieving economies of scale. The program office estimates that the economic order quantity purchases for lots 12-14 will save the program about $225.5 million. In addition, the program office and prime contractors have continued to invest in various initiatives to lower production costs. Specifically, the program office spent $320 million in efforts to improve manufacturing processes that it estimates could result in up to $10.5 billion in savings over the life of the program. The airframe contractor told us that it has invested $170 million as of January 2019 to further lower its production costs. The engine contractor also told us that it spent $33 million to potentially realize over $4 billion of cost savings. Airframe Production Trends The airframe contractor—Lockheed Martin—delivered 43 more aircraft in 2019 than in 2018, and as of October 2019, there were 229 aircraft in various stages of assembly worldwide. The contractor also delivered more aircraft on time in 2019. According to contractor officials, the improved rates of on-time delivery are partially a result of the contractor’s efforts to obtain a performance incentive fee that was added to the lot 11 production contract. The program intended the incentive fee to focus the contractor on improving its performance in the final assembly phase of production, which was expected to improve its on-time deliveries. To earn the incentive fee, contractor representatives told us they took several steps to improve production rates. For example, because the F-35Cs were taking longer to produce and all variants had to move through the same final assembly area, the contractor made a separate final assembly line for the F-35Cs so work could proceed without delaying the other variants. This step, according to program office officials, allowed the contractor to improve on-time deliveries for F-35As and F-35Bs. Figure 7 highlights progress in the contractor’s aircraft deliveries since 2016. Other production metrics associated with the airframe, however, demonstrated varied performance over the last two years as production increased. For example, the average number of hours needed to build an aircraft decreased slightly for the F-35A but increased for the B and C variants. Defense Contract Management Agency officials told us the increase was partly attributable to new personnel. In particular, since January 2017, the contractor has hired and trained nearly 1,700 new personnel to accommodate increased production rates—nearly doubling its workforce. New personnel take time to train and gain experience on the production line. According to contractor representatives, as these new employees become more experienced and produce more aircraft, they expect the metric to improve. The contractor’s amount of rework needed was also mixed. During the course of production, the contractor may identify issues with a part or a process, which, in turn, may lead to scrap, rework, and repair to replace or fix the issue. Between 2016 and 2017, most F-35 variants realized improvements in the amount of scrap, rework, and repair needed. In 2018 and 2019, however, only the F-35A continued to show improvements. Figure 8 shows the average total hours for scrap, rework, and repair for each variant since 2016. According to the program office, the increased production rate posed a challenge, and because the contractor has not built as many F-35Cs, this has added to the increase in scrap, rework and repair. To improve performance in this regard, the contractor put teams in place to focus on addressing the main drivers of scrap, rework, and repair. Engine Production Trends Similarly, the engine contractor—Pratt & Whitney—increased its production rate by roughly 51 percent in 2019. However, engine on-time delivery performance has continued to decline which officials attribute to production quality issues and parts delays. Specifically, in 2019, 91 percent of engines delivered were late. In 2019, the airframe contractor was able to work around the late engine deliveries to deliver the entire aircraft on time. Figure 9 shows the engine contractor’s on-time and late deliveries since 2016. In addition, the average number of quality notifications per engine— production defects indicating a quality issue—has increased by 16 percent in 2019. Figure 10 highlights the engine contractor’s quality notifications per engine over the last 4 years. According to the Defense Contract Management Agency’s performance reports, engine test failures, among other quality issues, have affected engine deliveries. According to an official from this agency, there have been 18 engine test failures in 2019, which is eight more than in 2018, each requiring disassembly and rework. The engine contractor stopped deliveries due to the test failures, which has slowed engine acceptance and reduced on-time deliveries. These issues are affecting engines built at the engine contractor’s production facility in West Palm Beach, Florida, which opened in 2014. To address this issue, the engine contractor has developed new tooling for the assembly line and has established a team to identify characteristics leading to the test failures. Plans are also in place for additional training for employees. F-35 Program Has Not Met All Manufacturing Leading Practices, Indicating Risk to Future Production While F-35 aircraft have been in production since 2007 and have reached a high level of manufacturing readiness per DOD guidance, the program is not meeting two of eight manufacturing leading practices GAO has identified as indicators of a program’s readiness for full-rate production, or milestone C review. To date, the program is meeting or plans to meet six leading practices for this milestone: Demonstrating processes on a pilot production line. Building and testing production-representative prototypes to demonstrate product in intended environment. Collecting statistical process control data. Conducting an independent cost estimate. Conducting an independent program assessment. Conducting major milestone decision review to begin production. However, we also found that the production processes are not in control according to the Process Capability Index. This index is a tool to measure how closely the production steps result in a part or subsystem that meets predefined standards. According to the leading practices, meeting these standards provides greater confidence that the contractor can produce a high quality product consistently, to minimize variation which results in fewer defects or the need for rework. Additionally, the F- 35 aircraft have not achieved their reliability goals through testing of production representative prototypes. These two leading practices focus on gathering sufficient knowledge to determine the relative ease of manufacturing and whether the product is of high quality and sufficiently mature to move forward into full-rate production. Our analysis of contractor data shows that the airframe contractor’s production processes are in flux. The contractor continues to change some of its production processes, and in other cases, is not following its own established processes well, which has led to several quality issues over the years. For example, in 2018, we reported that the contractor had halted deliveries of aircraft after the Air Force identified corrosion between the aircraft’s surface panels and the airframe because the contractor did not apply a primer when it attached the panels. We reported in 2019 that the program office, the contractor, and the F-35 Program Executive Officer reached a mutual agreement on the cost to resolve this issue, the details of which they did not disclose publicly. In November 2019, a mechanic identified titanium fasteners installed in an area of the aircraft where the design calls for a fastener stronger than titanium. According to the program office, the incorrect fasteners were installed on most already- fielded F-35 aircraft. That same month, the contractor started implementing its corrective action plan. As of March 2020, the F-35 program office had reviewed and approved the contractors’ analysis as well as its durability and damage reports on the use of these fasteners. We describe other key F-35 technical risks in appendix V. Over the years, the airframe contractor has continued to change and refine production processes, aiming to improve efficiency amidst concurrent development and production. For example, the airframe contractor identified a particular process that installs wiring harnesses into the aircraft wings as a driver of one of its production quality issues. To address this issue, the prime contractor developed a new tool that helps the installer route the wires more consistently. While process changes like these can improve the quality of the product, they also indicate that the overall production process are not in control less than a year before the program’s planned full-rate production decision, or milestone C review. In 2019, according to our analysis, the total number of key F-35 manufacturing processes identified in the final assembly phase increased 70 percent, to a total of over 10,000 critical processes. Furthermore, of these critical processes, only 30 percent are currently able to produce a product within predefined design standards. According to manufacturing leading practices, critical processes should be repeatable, sustainable, and consistent in producing parts within quality standards. Meeting these practices provides confidence that the contractor can produce the product within cost, schedule, and quality targets. Without processes in control, the program could face continued quality issues that will add to the overall cost of the program. Figure 11 shows the F-35 aircraft in the final assembly phase of production where some of these processes take place. Another leading practice that should be met before making a full-rate production, or milestone C decision, is to demonstrate that a production representative prototype can meet the program’s R&M goals. The R&M goals lay out specific quantitative goals aimed at ensuring that an aircraft will be available for operations as opposed to out of service for maintenance. We reported in April 2019 that the F-35 aircraft in service around the world were still not meeting all of their R&M goals and recommended the program take actions to ensure that the aircraft would meet those goals. Despite some improvement in 2019, the program is not meeting half of its R&M goals. Until the program does so, the warfighter will continue to accept aircraft for delivery that are less reliable and more costly to maintain than originally planned. For details on the F- 35’s R&M performance, see appendix IV. The program has not met these two leading manufacturing practices, in part, due to the changes the airframe contractor made and continues to make to the production line and the program’s concurrent approach to acquisition. We have repeatedly found that DOD programs that moved into full-rate production carrying manufacturing risks experienced billions of dollars in cost growth in production, and nearly two-thirds reported increases in average procurement unit costs. With the risks the F-35 program still faces, it may realize additional cost and schedule growth if these production risks are not evaluated. Despite these risks, the program has continued to push forward with increased production rates and has not taken actions to determine the potential impact of not meeting these leading practices may have on future production and overall life-cycle costs. Furthermore, according to a program official, the F-35 program has not completed a comprehensive assessment of production risks and does not plan to ahead of its full-rate production decision. However, according to DOD officials, the F-35 program office and prime contractor convene a monthly Joint Risk Management Board, which identifies and manages overall program risk, and has completed an independent technical risk assessment to support the full-rate production decision, which identified production risks. Title 10 section 2366c of the U.S. Code requires the milestone decision authority for a major defense acquisition program to provide Congress with a report that includes, among other things, a summary of any manufacturing risks associated with the program; however, this summary is not required until 15 days after the authority grants approval for the program to enter the production and deployment phase. The program currently plans to obtain this approval between September 2020 and March 2021. In this case, however, the F-35 program has not met all of the manufacturing leading practices that should be met before the full-rate production decision. Furthermore, the underlying risks, such as not meeting R&M goals, have persisted for years and the program has yet to take steps to fully address these risks. If an evaluation of these risks is not provided ahead of the full-rate production decision, Congress will not be fully aware of the risks the program is taking by committing to increased production rates. F-35 Supply Chain Challenges Continue and May Be Exacerbated by Turkey’s Suspension from the Program According to program officials, some suppliers for the F-35 struggled to meet increased production demands in 2019 and, as a result, the program witnessed increased rates of late deliveries or parts shortages. In particular, the number of parts delivered late to the airframe contractor, as well as parts shortages, have grown steadily over the past 2 years. According to the Defense Contract Management Agency: Between August 2017 and July 2019, the number of parts delivered late increased from under 2,000 to more than 10,000. Between July 2018 and July 2019, the parts shortages per month increased from 875 to over 8,000. According to contractor representatives, roughly 60 percent of parts shortages are attributable to 20 suppliers. To mitigate late deliveries and parts shortages—and deliver more aircraft on time—the airframe contractor has utilized methods such as reconfiguring the assembly line and moving planned work between different stations along the assembly line. According to the program office, such steps can cause production to be less efficient, which, in turn, can increase the number of labor hours necessary to build each aircraft. Airframe contractor representatives and a program office official cited measures they are taking to improve supplier performance in light of the upcoming full-rate production decision. For example, the contractor instituted action plans to help problematic suppliers, sent task teams to struggling suppliers to help resolve issues, and, in some cases, is seeking alternative sources. Additionally, the program office has established joint meetings with the prime contractor to monitor progress on a weekly basis and holds a semiannual review to achieve executive-level coordination. While prime contractor representatives told us that they have been actively managing underperforming suppliers for several years, some of their efforts are new and will need time before results materialize. These supply chain risks may compound as the program continues to produce, deliver, operate, and maintain more aircraft each year. For example, in April 2019, we found that fielded, operational F-35 aircraft were not meeting warfighter requirements, largely due to spare parts shortages and difficulty in managing and moving parts around the world. We recommended that the program assess what actions it should take to meet warfighter requirements, which could include adjusting the amount of spare parts acquired. DOD concurred and is working toward addressing the recommendation to identify warfighter gaps with regard to the supply chain. However, with the aircraft in production also facing significant shortages, this problem could get worse as the program prepares to further increase the production rate from 141 aircraft in 2019 to 169 in 2022. We found that Turkey’s recent suspension from the F-35 program is likely to compound these existing supply chain issues. In July 2019, Turkey was suspended from the F-35 program. In particular, the Under Secretary of Defense for Acquisition and Sustainment directed that the F-35 program establish alternative sources and to stop placing orders from Turkish suppliers after March 2020. According to an official with that office, Turkish suppliers will provide parts through the end of lot 14 deliveries (scheduled to take place through 2022), in part, to avoid disruptions to aircraft deliveries and additional cost growth from standing up new suppliers. The F-35 program office identified that Turkish companies supplied 1,005 parts for the F-35 airframe and engine and some of these parts have been provided by only one supplier. As of December 2019, the program has identified new suppliers for all of these parts, but it still needs to bring roughly 15 parts currently produced in Turkey up to the current production rate. During our review, the program reported that production through lot 14 should not be adversely affected if it continues to accept parts from Turkey until lot 14 aircraft are delivered, but risk remains with the transition to alternate sources. However, lots 12-14 still face some risk receiving parts from Turkey. According to program officials, some of these new parts suppliers will not be producing at the rate required until next year, as roughly 10 percent are new to the F-35 program. Airframe contractor representatives stated it would take over a year to stand up these new suppliers, with lead times dependent on several factors, such as part complexity, quantity, and the supplier’s production maturity. In addition, these new suppliers are required to go through qualification and testing to ensure the design integrity for their parts. According to an official with the Under Secretary of Defense for Acquisition and Sustainment, by accepting parts from Turkish suppliers through lot 14, the program will have additional time to ensure new suppliers can meet demands for parts. Additionally, the program reported that it intends to utilize alternative sources for parts currently made in Turkey for aircraft delivered under lots 13 and 14 contracts. Furthermore, according to a program office official, it is also not clear how the prices for parts that will be obtained from new suppliers after Lot 14 will compare with the prices under the contracts with the suppliers from Turkey, but the official noted that alternative sources could be more costly. Block 4 Reporting Requirement Expires Before Completion Date, and Cost Estimate Does Not Fully Reflect Leading Practices In its May 2019 report to Congress, DOD outlined its plans for Block 4 with a development cost estimate of $10.6 billion for activities through fiscal year 2024. Since the 2019 report, we found the program office has increased its estimate by about 14 percent, to $12.1 billion, primarily due to schedule delays. The program now expects to extend the delivery of Block 4 capabilities by 2 additional years, through 2026. In the meantime, DOD’s Block 4 annual reporting requirement to Congress is scheduled to end in 2023, 3 years before development is complete. Additionally, most of the capabilities the F-35 program planned to deliver in 2019 were delayed. Furthermore, we found that the program’s cost estimate used to support its report to Congress does not fully meet cost estimating leading practices. Block 4 Modernization Will Not Be Completed before Reporting Requirements Expire, in Light of Schedule Delays The Block 4 development cost and schedule have grown considerably since DOD’s last report to Congress. In 2016, GAO recommended that DOD manage Block 4 as a major defense acquisition program with its own reporting requirements, separate from the original F-35 development program. DOD did not concur with our recommendation, citing the F-35 as DOD’s most closely managed system and its existing F-35 program oversight. The NDAA for Fiscal Year 2017 required DOD to report annually on elements of a Block 4 baseline, such as development and retrofit cost estimates, beginning no later than one year after the award of the development contract for follow on modernization, until March 31, 2023. At that time, we reported that DOD had requested funding for the development and delivery of Block 4 through the end of 2022. However, over the last year, the program has revised its Block 4 schedule and now expects to field Block 4 capabilities into fiscal year 2026. As a result, there is no requirement for DOD to report on Block 4 progress for at least 3 years even though those efforts will be ongoing. In its May 2019 Block 4 report to Congress, DOD reported that the total cost to develop 66 Block 4 capabilities—both hardware and software— would be $10.6 billion for activities planned from fiscal years 2018 to 2024. The report also included the F-35 program office estimate of an additional $6.4 billion in fiscal year 2018 through 2024 funding to retrofit aircraft from the baseline F-35 configuration to a full Block 4 configuration. The F-35 program based the costs in this report on its Block 4 development cost estimate from July 2018. However, we found that reported Block 4 costs did not include all Block 4 costs. In particular, the report did not include Block 4 costs the program incurred prior to 2018 or costs that the effort will incur after 2024. Because the F-35 program office is not managing the Block 4 effort as a separate program, it has chosen to exclude the past and future costs in the Block 4 cost estimate it reported to Congress. Instead, the program reported on Block 4 costs for the future years defense program—which is DOD’s projected spending for the current budget year and the next four years. By excluding any costs prior to 2018 and those that would be incurred after 2024, the program did not report on the total costs of Block 4. In May 2019, the program also updated its Block 4 development cost estimate, increasing both the time and cost to complete the work, but this updated estimate was not included in its May 2019 report to Congress. The updated cost estimate reflects that the program office will be fielding Block 4 capabilities into fiscal year 2026. This new schedule adds 2 years to the costs DOD reported to Congress in May 2019. Additionally, our analysis of DOD’s updated cost estimate indicates the total cost of Block 4 development grew by $1.5 billion to a total of $12.1 billion for activities in fiscal years 2018 through 2026. Furthermore, in addition to the Block 4 development costs, the program also estimates it will need another $2.9 billion to develop other capabilities, such as upgrades to ALIS. Program officials attributed this schedule and cost growth to having better insight into the scope of work to develop and test Block 4 capabilities and noted that they would continue to refine and update these costs annually as modernization efforts progress further into development. Once the existing statutory reporting requirement expires in 2023, DOD will no longer be required to provide Congress key information that would be useful in making informed decisions regarding the Block 4 effort— which now extends until 2026. Furthermore, without a complete cost estimate for Block 4, inclusive of costs already incurred and those not yet incurred but estimated through completion, Congress is left without a complete picture of what DOD intends to spend on the total Block 4 effort. Without a complete picture of these costs, the Congress’s ability to assess the program’s cost and schedule performance in the future will be hindered. Program Office Delayed Delivery of Most 2019 Block 4 Capabilities to the 2020s The airframe contractor did not deliver the Block 4 capabilities it planned to deliver in 2019. Specifically, according to the plan outlined in its May 2019 report to Congress, the F-35 program was going to deliver eight Block 4 capabilities in 2019. However, the program delivered only one—a software capability called the auto ground-collision avoidance system. This capability enables the aircraft to perform an automatic recovery when it predicts that the aircraft will strike the ground. This was ahead of schedule as the program had originally planned to deliver this capability after 2019. According to program officials, the development of the other capabilities is taking longer than planned and, as a result, the program pushed their delivery schedule into 2020. Development and delivery of the capabilities within the Block 4 effort are complex, and the program does not consider development complete until the products for all elements of the F-35 air system are ready. In particular, full capability delivery occurs when the contractor delivers all of the software and hardware needed for all of the F-35 air system elements to support the planned capability. Program officials stated they are still working to put the processes in place to synchronize the delivery of the late capabilities for all of the F-35 air system elements. For example, the airframe contractor had planned to deliver a capability called the interim full motion video for the Marine Corps in 2019. The contractor developed the software needed, but it is late in developing the hardware needed for the software to operate and, as a result, the contractor did not deliver the capability in 2019 as planned. DOD test officials we met with at Edwards Air Force Base stated that in 2019, using the C2D2 approach, the contractor delivered other, partial Block 4 capabilities to be tested. However, test officials told us those capabilities were delivered later than expected. Since the program could not fully test those capabilities on the aircraft, the program office deferred them to the next incremental update scheduled for 2020. Changes such as these have contributed to the Block 4 cost and schedule growth. The program is also discovering issues during Block 4 testing, causing the testing to take longer than anticipated. According to a DOT&E official, Block 4 software changes caused issues with functionality of F-35 baseline aircraft capabilities that worked before the program installed new Block 4 software onto the aircraft. The program discovered issues with each new software version during flight testing and has been working to fix these issues in subsequent software updates. Testing and DOD officials stated that the contractor had not performed adequate testing of the software before delivering it to the test fleet as the reason for these issues. Contractor representatives acknowledged these issues and stated that they will conduct additional lab testing for future software releases to avoid such problems going forward. Block 4 Development Cost Estimate Does Not Fully Meet Leading Practices, Which Limits Congress’ Understanding of Costs We found that the F-35 program office’s Block 4 cost estimate did not fully meet the four key characteristics of GAO’s cost estimating leading practices when projecting Block 4 development costs. Table 2 presents key points from our assessment, and appendix II provides additional detail on our rationale. As reflected in table 2, our assessment of the F-35 Block 4 development cost estimate identified a number of missing elements. Specifically, the estimate does not rely on a product-oriented work breakdown structure (WBS), it does not address cost risk and uncertainty, it does not take into account risk related to technology maturity, and it does not have an independent cost estimate, as leading practices reflect. While the program office updates its cost estimate regularly, officials told us that they do not intend to address some of these missing elements in future updates. Work breakdown structure. According to cost estimating leading practices, the program should base its cost estimate on a program- level, product-oriented WBS that allows a program to track cost and schedule by defined deliverables, such as hardware or software components. The WBS ensures that the program does not leave out any portions of the work and makes it easier to compare it to similar systems and programs. According to program officials, the Block 4 cost estimate does not rely on a single WBS; rather, multiple, contractor-derived WBSs exist for the program. Without its own, program-office-level WBS, the program lacks a framework to develop a schedule and cost plan that it can use to track progress and accomplishments. Risk and uncertainty analyses. The program did not perform cost risk and uncertainty analyses. Program officials said they do not plan to conduct a formal risk analysis. The program office works jointly with the contractor to identify and manage risks for the F-35 program. For example, there are monthly Joint Risk Management Boards attended by both program office and contractor leadership. However, overall program risk management is different from quantitative cost risk and uncertainty analyses in that program risk management is not specific to costs and it is not used to assess the cost variance of the cost estimate itself. When planning for funding decisions for a program of this scale, analyzing program-level risks alone is inadequate. Without a risk analysis, the cost estimate will not be fully accurate or credible because it will not account for the effects of potential schedule slips or other risks that the program could realize. Technology maturity. A program office official stated that in developing the cost estimate they did not consider that technologies would not be mature, but rather assumed that most technologies needed to deliver each Block 4 capability would be mature before the program begins development for that capability. The official stated that the complexity of design, development, and testing based on the baseline program experience was reflected in the estimate, but the cost estimate did not identify if there were specific costs associated with maturing these technologies. The official further noted that Block 4 costs would increase if a capability takes longer than planned to design, integrate, and test due to its immaturity. In 2019, we recommended that the Secretary of Defense ensure that the F-35 program office completes an independent technology readiness assessment, as part of its business case for the initial Block 4 capabilities, before initiating additional development work. DOD did not concur with our recommendation. According to a program official, as of December 2019, the program office had not completed any technology readiness assessments even though the contractor has started development of over half of the capabilities within Block 4. Going forward, the program is considering holding incremental technology readiness assessments as it plans for and develops a new set of capabilities, in accordance with the C2D2 schedule. Program officials told us that, going forward, as they update the Block 4 cost estimate, they will consider the results of future technology readiness assessments. Until the program office does so, management cannot determine a reasonable level of additional resources that might be necessary to cover increased costs resulting from unexpected design complexity, incomplete requirements, technology uncertainty, and other uncertainties. Independent cost estimate. In 2019, we also recommended the F- 35 program office include an independent cost estimate as part of its business case for Block 4. As noted in table 2, the Block 4 effort still lacks an independent cost estimate. The program is planning for the Office of the Secretary of Defense, Cost Assessment and Program Evaluation to have a draft independent cost estimate for an interim program review scheduled for March 2020 and to have a complete independent cost estimate in June 2020. This estimate will evaluate the entire F-35 program, including Block 4. With these pieces currently missing, the Block 4 development cost estimate does not present a full picture of Block 4’s cost. Ultimately, without a complete understanding of Block 4 costs, the program could face additional cost growth, which will be hard to track without a complete cost baseline. The lack of a complete cost baseline hinders insight and oversight into the program’s costs, plans, and progress to date and going forward. Moreover, if a cost estimate does not fully or substantially meet all four characteristics of cost estimating leading practices, it cannot be considered reliable. Conclusions DOD plans for the F-35 to be central to the warfighter prevailing in future conflicts. However, the program has been behind schedule and over cost almost since its inception. DOD is slated to move into full-rate production despite several key challenges in the production of aircraft. We acknowledge that the current F-35 program’s production rates are more commonly associated with programs already in full-rate production. However, the F-35 aircraft in the field have not met standards for reliability and maintainability, indicating that the program is not delivering aircraft at the level of quality expected. Additionally, the program’s concurrent approach and the contractor’s continual changes to the production line indicate that the production line processes are not in control. Leading practices indicate that mature production lines—production lines ready for full-rate production—should meet metrics for consistency. Furthermore, to minimize production risk and potential cost growth, suppliers should routinely meet quality and delivery schedules, although this is not yet true of the F-35 program. Not meeting these leading practices poses risks that DOD and the international partners will not routinely receive the F-35’s they specified and need. The long-standing challenges with receiving parts on time and efforts underway to replace Turkish suppliers of parts for the F-35 compound these production challenges and may raise additional risks. Unless the program office assesses and reports on these manufacturing risks ahead of the milestone C review, Congress may not have key insights into the risks that remain with the program and to the overall effort to deliver F-35s to the warfighter. Since the F-35 program is not managing the Block 4 effort as a separate program with traditional oversight tools, we are particularly concerned as Block 4 efforts proceed through development and testing. Specifically, because of the delays to the program, after 2023, DOD will not be required to provide Congress information on Block 4’s development efforts as the current reporting requirements will end. Furthermore, the program’s cost estimate, as presented in its report to Congress, does not fully present all incurred and future costs for Block 4. Without this information, Congress may not have the insight it needs to assess Block 4 cost and schedule progress as well as to make informed oversight and budgeting decisions. In addition, the Block 4 development cost estimate does not fully meet leading practices, lacking a full reflection of all costs. Specifically, the cost estimate does not have a program office level work breakdown structure, a risk and uncertainty analysis, and consideration of technology readiness. Without a comprehensive and credible cost estimate, DOD and Congress lack a sound basis for informed investment decision making, realistic budget formulation, meaningful progress measurement, proactive course correction when warranted, and program and contractor accountability for results. Matter for Congressional Consideration Congress should consider revising Section 224(d) of the National Defense Authorization Act for Fiscal Year 2017, Pub. L. No. 114-328, to extend DOD’s Block 4 reporting requirement until all Block 4 capabilities are fielded to ensure that Congress is aware of cost and schedule growth beyond 2023. (Matter for Consideration 1) Recommendations for Executive Action We are making the following five recommendations to the Secretary of Defense to direct the Undersecretary of Defense for Acquisition and Sustainment (OUSD (A&S)). The OUSD (A&S) should direct the F-35 program office to provide information that is similar to that which is statutorily required after the milestone C review to Congress ahead of the milestone C review (full-rate production decision). This submission should include an evaluation of the production risks associated with critical production processes that are not in control, reliability and maintainability (R&M) targets that are not met, and supplier readiness—particularly for those replacing Turkish suppliers, along with the steps it is taking to address those risks. (Recommendation 1) The OUSD (A&S) should direct the F-35 program office to establish a Block 4 cost estimate baseline that includes all Block 4 costs, including incurred costs and future costs in its reports to Congress as required by the NDAA for Fiscal Year 2017, so that Congress has a complete understanding of all Block 4 costs and can compare this baseline to future cost estimates and performance. (Recommendation 2) The OUSD (A&S) should direct the F-35 program office to complete a program office level, product-oriented work breakdown structure for the next update to its Block 4 cost estimate to ensure that the estimate meets the comprehensive leading practices. (Recommendation 3) The OUSD (A&S) should direct the F-35 program office to conduct risk and uncertainty analyses for the next update to its Block 4 cost estimate to ensure that the estimate meets the credible leading practices. (Recommendation 4) The OUSD (A&S) should direct the F-35 program office to consider the results of its future technology readiness assessment of all Block 4 technologies and incorporate the cost and schedule risks of developing those technologies in the next update to its Block 4 cost estimate to ensure that the estimate meets the comprehensive leading practices. (Recommendation 5) Agency Comments and Our Evaluation We provided a draft of this report to DOD for review and comment. DOD provided written comments, which we have reproduced in appendix VI. DOD concurred with three of the recommendations related to the Block 4 modernization effort (recommendations 2, 4, and 5 above). While DOD did not concur with the other two recommendations, it outlined planned actions that we believe, if implemented, would meet the intent of our recommendations. DOD also provided technical comments, which we incorporated as appropriate. We will continue to monitor the program and evaluate implementation of these recommendations. DOD officials did not concur with the first recommendation, which, in the draft report, was to evaluate production risks and provide a statutorily required report to Congress ahead of the program’s full-rate production decision. While DOD did not concur with the draft recommendation, it agreed to keep the Congress apprised of these matters in its quarterly briefings to the defense committees. To clarify the actions we intended DOD to take to address our findings, we revised the recommendation to indicate that DOD should provide information to Congress on the production risks we identified in our report, ahead of the milestone C review. If the DOD provides a substantive assessment highlighting these production risks, as well as the steps it will take to mitigate them, during its quarterly briefing to Congress ahead of the milestone C review, it would address the intent of our recommendation. DOD also did not concur with our third recommendation for the F-35 program office to complete a program-level, product-oriented work breakdown structure (WBS) for the next update to its Block 4 cost estimate. DOD noted that its next scheduled update was due in April 2020, after we provided our report for comment. While DOD noted it would be unable to complete a program-level WBS by the April 2020 update, it agreed to evaluate moving to a program-level, product-oriented WBS in 2021. If the F-35 program office utilizes a program-level, product- oriented WBS for this cost estimate update, it would meet the intent of our recommendation. We are sending copies of this report to the appropriate congressional committees; the Secretary of Defense; and the Under Secretary of Defense for Acquisition and Sustainment, the Secretary of the Air Force, the Acting Secretary of the Navy, and the Commandant of the Marine Corps. In addition, the report is available at no charge on the GAO website at http://www.gao.gov. If you or your staff have any questions about this report, please contact me at (202) 512-4841 or ludwigsonj@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are listed in appendix VII. Appendix II: F-35 Block 4 Development Cost Estimate Analysis To assess the reliability of the F-35 Block 4 development cost estimate, we obtained and reviewed cost estimate documentation such as the Joint Program Office briefing on its May 2019 estimate, the Air System Procurement Playbook—a planning document for Block 4—and its cost estimate models. Additionally, we met with relevant staff in the F-35 program office and the Department of Defense’s Office of Cost Assessment and Program Evaluation. We analyzed this information and determined the extent to which the program office’s practices for developing the F-35 Block 4 development cost estimate were consistent with the leading practices identified in the GAO Cost Estimating and Assessment Guide. These practices have been found to be the basis for reliable cost estimates. We assessed each practice as being one of the following: Met—the agency provided data and documentation that satisfies the entire leading practice criterion. Substantially met—the agency provided data and documentation that satisfies a large portion of the leading practice criterion. Partially met—the agency provided data and documentation that satisfies about half of the leading practice criterion. Minimally met—the agency provided data and documentation that satisfies a small portion of the leading practice criterion. Not met—the agency provided data and documentation that does not satisfy any portion of the leading practice criterion. For our reporting needs, we collapsed GAO’s 18 leading practices into four general characteristics: comprehensive, well-documented, accurate, and credible. The assessment of each characteristic was based on an average of the F-35 program office’s scores for the leading practices included in that category. A second analyst verified the assessment and management reviewed the results. We determined the overall assessment rating by assigning each individual rating a number: Not met = 1, Minimally met = 2, Partially met = 3, Substantially met = 4, and Met = 5. Then, we took the average of the individual assessment ratings to determine the overall rating for each of the four characteristics. The resulting average becomes the Overall Assessment as follows: Not met = 1.0 to 1.4, Minimally met = 1.5 to 2.4, Partially met = 2.5 to 3.4, Substantially met = 3.5 to 4.4, and Met = 4.5 to 5.0. A cost estimate is considered reliable if the overall assessment ratings for each of the four characteristics are substantially or fully met. If any of the characteristics are not met, minimally met, or partially met, then the cost estimate does not fully reflect the characteristics of a high-quality estimate and cannot be considered reliable. See table 4 for a high level summary of each leading practice and the reasons for the overall scoring. Appendix III: Objectives, Scope, and Methodology This report fulfills two mandates. First, the National Defense Authorization Act for fiscal year 2015 included a provision for GAO to review the F-35 acquisition program annually until the program reaches full-rate production. This is the fifth report under that provision. Second, the National Defense Authorization Act for Fiscal Year 2020 includes a provision for GAO to review the program’s production and Block 4 progress annually through 2025. In this report, we (1) provide information on the program’s progress toward completing operational testing and resolving deficiencies found in testing; (2) assess the program’s production performance and manufacturing efficiency initiatives; and (3) assess the program’s modernization cost estimate and progress with Block 4 development efforts. To provide information on the program’s progress in operational testing and the resolution of deficiencies, we first reviewed the baseline program’s costs, schedule, and performance plans and compared the actual progress in each area with the goals established in its 2012 baseline to identify any significant trends. We reviewed progress on test events completed versus those that remain, test schedules, program briefings, and DOD briefings. We traveled to Edwards Air Force base to interview DOD test authorities and met with officials from the program office, DOD test authorities, and the contractor Lockheed Martin (the prime aircraft contractor), to discuss key aspects of F-35 development progress, including flight testing, future test plans, and recent findings from test events. Specifically, we obtained updates on key events that are required to complete testing according to the program office’s current schedule. We also interviewed the Director, Operational Test and Evaluation office and F-35 program developmental and operational test pilots. To provide information on the program’s progress resolving deficiencies, we interviewed the same officials mentioned above and discussed how the number of open and closed deficiencies changed in 2019. We reviewed program and contractor information on deficiency reports, mitigations, resolutions, and the deficiency resolution process. To assess the program’s production performance and manufacturing efficiency initiatives, we obtained and analyzed the production metrics from Lockheed Martin and Pratt & Whitney (the prime engine contractor) and their aircraft and engine delivery rates from 2012 through 2019. We reviewed metrics and briefings provided by the program office, Lockheed Martin, Pratt & Whitney, and the Defense Contract Management Agency to identify progress in improving manufacturing processes. We analyzed delivery dates for lot 11 aircraft delivered in 2019. We traveled to the production facility in Fort Worth, Texas to discuss reasons for any delivery delays and plans for improvements with officials from Lockheed Martin. We obtained cost investment and savings estimates and discussed cost and manufacturing efficiency initiatives, such as the economic order quantity purchases, with the contractors and program office officials to understand potential cost savings and plans. We collected and analyzed the extent to which the program has met leading practices identified by GAO for full-rate production. We also obtained and analyzed metrics on parts and aircraft quality through December 2019 and discussed steps taken to improve quality and deliveries with Lockheed Martin and Pratt & Whitney officials. We interviewed officials from Lockheed Martin, Pratt & Whitney, and Northrop Grumman (a key subcontractor) regarding the administration’s decision to suspend Turkey from the program and the implications of the suspension for the contractors. We determined that the contractors’ production metrics and delivery dates were sufficiently reliable for our purposes of determining production efficiency and deliveries. We collected and analyzed production and supply chain performance data from the program office, Lockheed Martin, and Pratt & Whitney. To assess the reliability of the May 2019 Block 4 development cost estimate, we evaluated documentation supporting the estimate, such as the cost estimating models, the F-35’s Air System Procurement Playbook, its updated acquisition strategy, the Decision Memorandum requirements document, and briefings provided to the DOD decision authority. We assessed the cost estimating methodologies, assumptions, and results against leading practices for developing a comprehensive, accurate, well- documented, and credible cost estimate, identified in GAO’s Cost Estimating and Assessment Guide. We also interviewed program officials responsible for developing and reviewing the cost estimate to understand their methodology, data, and approach for developing the estimate. We found that the cost estimate was not reliable. To assess progress with Block 4 development efforts, we interviewed DOD and program office officials, and contractor representatives regarding the program’s Block 4 planning, development, testing, and production activities to date. We reviewed other program documentation, such as the F-35’s fiscal year 2020 budget request, to identify costs associated with the Block 4 effort. We compared the program’s accomplishments in 2019 to its plans and identified what capabilities the program office delivered to the fleet. We reviewed the program office’s plans to develop and deliver additional Block 4 capabilities from 2020 through 2025. We conducted this performance audit from June 2019 to May 2020 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives. Appendix IV: The F-35’s Reliability and Maintainability Metrics The Joint Strike Fighter Operational Requirements Document, which outlines the requirements Department of Defense and the military services agreed the F-35 should meet, defines all eight reliability and maintainability (R&M) metrics. Table 5 shows each F-35 variants’ performance against these metrics’ targets, as of August 2019. Appendix V: Status of Selected F-35 Technical Risks Newly Identified Technical Risks F-35B Thrust Cutback: An F-35B aircraft can experience an unanticipated cutback in thrust during vertical landings (hover). The contractor put hover weight restrictions in place to mitigate the effect and has identified the root cause. The contractor is developing software and hardware fixes. F-35C Nose Landing Gear: During shipboard landings, the F-35C can experience bending stress, which causes cracking of the coating on a part in the nose landing gear. In the short term, this part will be inspected for damage every 400 flying hours. The contractor is also redesigning the part that is cracking and expects to test it between early 2020 to June 2021. F-35B Three Bearing Swivel Module: The module is mounted at the back of the aircraft and allows the thrust from the engine to be vectored from straight aft for conventional flight to straight down for short takeoff and vertical landing operations. In June 2019, an F-35B experienced a warning indicator in its short takeoff mode due to a hardware component. However, according to the contractor, this component should not cause a warning indicator or loss of functionality for the aircraft. The contractor has identified the root cause of the hardware issue and a gap in the software’s logic that led to the warning. As a result, the contractor is making manufacturing changes to the hardware and implementing software changes to address the issue. Technical Risks Identified in Our Previous Reports Canopy Coating Delaminations: The F-35 fleet has experienced over 50 incidents of the canopy transparencies delaminating after less than 100 flight hours since August 2017. This is over 30 more than we reported in 2019. The contractor tested solutions for the delaminations in 2019 and implemented a solution of adding a vent to the canopy’s frame. Since October 2019, the contractor has added a vent to 146 canopies with one subsequent delamination. Helmet Mounted Display: During low-light flights, the Helmet Mounted Display’s technology cannot display pure black images, instead presenting a green glow on the screen, which makes it difficult to see the full resolution of the night vision video feed. The contractor developed a new display to avoid this effect. According to F-35 program officials, they placed an initial order of 62 displays with 35 delivered by December 2019 to support U.S. Marine Corps and Navy F-35C fleet operations. Three F- 35C pilots completed initial day and night testing using the new display in July 2019 on a carrier. The contractor expects to have a fully qualified redesign by August 2021 and will incorporate it into the production of lot 12 aircraft. Appendix VII: GAO Contact and Staff Acknowledgments GAO Contact Staff Acknowledgments In addition to the contact named above, the following staff members made key contributions to this report: Justin Jaynes and Alissa Czyz (Assistant Directors); Diana Maurer, Desirée E. Cunningham (Analyst-in- Charge), Jillena Roberts, Tim Moss, Rose Brister, Juaná Collymore, Emile Ettedgui, Jennifer Leotta, and Jeff Hubbard, Other staff who contributed include Leslie Ashton, Priyanka Sethi Bansal, Vinayak Balasubramanian, Julia DiPonio, Christine Pecora, Ralph Roffo, Roxanna Sun, Jessica Waselkow, and Alyssa Weir. Related GAO Products Weapon System Sustainment: DOD Needs a Strategy for Re-Designing the F-35’s Central Logistics System. GAO-20-316. Washington, D.C.: March 6, 2020. F-35 Aircraft Sustainment: DOD Faces Challenges in Sustaining a Growing Fleet. GAO-20-234T. Washington, D.C.: November 13, 2019. Defense Acquisitions: Observations on the F-35 and Air Force’s Advanced Battle Management System. GAO-19-456T: Washington, D.C.: May 2, 2019. F-35 Joint Strike Fighter: Action Needed to Improve Reliability and Prepare for Modernization Efforts. GAO-19-341. Washington, D.C.: April 29, 2019. F-35 Aircraft Sustainment: DOD Needs to Address Substantial Supply Chain Challenges. GAO-19-321. Washington, D.C.: April 25, 2019. F-35 Joint Strike Fighter: Development is Nearly Complete, but Deficiencies Found in Testing Need to Be Resolved. GAO-18-321. Washington, D.C.: June 5, 2018. Warfighter Support: DOD Needs to Share F-35 Operational Lessons Across the Military Services. GAO-18-464R. Washington, D.C.: April 25, 2018. F-35 Aircraft Sustainment: DOD Needs to Address Challenges Affecting Readiness and Cost Transparency. GAO-18-75. Washington, D.C.: October 26, 2017. F-35 Joint Strike Fighter: DOD’s Proposed Follow-on Modernization Acquisition Strategy Reflects an Incremental Approach Although Plans Are Not Yet Finalized. GAO-17-690R. Washington, D.C.: August 8, 2017. F-35 Joint Strike Fighter: DOD Needs to Complete Developmental Testing Before Making Significant New Investments. GAO-17-351. Washington, D.C.: April 24, 2017. F-35 Joint Strike Fighter: Continued Oversight Needed as Program Plans to Begin Development of New Capabilities. GAO-16-390. Washington, D.C.: April 14, 2016. F-35 Sustainment: DOD Needs a Plan to Address Risks Related to Its Central Logistics System. GAO-16-439. Washington, D.C.: April 14, 2016. F-35 Joint Strike Fighter: Preliminary Observations on Program Progress. GAO-16-489T. Washington, D.C.: March 23, 2016. F-35 Joint Strike Fighter: Assessment Needed to Address Affordability Challenges. GAO-15-364. Washington, D.C.: April 14, 2015. F-35 Sustainment: Need for Affordable Strategy, Greater Attention to Risks, and Improved Cost Estimates. GAO-14-778. Washington, D.C.: September 23, 2014. F-35 Joint Strike Fighter: Slower Than Expected Progress in Software Testing May Limit Initial Warfighting Capabilities. GAO-14-468T. Washington, D.C.: March 26, 2014. F-35 Joint Strike Fighter: Problems Completing Software Testing May Hinder Delivery of Expected Warfighting Capabilities. GAO-14-322. Washington, D.C.: March 24, 2014. F-35 Joint Strike Fighter: Restructuring Has Improved the Program, but Affordability Challenges and Other Risks Remain. GAO-13-690T. Washington, D.C.: June 19, 2013. F-35 Joint Strike Fighter: Current Outlook Is Improved, but Long-Term Affordability Is a Major Concern. GAO-13-309. Washington, D.C.: March 11, 2013. Fighter Aircraft: Better Cost Estimates Needed for Extending the Service Life of Selected F-16s and F/A-18s. GAO-13-51. Washington, D.C.: November 15, 2012.