Background F-35 Program The F-35 Lighting II program is a joint, multinational acquisition program intended to develop and field a family of next-generation strike fighter aircraft for the U.S. Air Force, Navy, and Marine Corps (hereinafter referred to as the services); seven international partners; and four foreign military sales customers (collectively hereinafter referred to as program participants). The program has developed and is delivering three variants of the F-35 aircraft: F-35A conventional takeoff and landing variant for the Air Force. (see fig. 1) F-35B short takeoff and vertical landing variant for the Marine Corps. F-35C carrier-suitable variant for the Navy. The characteristics of the services’ variants are similar in that each is intended to be a multi-role, stealthy strike aircraft, but each service’s variant also has unique operating requirements. For example, the Marine Corps requires that the F-35B be capable of operating from aircraft carriers, amphibious ships, and main and austere operating bases alike, requiring the ability to conduct short take offs and vertical landings. DOD initiated the F-35 program in October 2001. Since then, the Marine Corps and Air Force declared initial operational capability in 2015 and 2016, respectively, while the Navy declared initial operational capability in February 2019. Operational testing of the F-35 aircraft began in December 2018 and is currently scheduled to be completed late 2020. At that time, DOD will make a decision on whether to proceed with plans to begin full-rate production of the aircraft. DOD has, concurrently, been fielding and operating a growing fleet of aircraft as part of low-rate initial production. As of October 2019, more than 435 U.S. and international aircraft had been fielded and were operating from 19 sites worldwide. By 2023, the global F-35 fleet is expected to expand to more than 1,100 aircraft across 43 operational sites. In total, the program participants plan to purchase more than 3,300 F-35 aircraft, with the U.S. services planning to purchase nearly 2,500 of those aircraft. See Figure 2 for a timeline of anticipated worldwide fleet growth in the F-35 program. DOD has two primary contractors for the F-35 program: Lockheed Martin for the overall aircraft system and Pratt & Whitney for the engine. As the prime contractor for the overall aircraft system, Lockheed Martin (hereinafter referred to as the prime contractor) is responsible for managing the F-35 supply chain, depot maintenance, and pilot and maintainer training, as well as for providing engineering and technical support. Currently, DOD is contracting for this support with the prime contractor largely through annual contracts. It plans to transition to multiple-year, fixed-price, performance-based sustainment contracts when the program achieves certain condition-based criteria, including the establishment of critical sustainment capabilities and the government’s ability to collect and more fully assess performance and cost data. In addition, the U.S. Air Force, Navy, and Marine Corps have each established an F-35 integration office or similar construct focused on how the services will operate and afford the F-35, among other things. Figure 3 depicts how these key stakeholders provide support to the F-35 program participants across the three aircraft variants. Autonomic Logistics Information System The Autonomic Logistics Information System (ALIS) is a system of systems that serves as the primary logistics tool to support F-35 operations, mission planning, and sustainment. ALIS is intended to help maintainers manage tasks including aircraft health and diagnostics, supply-chain management, and other maintenance events. ALIS functionality is intended to support many of the F-35 program’s key performance parameters such as: Increase sortie generation rate: Number of aircraft sorties launched in a flight day. Increase mission reliability: The probability that a system will perform mission essential functions for a period of time. Reduce logistics footprint: The size of in-theater logistics support needed to move and sustain a warfighting force. The footprint includes all the necessary support needed to maintain the force such as fuels, parts, support equipment, transportation, and people. According to DOD officials, ALIS is integral to supporting F-35 operations. Figure 4 shows some of the key intended capabilities of ALIS. These capabilities reside in multiple software applications within the system that perform specific functions for maintainers, pilots, supply personnel, and data analysts. Lockheed Martin is the prime contractor for ALIS and has been responsible for developing and managing the capabilities of the system, as well as developing training materials for F-35 pilots, maintainers, and supply personnel. ALIS is co-located with F-35 aircraft both at U.S. military installations and in theater to support missions and assist with maintenance and resource allocation. ALIS consists of the overarching system, the applications housed within it, and the network infrastructure required to provide global integrated and autonomic support of the F-35 fleet. It comprises both hardware and software, and supports the flow of unclassified and classified aircraft-related data. As a system of systems, major components of ALIS consist of: The Autonomic Logistics Operating Unit (ALOU). The ALOU is the central computer unit that all F-35 data are sent through. As part of the unit, the ALOU consists of two servers that process and store classified and unclassified data respectively. There is only one ALOU, and it is owned by the prime contractor. The Central Point of Entry (CPE). The CPE is a server unit configured to provide software and data distribution for a country’s entire F-35 fleet. It is the node between the ALOU and each country’s Standard Operating Units (generally housed at F-35 installations). The CPE consists of two servers that process and store classified and unclassified data respectively. There is typically one operational CPE per country, although the United States has separate CPEs for its operational commands and training sites. The Standard Operating Unit (SOU). The SOU is a server that is intended to provide all ALIS capabilities to support flying, maintenance, and training at F-35 installations. Typically, each F-35 squadron has at least one SOU. It is the node local to each F-35 squadron. There are two types of SOUs: a classified SOU that supports the flow of classified aircraft-related data and an unclassified SOU that supports the flow of unclassified aircraft-related data. The Portable Memory Device (PMD). The PMD is informally referred to as the “brick” that F-35 pilots use to upload information such as mission planning data. F-35 personnel use the PMD to store mission and maintenance data generated during flight which may then be downloaded into the ALIS SOU to support maintenance and mission debrief activities. The Portable Memory Device Reader (PMD Reader). The PMD Reader is a device intended to be used to remove maintenance data, including health-related codes, off of the Portable Memory Device and load into the SOU. The Portable Maintenance Aid (PMA). The PMA is an unclassified ruggedized laptop used by F-35 maintainers and flight-line supervisors to view unclassified technical data, and perform and document maintenance activities. According to the F-35 program office, the purpose of the server construct is to support the exchange of information necessary to support the F-35 sustainment enterprise. As of September 2019, according to program officials, there was one operational ALOU and CPE within the United States. Each F-35 site in the United States has a varying number of SOUs depending on the site’s number of aircraft and squadrons. The SOU was designed to have its components fit into transit cases that can be carried by two personnel, with each case weighing up to 200 pounds. The PMDs, PMD Readers, and PMAs reside at the squadron and support the collection and transfer of unclassified and classified aircraft-related data. Figure 5 shows how unclassified ALIS data are collected and transferred from component to component. As we have previously reported, ALIS has experienced recurring developmental issues and schedule delays. The development of ALIS originated in 2002, a year after the start of the F-35 program. However, the first major ALIS release was not fielded until October 2009, nearly 7 years after initial development began. DOD officials had originally planned for the version of ALIS that would include all of the capabilities required to complete developmental testing of the program to be finalized in 2010. However, this milestone was reached in September 2018, nearly 8 years behind the original schedule. Figure 6 shows the timeline of major ALIS software version releases and other significant ALIS-related milestones. DOD Has Made Some Improvements to ALIS, but Users Continue to Report Significant Challenges ALIS users from all 5 F-35 locations we visited reported that ALIS has improved in some aspects over the last 5 years. However, these users continue to report significant challenges with ALIS that are affecting the day-to-day operations of the aircraft. DOD is currently unable to assess the overall performance of ALIS because it has not developed performance metrics. Additionally, DOD is unaware of how challenges with ALIS are affecting F-35 fleet-wide readiness. Users Report Some Improvements with ALIS According to pilots, maintainers, supply personnel, and contractors at 5 U.S. F-35 locations, ALIS is generally performing better than it was 5 years ago. Specifically, users at all 5 locations stated that data processing, downloading of information, and screen navigation were generally faster than previous years. According to users at 1 location, in previous releases of ALIS, it could take several minutes to complete a simple function like a screen download. Further, some users also reported minor functionality improvements within certain ALIS applications, such as the Computerized Maintenance Management System, leading to reduced time required to perform actions within those applications. We reported in April 2016 that ALIS users had problems accessing data in ALIS to produce service-specific reports for their squadrons. Users we spoke to at 4 locations for this report stated that they can now access some data within ALIS and can generate reports that they previously could not. For example, users at 1 location said that it was now easier to export aircraft-related maintenance information from ALIS and put it into an external spreadsheet. Additionally, in December 2015, the F-35 program began deploying software “fixes” to address minor defects in ALIS at F-35 locations in between major ALIS software version releases, which users at 1 location said have made improvements to the system. According to the F-35 program office, these software releases, referred to as service packs, have focused on improving user interface-related flaws that were discovered during major releases. Service packs provide users more frequent functionality fixes to the system, preventing them from having to wait, in most cases, over a year for a major ALIS software release. Users Continue to Report Significant Challenges Using ALIS While users at all 5 F-35 locations we visited said that ALIS is performing better than it was 5 years ago, they also stated that the system still posed significant challenges to day-to-day F-35 operations. Specifically, users across the 5 locations we visited stated that seven significant challenges still exist with ALIS, as shown in table 1. Many of the challenges cited above are similar to those we reported in April 2016, including deployability, inefficient issue resolution process, and data inaccuracies. We recommended at that time that DOD develop a plan to prioritize and address ALIS issues. DOD concurred and in 2016 developed a plan that identified key areas for system modernization and sustainment, which included prioritizing issues related to ALIS. While DOD’s development of this plan is a positive step, significant user issues persist today, which are discussed in more detail below. Continued attention on ALIS is needed to make improvements to the system, reduce the burden on its users, and mitigate risks to operations and maintenance. Inaccurate or Missing Data Users at all 5 F-35 locations we visited expressed concern about data integrity issues related to inaccurate or missing data within ALIS. For example, users at all the locations said they have had consistent problems with data related to aircraft parts. Certain F-35 parts have an associated electronic record, which is used to track the remaining time before the part must be replaced, among other things. To be cleared for flight, F-35 policy states that an aircraft must be electronically “complete” in ALIS, meaning that all of the electronic records from each installed F- 35 part must be entered into ALIS. However, users at all 5 of the locations we visited told us that electronic records are frequently incorrect, corrupt, or missing, resulting in ALIS signaling that the aircraft should be grounded, often in cases where maintainers know that the parts have been correctly installed and are safe for flight. Users at 1 location said that within a 6-month period in 2019, they experienced anywhere between 0 and 400 issues per week related to inaccurate or missing electronic records. These same users said that it is common for their squadron leadership to elect to allow an aircraft to fly with over 20 inaccurate or missing electronic records that ALIS signals to ground. According to users at all 5 locations we visited, squadron leadership (e.g., DOD personnel designated by maintenance squadron commanders) may decide to fly an aircraft with inaccurate or missing electronic records, but we found that this practice varies by location and type of part. In June 2019, the Department of Defense Inspector General published a report on missing electronic records on F-35 spare parts. The report found that since 2015, F-35 locations have been consistently receiving spare parts without requisite electronic records. For example, of the 263 spare parts delivered to one location in June 2018, 213 spare parts (81 percent) did not have electronic records. Due in part to the unreliability of the data in ALIS, users at all 5 F-35 locations we visited have been collecting and tracking information outside of the system that should be automatically captured in ALIS. Although not a requirement, users said they need to track information outside of the system because they do not always trust the data that reside in ALIS. Users provided examples of critical aircraft data that they are tracking outside of ALIS—such as aircraft performance data and maintenance inspection deadlines—and said that manually tracking this information is a time-intensive process that pulls maintainers away from completing other aircraft maintenance-related responsibilities. For example, users at 1 location estimated that they spend an average of 5,000 to 10,000 hours per year manually tracking information that should be automatically and accurately captured within ALIS. In addition, there may be risks associated with using information tracked outside of the system of record to make decisions about the safety and operational health of aircraft. For example, users at one location said that there is a danger of overlooking a critical piece of information when key aircraft data used to determine an aircraft’s status must be tracked manually using Excel spreadsheets. Users also said that by continuously ignoring alerts in ALIS caused by missing or inaccurate data, squadrons could be at risk of ignoring an alert for a legitimate aircraft issue. Finally, one commander we spoke with said that while his policy is to generally require maintainers to resolve data issues before releasing an aircraft for flight, in a wartime scenario, his squadron will carry out missions with inaccurate or missing ALIS data and assume the subsequent risk that this may entail. Challenges Deploying Users at all 5 F-35 locations we visited cited challenges deploying with ALIS to forward locations. Users stated that the required hardware for ALIS is bulky, can be cumbersome to transport, and, when necessary, difficult to store on a ship. For example, the unclassified and classified Standard Operating Unit (SOU) servers that are required for collecting and analyzing aircraft data in ALIS are broken up into a series of transportable cases. These cases each weigh approximately 200 pounds and require at least two people to lift. Users from 1 location told us that they have taken several separate SOU-related cases to support ALIS on deployments. These servers, as shown in figure 7, require dedicated transportation to transport them to forward locations, and heavy-duty equipment to load them on and off of ships. Some users stated that it was challenging to find space on the ship to store these servers since they typically require an entire room to function, as well as specific power and environmental controls. Additionally, users at all 5 locations stated that limited internet connectivity can make deployments challenging. Although SOU servers are critical ALIS hardware components, due to their size, squadrons will not always take them on deployments. In these instances, internet connectivity is important to access critical aircraft data from the forward location and send it back to the squadron’s SOU for processing. However, internet connectivity can be slow or non-existent at these locations. In 2018, we recommended that the F-35 Program Executive Officer should test operating the F-35 disconnected from ALIS for extended periods of time in a variety of scenarios to assess the risks related to operating and sustaining the aircraft. DOD concurred with the recommendation, but as of December 2019, DOD had still not determined how long the aircraft can safely fly without connectivity to ALIS. Finally, users at 2 locations stated that contractor support is critical to supporting deployments. For example, at one location, due to inaccuracies with parts data in ALIS, the prime contractor prefers to match every requisite electronic record with its respective spare part prior to a deployment, which requires significant time and advanced planning. Furthermore, according to users at another location, due to the complexities and functionality issues related to ALIS, contractor support is required on deployments; however, deploying with contractors could become problematic in a combat scenario. Overall, users at all 5 locations said that they have completed deployments using ALIS. However, deployments are challenging and the current deployment preparation process for ALIS inhibits a military service’s ability to deploy on short notice. Increasing Personnel Needs Users at 4 of 5 F-35 locations we visited stated that ALIS requires more contractor or military personnel support than originally planned. According to the F-35’s Operational Requirements Document—the document that outlines the overall requirements for the F-35 program— ALIS is supposed to help reduce the logistics footprint for the F-35. However, a 2013 DOD-commissioned study on reducing F-35 costs stated that the current ALIS support plan already uses 30 percent more administrators across squadrons and bases than a similarly-scaled IT implementation would normally require. In addition, current ALIS users at these 4 locations are finding that as ALIS becomes more mature, even more personnel are required to support the system’s operations. For example, according to users at 1 Air Force location, the Air Force currently relies on about 8 contractor employees to support each ALIS SOU server, but has determined that this is not sufficient. Users at 2 Air Force locations stated that until the Air Force can train more military personnel to support ALIS-related issues, they will need to increase the number of contractor employees per squadron to support F-35 operations. Further, users from 1 Air Force location said they have had to assign full- time “ALIS Expeditor” responsibilities to military personnel within the squadrons to keep track of ALIS-related issues and pressure the contractor for resolution. Since these roles are not official billets, their resulting responsibilities are adding to the military personnel’s existing, non-ALIS related responsibilities on the flight line. Air Force users from 1 location reported that due to inconsistencies within ALIS, they now have 20 full-time ALIS Expeditors to track ALIS-related issues and help ensure safety of flight for the aircraft. The Marine Corps had originally planned to maintain ALIS using only military personnel; however, as the numbers of aircraft and requisite SOUs increased, users at 1 Marine Corps location said that it was too difficult to develop and retain personnel with ALIS- specific expertise. According to these users, this has resulted in the Marine Corps needing increased numbers of contractor personnel to support its squadron operations. Inefficient F-35 Issue Resolution Process Users at all 5 F-35 locations we visited said that the process for resolving F-35 issues within ALIS remains problematic and inefficient. The Action Request (AR) process requires personnel to use an application within ALIS to submit an AR about any F-35 problem, including those about ALIS itself, to the contractor for triaging and ultimate resolution. In April 2016, we reported that ALIS users thought the AR process did not allow for the effective reporting and resolution of F-35 aircraft and ALIS issues. Specifically, users stated that the process did not provide transparency to all ARs submitted across F-35 locations and placed responsibility for resolving the requests primarily on the contractor. ALIS users at 4 locations stated that this remains the case. Users from 3 locations stated that the overall process would be more efficient if they were able to search ARs submitted by other squadrons across the fleet to determine if a solution to the problem already exists. Without this ability, users must submit an AR for every issue and wait for a response that can sometimes take months. For example, 1 location reported that from October 2018 through September 2019, F-35 aircraft were grounded for 9,262 hours or 9 percent of possible flight hours, due to unresolved ALIS- related ARs attributed mainly to missing and inaccurate electronic parts records. Officials from another location reported that during a 6-month period they had to ground aircraft for 2,200 hours as a result of waiting for contractors to resolve parts-related ARs. Users from a third location stated that more transparency in the AR process could reduce reliance on contractor support, provide a way to address F-35 problems more efficiently, and reduce costs to the program since DOD incurs a fee each time an AR is submitted. Poor User Experience Users at all 5 F-35 locations we visited stated that ALIS is not user- friendly or intuitive. While users stated that there have been some limited improvements to ALIS over the past years, as previously discussed, in general, users at all 5 locations described ALIS applications as difficult to navigate. For example, users from 1 location stated that it is more difficult and time-consuming to search for information on parts in ALIS than in legacy logistics systems because the information is located in multiple locations within ALIS. Additionally, users from all 5 locations said that some of the applications within ALIS have very slow processing speeds. According to users at 1 location, in some instances, ALIS’s slow applications require maintainers to work additional hours to complete required maintenance tasks. During a demonstration of ALIS and its Joint Technical Data application at one of the locations we visited, we observed maintainers deal with a slow log-in process, problems filtering and searching for data in an application, and ultimately having the application freeze and kick them out. Figure 8 shows a maintainer using a PMA to work in ALIS. Immature Applications Users at all 5 F-35 locations we visited stated that the training and mission planning applications within ALIS remain immature. Users at all 5 locations said they are not using the Training Management System (TMS), an application designed for pilots and maintainers to track training qualifications and assign personnel to carry out specific tasks, for its intended purpose. Users from 4 locations said that because of the ongoing issues with TMS, they are using legacy systems in its place. For example, one Air Force command released a memorandum in January 2018 allowing some squadrons to use an external legacy system in place of the TMS application due to shortfalls in TMS functionality, which it stated had caused excessive work to execute normal operations and become an unacceptable burden. Marine Corps and Navy users from 2 locations we visited said that they are using other legacy systems to circumvent the TMS application as well. Additionally, pilots at 4 locations stated that the Off-Board Mission Support (OMS) application within ALIS is immature and remains non- intuitive, time consuming, and difficult to navigate. The OMS application is a key application for pilots to conduct mission planning and debriefing. Pilots at 2 locations said that they rely on contractors to help them complete tasks in the application. Ineffective Training Users at all 5 F-35 locations we visited stated that training to learn how to use ALIS does not provide adequate knowledge or information to fully prepare users to operate the system. Specifically, users at 3 locations we visited stated that the training for ALIS does not reflect a realistic operational environment. Instead, users at all 5 locations stated that training materials are usually in the form of PowerPoint slides and that knowledge of ALIS and its functionality is primarily obtained at the squadron level through on-the-job-training. In April 2016, we reported that almost every user in the F-35-related focus groups we conducted at that time noted that they did not learn how to operate any ALIS applications until on-the-job training began on the flight line. Users stated that this remains true today. Users at 1 of the locations we visited stated that learning how to use ALIS in this manner has caused people to develop their own unique way of operating the system, which creates an F-35 fleet environment that is using its primary logistics tool in different ways. DOD Is Unable to Assess the Performance of ALIS or How the System Is Impacting F-35 Fleet Readiness DOD Has Not Developed Performance Metrics for ALIS Although DOD and F-35 program officials agreed that ALIS continues to provide challenges for users and is generally not performing well, DOD still has not determined how it wants the system to perform. For example, officials from the Joint Strike Fighter Integrated Test Force told us that testing for individual ALIS software version releases focuses primarily on whether the new version is performing “better” than the previous version. Specifically, ALIS testers have developed criteria to determine if the newest version of ALIS is functioning more efficiently than the previous version by comparing such tasks as screen download times. However, according to these officials, these tests are not determining if the ALIS system is performing to a specified standard because DOD has not defined this standard. In September 2014, we recommended that DOD develop a performance- measurement process for ALIS that includes, but is not limited to, performance metrics and targets that (1) are based on the intended behavior of the system in actual operations and (2) tie system performance to user requirements. The DOD Systems Engineering Guide for Systems of Systems states that to fully understand performance of systems of systems (such as ALIS), it is important to have a set of metrics that assess the system’s performance and trace back to user requirements because the system will likely evolve based on incremental changes—similar to ALIS’s incremental fielding. These metrics should measure the intended behavior and performance of the system in actual operations versus the progress of the development of the system, allowing an assessment of system capabilities based on user requirements. After over 5 years, and more than 400 aircraft fielded, DOD has not yet established a performance-measurement process for ALIS. DOD concurred with our 2014 recommendation, and repeated its commitment to develop performance metrics for ALIS after the release of our 2016 report on ALIS risks. In September 2019 program officials told us that DOD remains in the process of developing these metrics and has no set timeline for their completion. Without a performance-measurement process, the F-35 program does not have critical information about ALIS performance across F-35 locations. Such information could help address current and future ALIS performance issues and systematically measure ALIS functionality compared to intended performance. Problems with ALIS Could Be Affecting Overall F-35 Fleet Readiness Users at all 5 F-35 locations we visited also stated that problems with ALIS are affecting the overall readiness of the F-35 fleet; however, they were unable to tell us the degree to which this is the case. Overall F-35 fleet-wide performance has been falling short of warfighter requirements—that is, aircraft cannot perform as many missions or fly as often as required. Figure 9 shows F-35 fleet aircraft performance from October 2018 through September 2019. Full mission capability, or the percentage of time during which the aircraft can perform all of its tasked missions, was 31.6 percent across the fleet, as compared with the warfighter minimum target of 60 percent. Mission capability, or the percentage of time during which the aircraft can safely fly and perform at least one tasked mission, was 59.5 percent across the fleet, as compared with the warfighter minimum target of 75 percent. Furthermore, citing less than desirable aircraft performance, in September 2018, the Secretary of Defense directed the military services to achieve and maintain 80 percent mission capability rates for their critical aviation platforms, including the F- 35 fleet, by the end of fiscal year 2019. Two F-35 locations have started tracking information on how ALIS is affecting F-35 aircraft performance at their locations. Officials from one location told us that from October 2018 through September 2019, F-35 aircraft were grounded and thus non-mission capable for 16,221 hours, or 2 percent of possible flight hours, as a direct result of issues with ALIS— such as inaccurate or missing electronic records. However, according to officials at this location, this number does not capture all scenarios in which ALIS is affecting aircraft performance because sometimes squadron commanders make decisions to fly an aircraft when ALIS signals that they should not, in order to fulfill mission requirements. Officials from another location reported that in fiscal year 2018, ALIS- related issues caused the F-35 aircraft to be non-mission capable for 3,246 hours, or .5 percent of possible flight hours; however, as was the case with the previous location, officials said that this number also did not capture all scenarios in which ALIS is affecting aircraft performance. These limited efforts represent squadron-specific initiatives, as no other F-35 location has tracked similar ALIS-related data. Further, the data collected by the two locations only capture non-mission capability rates when ALIS signals to ground the aircraft and makes the aircraft incapable of completing a mission. The data do not account for the workarounds users said they are routinely performing to circumvent a non-functioning aspect of ALIS in order to get an aircraft ready to fly, or the times when squadron leadership decides to fly the aircraft when ALIS signals otherwise. Different factors can play a role in reducing F-35 aircraft readiness. For example, in April 2019, we reported that reduced aircraft performance was due largely to spare parts shortages. This conclusion was drawn from data that had been collected and tracked by both the contractor and DOD across the entire fleet to determine non-mission capability rates due to supply issues. Further, the F-35 program collects data on the degree to which maintenance issues are affecting F-35 mission capability. And, there are ongoing efforts to improve F-35 fleet readiness that are specifically targeted at supply and maintenance issues that are causing the significant mission-capability degradation. However, users and program officials stated that recurring issues with ALIS could also be affecting aircraft performance and noted that data on these issues are not being collected by the contractor or DOD. Although users reported multiple instances when ALIS-related issues grounded aircraft, these issues are being captured and categorized as either supply or maintenance-related issues, thus masking ALIS’s effect on fleet-wide readiness. DOD Instruction 5000.02T, “Operation of the Defense Acquisition System,” states that the program manager will use technical performance measures and metrics to assess program progress. It further states that the analysis of technical performance measures and metrics, in terms of progress against established plans, will provide insight into the technical progress and risk of a program like the F-35. In the case of ALIS, the F-35 program does not have a fleet-wide process for measuring, collecting, and tracking information on how ALIS is affecting the performance of the F-35 aircraft, such as fleet-wide mission capability rates. Without such a process, the F-35 program may be limited in its ability to identify all of the drivers of reduced aircraft performance and appropriate target solutions. Further, as we previously reported, DOD plans to enter into multi-year, performance-based F-35 sustainment contracts with the prime contractor, but may not be well positioned to enter into such contracts because, in part, it does not fully understand the technical characteristics of the aircraft. ALIS may or may not be having a notable effect on mission capability rates for the F-35 fleet. However, without understanding how or the extent to which ALIS is affecting the performance of the aircraft, DOD risks entering into long-term, performance-based logistics contracts without fully understanding all of the factors currently affecting aircraft operations. This could hinder DOD’s ability to effectively negotiate performance-related terms of the contract. Finally, without understanding how ALIS is affecting the performance of the aircraft, DOD risks developing a performance-measurement process for ALIS that is not tied to the overall performance goals of the program. DOD Is Pursuing Actions to Enhance the Long-Term Viability of ALIS, but It Has Not Established a Strategy for the Future System Re- Design DOD is taking actions to enhance the long-term viability of ALIS. Limited DOD attention on ALIS has resulted in a troubled history with the system. As a result, multiple efforts are currently underway to re-design and attempt to improve ALIS. However, key technical and programmatic uncertainties hinder these efforts. Furthermore, DOD does not have an overarching strategy for the future redesign of ALIS. Limited DOD Attention Has Resulted in a Troubled History with ALIS As originally envisioned, ALIS was intended to be a first-of-its-kind, fully autonomic system that would provide users access to data on a range of capabilities—including operations, maintenance, prognostics, supply chain, customer support services, training, and technical data—in one logistics system to support aircraft operations. According to Joint Strike Fighter Integrated Test Force officials, previous DOD aircraft logistics systems were much simpler, not fully autonomic, and generally included data related to fewer major capabilities. However, the F-35 program office did not clearly specify what it required from ALIS from the warfighter’s perspective beyond the broad capabilities to be included in the system. Air Force officials stated that instead, the F- 35 program office relied on the prime contractor to take the lead in managing the development of the system. For example, the F-35 Operational Requirements Document provides only overarching, high- level requirements for ALIS and does not include specific, user-related requirements or requirements to adapt and modernize the system over time. DOD officials acknowledged that historically, DOD has prioritized other aspects of the F-35 program, such as the development of the airframe, over its logistics system. In addition, DOD’s focus with ALIS development over the last 5 years has largely centered on adding capabilities required to complete developmental testing for the F-35. As issues with the fielded system have arisen, DOD and the prime contractor’s approach has generally been to resolve these issues on a case-by-case basis as available resources allowed, as opposed to making more costly and time-intensive improvements to the system’s underlying design and functionality. DOD contracting officials and prime contractor representatives stated that the need to balance a limited number of software development personnel between efforts to stabilize the current system and add new features has negatively affected the development of ALIS. In a 2017 report, the Air Force Digital Service recommended that the F-35 program office cease adding new capabilities in order to re-evaluate ALIS-related design choices and improve software development processes and procedures. According to the report, many of the issues with ALIS have known root causes that are directly related to software and hardware design choices that are 15 years old. For example, ALIS is made up of siloed applications that each have their own, sometimes conflicting, databases. Further, according to the Air Force Digital Service report, efforts to upgrade ALIS from an out-of-date operating system have not been prioritized by the F- 35 program office. Finally, ALIS hardware is cumbersome, consisting of heavy servers as well as laptops that were originally designed in the mid- 1990s. The current approach to developing ALIS has generally led to scheduling delays and challenges addressing a backlog of ALIS deficiencies. For example, the ALIS version required to complete developmental testing for the F-35 was not released until 2018—8 years after the originally planned release date. F-35 program office officials emphasized that in general, the timeframe for releasing major software updates for ALIS—up to 18 months—has been long. Further, based on data from the prime contractor, as of September 2019, there were about 4,700 open ALIS deficiencies, which are used by the prime contractor to track and manage issues with the system. According to an F-35 program office official, ALIS deficiencies may be identified in the field by F-35 users, in the prime contractor’s testing laboratory, or during DOD-led developmental and operational testing of the F-35 and ALIS. Of these 4,700 deficiencies, about 34 percent were identified in 2017 or earlier and 22 percent were category 1 or category 2 deficiencies. Category 1 deficiencies are considered critical and could jeopardize safety, security, or another requirement; category 2 deficiencies are those that could impede or constrain successful mission accomplishment. As shown in figure 10, the total number of open deficiencies has generally increased over the last 2 years. In addition, the number of open category 1 through category 3 deficiencies, which are considered critical or have an adverse effect on mission accomplishment, generally increased during this period. While the rate at which the prime contractor closed deficiencies during this period increased, the rate of increase was generally lower than the rate at which new deficiencies were identified. Officials from the Joint Strike Fighter Integrated Test Force and Office of the Director of Operational Test and Evaluation expressed concerns about the number and nature of the ALIS-related deficiencies they have identified during developmental and operational testing. For example, F- 35 testers identified a number of deficiencies with the most recent ALIS software version, ALIS 3.5, including eight category 1 deficiencies. ALIS 3.5 is referred to as the “stabilization” release because it was intended to address longstanding issues with ALIS. In addition, F-35 testers stated that since 2016, they have identified a number of cyber-related ALIS deficiencies, most of which remain open today. While officials said that the number of cyber deficiencies is consistent with other DOD weapons systems, they stressed that a vulnerable ALIS is particularly problematic because of how interconnected the system is with the F-35 aircraft and its operations. Multiple Efforts Are Underway to Re-Design ALIS DOD and the prime contractor have acknowledged ALIS’s troubled history and have established three initiatives to re-design and fix ALIS. At a November 2019 congressional hearing, the F-35 Program Executive Officer stressed that significant additional work is required to improve ALIS functionality and that this work cannot be done in old and outdated ways. Table 2 summarizes the three initiatives, led by the F-35 program office, Air Force, and prime contractor respectively. Key Technical and Programmatic Uncertainties Hinder Efforts to Re-Design ALIS According to the F-35 program office, the three initiatives are complementary and will eventually be integrated in a final redesign of ALIS. However, we found that DOD lacks clarity on how it will address key technical and programmatic uncertainties about the future of the system (see figure 11). These uncertainties relate to complex aspects of ALIS that will significantly impact the future design of the system and how it will be managed. Further, there are divergent views among officials involved with the various initiatives in terms of how DOD should approach key aspects of the re-design, highlighting the uncertainty that exists about the future of ALIS. ALIS Capabilities DOD has not fully determined what capabilities will be included in the ALIS re-design. After years of focusing on adding new capabilities with each major ALIS software version release, DOD officials agreed that that their current goal is to streamline and simplify ALIS. For example, the Mad Hatter initiative is designing applications based on the minimum capabilities required by maintainers to quickly release an aircraft for flight. Similarly, the ALIS Next initiative is working to optimize functions in ALIS by identifying aspects of the current design that could be slowing down the system—for example, transferring an aircraft’s entire digital history each time the jet is transferred from one SOU to another. However, officials from the Office of the Director of Operational Test and Evaluation indicated that there continues to be uncertainty about the capabilities— both classified and unclassified—that will be included in the re-design. Further, as discussed previously, the F-35 program office has not formally established how it expects ALIS to perform in operations or developed a performance-measurement process for ALIS. Program officials indicated the need for discussions with the services and international partners about aspects of the current system that are not consistently being used and may therefore not be required (such as the Training Management System) through an updated process for establishing ALIS-related requirements. This process, which requires coordination across all military services and international partners, has proven to be challenging in the past. According to a 2017 Air Force Digital Service report, the F-35 program office faces challenges identifying and prioritizing ALIS capabilities across multiple services and international partners, and this has negatively affected the development of the system. Software Development Model DOD is unclear about the extent to which it can adopt a more flexible software development model known as Agile. As we reported in April 2019, the F-35 program as a whole is pursuing a faster and more incremental approach for delivering new aircraft capabilities to the warfighter in order to more flexibly address evolving threats. One approach to software development that helps facilitate such incremental delivery is Agile, which calls for the delivery of software in small, short increments rather than in the typically long, sequential phases of a traditional software development approach. More a philosophy than a methodology, Agile emphasizes early and continuous software delivery, as well as using collaborative teams, and measuring progress with working software. According to some F-35 program office officials, adopting Agile could result in a more secure system because it involves continually testing software for security vulnerabilities. Further, we have previously reported that following an incremental development approach, such as Agile, gives agencies the opportunity to obtain additional feedback from users, which increases the probability that each successive increment will meet user needs. The Mad Hatter initiative is experimenting with an Agile approach and has had some initial successes using this model. For example, in July 2019, we observed a demonstration of a Mad Hatter-developed application that allows the user to quickly and easily search through Joint Technical Data, an application within ALIS that has been reported by some users as being extremely difficult to navigate. However, the Mad Hatter initiative has operated outside of F-35 program office policies and processes and its applications are currently not integrated with the fielded ALIS system. Further, Mad Hatter and F-35 program office officials said that they have faced challenges communicating the value of their approach with one another, and according to a senior Air Force official associated with the Mad Hatter initiative, the F-35 program office has not clarified the role of Mad Hatter representatives in current planning efforts aimed at scaling the results of the Mad Hatter initiative to the entire F-35 enterprise. Separately, as part of its own ALIS initiative, prime contractor officials said that their company recently began taking steps to adopt best practices for delivering new ALIS software using an Agile model. However, these efforts are new, and the F-35 program office has not developed standards for software developed by the prime contractor using this model. DOD officials we spoke with expressed differing views on the extent to which DOD should adopt an Agile software delivery model for ALIS. For example, in a 2018 memorandum establishing the Mad Hatter pilot, a senior Air Force acquisition official stated that the F-35 program should embrace the tenets of this type of model in order to innovate and rapidly deliver useful capability through ALIS. Similarly, Air Force, Office of the Secretary of Defense, and some F-35 program office officials stated that modernizing ALIS will require DOD to adopt industry best practices by making decisions quickly, delivering usable products early and often, and revising plans to reflect experience from completed software iterations. In contrast, Marine Corps and some F-35 program office officials indicated that DOD should carefully consider different commercially-available software tools, as well as DOD-specific constraints, before delivering new ALIS capabilities. For example, F-35 program office officials associated with the ALIS Next initiative stated that they conducted an assessment of the commercial software tools that could be used for new ALIS software development. These officials said that some of the tools that were initially being used by the Mad Hatter initiative to develop applications make software development easier in the short-term but more difficult to switch toolsets and/or contractors in the long-term. Marine Corps and some F-35 program officials also noted that current DOD processes and procedures—such as the software certification and cost-estimating processes—may not be able to support quick software releases. While an Agile software delivery model has been identified as having the potential to improve the way in which the federal government develops and implements IT, we previously reported that this type of model requires significant procedural and organizational changes in order to be implemented successfully. The Cloud Environment DOD has not made a decision about the extent to which the ALIS re- design will be hosted in the cloud as opposed to onsite servers at the squadron level. In April 2019, we reported that cloud computing allows federal agencies to access on-demand, shared computing resources with the goal of delivering services more quickly and at a lower cost. More specifically, purchasing IT services through a provider enables agencies to avoid paying for all of the computing resources (e.g., hardware, software, networks) that would typically be needed to provide such services. This approach offers federal agencies a means to buy the services faster and possibly at less cost than building, operating, and maintaining these computing resources themselves. However, National Institute of Standards and Technology guidance states that public cloud computing represents a significant shift from the norms of on-site data centers and should therefore be approached carefully with consideration to the sensitivity of data. While the Mad Hatter initiative has embraced hosting ALIS in the cloud, including at the squadron level, ALIS Next is conducting an assessment of the extent to which a cloud-based system is the best option for ALIS. Further, as part of its internal ALIS investment, the prime contractor has designed an alternative model to the current system that includes an onsite server at each F-35 squadron. Office of the Secretary of Defense, Air Force, and F-35 program office officials we talked to agreed that the ALIS re-design will involve migrating some portions of ALIS from onsite servers to the cloud. For example, these officials agreed that DOD should explore options for migrating the ALOU and U.S. CPE to the cloud. However, these officials disagreed about how much of the future system should be cloud-based at the squadron level. For example, Air Force, Office of the Secretary of Defense, and some F-35 program office officials stressed that for day-to- day maintenance at U.S. bases, F-35 squadrons should be able to access ALIS using Wi-Fi, and that the reliance on onsite servers should therefore be minimal and limited to deployed scenarios. According to these officials, DOD can achieve significant cost savings by moving ALIS to the cloud. These officials also indicated that DOD’s hesitation about moving from onsite servers to the cloud is mostly cultural and the result of a lack of understanding about what the cloud is. One senior Office of the Secretary of Defense official with software expertise stated that warfighters should be able to deploy with a minimal amount of ALIS hardware (for example, only a high-powered laptop). In contrast, other F- 35 program office officials told us that the F-35 program office is restricted in the extent to which it can migrate to cloud-based SOUs due to connectivity and security restrictions. Further, at an ALIS Next conference, some partner country representatives expressed concerns about hosting ALIS in the cloud, stating that stringent security requirements would likely prevent their governments from accepting a cloud-based solution for ALIS. User Feedback DOD does not have a plan for incorporating users early and often in the development of new ALIS software across the F-35 enterprise. Previous GAO reports as well as other DOD studies have found that giving users the opportunity to provide feedback on actual working software early and often in the software development process, and incorporating that feedback in subsequent development, is critical to the success of any software development effort. For example, in March 2019, we reported that obtaining frequent feedback is linked to reducing risk, improving customer commitment, and improving technical staff motivation. Historically, user feedback has not been prioritized in the ALIS software development process. According to users we talked to, working groups do exist that serve as a venue for voicing user-related issues; however, users stated that these working groups meet infrequently and often do not lead to desired changes. Further, prime contractor representatives told us that while they recently began soliciting user feedback as part of their ALIS initiative, the F-35 program office has not contractually required incorporating user feedback in the ALIS software development process. The Mad Hatter initiative is currently incorporating user feedback into new software development for ALIS and has established a process whereby F-35 users and Mad Hatter software developers can communicate directly about the Mad Hatter applications that are in development. As part of this process, Mad Hatter product teams develop simple applications, field the applications to users, and then use feedback from users—obtained by email or videoconferences—to adjust and enhance the applications. Although Mad Hatter’s process for incorporating user feedback aligns with the practice of incorporating feedback early and often, the initiative is being executed at one F-35 installation, with one military service. Further, while the F-35 program office intends to eventually scale the results of Mad Hatter’s experimentation to the rest of the F-35 enterprise, it has not formally outlined how it will institutionalize the initiative’s process for incorporating user feedback across multiple services and international partners. Primary ALIS Owner DOD has not determined the roles of DOD and the prime contractor in future ALIS development and management. DOD officials stressed that historically, the department has relied heavily on the prime contractor to develop and manage ALIS. Officials also said that moving forward, DOD will need to play a more active role in the management of ALIS. For example, Air Force, Office of the Secretary of Defense, and F-35 program office officials all said that DOD should serve as the primary owner of the ALIS software system, with the prime contractor and other firms developing applications that will feed into DOD’s software pipeline. However, the F-35 program office has not officially named DOD as the prime ALIS owner, or specified how it will coordinate software development across these multiple entities. Further, while one of the long- term objectives of the Mad Hatter initiative is to build DOD’s capacity to manage and develop new ALIS software itself, Air Force officials involved in this initiative stated that DOD has not yet fully developed this capacity. As the original ALIS developer, prime contractor representatives stated that their company is in the best position to modernize ALIS. F-35 program office officials acknowledged that because the prime contractor plays such a critical role in the development and sustainment of the F-35, it will be necessary for DOD to work closely with the contractor, regardless of the direction DOD decides to take. For example, DOD officials said they have faced challenges obtaining key technical data from the prime contractor that would be required by DOD to lead ALIS software development, such as the underlying source code for current ALIS software, and that they were uncertain about the extent to which they would be able to obtain these data in the future. At a November 2019 congressional hearing, the Under Secretary of Defense for Acquisition and Sustainment stressed that many of the challenges with ALIS stem from the fact that ALIS data are fed back through prime- contractor computers, and there is resulting ambiguity over the ownership of that data. As we previously reported, DOD continues to lack clarity about the technical data it owns and the additional data it would require to maintain flexibility in the sustainment of the F-35. Current ALIS Software DOD has not agreed on the extent to which the ALIS re-design will incorporate current ALIS software—consisting of 8 million lines of code. As part of the ALIS Next initiative, F-35 program office officials said they intend to review the underlying source code for ALIS to determine which aspects of the current software should be integrated in the re-design. These officials explained that redesigning ALIS software from scratch will take too long and the future ALIS system will therefore need to incorporate, to some extent, current ALIS software. In contrast, a senior Air Force official associated with the Mad Hatter initiative stated that the initiative intends to replace most current ALIS applications with commercial or new custom applications, retaining only those ALIS applications that can be cost-effectively modernized. Further, officials from the Air Force, Office of the Secretary of Defense, and F-35 program office indicated that because most of the ALIS source code has not been updated in years and contains numerous security vulnerabilities, the software should be completely re-designed. DOD Does Not Have a Strategy for the Future Re- Design of ALIS DOD is unclear about how it will approach the key technical and programmatic uncertainties surrounding ALIS because the department has not developed a strategy for the future re-design of the system. DOD guidance for program managers states that a sound strategy requires, among other things, a clear articulation of program goals as well as an understanding of the risks or uncertainties and costs associated with achieving those goals. While DOD and the prime contractor have established various initiatives to re-design ALIS, DOD has not developed a strategy for the future of ALIS that clearly identifies and assesses goals, key risks or uncertainties, and associated costs. For example, as discussed previously, DOD lacks clarity about the goals of the re-design, such as the capabilities that will be included in the future system and the extent to which ALIS will be hosted in the cloud. In addition, DOD has not fully assessed key risks or uncertainties, including the extent to which DOD can adopt an Agile software development approach or manage the system itself. Finally, because it has not answered key questions about the future of the system, such as the extent to which the re-design will incorporate current ALIS software, DOD has not been able to develop accurate cost estimates for the ALIS re-design. In the past, DOD has faced challenges estimating and tracking ALIS costs. For example, in 2016 we reported that while DOD had estimated that ALIS would cost approximately $17 billion, the estimate was not fully credible because DOD had not performed uncertainty and sensitivity analyses as part of the cost-estimating process. Further, for this review, the F-35 program office was not able to provide us with historic costs showing how much the department has spent on ALIS over the years. DOD officials stated that historically, the department has faced challenges allocating scarce resources across competing priorities, and that the F-35 air vehicle has generally been prioritized over ALIS. With the completion of F-35 developmental testing in April 2018, program officials said they are now in a better position to focus on ALIS and address long-standing issues with the system. However, efforts to correct ALIS are relatively new and have not been fully developed. Without a strategy to guide the re-design of ALIS, DOD will not be able to effectively plan for the transition from the current system to a future one. For example, according to F-35 program office officials, DOD recently procured additional hardware for the current system, which officials said may not be required if DOD is able to develop and field a re-designed ALIS in the near term. Officials from the Office of the Director of Operational Test and Evaluation stressed that effectively transitioning from the current system to a future one will be particularly challenging for DOD given the need to continue sustaining the more than 400 aircraft that have already been fielded with current ALIS. Further, as discussed above, there are divergent views in terms of how DOD should approach key technical and programmatic aspects of the re-design, and integrating the different efforts that are underway to fix ALIS—led by the F-35 program office, Air Force, and prime contractor—will therefore require significant direction and leadership. Without a strategy, DOD may not be able to effectively coordinate and leverage the different ALIS initiatives that are underway, potentially leading to inefficiencies. DOD also risks repeating history by failing to clearly articulate what it expects from ALIS and how it will play a more active role in the management of the system going forward. Conclusions The F-35 aircraft, with its advanced warfighting capabilities, provides critical tactical aviation for the Department of Defense. However, DOD will need to overcome substantial challenges related to ALIS if it wants to find successes in both sustainment and operations of the aircraft. Current ALIS users continue to report significant challenges with the system that are affecting day-to-day operations of the aircraft, adding additional flight line-related responsibilities, and, in some instances, causing squadron leadership to assume the risk of flying aircraft when ALIS tells them to stay on the ground. Although ALIS is not currently performing well, over 5 years after we recommended it, DOD has yet to establish a performance- measurement process that would define how ALIS should perform. In the absence of such a process, DOD will be challenged to address current and future ALIS-performance issues because it cannot measure ALIS functionality compared to intended system performance. Furthermore, ALIS users collectively agree that the issues with ALIS are affecting the readiness of the aircraft; however, the degree to which this is true remains unknown. Fleet-wide mission capability rates for the F-35 are still below the warfighter’s minimum targets, but DOD does not have a process for measuring, collecting, and tracking information on how ALIS is affecting these rates. Without such a process, DOD may not understand all of the factors behind the reduced aircraft performance, thus limiting its ability to target appropriate solutions. DOD officials have acknowledged the ongoing challenges with ALIS and know that the system, as it stands today, cannot be sustained into the future; therefore, it is positive that the department has embarked on efforts to re-design and fix ALIS, as well as take on a more active role in the management of the system. However, DOD faces a significant challenge as there are several complex technical and programmatic uncertainties that will need to be resolved before any future ALIS solution can be realized. Additionally, there are divergent views among ALIS stakeholders about how to go about addressing these complex issues. The future of ALIS remains unclear because the department has not developed a strategy for the re-design of the system that would identify, among other things, what the system should look like, how will it be developed and managed, how it will address key risks, and how much it will ultimately cost. Without such a strategy, DOD will not be able to effectively plan for the transition from the current ALIS system, which is already embedded in over 400 aircraft across the global F-35 fleet, to whatever solution is determined. Furthermore, a strategy would help align what is currently a chorus of divergent views within the department on how to address the future of ALIS. With the worldwide fleet expected to grow to over 1,000 aircraft over the next four years, and with the U.S. services becoming increasingly reliant on the F-35’s capabilities to support their operational strategies, it will be imperative for DOD to address the ongoing issues related to the F-35’s logistics system. Matter for Congressional Consideration Congress should consider legislation requiring the Department of Defense to establish a performance-measurement process for ALIS that includes, but is not limited to, performance metrics and targets that (1) are based on intended behavior of the system in actual operations and (2) tie system performance to user requirements. (Matter for Consideration 1) Recommendations for Executive Action We are making the following two recommendations to DOD: The Secretary of Defense should ensure the Under Secretary of Defense for Acquisition and Sustainment, in consultation with the F-35 Program Executive Officer, develops a program-wide process for measuring, collecting, and tracking information on how ALIS is affecting the performance of the F-35 fleet to include, but not be limited to, its effects on mission capability rates. (Recommendation 1) The Secretary of Defense should ensure the Under Secretary of Defense for Acquisition and Sustainment, in consultation with the F-35 Program Executive Officer, develops and implements a strategy for the re-design of ALIS. The strategy should be detailed enough to clearly identify and assess the goals, key risks or uncertainties, and costs of re-designing the system. (Recommendation 2) Agency Comments We provided a draft of this report to DOD for review and comment. In its written comments, reproduced in appendix II, DOD concurred with our recommendations and identified actions that it was taking or planned in response. We agree that DOD is taking positive steps in addressing issues with ALIS, including the decision to replace ALIS with a future system that it has named the F-35 Operational Data Integrated Network (ODIN). According to DOD, the department is currently developing a strategy that will guide ODIN’s development. As DOD proceeds with replacing ALIS with ODIN, it will be imperative for the department to carefully consider and assess the key technical and programmatic uncertainties discussed in this report. These issues—including how much of ALIS will be incorporated in ODIN and the extent to which DOD has access to the data it needs to play a more active role in the management of the system—are complex, and will require significant direction and leadership to resolve. As agreed with your office, unless you publicly announce the contents of this report earlier, we plan no further distribution until 7 days from the report date. At that time, we will send copies of this report to congressional requesters; the Secretary of Defense; the Under Secretary of Defense for Acquisition and Sustainment; the F-35 Program Executive Officer; the Secretaries of the Air Force and Navy; and the Commandant of the Marine Corps. In addition, the report will be available at no charge on the GAO website at http://www.gao.gov. If you or your staff have any questions about this report, please contact me at (202) 512-9627 or maurerd@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. Staff members making key contributions to this report are listed in appendix III. Appendix I: Scope and Methodology For each of our objectives, we reviewed relevant F-35 sustainment and the Autonomic Logistics Information System (ALIS)-related data, plans, program briefs, guidance, and other documentation and collected information by interviewing officials from the Office of the Under Secretary of Defense for Acquisition and Sustainment, the F-35 Joint Program Office, the Director, Operational Test and Evaluation, the Defense Contract Management Agency, the U.S. Air Force, the U.S. Navy, the U.S. Marine Corps, the Air Force Digital Service, and the prime contractor, Lockheed Martin. To interview officials and observe ALIS- related operations, we conducted site visits to five F-35 locations—Luke Air Force Base, Arizona; Edwards Air Force Base, California; Nellis Air Force Base, Nevada; Marine Corps Air Station Yuma, Arizona; and Naval Air Station Lemoore, California. We selected these locations to obtain perspectives from ALIS-users (i.e. maintainers, pilots, supply personnel, contractors) from all U.S. services participating in the F-35 program, including from operational, training, and testing locations. Additionally, we developed a data collection instrument to collect ALIS-related inputs and data from ALIS-users (i.e. maintainers, pilots, supply personnel, contractors) at all 10 U.S. F-35 locations—Luke Air Force Base, Arizona; Edwards Air Force Base, California; Nellis Air Force Base, Nevada; Marine Corps Air Station Yuma, Arizona; Naval Air Station Lemoore, California; Hill Air Force Base, Utah; Naval Air Station Patuxent, Maryland; Eglin Air Force Base, Florida; Marine Corps Air Station Beaufort, South Carolina; and Marine Corps Air Station Iwakuni, Japan. Finally, we met with officials from the F-35 Joint Program Office, Massachusetts Institute of Technology (MIT) Lincoln Labs, Lockheed Martin Rotary and Mission Systems, Air Force Digital Service, Kessel Run (Air Force), and others to discuss ALIS-related improvement efforts. In support of our objectives, we gathered data from fiscal year 2019 (the most recent full fiscal year of data available at the time of our review) from the prime contractor on the performance of the F-35 fleet such as the full and mission capability rates. We also collected the most recent available information on ALIS software deficiencies. To determine the reliability of these data, we collected information on how the data were collected, managed, and used through a questionnaire and interviews. Although we identified some limitations in the way that certain data are being collected and reported—such as data related to aircraft performance like mission capability rates—we determined that they are sufficiently reliable for the way in which we reported them and our purposes of providing information on the progress and challenges within the program. All the performance data presented in our report are sufficiently reliable to provide a general comparison of capabilities to minimum targets. To assess the extent to which there have been improvements as well as key challenges with ALIS over the last 5 years, we interviewed officials and examined guidance and briefing documents from the Office of the Under Secretary of Defense for Acquisition and Sustainment, the U.S. Services, the F-35 Joint Program Office, the Defense Contract Management Agency and Lockheed Martin Rotary and Mission Systems officials to discuss the current status of the system and plans for mitigating risks. To determine user views on risks to (or issues with) ALIS, we interviewed officials at our 5 selected bases, conducted a short data collection instrument of the other 5 bases, interviewed officials at Air Force headquarters and the contractor, and reviewed relevant documents. At the 5 bases, we interviewed groups of pilots, maintainers and supply personnel about ALIS performance, challenges, and possible improvements. In addition, we posed several targeted questions based on risks found in our last report. In total, we received input from more than 160 users at the 5 bases we visited through group discussions or interviews. We analyzed the responses provided in these group interviews, and identified the issues/risks that at least one set of users reported at each of the 5 bases. We also considered any improvements that were described as having occurred during the last few years. We also compared the responses from the interviews at the 5 bases with our data collection responses, and the other testimonial and documentary evidence we obtained. The list of issues/risks we identified contains some that were reported in our 2016 report as well as some new ones. While this list summarizes the types of issues/risks described at the 5 bases, and also in other interviews and document review, individual user views and experiences could vary by base and user group. We also interviewed officials and reviewed reports from the Air Force Audit Agency, the Director, Operational Test and Evaluation, and the Department of Defense Inspector General to identify improvements as well as any functionality issues with ALIS. We interviewed and gathered information from DOD officials on testing for ALIS, metrics on ALIS’s performance, and the operations of the system. As discussed previously, we collected and analyzed data for fiscal year 2019 that we obtained from the prime contractor on the overall aircraft performance such as the full mission capability and mission capability rates. We analyzed and compared information obtained from interviews, site visits, data collection instruments, and documents with guidance such as DOD’s System Engineering Guide for System of Systems to determine the extent to which DOD has an effective procedure for addressing and mitigating specific risks and challenges that may be associated with a major weapon system. We also compared this information with previous GAO reports from 2014, 2016, and 2018 to determine the extent to which DOD has addressed our prior recommendations on ALIS-related issues. To assess the extent to which the F-35 program has addressed issues with ALIS, we gathered and analyzed data from the prime contractor on open and closed ALIS deficiencies identified from November 2017 through October 2018. We selected this timeframe because it included the most recent data on ALIS deficiencies at the time of our review and also allowed us to observe trends in ALIS deficiencies over a two-year period. The data we received included summary information on the total number of open deficiencies, the total number of closed deficiencies, the number of newly closed deficiencies, the number of newly identified deficiencies, and the total number of open category 1 through category 3 deficiencies (considered critical or adverse) for each month during the two-year period. To determine the reliability of these data, we conducted electronic tests to identify any internal inconsistencies with the data. We also reviewed documentation from the prime contractor on the management of ALIS deficiency data and collected information on how the data were collected, managed, and used through a questionnaire. Specifically, we asked questions about inconsistencies we identified through electronic testing of the data, the extent to which the prime contractor’s system for collecting deficiency information includes edit checks or controls to help ensure the data are entered accurately, and limitations related to the accuracy or completeness of the data. As a result, we determined the data to be sufficiently reliable for the purpose of reporting trends in the number of open and closed ALIS deficiencies over time. To determine the extent to which DOD is taking actions to enhance the long-term viability of the system, we interviewed officials and reviewed guidance and/or planning documents from the Office of the Under Secretary of Defense for Acquisition and Sustainment, the F-35 Joint Program Office, and the Office of the Assistant Secretary of the Air Force for Acquisition, Technology, and Logistics. We interviewed officials from the prime contractor to determine their role in helping DOD mitigate risks regarding the long-term viability for ALIS. Additionally, we examined briefing documents from the MIT-Lincoln Labs, a federally-funded research and development center assisting the F-35 Joint Program Office, on plans, timelines, and risks for modernizing the hardware and software. We interviewed officials from the Air Force’s Kessel Run team to discuss their Mad Hatter initiative (intended to improve ALIS functionality), the viability of current ALIS software, and any risks associated with the future of ALIS. We conducted a site visit to Nellis Air Force Base to observe the Mad Hatter initiative and discuss its results and the future of ALIS software. Further, as discussed previously, we analyzed data from November 2017 through October 2019 on ALIS deficiencies. We reviewed reports and interviewed officials from the Air Force Digital Service and the Director, Operational Test and Evaluation on the future viability of these long-term initiatives for ALIS. Finally, we analyzed and compared information obtained from interviews, site visits, and documents with applicable guidance to determine the extent to which DOD has an effective long-term plan for ALIS that addresses operational and financial risks. In support of our work, we interviewed officials from the following DOD organizations and other organizations during our review. We selected these organizations based on their oversight, planning, and/or execution roles related to F-35 ALIS operations. Office of the Under Secretary of Defense for Acquisition and Sustainment, Arlington, Virginia Office of the Director for Operational Test and Evaluation, Arlington, Defense Contract Management Agency Lockheed Martin, Orlando, F-35 Joint Program Office, Arlington, Virginia Office of the Assistant Secretary of the Air Force for Acquisition, Air Force F-35 Integration Office, Arlington, Virginia Kessel Run Team, Hanscom Air Force Base, Massachusetts Luke Air Force Base, Arizona 56th Maintenance Group 61st Aircraft Maintenance Unit 62nd Aircraft Maintenance Unit Edwards Air Force Base, California Nellis Air Force Base, Nevada 57th Aircraft Maintenance Squadron Navy F-35 Integration Office, Arlington, Virginia Naval Air Station Lemoore, California Strike Fighter Wing Pacific Strike Fighter Squadron 125 Strike Fighter Squadron 147 Marine Corps F-35 Integration Office Marine Corps Air Station Yuma, Arizona Marine Aircraft Group 13 Marine Aviation Logistics Squadron 13 Marine Fighter Attack Squadron 211 Marine Fighter Attack Squadron 122 Air Force Digital Service, Arlington, Virginia Lockheed Martin Rotary and Mission Systems, Orlando, Florida MIT Lincoln Laboratory, Lexington, Massachusetts We conducted this performance audit from August 2018 to March 2020 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides and reasonable basis for our findings and conclusions based on our audit objectives. Appendix II: Comments from the Department of Defense Appendix III: GAO Contact and Staff Acknowledgments GAO Contact Staff Acknowledgments Diana Maurer, (202) 512-9627, maurerd@gao.gov In addition to the contact named above, Alissa Czyz (Assistant Director), Matthew Bader, Vincent Buquicchio, Tracy Burney, Juana Collymore, Martin De Alteriis, Michael Holland, Jeff Hubbard, Clarice Ransom, and Elisa Yoshiara made key contributions to this report. Related GAO Products F-35 Aircraft Sustainment: DOD Faces Challenges in Sustaining a Growing Fleet. GAO-20-234T. Washington, D.C.: November 13, 2019. Space Command and Control: Comprehensive Planning and Oversight Could Help DOD Acquire Critical Capabilities and Address Challenges. GAO-20-146. Washington, D.C.: October 30, 2019. F-35 Joint Strike Fighter: Action Needed to Improve Reliability and Prepare for Modernization Efforts. GAO-19-341, Washington, D.C.: April 29, 2019. F-35 Aircraft Sustainment: DOD Needs to Address Substantial Supply Chain Challenges.GAO-19-321. Washington, D.C.: April 25, 2019. Cloud Computing: Agencies Have Increased Usage and Realized Benefits, but Cost and Savings Data Need to Be Better Tracked. GAO-19-58, Washington, D.C.: April 4, 2019. DOD Space Acquisitions: Including Users Early and Often in Software Development Could Benefit Programs. GAO-19-136. Washington, D.C.: March 18, 2019. F-35 Joint Strike Fighter: Development Is Nearly Complete, but Deficiencies Found in Testing Need to Be Resolved. GAO-18-321. Washington D.C.: June 5, 2018. Warfighter Support: DOD Needs to Share F-35 Operational Lessons Across the Military Services. GAO-18-464R, Washington D.C.: April 25, 2018. Military Aircraft: F-35 Brings Increased Capabilities, but the Marine Corps Needs to Assess Challenges Associated with Operating in the Pacific. GAO-18-79C. Washington, D.C.: March 28, 2018. Information Technology Reform: Agencies Need to Improve Certification of Incremental Development. GAO-18-148. Washington, D.C.: November 7, 2017. F-35 Aircraft Sustainment: DOD Needs to Address Challenges Affecting Readiness and Cost Transparency. GAO-18-75. Washington D.C.: October 26, 2017. F-35 Joint Strike Fighter: DOD’s Proposed Follow-on Modernization Acquisition Strategy Reflects an Incremental Approach Although Plans Are Not Yet Finalized. GAO-17-690R. Washington, D.C. August 8, 2017. F-35 Joint Strike Fighter: DOD Needs to Complete Developmental Testing Before Making Significant New Investments. GAO-17-351. Washington, D.C.: April 24, 2017. F-35 Joint Strike Fighter: Continued Oversight Needed as Program Plans to Begin Development of New Capabilities. GAO-16-390. Washington, D.C.: April 14, 2016. F-35 Sustainment: DOD Needs a Plan to Address Risks Related to Its Central Logistics System. GAO-16-439. Washington, D.C.: April 14, 2016. F-35 Joint Strike Fighter: Preliminary Observations on Program Progress. GAO-16-489T. Washington, D.C.: March 23, 2016. F-35 Joint Strike Fighter: Assessment Needed to Address Affordability Challenges. GAO-15-364. Washington, D.C.: April 14, 2015. F-35 Sustainment: Need for Affordable Strategy, Greater Attention to Risks, and Improved Cost Estimates. GAO-14-778. Washington, D.C.: September 23, 2014. F-35 Joint Strike Fighter: Slower Than Expected Progress in Software Testing May Limit Initial Warfighting Capabilities. GAO-14-468T. Washington, D.C.: March 26, 2014. F-35 Joint Strike Fighter: Problems Completing Software Testing May Hinder Delivery of Expected Warfighting Capabilities. GAO-14-322. Washington, D.C.: March 24, 2014. F-35 Joint Strike Fighter: Restructuring Has Improved the Program, but Affordability Challenges and Other Risks Remain. GAO-13-690T. Washington, D.C.: June 19, 2013. F-35 Joint Strike Fighter: Program Has Improved in Some Areas, but Affordability Challenges and Other Risks Remain. GAO-13-500T. Washington, D.C.: April 17, 2013. F-35 Joint Strike Fighter: Current Outlook Is Improved, but Long-Term Affordability Is a Major Concern. GAO-13-309. Washington, D.C.: March 11, 2013. Software Development: Effective Practices and Federal Challenges in Applying Agile Methods. GAO-12-681. Washington, D.C.: July 27, 2012. Joint Strike Fighter: DOD Actions Needed to Further Enhance Restructuring and Address Affordability Risks. GAO-12-437. Washington, D.C.: June 14, 2012. Joint Strike Fighter: Restructuring Added Resources and Reduced Risk, but Concurrency Is Still a Major Concern. GAO-12-525T. Washington, D.C.: March 20, 2012. Joint Strike Fighter: Implications of Program Restructuring and Other Recent Developments on Key Aspects of DOD’s Prior Alternate Engine Analyses. GAO-11-903R. Washington, D.C.: September 14, 2011. Joint Strike Fighter: Restructuring Places Program on Firmer Footing, but Progress Is Still Lagging. GAO-11-677T. Washington, D.C.: May 19, 2011. Joint Strike Fighter: Restructuring Places Program on Firmer Footing, but Progress Still Lags. GAO-11-325. Washington, D.C.: April 7, 2011. Joint Strike Fighter: Restructuring Should Improve Outcomes, but Progress Is Still Lagging Overall. GAO-11-450T. Washington, D.C.: March 15, 2011.