Background According to the Senate Committee on Homeland Security and Governmental Affairs report concerning PMIAA, the purpose of PMIAA is to improve program and project management in certain larger federal agencies. The act includes requirements for OMB, OPM, and the 24 agencies listed in the CFO Act. PMIAA requires OMB’s Deputy Director for Management or the designee to, among other things: adopt government-wide standards, policies, and guidelines for program and project management for executive agencies; engage with the private sector to identify best practices in program and project management that would improve federal program and project management; conduct portfolio reviews of agency programs not less than annually, to assess the quality and effectiveness of program management, in coordination with Program Management Improvement Officers (PMIO); establish a 5-year strategic plan for program and project conduct portfolio reviews of programs on our High-Risk List. The two types of portfolio reviews required by PMIAA—the portfolio reviews of agency programs and the portfolio reviews of programs identified as high risk on our High-Risk List—are separate requirements. For purposes of this report, we define programs, projects, and portfolios consistent with how those terms are defined in OMB’s PMIAA strategic plan. OMB defines program as the functions or activities which agencies are authorized and funded by statute to administer and enforce. Programs typically involve broad objectives. OMB views projects as temporary efforts with defined scopes to create products or services to improve the efficient and effective implementation of programs. Because programs are comprised of projects, programs inherently address the projects subsumed within them. Consequently, our discussions of programs throughout this report also pertain to projects. Finally, OMB defines portfolios as organized groupings of programs whose coordination in implementation enables agencies to achieve their objectives. The act also established the Program Management Policy Council (PMPC), an interagency forum for improving agency practices related to program management. OMB’s Deputy Director for Management chairs the PMPC. The PMPC responsibilities include advising OMB on the development and applicability of government-wide standards for program management transparency. Furthermore, the act requires PMPC members “to discuss topics of importance to the workforce,” such as workforce development needs and major challenges across agencies in managing programs. As chair of the PMPC, OMB’s Deputy Director is required to preside at meetings, determine agendas, direct the work, and establish and direct its subgroups, as appropriate. The act requires the PMPC to meet not less than twice per fiscal year. Additionally, OPM’s Director, in consultation with OMB’s Director, is required to issue regulations that: identify key skills and competencies needed for a program and a project manager in an agency; establish a new job series, or update and improve an existing job series, for program and project management within agencies; and establish a new career path for program and project managers within an agency. Overall, OPM’s role in implementing PMIAA is to establish a new job series or update an existing job series by providing the occupational standards that agencies will need to develop a trained and competent workforce with the program and project management experience, knowledge, and expertise to solve management challenges and support agency decision-making. The act requires OPM to establish new—or revise existing—occupational standards in consultation with OMB. Occupational standards are included within OPM’s classification guidance, which is provided to agencies to assist in classifying positions. This guidance helps agencies to determine the proper occupational series, position title, and grade of each position. The act requires OMB’s Deputy Director of Management to oversee implementation of the standards, policies, and guidelines for executive agencies. OMB implemented some PMIAA requirements using existing processes put in place to implement GPRAMA. We previously reported that GPRAMA provides important tools that can help decision makers address challenges facing the federal government, such as the annual reviews of progress on agency strategic objectives conducted during strategic reviews and the implementation of federal government priority goals. Federal government priority goals, also known as cross-agency priority (CAP) goals, are written by OMB in partnership with agencies. GPRAMA requires OMB to coordinate with agencies to develop CAP goals, which are 4-year outcome-oriented goals covering a number of complex or high-risk management and mission issues. For example, OMB directed agencies to align their noninformation technology major acquisition programs with relevant strategic objectives so they could assess progress for the PMIAA required program portfolio reviews concurrent with required GPRAMA strategic reviews. GPRAMA also requires OMB to present a program inventory of all federal programs by making information available about each federal program on a website. Finally, GPRAMA required OMB to establish a number of CAP goals intended to cover areas where increased cross-agency collaboration is needed to improve progress towards shared, complex policy or management objectives across the federal government. OMB uses CAP goals to address issues outlined in the President’s Management Agenda. For example, OMB wrote a CAP goal to improve management of major acquisitions across the government which complements PMIAA and its required activities. PMIAA requires the OMB Deputy Director, as chair of the PMPC, to conduct portfolio reviews of programs from our High-Risk List. The PMPC is also required to review programs we identify as high risk and to make recommendations for actions to be taken by the Deputy Director for Management of OMB or a designee. See figure 1 below for an overview of roles and responsibilities of OMB, OPM, the PMPC, and agencies. OMB, OPM, and CFO Act Agencies Have Taken Steps to Implement PMIAA, but Some Program and Project Management Capacity Limitations Exist Agencies responsible for PMIAA implementation have taken steps to complete some requirements, but actions remain to fully implement the law (see Table 1). OMB’s PMIAA Strategic Plan Incorporated Leading Practices OMB met the PMIAA requirement “to establish a five-year strategic plan for program and project management.” The plan OMB developed details three key strategies to implement PMIAA: (1) coordinated governance, (2) regular OMB and agency engagement reviews, and (3) strengthening program management capacity to build a capable program management workforce. The three strategies focus on areas such as clarifying key roles and responsibilities, identifying principles-based standards, and identifying plans for enhancing workforce capabilities. The plan describes the roles and functions of the PMIOs, the PMPC, and the requirements of the agency implementation plans. It outlines a phased approach for implementing PMIAA actions with milestones occurring throughout the 5- year period. We found that OMB followed several strategic planning leading practices in the creation of the PMIAA strategic plan. First, the plan incorporates general goals and objectives for agencies’ implementation of PMIAA with three corresponding strategies explaining OMB’s overall approach. OMB followed a second leading practice by gathering input from stakeholders. OMB staff told us they solicited input from congressional staff, and members of external organizations like the Federal Program and Project Management Community of Practice (FedPM CoP). Agencies’ staff also confirmed to us that they had input into the OMB plan. Third, OMB demonstrated interagency collaboration in its efforts to establish and lead the PMPC and its efforts to work with the FedPM CoP to address any issues identified by agencies. Finally, the plan included a timeline with quarterly milestones to track completion of PMIAA’s activities and to gauge progress toward achieving the desired results of PMIAA. OMB’s Program and Project Management Standards Are Less Detailed Compared with Accepted Program and Project Management Standards PMIAA required OMB to establish standards and policies for executive agencies consistent with widely accepted standards for program and project management planning and delivery. A consistent set of government-wide program management standards and policies is important because it helps ensure that agencies utilize key program management practices to improve the outcomes of government programs. OMB published in June 2018 a set of standards for program and project management as part of OMB’s PMIAA strategic plan. OMB’s strategic plan directed agencies to apply these 15 standards to internal management processes for planning, implementing, and reviewing the performance of programs and activities. OMB staff told us they decided to develop this set of standards rather than adopt an existing set of consensus-based standards, such as the widely accepted standards for program and project management from the Project Management Institute (PMI). PMI is a not-for-profit association that provides global standards for, among other things, project and program management. The PMI standards are utilized worldwide and provide guidance on how to manage various aspects of projects, programs, and portfolios and are approved by the American National Standards Institute (ANSI). OMB staff told us that they decided not to specifically adopt the PMI standards because they wanted to allow agencies to use a range of standards that agencies had already developed and were using to manage their programs, such as standards developed in-house by NASA for their space flight programs. OMB further directed CFO Act agencies that the 15 standards and application of them should be incorporated or aligned with existing agency-specific program management policies and practices, and tailored to reflect program characteristics. OMB staff told us that they chose the approach to provide more principle-based standards, as opposed to specific standards, to be flexible enough for a range of government agencies to apply them. OMB’s standards are similar in definition to PMI standards, but they are less detailed by comparison. Our analysis of OMB’s standards shows that OMB uses similar definitions for all 10 of PMI’s program management standards and nine out of 10 of PMI’s project management standards, such as risk management and change management. However, OMB program and project management standards are less detailed when compared to PMI’s standards in the following ways: OMB standards do not provide a minimum threshold against which agencies can gauge to what extent they have met each standard. PMI’s Standard for Program Management provides the definition of a standard but also what components are required for an entity to confirm that the standard has been met. For example, meeting the program financial management standard in PMI requires a financial management plan to be developed, along with its related activities. This plan allows entities applying the standard to confirm whether they have met the standard for program financial management or not. OMB’s standards do not distinguish between how the standards apply differently to programs and projects while PMI has separate detailed standards for program management and for project management. The project management standards from PMI provide details on how the standards apply to more granular tasks, such as establishing a quality management or communication plan for a specific project. OMB’s standards do not distinguish between how the standards relate to each other during a program or project while PMI’s Standard for Program Management details how project standards help build on each other during a program. For example, a program scope management plan is needed to determine the type of schedule management planning that is necessary to accomplish the delivery of the program’s outputs and benefits. OMB provides minimal guidance on how standards apply differently across the life cycle of a program or project while PMI’s Standard for Program Management provides information detailing when a specific standard should be utilized in different ways during the life cycle of a program. For example, in the beginning of a program, risk management should be planned and an initial risk assessment created. Later, during program implementation, risk management tasks focus on monitoring, analyzing risk, and responding to risk. If the standards had the additional detail, it would be possible to determine if agencies are meeting them and properly applying them to programs and projects. OMB Does Not Have a Detailed Governance Structure for Further Developing Program Management Standards Our work on the Digital Accountability and Transparency Act of 2014 (DATA Act) standards has emphasized the necessity for a governance structure with a clear set of policies and procedures for developing and maintaining standards over time that are consistent with leading practices. A governance structure is important because it helps ensure that the standards are developed, maintained, adjusted, and monitored over time. The DATA Act is similar to PMIAA because PMIAA gives OMB responsibility to develop standards for program management, and the DATA Act gives OMB and the Department of the Treasury responsibility for establishing data standards for the reporting of federal funds. These standards specify the data to be reported under the DATA Act and define and describe what is to be included in each element with the aim of ensuring that information will be consistent and comparable. Several governance models exist that could inform OMB’s efforts to help ensure that the standards are developed, maintained, adjusted, and monitored over time. These models define governance as an institutionalized system of decision rights and accountabilities for planning, overseeing, and managing standards. Many of these models promote having a common set of key practices that include establishing clear policies and procedures for developing, managing, and enforcing standards. A common set of key practices endorsed by standards setting organizations including the National Institute of Standards and Technology, ANSI, and the American Institute of Certified Public Accountants recommend that governance structures should include the key practices shown in the text box below. Key Practices for Governance Structures 1. Delineating roles and responsibilities for decision-making and accountability, including roles and responsibilities for stakeholder input on key decisions. 2. Obtaining input from stakeholders and involving them in key decisions, as appropriate. 3. Developing and approving standards. 4. Making decisions about changes to existing standards and resolving conflicts related to the application of standards. 5. Managing, controlling, monitoring, and enforcing consistent application of standards. OMB staff told us they did not have any additional documentation about the governance structure used to develop the program management standards and how OMB will further develop and maintain them. We compared available information about OMB’s governance structure for developing and maintaining program management standards to the five key practices on governance structures and found OMB’s governance structure is incomplete in each of the five key practices. OMB has not delineated roles and responsibilities for decision-making and accountability, including responsibilities for stakeholder input on key decisions. OMB’s strategic plan notes that one role of the PMPC is to help further develop the program management standards. However, OMB has not provided information on how roles and responsibilities will be assigned to continue developing standards in the future. Without clearly delineated roles and responsibilities, there is a risk of confusion which could impede action and accountability for future improvements to program management standards. Further, having clearly delineated roles and responsibilities is particularly important during periods of transition when administrations change. OMB has an incomplete plan for how it will obtain input from stakeholders and involve them in decision-making. OMB received input from stakeholders on the standards it developed in 2018, though the strategic plan states that standards will be further developed with the PMPC in the fourth quarter of fiscal year 2020. However, the strategic plan does not give details on how the PMPC and others will further develop standards. Without robust and comprehensive outreach to individuals who will use or otherwise be affected by the standards, the opportunity to learn from stakeholder experience and perspectives, or anyone who will use or otherwise be affected by the standards, may be diminished. OMB has an incomplete process for developing and approving program management standards. OMB developed and approved the existing standards by obtaining stakeholder input and releasing their approved standards in its strategic plan. However, the strategic plan does not provide documentation on how that process was structured and how it will function in the future. Thus, it is unclear how OMB plans to further develop the standards and what responsibilities and resources will be required from OMB, the PMPC, and agencies under the leadership of the agency PMIOs. OMB has not defined a process for making decisions about changes to existing standards and describing how conflicts related to the application of standards would be resolved. Therefore, it is unclear if or how the standards will be periodically reassessed and updated as circumstances change and leading practices in program and project management are identified. Also, lack of consensus on standards and conflict over how to use them can lead to weakened acceptance and inconsistent application. OMB has not defined a process for managing, controlling, monitoring, and enforcing consistent application of standards. OMB has not developed or directed any type of review or oversight process to determine the adequacy of existing or newly developed standards agency use to manage programs. Having such a process could help agencies to achieve a balance between consistent application of standards and flexible application to account for differences in programs, agency missions, and other factors. However, OMB staff told us that they consider the PMIAA program portfolio review process as a way to help monitor and enforce program standards, as they have a view into how each agency is applying standards for their particular portfolio of programs. Additionally, OMB has given agencies flexibility in using existing agency standards and flexibility to adopt or develop new ones. Without a review mechanism, OMB lacks reasonable assurance that agencies’ efforts to use existing standards or develop new ones will align with government-wide efforts to improve program and project management. Also, establishing an approach to monitoring agencies’ efforts would help identify opportunities to improve program management standards. Without having a governance structure for the program standards, the potential exists that standards will develop in an ad hoc manner, may be applied inconsistently or not at all, and may not be updated to reflect new developments in program management. Further, having a governance structure for managing efforts going forward better positions OMB to sustain progress on program standards as they change over time. OMB Leveraged Existing Performance Reviews, but Reviews Are Limited to Major Acquisitions PMIAA requires agencies and OMB to regularly review portfolios of programs to assess the quality and effectiveness of program management and identify opportunities for performance improvement. To conduct these portfolio reviews, OMB Circular A-11 notes that agencies and OMB are to use a set of broadly applicable program management principles, practices, and standards associated with successful program outcomes, in addition to more specific standards based on the type of program under review. As a way to help agencies acclimate to the requirements of PMIAA, OMB leveraged two components of the GPRA Modernization Act of 2010 (GPRAMA): the strategic review and a cross-agency priority (CAP) goal. OMB guidance stated that agencies’ portfolio reviews of programs would be conducted and integrated to the extent practical with strategic reviews. Furthermore, OMB staff told us that the implementation of PMIAA and the CAP goal for improving management of major acquisitions (CAP Goal 11) shared complementary goals and strategies. For example, the CAP Goal 11 action plan includes the routine monitoring of federal program management progress. Consequently, OMB staff said they decided that the first PMIAA program portfolio reviews would focus on major acquisitions. Excerpt from OMB Cross-agency Priority Goal 11 from 2018 President’s Management Agenda: Improve Management of Major Acquisitions Federal agencies will ensure that contracts supporting transformative and priority projects meet or beat delivery schedules, provide exceptional customer service, and achieve savings or cost avoidance for the taxpayer. The Challenge: Major acquisitions—which vary in size by agency but often exceed $50 million—account for approximately one-third of annual federal spend on contracts. These large contracts frequently support projects meant to transform areas of critical need. Yet major acquisitions often fail to achieve their goals because many federal managers lack the program management and acquisition skills required to successfully manage and integrate large and complex acquisitions into their projects. These short- comings are compounded by complex acquisition rules that reward compliance over creativity and results. The Strategies: Agencies will pursue three strategies: 1) strengthen program management capabilities in the acquisition workforce; 2) use modern and innovative acquisition flexibilities; and 3) track investments using portfolio, program, and project management principles. OMB Reported Lessons Learned from Pilot, but Did Not Follow Most Leading Practices for Pilot Design In 2018, OMB conducted a pilot project involving program portfolio review focused on noninformation technology (IT) major acquisition programs. According to OMB staff, the pilot project gave agencies the opportunity to complete “dry runs” for the PMIAA-required portfolio reviews and to provide lessons learned in anticipation of the fiscal year 2019 portfolio reviews. OMB planned for the results from the pilot to provide information for internal dialogue and decision-making about subsequent portfolio reviews. Further, according to OMB’s strategic plan, the purpose of the pilot was (1) to determine how well agency program portfolios of non-IT major acquisitions were performing throughout the life cycle of the investment using a set of standards and practices, and (2) to refine the process of coordinating program portfolio reviews as a component of OMB agency strategic reviews. For the pilot, OMB staff directed agencies to assess the cost, schedule, and performance of agency-selected acquisition portfolios. One result from the pilot was that agencies demonstrated a range of maturity in their abilities to collect data for these required program portfolio measures from their various departments and program types. OMB staff told us pilot agencies found it easier to compile data on major construction projects compared to service contracts. Consequently, an agency doing many of these projects might be more advanced than an agency for which major acquisitions focus on services. Department of Veterans Affairs (VA) staff shared their lessons learned from their participation in pilot portfolio reviews, as seen in the text box below. OMB staff said that they determined that the portfolio review process worked sufficiently well for the pilot agencies and continued their planned strategy of focusing solely on non-IT major acquisition programs for fiscal year 2019 portfolio reviews. Example of Department of Veteran Affairs (VA) Lessons Learned from Pilot Portfolio Review The VA looked at the effectiveness of portfolio management during the Office of Management and Budget noninformation technology major acquisition pilot portfolio review by focusing on the agency’s adherence to best practices in assessing project performance and progress. VA officials said this pilot informed their decision-making and was successful in the following ways: 1. The pilot helped VA determine logical ways to manage a portfolio by showing what data were helpful to make impactful decisions. 2. VA learned how best to display the data on cost, schedule, scope, and quality of outcomes on a dashboard to make it accessible and comparable across the agency. 3. VA learned that it needs to collect better quality data so that project management principles can be instituted and aligned across the agency. A well-developed and documented pilot program can help ensure that agency assessments produce information needed to make effective program and policy decisions. Such a process enhances the quality, credibility, and usefulness of evaluations in addition to helping to ensure the effective use of time and resources. We have identified five leading practices that, taken together, form a framework for effective pilot design, as seen in the text box below. OMB fulfilled the first leading practice of establishing objectives in its design of the PMIAA pilot program portfolio review. OMB’s PMIAA strategic plan and the CAP Goal 11 Action Plan stated the objectives of the pilot. In addition to the two objectives listed in the PMIAA strategic plan, the CAP Goal 11 Action Plan lists seven pilot objectives, as seen in the text box below. PMIAA Pilot Program Portfolio Review Objectives 1. Perform portfolio management preparation activities 2. Identify first portfolio of major acquisitions 3. Align portfolio with agency strategic goals 4. Collect performance data for each item in the portfolio 5. OMB officials said that they did not structure the pilot to follow the remaining four leading practices for effective pilot design. However, OMB said that it learned that the pilot agencies demonstrated several program management capabilities. They also learned that it would be important to tailor portfolio reviews to the agency and the program to account for significant differences in the types of acquisitions and the level of program management maturity. Despite identifying lessons learned from its pilot program portfolio review, in neglecting to fully follow leading practices, OMB may have missed opportunities to make additional improvements for fiscal year 2019 portfolio reviews. Going forward, as OMB expands the portfolio reviews to other types of program areas beyond non-IT major acquisitions, it has the opportunity to develop and learn from additional pilots. Although OMB staff have not yet determined if they will do additional pilots for program management in the future, they could decide to pilot the portfolio reviews of grants that they plan to initiate in fiscal year 2020. OMB Limited Its Portfolio Reviews to Non-IT Major Acquisition Programs For fiscal year 2019, OMB directed all agencies to select portfolios of non-IT acquisition programs and align them with relevant strategic objectives as part of their internal agency strategic review processes. In spring 2019, OMB expected agencies to discuss one to two of these major-acquisition portfolio reviews during their strategic reviews with OMB. OMB expected agencies to track the cost, schedule, and performance of their selected major acquisition programs. However, OMB reports that not all agency program portfolio reviews were completed because OMB was behind in scheduling the reviews due to the partial government shutdown. According to documents we reviewed and what OMB staff told us, in October 2019 OMB completed agency program portfolio reviews with ten agencies: the Departments of Commerce, Homeland Security, Housing and Urban Development, Labor, and Transportation; the General Services Administration, the Social Security Administration, NASA, the National Science Foundation, and the US Agency for International Development. OMB staff also told us that they also held preparatory meetings with agencies to set expectations for future portfolio reviews. OMB reported that these one-on-one meetings were held with 12 agencies as of October 2019 to discuss their initial portfolio structures and other transformative initiatives. Portfolio reviews in 2020 are to expand in scope to include grants, and also will continue acquisition portfolio reviews as part of the agency’s routine management processes. However, OMB has not yet identified other program areas, such as research and development or benefit programs, to be included in future portfolio reviews. Standards for Internal Control in the Federal Government states that effective information and communication are vital for an entity to achieve its objectives. Specifically, management should externally communicate necessary quality information to achieve its objectives. Increasing communication to agencies about specific program areas, portfolio review procedures, and expectations beyond 2020 could help ensure continued progress to implement PMIAA more broadly. Furthermore, communicating such procedures with specific time frames could help agencies better direct their efforts to improve the portfolio review processes. OMB Has Not Fully Implemented an Inventory of All Federal Programs GPRAMA requires OMB to make a list of all federal programs identified by agencies publicly available, on a central government-wide website. The implementation of the program inventory is a critical tool to help decision makers better identify and manage programs across the federal government. Among other things, the completion of the program inventory would provide agencies and Congress with a comprehensive list of programs, so it would be clear how many programs agencies are managing and how they relate to their strategic objectives and portfolios of programs at each agency. Having a program inventory could also help ensure a match between the number of agency programs and needed program manager resources. Agencies continue to struggle with challenges defining their programs. Officials from three of the five selected agencies we spoke with told us that they have not yet identified all of their programs and projects. In our first report on the program inventory in October 2014, we noted similar issues. For example, agencies were not using the same program definition approach across their subcomponents or offices, which limited comparability of their own programs. We made eight recommendations in that report to the Director of OMB to update relevant guidance to help develop a more coherent picture of all federal programs and to better ensure information is useful for decision makers. As of October 2019, OMB had not taken any actions in response to the eight recommendations. While OMB has provided a timetable for action in its June 2019 A-11 guidance, this does not complete the recommendation. In September 2017, we made two recommendations to OMB to make progress on the federal program inventory. First, we recommended that OMB consider using a systematic approach for the program inventory, such as the one we developed from principles of information architecture. Information architecture—a discipline focused on organizing and structuring information—offers an approach for developing a program inventory to support a variety of uses, including increased transparency for federal programs. OMB staff told us that they considered our information architecture approach and noted that a structured information architecture format is used on USASpending.gov. However, OMB staff told us they had not yet determined how the information architecture format of USASpending.gov—which is focused on spending data—could be used to meet additional information reporting requirements and our past recommendations related to the inventory. We made a second recommendation that OMB should revise and publicly issue OMB guidance—through an update to its Circular A-11, a memorandum, or other means—to provide time frames and associated milestones for implementing the federal program inventory. As mentioned above, OMB did provide a timetable but it does not have milestones. According to the timetable, beginning with the 2021 budget cycle, agencies’ program activities will be used for the inventory’s program-level reporting requirements. This will allow OMB and agencies to present program-level spending data by leveraging what is reported on USASpending.gov as required by the DATA Act. However, OMB’s guidance does not cover other inventory information reporting requirements, or the actions we recommended in October 2014. We will continue to monitor progress. We continue to believe it is important for OMB to implement our program inventory recommendations. Such an inventory could be a critical tool to help decision makers better identify and manage fragmentation, overlap, and duplication across the federal government. Additionally, fully taking action on these recommendations would assist agencies in identifying programs, better prepare for future PMIAA portfolio reviews, and help match resources to agencies’ program management needs. Further, OMB developed three different definitions for what constitutes a “program” or “program activity” that it provided to agencies in its PMIAA, GPRAMA, and DATA Act guidance, respectively. OMB developed each of these definitions independently and in response to three different statutory requirements. OMB staff told us that these three requirements differ in their legislative intent. The definitions and their associated guidance are in the table below. OMB has not reconciled these overlapping, yet divergent, definitions of what constitutes a “program” or “program activity.” According to Standards for Internal Control in the Federal Government, management should ensure that specific terms are fully and clearly set forth so they can be easily understood. Standards for Internal Control in the Federal Government also states that management should design processes that use entities’ objectives and related risks to identify information requirements needed to achieve objectives and address risks. OMB has defined what constitutes a “program” or “program activity” in PMIAA, GPRAMA, and the DATA Act each, but its three different program definitions and approaches to determining what is a “program,” could cause confusion for agencies. Agency officials from the Department of Energy told us they are already experiencing confusion over how to appropriately apply the applicable program definition to identify their programs for PMIAA. Agency officials from Treasury told us that different definitions for programs could contribute to confusion as they work to implement PMIAA within the Department. The inconsistent approaches may increase the burden on agencies as they work to identify, maintain, and report on three sets of differently defined programs. Conversely, clarifying the definitions could help agencies and OMB identify synergies across the three laws and increase transparency. For example, providing explanations of how the term “program” or “program activity” is used across the three statutory definitions and developing a crosswalk to show similarities and differences could provide more clarity for agencies. Then, spending and performance data can be aligned with agency strategic goals, which could be monitored, reviewed, and reported in a streamlined manner. OPM Meeting Workforce Requirements of PMIAA OPM followed PMIAA requirements to create policy and guidance. Specifically, according to documents we reviewed, OPM (1) worked with subject matter experts to develop program and project management skills and competencies, (2) updated the program management 0340 job series and created guidance for identifying project management positions, (3) plans to release a career path for program and project managers by the end of calendar year 2019, and (4) plans to create a unique job identifier code that can be used to pinpoint program and project managers in any job series. These efforts will form the foundation needed by agencies to strengthen resource and talent management. Competency modeling. Since enactment of PMIAA, OPM identified skills and competencies which will be required for program and project managers. According to documents we reviewed, OPM met with subject matter experts and human capital staff in agencies to help identify the skills needed to develop the competency model. OPM also conducted a literature review looking at prior competency studies and industry practices to help identify and support program and project management competencies. OPM also drew from Project Management Institute resources, such as the Project Management Body of Knowledge and the Standard for Program Management, as part of identifying its competencies. The resulting competencies are in two categories: general and technical. General competencies focus on interpersonal or general on-the-job skills such as teamwork and problem solving. Technical competencies more narrowly focus on particular skills needed to run programs and projects, such as risk management and cost-benefit analysis. OPM documents stated that agencies will need to determine the applicability of these competencies to positions within their agency. Agencies must determine if staff meet the competencies, and if not, staff will have the opportunity to develop them or must move to a different job series, according to OPM staff. OPM staff also said additional competency assessment steps are needed to finalize the model. Agencies will be given time to consider the competency model. In addition, OPM will use subject matter expert panels to further develop the model, according to OPM documents we reviewed. Updated job series. To implement job series requirements in PMIAA, OPM staff conducted an occupational study and determined that pre- existing classification policy was sufficient for classifying program management work rather than creating a new job series classifying program management positions, according to OPM staff. Prior to OPM updating the program management 0340 job series for PMIAA, the classification standard was not developed, as it did not contain competencies describing what qualifications staff were required to meet as a program manager. In May 2019, OPM released the updated job series classification guidance designed to assist agencies in determining which employees fit in the job series. OPM also released guidance for classifying project managers to help agencies specifically identify project managers in any occupational job series. According to the memorandum sent by the Acting Director of OPM to agencies with the OPM classification guidance, agencies are required to implement the policy and guidance to covered positions by May 1, 2020. Career path. OPM staff told us that they have developed a career path for program and project managers that is currently in internal review. They said that the value of the updated career path is that it will highlight training and skills needed to progress in a program management career. According to the presentation given by OPM at the 2019 April PMPC meeting, the career path will contain: (1) a career progression outline for employees to move among and across jobs in program and project management, (2) help for employees and supervisors to plan and sequence appropriate career training and development for each general and technical competency, and (3) a list of common degrees and certifications completed by program and project managers, among other things. Staff told us they plan to release the program and project management career path for agency comment by the end of calendar year 2019. Job identifier for program managers and project managers. Because program and project managers are found in other job series outside the 0340 program management series, OPM is developing a job identifier code that can be attached to any job series for the purposes of identifying program and project managers. OPM staff told us that program managers classified to the 0340 series means that the position does not have a specialization. If the position requires specialized expertise, the position would be classified to a specialized occupational series but would also have a program management job identifier code. For example, since a grants managers is also a program manager, “grants manager (program management)” would be his or her official title. Project management positions will also use a job identifier to identify project managers in any occupational series. The job identifier will allow employees with a specialization to be designated program and project managers, while still maintaining their original career path. OPM staff told us they plan to complete this project in 2020. OPM and Agencies in Early Stages of Workforce Planning Our analysis of OPM Enterprise Human Resources Integration data shows that the 0340 job series included about 15,000 employees across all 24 CFO Act agencies in fiscal year 2018. However, OPM reported that not all employees in this job series are actually program and project managers; conversely, many program and project managers are working outside of the 0340 job series. Selected agencies reported varying degrees of difficulty identifying program and project managers. For example, NASA staff reported that they were able to identify almost all their program and project managers. In contrast, the Department of the Treasury reported that it faces challenges identifying the number of program and project managers outside of the program management job series, as this would require a resource-intensive manual effort, made more challenging by the agency’s large, complex, and decentralized structure. The Department of Energy (DOE) staff said they have not completed the count of their program managers. The Departments of Commerce and Veterans Affairs also report they do not know the number of program and project managers in their departments, respectively. The Department of Commerce staff told us that they cannot accurately identify the number of program and project managers until they can use the job identifier that they expect OPM to release in 2020. Further, Commerce officials told us they are also continuing to work to identify program managers and engaged the Project Management Institute (PMI) to request a list of those within Commerce who have the Project Management Professional (PMP) certification. PMI was able to provide Commerce details about the numbers of PMPs at Commerce, but PMI declined to share the names of those individuals with the PMP certification. In OPM’s 2018 Federal Workforce Priorities report, OPM recognizes that not all agencies have adequately analyzed workload demands, staffing levels, or current and future skills needs—all steps in workforce planning. As part of the OPM human capital framework, agencies are required to develop a human capital operating plan which is an agency’s human capital implementation document. These plans are to describe how agencies will execute the human capital strategies needed to implement the agency’s strategic plan and Annual Performance Plan (APP). Agencies are also required to include program specific strategies (e.g., hiring, closing skills gaps, etc.) in the APPs as appropriate. Effective workforce planning can help agencies focus on determining how many program and project managers they have, how many they may need, what skills gaps exist, and what training and other strategies can help address skills gaps. OPM’s workforce planning model is comprised of five steps: 1. Set strategic direction; 2. Analyze workforce, identify skills gaps, and conduct workforce 3. Develop action plan; 4. Implement action plans; and 5. Monitor, evaluate, and revise. The discussion below describes how OPM and agencies are working to strengthen the program management workforce in the context of OPM’s workforce planning model. Some activities may span more than one phase of workforce planning. Set strategic direction. The PMIAA strategic plan establishes direction for agencies to build its program management capacity and capability with its third strategy, “Strengthening Program Management Capacity to Build a Capable Program Management Workforce.” Setting strategic direction also involves linking work activities to the objectives of a strategic plan. OPM’s planned activities, such as updating the classification standards and creating a job identifier, are critical to executing this strategy so agencies can identify their workforce and build program management capacity through training, career paths, and mentorship opportunities. Analyze workforce, identify skills gaps, and conduct workforce analysis. OPM and agencies are in the early stages of identifying who their program and project managers are and what human capital strategies might be needed to address agencies’ needs. Documents we reviewed showed that OPM also worked with the Chief Human Capital Officers Council, the Chief Administrative Officers Council and others to develop competencies. These competencies provide a foundation for the subsequent assessment of program and project manager skills. Develop action plan. In their PMIAA implementation plans, some agencies have identified available training and possible recruitment and hiring strategies. In OPM’s model, agencies need to complete their workforce analysis before they can develop their action plans. Implement plan. This step is dependent on agencies developing action plans. However, OPM and agencies have already started to develop staff in the absence of plans. For example, OPM is working with agencies to identify program management training matching desired competencies to be placed in an online training repository that will be accessible to all agencies. OPM staff told us that agencies would provide the trainings from their learning management systems and offer them for interagency access. OPM is developing this training and development repository which will house agency-owned courses and also identify mentors in project and program management, according to OPM staff. OPM will house the repository on its training and development policy wiki at https://www.opm.gov/wiki/training/index.aspx. Each PMIO is to also establish a website with agency-specific program management tools and resources. Additionally, OMB recognized that the Federal Program and Project Management Community of Practice (FedPM CoP), scaled up from a community of practice housed in DOE, could be an important partner in supporting PMIAA implementation. As of April 2019, more than 1,000 managers had joined the FedPM CoP as indicated in its briefing to the PMPC. The FedPM CoP has identified several project management-related documents that are now available on the PMIAA portal. To further develop program managers, OMB is working with agencies to improve mentoring and recognition efforts. To improve mentoring government-wide, OMB reports that PMIOs will work with agency chief human capital offices to develop and implement a mentoring strategy for agency program managers. OMB also plans to take existing mentorship programs established in more functionally aligned-management fields (e.g., information technology, acquisition) and expand them to include a broader range of management career paths. To improve recognition efforts in acquisitions, the Chief Acquisition Officer Council plans to establish an annual award to recognize federal program manager excellence. Monitor, evaluate, and revise. This step cannot begin until agencies develop and implement their workforce action plans. As agencies begin to monitor their implementation of these plans, they will need to determine if any skills gaps exist in the program and project manager occupational series. OPM regulations require agencies to describe in their human capital operating plans agency-specific skills and competency gaps that must be closed through the use of agency selected human capital strategies. Agencies must also have policies and programs that monitor and address skills gaps within government wide and agency-specific mission-critical occupations. OPM has not yet determined if program and project management occupations are experiencing mission-critical skills gaps across the government, and OPM staff noted that agencies are not specifically required to report program and project manager skills gaps in their annual human capital operating plans. OMB and OPM Completed Some PMIAA Requirements Late OMB and OPM both missed statutory deadlines to fulfill requirements in PMIAA. In June 2018, OMB issued the required PMIAA agency implementation guidance in the PMIAA strategic plan, 6 months after the statutory deadline of December 2017. According to OMB staff, this delay was due to their own research project to (1) build sufficient knowledge in program and project management; and (2) increase stakeholder support in Congress and with agencies for its approach. Specifically, OMB met with experts from PMI, academics, consulting firms, federal chief senior level officer (CXO) councils, and other agency officials to increase its own understanding of program and project management principles. OMB staff told us that they used the collected information to draft initial guidance, which they then shared with congressional stakeholders and executive branch agency officials to obtain feedback and incorporate changes. OMB staff also told us that it was a transition year from one administration to another administration, and this transition was an additional factor in delaying completion of the guidance. None of the selected agencies’ staff identified an impact from the delayed guidance. OPM officials told us they missed the statutory deadline to complete their required activities after the issuance of OMB guidance. The release of the policy and guidance was due to the partial government shutdown from December 22, 2018 to January 25, 2019, along with a 3-month delay due to OPM’s own internal review and clearance process. As a result, OPM released the key skills and competencies needed for program and project management on April 5, 2019, and the classification guidance for the program manager job series 0340 and project manager interpretative guidance on May 2, 2019. OPM officials told us that agencies have 1 year from the date of issuance to comment on any language in the guidance. None of the selected agencies’ staff identified an impact from OPM’s delays, although one agency expressed concern that the pace of their efforts to identify program and project managers is dependent on OPM completing the job identifier. Figure 2 shows the delays in releasing OMB and OPM guidance. PMPC Has Met Three Times and CFO Act Agencies Have Started to Implement PMIAA Requirements OMB officials established the PMPC in 2018 and fulfilled requirements that it meet at least twice per year. By September 2018, the 24 CFO Act agencies had all appointed a PMIO and held three PMPC meetings, in September 2018, April 2019, and September 2019. Selected agenda items for these PMPC meetings included: status updates on OPM completing program and project manager competencies, job series, and career path; breakout sessions to discuss PMIAA implementation approaches with discussion of PMPC priorities and focus for 2020. At the April 2019 PMPC meeting, for example, staff from the Department of Veterans Affairs and the National Science Foundation shared some best practices, such as how to improve the tracking performance of portfolios, programs, and projects. According to OMB documents we reviewed, OMB plans to: convene the PMPC in the first quarter of each calendar year to prepare for upcoming OMB and agency strategic review meetings; use the PMPC meeting in the third quarter of the calendar year to review findings and outcomes from the most recent strategic review; update program and project management standards based on its findings and feedback at the PMPC meeting in the fourth quarter of 2020; use the PMPC to develop revised strategies, initiatives, and priorities to be reflected in an updated 5-year strategic plan at the PMPC meeting in the fourth quarter of 2021; and use the PMPC to focus on improving our high-risk areas at some future point. At the September 2019 PMPC meeting, OMB informed agencies of PMIAA implementation resources placed on OMB’s online portal for PMIAA and discussed OMB’s observations on portfolio reviews completed in 2019. One observation was the need to reinforce better visualization of performance data. In addition, OPM updated the PMPC on the status of its required PMIAA workforce efforts. The PMPC decided its primary focus for the year 2020 should be on the third strategy of the PMIAA strategic plan to build a capable workforce. Officials from the selected agencies that we interviewed provided us some suggestions on how OMB can improve the functionality of the PMPC. Table 3 illustrates the range of these suggestions: The PMPC met twice in 2019, as required by PMIAA, and has not established any working groups to help execute its significant responsibilities to share leading practices, develop standards, and help improve the workforce. Agencies have taken initial steps to incorporate requirements into program efforts. According to OMB guidance, agencies were to report in implementation plans how they are institutionalizing PMIAA efforts— especially PMIO responsibilities—into existing program and project management practices. OMB requested that agencies include 10 specific elements in their implementation plans, such as: identification of the agency PMIO, identification of major acquisition portfolios, and strategies and actions for enhancing training and improving recruitment and retention of program and project managers. These plans were due to OMB by November 30, 2018. We reviewed PMIAA draft implementation plans for 22 of the 24 CFO Act agencies and determined the extent to which agencies included the required elements in their plans. In its PMPC meeting in April 2019, OMB reported that a majority of agencies only partially included OMB requirements in their draft implementation plans. OMB told us they have not directed agencies to address missing requirements nor have they required agencies to finalize their draft implementation plans. They told us that they view the implementation plans as an opportunity for each agency to engage with OMB and discuss how they will implement PMIAA. OMB staff told us that their view is that if implementation plans provide value to agencies, they may stay in draft form and do not need to be final. Overall, draft implementation plans for these agencies provided some but not all information required to fully meet the directives from OMB. Our analysis of the plans shows that on average, agencies fully met six out of 10 requirements for their implementation plans. For example, almost all agencies met the requirements for identifying the PMIO (21 out of 22). However, 11 out of 22 agencies did not provide complete information on major acquisition portfolios. Table 4 shows how agencies’ implementation plans varied in meeting the requirements. Seven of 24 agencies reported in our questionnaire that they were creating either task forces or new or restructured offices to direct PMIAA implementation within their agencies. For example, DOE reported establishing a new office to support its PMIO. The Department of the Treasury and NASA reported creating an intra-agency cross-functional core team to discuss and design PMIAA implementation strategies. OPM reported establishing an enterprise program management office to drive the standardization of program and project management processes internally. Agencies selected PMIOs in existing leadership positions to leverage resources and agency processes to implement PMIAA. All agency PMIOs reported having additional leadership responsibilities beyond their PMIO roles. OMB documentation and information gathered from CFO Act agencies shows: every PMIO has at least one additional CXO role within its agency; thirty-eight percent of PMIOs have an additional performance management role; eight of 24 PMIOs have an additional budgetary role; and four of the 24 PMIOs have an explicit additional program or acquisition role. OMB Has Taken Limited Steps to Address Areas on Our High-Risk List In the past, we have met with senior management officials from OMB and applicable agencies to discuss where additional management attention could be beneficial to addressing high-risk areas identified on our High- Risk List. We also reported that these trilateral meetings, which began in 2007 and pre-dated PMIAA’s 2016 enactment, have continued across administrations and have been critical for progress that has been made in addressing high-risk areas. According to PMIAA, OMB’s Deputy Director of Management is to conduct annual portfolio reviews of the most at-risk agency programs, as designated by our High-Risk List. OMB officials view the trilateral meetings as their method for holding the portfolio review meetings for high-risk areas as required under PMIAA. Our High-Risk List is comprised of programs as well as functions and operations. Consequently, in our assessment of OMB’s implementation of PMIAA, we consider programs, functions, and operations on our High-Risk List as relevant for OMB’s portfolio review of areas on our High-Risk List. OMB used three strategies intended to meet PMIAA’s high-risk requirements. OMB (1) expanded its strategic reviews in 2018 to include a review of some high-risk areas, (2) continued to use the long-standing trilateral meetings to review high-risk areas with agency leaders and with us, and (3) held ad hoc meetings with agencies outside of the strategic review and trilateral meetings. OMB Discussed High-Risk Areas with Some Agencies during Strategic Review Meetings In preparation for the 2018 strategic reviews, OMB issued Memorandum M-18-15 directing agencies to provide several items in advance of their strategic review meetings with OMB. Requested items included updates from agencies on areas identified on our High-Risk List in which agencies disagreed with our recommendations or faced implementation barriers preventing progress. These materials were to be discussed during strategic review meetings. Thirteen CFO Act agencies reported submitting high-risk updates to OMB prior to these meetings, and eight agencies reported discussing their high-risk areas with OMB during the meetings. OMB guidance from June 2019, communicated in OMB’s Circular No. A- 11, did not include the statement from Memorandum M-18-15 that high- risk areas would be discussed during strategic review meetings. OMB staff felt that a broader approach could yield better results for addressing high-risk areas. Guidance in Circular No. A-11 maintained that agencies should submit updates about high-risk programs to OMB for the Deputy Director’s high-risk portfolio review, but it did not specify what should comprise agency updates about high-risk programs. Also, OMB staff told us that they requested that agencies provide topics for discussion at strategic review meetings, and that agencies could provide agenda items related to our High-Risk List. OMB staff said they addressed only a few of the high-risk issues during strategic reviews, both during the review process and the strategic review meetings. Discussions about high-risk issues during strategic review meetings generally focused on government-wide high-risk areas, if relevant, such as “Ensuring the Cybersecurity of the Nation” and “Improving the Management of Information Technology (IT) Acquisitions and Operations.” However, OMB and agencies also discussed high-risk areas in instances when agencies provided strategic review meeting agenda topics related to our High-Risk List. For example, Treasury staff told us they spoke with OMB this year about high-risk areas as part of the strategic review process. Treasury is directly responsible for the Enforcement of Tax Laws high-risk area and shares responsibility with other agencies for other high-risk areas, such as the government-wide areas on cybersecurity and strategic human capital. OMB Held Trilateral Meetings on Five of 35 High-Risk Areas OMB has held a limited number of trilateral meetings with agencies and us about high-risk areas as part of the high-risk portfolio reviews. Between March 2018 and October 2019, OMB addressed the following five high-risk areas in trilateral meetings with applicable agencies and us: 2020 Decennial Census, Managing Federal Real Property, Government-wide Personnel Security Clearance Process, Ensuring the Cybersecurity of the Nation, and NASA Acquisition Management. OMB has not held meetings to address the remaining 30 high-risk areas on our High-Risk List. OMB staff told us they plan to hold additional meetings in the next year but that they are unlikely to be able to schedule all remaining meetings within our 2-year cycle for updating the High-Risk List. OMB staff said that it is sometimes challenging to coordinate and convene trilateral meetings given the high-ranking officials who must attend and finding available times across schedules. OMB also told us that they plan to meet with agencies for all high-risk areas eventually, but that they prioritize meetings aligned with our priority areas and the President’s Management Agenda. We evaluate progress made on high-risk areas every 2 years to determine if new areas should be added to our High-Risk List and if areas on the list should be removed due to progress to address the risks. Top leadership commitment is one of the five criteria we use to assess whether progress is being made to address and ultimately remove areas from our high-risk list. As we have reported in our March 2019 High-Risk Series report, leadership commitment is the critical element for initiating and sustaining progress, and leaders provide needed support and accountability for managing risks. Leadership commitment is vital if agencies are to adequately address high-risk areas, and trilateral meetings have been critical in focusing leadership attention in the past. Because OMB officials have met on only five of 35 high-risk areas, it remains to be seen if they will meet on all high-risk areas in the future. Convening the trilateral meetings on all high-risk areas in the 2-year reporting cycle, would better position OMB to enhance the leadership commitment needed to make greater progress on the remaining high-risk areas. OMB Occasionally Discussed High-Risk Areas with Some Agencies throughout 2018 and 2019 beyond Trilateral and Strategic Review Meetings Staff from OMB said that they sometimes have briefings related to agencies’ high-risk areas separate from the annual strategic review meetings and high-risk trilateral meetings. These meetings happen on an ad hoc basis and are typically initiated by agency officials. Officials from some of our selected agencies corroborated that the discussion at the strategic review meetings and trilateral meetings is not the full extent of OMB’s interaction with agencies about high-risk areas throughout the year. For example, VA officials said that high-risk areas are frequently agenda items in meetings with OMB. NASA officials said they spoke with OMB about NASA’s high-risk areas after submitting material as part of the strategic review process. Program Management Policy Council Has Not Made Recommendations to Address High-Risk Areas The PMPC, chaired by the Deputy Director for Management of OMB, did not address our High-Risk List during its three meetings nor did it make recommendations to OMB about addressing high-risk areas, as required. The PMPC meetings have lasted 60 to 90 minutes each and the High-Risk List has not appeared as an item on any of the PMPC meeting agendas. OMB staff said PMPC meetings at this point in PMIAA implementation primarily act as forums in which agencies can share program management practices. Rather than focusing meeting time on high-risk areas, OMB staff asserted that the best use of the PMPC is primarily as a forum for agencies to share program and project management best practices. Consequently, the PMPC has not satisfied all PMPC requirements as delineated in PMIAA, including for high-risk areas to be addressed. OMB Identified Measures to Assess Results of Portfolio Reviews, but Has Been Limited by Agency Data Quality OMB Established a Prototype Dashboard to Help Track Portfolio Program Management Measures of Cost, Schedule, and Performance OMB created a dashboard to identify measures of cost, schedule, and performance that agencies should use to track their selected non-IT major acquisition programs for the first PMIAA program portfolio review. OMB partnered with the General Services Administration to complete a prototype of a dashboard to show cost, schedule, and performance data from each program or project within a portfolio of programs. The dashboard also provides a short description of each program or project and its strategic alignment to the agency’s relevant strategic goal. Staff from OMB’s Office of Federal Procurement Policy said the dashboard could provide them with some visibility and improved transparency for major acquisitions programs. According to the PMIAA strategic plan, the dashboard would display the agency portfolio and summarize performance for each item in the portfolio, similar to the portfolio reviews of IT programs required by the Federal Information Technology Acquisition Reform Act. Initially, according to OMB, it plans to request summary information for each portfolio, and restrict the dashboard to authorized government employees. Moving forward, OMB staff said that as the portfolio management process matures, a portion of the dashboard may be available to the public, similar to the IT dashboard. OMB staff told us they are in conversation with agencies about how to overcome difficulties in collecting data for the dashboard. According to OMB, the results from the pilot portfolio review showed that agencies experienced challenges with collecting high-quality data. OMB staff said there will likely be more metrics for large construction projects because management practices for them are more mature than for other types of programs, such as services. OMB is working with agencies to see how they can retrieve cost, schedule, and performance data that could provide early warning indicators of potential problems with programs. Agencies Plan a Range of Ways to Measure PMIAA Agencies reported in our questionnaire they are considering various ways to measure implementation of PMIAA. A little more than half of agencies responding to our PMIAA questionnaire provided ideas on how to measure implementation of PMIAA, such as tracking completion of their identified PMIAA milestones, developing their own survey as a baseline measure, or using their agency implementation plan outcomes to measure results. Six agencies’ questionnaire responses noted that they are planning to use existing metrics to assess program performance, either through internal processes or their annual strategic review process. For example, Treasury plans to focus in the near term on tracking completion of milestones of PMIAA implementation, such as major program and project alignment to department strategic objectives, development of an information-sharing site for program and project management resources, and workforce capabilities, among other things. VA anticipates developing outcome measures associated with successful program execution and is leveraging measures from existing plans, such as their Acquisition Human Capital Plan. OMB staff told us that they have no plans to identify measures to assess outcomes of PMIAA because it is too early and agencies are in the early stages of implementation. Rather than tracking anything specific, they told us that OMB looks at whether agencies’ PMIOs are engaged, if agencies are using training material and mentorship programs, the involvement of chief senior level officers, and if there is funding in the budget for program management certificate programs. However, OMB has not identified specific measures to track any of these areas. In collaboration with OMB, VA developed a program management maturity model survey identify capability gaps, obtain insights, and enable benchmarking of program management capabilities. It surveyed agencies’ level of maturity on a range of program management capabilities, such as talent management, governance, and portfolio management. Maturity assessment surveys can be useful tools for measuring progress to develop capacity in areas such as program management, according to subject matter specialists. Periodically measuring maturity can help agencies institutionalize continuous assessment and improvement. PMI also supports using such tools to identify trends that can help pinpoint actions needed and opportunities to learn from more mature organizations. We have found that ongoing performance measurement can serve as an early warning system to management and as a vehicle for improving accountability to the public. We have previously reported that providing baseline and trend data can help to assess an agency’s performance more fully because the data show progress over time and decision makers can use historical data to assess performance. As OMB and agencies move forward with PMIAA implementation, it will be critical to measure how agencies are maturing or building their capacity in the areas of program and project management. Such measures could include showing how OMB’s program management standards and principles are integrated into agencies’ programs and policies, the improvement of data quality used to track agency program outcomes in the program portfolio reviews, and improvement in program manager skills. Although not required by PMIAA, it is a good practice for OMB and agencies to consider ways to measure the effects of the act. Without establishing such measures to assess PMIAA outcomes, it will be challenging to gauge how agencies are making progress to identify trends, or to help agencies improve data quality. Conclusions The program and project management standards OMB developed are less detailed than accepted standards and are missing several elements that would have made them more useful. For example, the OMB standards do not provide a minimum threshold against which agencies can gauge to what extent they have met each standard. Further, OMB’s current governance structure is insufficient for further developing and maintaining program management standards. Although OMB received input from stakeholders to develop the standards and plans to update them in partnership with the PMPC in 2020, OMB does not have a governance structure that assigns roles and responsibilities to further develop, approve, maintain, or monitor standards. Having such a governance structure for managing efforts going forward could help sustain the program standards as they change over time. OMB did not follow most leading practices for designing pilots and may have missed opportunities to make improvements for fiscal year 2019 portfolio reviews. OMB has not determined if it plans to conduct additional pilot efforts. Going forward, as OMB expands the portfolio reviews to other types of program areas beyond non-IT major acquisitions, it has the opportunity to develop and learn from additional pilots. Although OMB staff have not yet determined if they will do additional pilots for program management in the future, they could decide to pilot the portfolio reviews of grants that they plan to initiate in fiscal year 2020. OMB has not identified other program areas beyond non-IT major acquisitions and grants to be included in future portfolio reviews. Communicating to agencies about specific program areas, portfolio review procedures, time frames, and expectations beyond 2020 could help agencies better direct their efforts to improve the portfolio review processes and help ensure continued progress to implement PMIAA more broadly. As of October 2019, OMB had not taken any actions in response to the recommendations in our September 2017 report and has not yet fully established an inventory of federal programs. Such an inventory of programs could be a critical tool to help agency officials identify and manage programs across the federal government. Furthermore, if OMB were to fully implement our recommendations and complete the required inventory of federal programs, it would assist agencies to match resources to agencies’ program management needs and assist agencies in preparing for future PMIAA portfolio reviews. Furthermore, OMB provides three different definitions for a “program” in its guidance for PMIAA, GPRAMA, and the DATA Act. Having different definitions of what constitutes a program could lead to confusion among agencies. It could also cause increased burden on agencies as they work to identify, maintain, and report on three sets of differently defined programs. Meetings between OMB, relevant agencies, and us have been critical for past progress on high-risk areas. However, OMB has held these trilateral meetings to address only five of 35 high-risk areas since it began implementing PMIAA. These meetings could both demonstrate and improve the commitment of agency leadership to high-risk areas across the federal government. As we have reported, leadership commitment is a key tenet in agencies’ ability to address high-risk areas. Without convening trilateral meetings on each high-risk area, OMB might miss opportunities to make progress toward addressing high-risk areas by improving leadership commitment to addressing them. The PMPC did not address our High-Risk List during its meetings nor has it made recommendations to OMB about high-risk areas. The High-Risk List has not appeared as an item on any of the PMPC meeting agendas. OMB staff asserted that the best use of the PMPC’s limited meeting time is as a forum for agencies to share program management best practices. In choosing to focus on program management practices rather than high- risk areas, the PMPC has not satisfied all PMPC requirements as delineated in PMIAA. Having measures to assess outcomes of PMIAA, such as establishing a baseline of information on programs or collecting trend data, can help OMB ensure that it has established a framework to effectively guide and assess PMIAA’s implementation. Assessment measures would also allow OMB to better target efforts to improve project management and the capabilities of managers. Recommendations for Executive Action We are making a total of eight recommendations to OMB. Specifically: The Deputy Director for Management of OMB, in conjunction with the PMPC, should develop program and project management standards to include (1) a minimum threshold for determining the extent to which agencies have met the standards, (2) how standards apply differently at the program and project levels, (3) how standards are interrelated to work in a synchronized way, and (4) how standards should be applied across the life cycle of a program or project. (Recommendation 1) The Deputy Director for Management of OMB, in conjunction with the PMPC, should create a governance structure to further develop and maintain program and project management standards that fully aligns with key practices for governance structures. (Recommendation 2) The Deputy Director for Management of OMB should, when expanding PMIAA to additional program types, design pilot efforts to follow leading practices so that OMB can optimize its efforts to improve and broaden portfolio reviews across a full range of program types. (Recommendation 3) The Deputy Director for Management of OMB should communicate program areas and timeframes, and expectations pertinent to annual program portfolio reviews, to be reviewed in future program portfolio reviews. (Recommendation 4) The Deputy Director for Management of OMB should clarify for agencies how the different definitions of a “program” relate to each other in OMB guidance. (Recommendation 5) The Deputy Director for Management of OMB should convene trilateral meetings between OMB, relevant agencies, and us for addressing all high-risk areas during each two-year high-risk cycle (Recommendation 6). The Deputy Director for Management of OMB, in conjunction with PMPC, should ensure PMPC meeting agendas include time for discussing high- risk areas during meetings and provide time for the PMPC to make recommendations to OMB about addressing high-risk areas. (Recommendation 7) The Deputy Director for Management of OMB, in conjunction with PMPC, should establish measures to assess outcomes of PMIAA, such as establishing a baseline of information on programs or collecting trend data. (Recommendation 8) Agency Comments and Our Evaluation We provided a draft of this product for comment to OMB, OPM, and the five selected agencies. OMB neither agreed nor disagreed with the recommendations and stated that it would take them into consideration when making future updates to its policies and guidance for agencies for improving program and service delivery. In addition, OMB, OPM, Commerce, NASA, Treasury, and Veterans Affairs provided technical comments which we incorporated as appropriate. Energy responded that it had no comments. We are sending copies of this report to congressional committees, the Acting Director of OMB and Director of OPM, The Secretaries of the Departments of Commerce, Energy, Treasury, and Veterans Affairs, the Administrator of NASA, and other interested parties. In addition, the report is available at no charge on the GAO website at http://www.gao.gov. If you or your staff have any questions about this report, please contact me at (202) 512-6806 or Jonesy@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are listed in appendix II. Appendix I: Objective, Scope, and Methodology This report examines: (1) the steps taken by the Office of Management and Budget (OMB), the Office of Personnel Management (OPM), and the Chief Financial Officer Act of 1990 (CFO Act) agencies to implement the Program Management Improvement Accountability Act (PMIAA); (2) the extent to which OMB is using or planning to use portfolio reviews required in PMIAA to address issues on our High-Risk List; and (3) the extent to which OMB provided methods for agencies to assess the results of PMIAA. To examine the steps taken by OMB, OPM, and CFO Act agencies to implement PMIAA, we reviewed agency documents, designed and disseminated a questionnaire to the 24 CFO Act agencies, and analyzed their responses. We also selected five PMIAA CFO Act agencies as case studies. We reviewed documentation from OMB, including the OMB PMIAA strategic plan and actions taken, as well as Cross Agency Priority goal 11 quarterly reports, and screen shots of PMIAA documents on OMB Max portal. We interviewed OMB staff to gain insight into their approach to implementing PMIAA. To examine the OMB standards for program and project management, we used criteria from the Project Management Institute (PMI) for Standard for Program Management and the Project Management Body of Knowledge. In addition, we reviewed documentation from OPM regarding their PMIAA plans and documents for the update of the 0340 job series. We further analyzed Enterprise Human Resources Integration (EHRI) data from fiscal year 2018 from OPM to identify employees in current program management 0340 occupational series. We also interviewed OPM officials regarding their role in implementing PMIAA. We interviewed outside subject matter specialists to provide their views on federal program and project management. Specifically, we met with staff from PMI and Professor Janet Weiss from the University of Michigan—who had conducted a study on how to improve federal program management—as she had been recommended by the Congressional Research Service, OMB, and the IBM Center for the Business of Government. To examine the steps agencies had taken, we requested PMIAA implementation plans from all 24 CFO Act agencies. CFO Act agencies were to submit PMIAA implementation plans to OMB by November 30, 2018. We collected implementation plans between November 29, 2018, and April 16, 2019. We received 22 out of 24 implementation plans. We did not review plans from the Department of Health and Human Services or the Environmental Protection Agency because they had not completed their plans at the time of our review. Two analysts independently reviewed separate implementation plans. These reviews were then verified by another analyst. Implementation plans were evaluated on whether they fully met, partially met, or did not meet the 10 requirements provided in the OMB implementation guidance, such as how the major acquisition portfolios aligned to relevant strategic objectives, or whether the agency had existing training for program and project managers. We also disseminated a questionnaire to all CFO Act agencies to collect information on PMIAA implementation. This questionnaire was pre-tested by two CFO Act agencies and two members of the Federal Program and Project Management Community of Practice and revised for clarity. The questionnaire was sent to all 24 CFO Act agencies on February 4, 2019, and responses collected between February 11 and April 22, 2019. All 24 agencies responded to the questionnaire. Agency officials were asked questions on: 1. the steps their agency has taken to implement PMIAA, 2. the challenges their agency faces in implementing PMIAA, 3. efforts to address high-risk issues, and 4. plans to measure PMIAA outcomes, if any. We selected five agencies for case studies and analyzed further documentation and interviewed agency officials to provide illustrative examples of PMIAA implementation at the agency level. We assessed whether: agencies had responsibility for a program, function, or operation on our 2019 High-Risk List; OMB considered them further along in PMIAA implementation compared to other agencies; the agency reported it was selected for the OMB pilot of noninformation technology acquisition program portfolio reviews; agency officials reported actions taken to direct internal program management training or workforce development in their questionnaire responses or OMB required implementation plans; and agency officials reported any actions to implement PMIAA beyond the requirements listed in the OMB PMIAA strategic plan. To achieve of a range of PMIAA experiences, we selected five agencies that met varying numbers of the criteria. The Department of Commerce was chosen because all four selection criteria were met, the Department of Energy met three, the Department of Veterans Affairs met two, and the Department of the Treasury and the National Aeronautics and Space Administration each met one. We interviewed and reviewed documents from each of the agencies. We asked questions about steps agencies were taking and their interactions with OMB and OPM to help them implement PMIAA. We also asked these agencies to suggest any ways in which OMB and OPM could improve implementation. To assess the OMB PMIAA strategic plan, we reviewed leading practices on strategic planning from our body of work. We also considered testimonial evidence from OMB staff. Specifically, we reviewed prior reports on leading strategic planning practices and requirements for agencies to use in strategic planning. We selected relevant criteria from the Government Performance and Results Act of 1993 (GPRA) and the GPRA Modernization Act, that not only pertained to agency strategic plans, but also were relevant as for strategic planning principles. Specifically, we selected criteria from the following categories: (1) mission statement; (2) general goals and objectives; (3) strategies for accomplishing goals and objectives; (4) input from stakeholders; (5) interagency collaboration; 6) milestones and metrics to gauge progress. To determine the extent to which the leading practice was included in the strategic plan, we assessed documentary evidence from the PMIAA strategic plan and testimonial evidence from OMB staff as defined below: A practice was categorized as fully met if the evidence fulfilled all aspects of the definition. A practice was categorized as partially met if the evidence fulfilled some, but not all, aspects of the definition, or if the evidence was judged to fulfill the general meaning of the definition, while not technically meeting it fully. A practice was categorized as not met if no evidence was found relevant to the criterion, or if evidence did not fulfill any aspects of the definition. In addition, we reviewed documents from and interviewed selected agencies on what measures OMB was developing for evaluating PMIAA implementation. We also asked these agency officials what kinds of evaluative measures would be useful to monitor the successful implementation of PMIAA from their perspective. In addition, we assessed the pilot of the required PMIAA program portfolio reviews against the five leading practices we identified from our work on designing pilots. We determined that the design fully met the criteria when we saw evidence that all aspects of a leading practice were met. When we were unable to assess whether all aspects of a leading practice were met without additional information, we determined that the design partially met the criteria. Finally, when we saw no evidence of a leading practice, we determined that the criteria were not met. To examine OMB’s standards for program and project management, we selected two sets of criteria for program and project management criteria from PMI. PMI standards are generally recognized as leading practices for program and project management. To select program management standards, we identified 10 PMI program management activities. To select project management standards, we identified 10 project management knowledge areas. Further, PMI’s leading practices were selected to explain how program and project management standards apply differently, and how both set of standards relate to the lifecycle of a program or project. We then compared the definition of these 10 PMI program and 10 PMI project management standards to the definition of OMB’s initial 15 program and project standards released for PMIAA implementation. In addition, OMB’s initial standards were compared to PMI leading practices that distinguish the relationship between programs and projects and leading practices on applying standards across the life cycle of a program or project. We also applied leading practices we identified from our previous work on data governance standards to assess the governance process OMB used to develop, maintain, and monitor program management standards. Our past work identified common key practices for establishing effective data governance structures. This work selected a range of organizations, including domestic and international standards-setting organizations, industry groups or associations, and federal agencies, to ensure we had comprehensive perspectives of data governance key practices across several domains. Two analysts compared the five key practices on the data governance structures to OMB plans and documented practices. We assessed the reliability of OPM’s EHRI data through electronic testing to identify missing data, out of range values, and logical inconsistencies for employees classified as 0340s. We believe the EHRI data we used are sufficiently reliable for the purpose of this report. To examine the extent to which OMB is using or planning to use portfolio reviews to address our High-Risk-List, we reviewed documentation from OMB and 24 CFO Act Agencies. As part of our questionnaire, we asked 24 CFO Act agencies to provide any of our High-Risk List summary and detailed analyses that the agencies were required to submit to OMB as part of the 2018 strategic review process. We analyzed this information to determine the extent to which agencies provided information to OMB during their 2018 strategic review process. We also selected criteria from the Standards for Internal Control in the Federal Government on maintaining documentation of the internal control system to assess steps that OMB had taken related to its responsibilities for conducting high-risk portfolio reviews and the management of the Program Management Policy Council. Specifically, we selected information and communication which states that management should externally communicate the necessary quality information that an entity needs to achieve its objectives. We conducted this performance audit from June 2018 to December 2019 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives. Appendix II: GAO Contact and Staff Acknowledgments GAO Contact Yvonne D. Jones, (202) 512-6806, or jonesy@gao.gov. Staff Acknowledgments In addition to the contact named above, William Reinsberg (Assistant Director), Carole J. Cimitile (Analyst in Charge), Jacqueline Chapin, Martin J. De Alteriis, Emily Gamelin, Jaeyung Kim, Matthew L. McKnight, Robert Robinson, Dylan Stagner, Andrew J. Stephens, and John Villecco made key contributions to this report.