GAO Cost Estimating and Assessment Guide: Best Practices for Developing and Managing Capital Program Costs (GAO-09-3SP).

440 Pages·2009·4.49 MB·English

by  Carol Cha, U.S.G.A.O.

United States Government Accountability Office
GAO Applied Research and Methods
GAO COst EstimAtinG
And AssEssmEnt GuidE
Best Practices for developing and managing
Capital Program Costs
March 2009
GAO-09-3SP

PrefAce
The U.S. Government Accountability Office is responsible for, among other things, assisting the
Congress in its oversight of the federal government, including agencies’ stewardship of public funds.
To use public funds effectively, the government must meet the demands of today’s changing world by
employing effective management practices and processes, including the measurement of government
program performance. In addition, legislators, government officials, and the public want to know whether
government programs are achieving their goals and what their costs are. To make those evaluations,
reliable cost information is required and federal standards have been issued for the cost accounting that
is needed to prepare that information.1 We developed the Cost Guide in order to establish a consistent
methodology that is based on best practices and that can be used across the federal government for
developing, managing, and evaluating capital program cost estimates.
For the purposes of this guide, a cost estimate is the summation of individual cost elements, using
established methods and valid data, to estimate the future costs of a program, based on what is known
today.2 The management of a cost estimate involves continually updating the estimate with actual data
as they become available, revising the estimate to reflect changes, and analyzing differences between
estimated and actual costs—for example, using data from a reliable earned value management (EVM)
system.3
The ability to generate reliable cost estimates is a critical function, necessary to support the Office of
Management and Budget’s (OMB) capital programming process.4 Without this ability, agencies are at risk
of experiencing cost overruns, missed deadlines, and performance shortfalls—all recurring problems that
our program assessments too often reveal. Furthermore, cost increases often mean that the government
1
Federal Accounting Standards Advisory Board, Statement of Federal Financial Accounting Standards No. 4: Managerial Cost
Accounting Standards and Concepts (Washington, D.C.: July 1995).
2
In the context of the Cost Guide, a program refers to all phases in a capital asset’s life cycle—that is, concept analysis, technology
definition, requirements planning, acquisition, and operations and maintenance.
3
EVM is a project management tool that integrates the technical scope of work with schedule and cost elements for investment
planning and control. It compares the value of work accomplished in a given period with the value of the work expected in that
period. Differences in expectations are measured in both cost and schedule variances. The Office of Management and Budget
(OMB) requires agencies to use EVM in their performance-based management systems for the parts of an investment in which
development effort is required or system improvements are under way.
4
Office of Management and Budget, Preparation, Submission, and Execution of the Budget, Circular No. A-11 (Washington,
D.C.: Executive Office of the President, June 2006); Management of Federal Information Resources, Circular No. A-130 Revised
(Washington, D.C.: Executive Office of the President, Nov. 28, 2000); and Capital Programming Guide: Supplement to Circular
A-11, Part 7, Preparation, Submission, and Execution of the Budget (Washington, D.C.: Executive Office of the President, June 2006).
www.whitehouse.gov/omb/circulars/index.html.
GAO-09-3SP Preface i
cannot fund as many programs as intended or deliver them when promised. The methodology outlined in
this guide is a compilation of best practices that federal cost estimating organizations and industry use to
develop and maintain reliable cost estimates throughout the life of a government acquisition program. By
default, the guide will also serve as a guiding principle for our auditors to evaluate the economy, efficiency,
and effectiveness of government programs.
The U.S. Government Accountability Office, the Congressional Budget Office (CBO), and others
have shown through budget simulations that the nation is facing a large and growing structural
deficit in the long term, primarily because the population is aging and health care costs are rising. As
Comptroller General David Walker noted, “Continuing on this unsustainable path will gradually erode,
if not suddenly damage, our economy, our standard of living and ultimately our national security.”5
New budgetary demands and demographic trends will place serious budgetary pressures on federal
discretionary spending, as well as on other federal policies and programs, in the coming years.
As resources become scarce, competition for them will increase. It is imperative, therefore, that
government acquisition programs deliver as promised, not only because of their value to their users but
also because every dollar spent on one program will mean one less available dollar to fund other efforts.
To get better results, programs will need higher levels of knowledge when they start and standardized
monitoring metrics such as EVM so that better estimates can be made of total program costs at
completion.
5
GAO, 21st Century Challenges: Reexamining the Base of the Federal Government, GAO-05-325SP (Washington, D.C.: February 2005), p. 1.
ii Preface GAO-09-3SP
cOntentS
Preface i
Abbreviations xv
Introduction 1
The Guide’s Case Studies 3
The Cost Guide in Relation to Established Standards 3
The Guide’s Readers 3
Acknowledgments 4
Chapter 1
The Characteristics of Credible Cost Estimates and a Reliable Process for Creating Them 5
Basic Characteristics of Credible Cost Estimates 6
A Reliable Process for Developing Credible Cost Estimates 8
Chapter 2
Why Government Programs Need Cost Estimates and the Challenges in Developing Them 15
Cost Estimating Challenges 17
Earned Value Management Challenges 23
Chapter 3
Criteria for Cost Estimating, EVM, and Data Reliability 25
Determining Data Reliability 29
Chapter 4
Cost Analysis Overview 31
Differentiating Cost Analysis and Cost Estimating 31
Main Cost Estimate Categories 31
The Overall Significance of Cost Estimates 36
Cost Estimates in Acquisition 37
The Importance of Cost Estimates in Establishing Budgets 38
Cost Estimates and Affordability 39
Evolutionary Acquisition and Cost Estimation 42
GAO-09-3SP contents iii
Chapter 5
The Cost Estimate’s Purpose, Scope, and Schedule 47
Purpose 47
Scope 47
Schedule 48
Chapter 6
The Cost Assessment Team 51
Team Composition and Organization 52
Cost Estimating Team Best Practices 53
Certification and Training for Cost Estimating and EVM Analysis 54
Chapter 7
Technical Baseline Description Definition and Purpose 57
Process 57
Schedule 58
Contents 58
Key System Characteristics and Performance Parameters 61
Chapter 8
Work Breakdown Structure 65
Best Practice: Product-Oriented WBS 65
Common WBS Elements 68
WBS Development 71
Standardized WBS 71
WBS and Scheduling 75
WBS and EVM 75
WBS and Risk Management 75
WBS Benefits 76
Chapter 9
Ground Rules and Assumptions 79
Ground Rules 79
Assumptions 79
Global and Element-Specific Ground Rules and Assumptions 80
Assumptions, Sensitivity, and Risk Analysis 82
Chapter 10
Data 89
Data Collection 89
Types of Data 91
Sources of Data 92
iv contents GAO-09-3SP
Data Applicability 95
Validating and Analyzing the Data 96
EVM Data Reliability 98
Data Normalization 100
Recurring and Nonrecurring Costs 101
Inflation Adjustments 102
Selecting the Proper Indexes 103
Data Documentation 104
Chapter 11
Developing a Point Estimate 107
Cost Estimating Methods 107
Production Rate Effects on Learning 121
Pulling the Point Estimate Together 122
Chapter 12
Estimating Software Costs 125
Unique Components of Software Estimation 126
Estimating Software Size 127
Estimating Software Development Effort 131
Scheduling Software Development 133
Software Maintenance 134
Parametric Software Estimation 135
Commercial Off-the-Shelf Software 136
Enterprise Resource Planning Software 137
Software Costs Must also Account for Information Technology Infrastructure and Services 139
Unique Components of IT Estimation 139
Chapter 13
Sensitivity Analysis 147
Sensitivity Factors 148
Steps in Performing a Sensitivity Analysis 149
Sensitivity Analysis Benefits and Limitations 150
Chapter 14
Cost Risk and Uncertainty 153
The Difference Between Risk and Uncertainty 153
Point Estimates Alone Are Insufficient for Good Decisions 154
Budgeting to a Realistic Point Estimate 155
Developing a Credible S Curve of Potential Program Costs 159
Risk Management 175
GAO-09-3SP contents v
Chapter 15
Validating the Estimate 179
The Cost Estimating Community’s Best Practices for Validating Estimates 179
Chapter 16
Documenting the Estimate 191
Elements of Cost Estimate Documentation 193
Other Considerations 195
Chapter 17
Presenting the Estimate to Management 197
Chapter 18
Managing Program Costs: Planning 199
Linking Cost Estimation as the Foundation for EVM Analysis 199
The Nature and History of EVM 202
Implementing EVM 204
Federal and Industry Guidelines for Implementing EVM 211
The Thirteen Steps in the EVM Process 215
Integrated Baseline Reviews 231
Award Fees 233
Progress and Performance-Based Payments Under Fixed-Price Contracts 234
Validating the EVM System 236
Chapter 19
Managing Program Costs: Execution 243
Validating the Performance Measurement Baseline with an IBR 243
Contract Performance Reports 249
Monthly EVM Analysis 255
Project Future Performance 265
Provide Analysis to Management 269
Continue EVM until the Program Is Complete 269
Chapter 20
Managing Program Costs: Updating 273
Incorporating Authorized Changes into the Performance Measurement Baseline 273
Using EVM System Surveillance to Keep the Performance Measurement Baseline Current 276
Overtarget Baselines and Schedules 284
Update the Program Cost Estimate with Actual Costs 293
Keep Management Updated 294
vi contents GAO-09-3SP
A
ppendixes
Appendix I
Auditing Agencies and Their Web Sites 297
Appendix II
Case Study Backgrounds 298
Appendix III
Experts Who Helped Develop This Guide 305
Appendix IV
The Federal Budget Process 308
Appendix V
Federal Cost Estimating and EVM Legislation, Regulations, Policies, and Guidance 310
Appendix VI
Data Collection Instrument 321
Appendix VII
Data Collection Instrument: Data Request Rationale 323
Appendix VIII
SEI Checklist 327
Appendix IX
Examples of Work Breakdown Structures 328
Appendix X
Schedule Risk Analysis 353
Appendix XI
Learning Curve Analysis 367
Appendix XII
Technology Readiness Levels 373
Appendix XIII
EVM-Related Award Fee Criteria 374
Appendix XIV
Integrated Baseline Review Case Study and Other Supplemental Tools 377
Exhibit A – Program X: Subject Contractor 382
Exhibit B – IBR Discussion Assessment Forms 383
Exhibit C – IBR Action Item Forms 385
Exhibit D – Sample IBR Discussion Questions 387
Appendix XV
Common Risks to Consider in Software Cost Estimating 393
Appendix XVI
Contacts and Acknowledgments 404
References 405
Image Sources 420
GAO-09-3SP contents vii
L f
istof igures
Figure 1: The Cost Estimating Process 8
Figure 2: Challenges Cost Estimators Typically Face 17
Figure 3: Life-Cycle Cost Estimate for a Space System 33
Figure 4: Cone of Uncertainty 38
Figure 5: An Affordability Assessment 40
Figure 6: Typical Capital Asset Acquisition Funding Profiles by Phase 41
Figure 7: Evolutionary and Big Bang Acquisition Compared 43
Figure 8: Incremental Development 44
Figure 9: Disciplines and Concepts in Cost Analysis 51
Figure 10: A Product-Oriented Work Breakdown Structure 66
Figure 11: A Work Breakdown Structure with Common Elements 70
Figure 12: A Contract Work Breakdown Structure 70
Figure 13: A Learning Curve 120
Figure 14: A Sensitivity Analysis That Creates a Range around a Point Estimate 150
Figure 15: Changes in Cost Estimate Uncertainty across the Acquisition Life Cycle 155
Figure 16: A Cumulative Probability Distribution, or S Curve 157
Figure 17: A Risk Cube Two-Dimensional Matrix 164
Figure 18: The Distribution of Sums from Rolling Two Dice 167
Figure 19: A Point Estimate Probability Distribution Driven by WBS Distributions 168
Figure 20: Integrating Cost Estimation, Systems Development Oversight, and Risk
Management 199
Figure 21: Integrating EVM and Risk Management 201
Figure 22: Inputs and Outputs for Tracking Earned Value 207
Figure 23: WBS Integration of Cost, Schedule, and Technical Information 216
Figure 24: Identifying Responsibility for Managing Work at the Control Account 217
Figure 25: An Activity Network 219
Figure 26: Activity Durations as a Gantt Chart 221
Figure 27: Earned Value, Using the Percent Complete Method, Compared to
Planned Costs 227
Figure 28: The Genesis of the Performance Measurement Baseline 228
Figure 29: The Time-Phased Cumulative Performance Measurement Baseline 229
Figure 30: A Performance-Based Payments Structured Contract 235
Figure 31: The EVM System Acceptance Process 236
Figure 32: IBR Control Account Manager Discussion Template 248
Figure 33: Monthly Program Assessment Using Earned Value 250
Figure 34: Overall Program View of EVM Data 251
viii contents GAO-09-3SP