DTIC ADA487645: Price Enumeration and Probabilistic Evaluation in System Acquisition

15 Pages·0.28 MB·English

by  Defense Technical Information Center

Acquisition Review Quarterly — Fall 2002
Photo: Courtesy of the United States Air Force
260
Report Documentation Page Form Approved
OMB No. 0704-0188
Public reporting burden for the collection of information is estimated to average 1 hour per response, including the time for reviewing instructions, searching existing data sources, gathering and
maintaining the data needed, and completing and reviewing the collection of information. Send comments regarding this burden estimate or any other aspect of this collection of information,
including suggestions for reducing this burden, to Washington Headquarters Services, Directorate for Information Operations and Reports, 1215 Jefferson Davis Highway, Suite 1204, Arlington
VA 22202-4302. Respondents should be aware that notwithstanding any other provision of law, no person shall be subject to a penalty for failing to comply with a collection of information if it

does not display a currently valid OMB control number.
1. REPORT DATE 3. DATES COVERED
2002 2. REPORT TYPE 00-00-2002 to 00-00-2002
4. TITLE AND SUBTITLE 5a. CONTRACT NUMBER
Price Enumeration and Probabilistic Evaluation in System Acquisition
5b. GRANT NUMBER
5c. PROGRAM ELEMENT NUMBER
6. AUTHOR(S) 5d. PROJECT NUMBER
5e. TASK NUMBER
5f. WORK UNIT NUMBER
7. PERFORMING ORGANIZATION NAME(S) AND ADDRESS(ES) 8. PERFORMING ORGANIZATION
ESC/JS,Hanscom AFB,MA,01731 REPORT NUMBER
9. SPONSORING/MONITORING AGENCY NAME(S) AND ADDRESS(ES) 10. SPONSOR/MONITOR’S ACRONYM(S)
11. SPONSOR/MONITOR’S REPORT
NUMBER(S)
12. DISTRIBUTION/AVAILABILITY STATEMENT
Approved for public release; distribution unlimited
13. SUPPLEMENTARY NOTES
Acquisition Review Quarterly, Fall 2002
14. ABSTRACT

15. SUBJECT TERMS
16. SECURITY CLASSIFICATION OF: 17. LIMITATION OF 18. NUMBER 19a. NAME OF
ABSTRACT OF PAGES RESPONSIBLE PERSON
a. REPORT b. ABSTRACT c. THIS PAGE Same as 14
unclassified unclassified unclassified Report (SAR)
Standard Form 298 (Rev. 8-98)
Prescribed by ANSI Std Z39-18
Price Enumeration and ProbaTbUilTisOtiRc IEAvaLluation in System Acquisition
PRICE ENUMERATION AND
PROBABILISTIC EVALUATION
IN SYSTEM ACQUISITION
Maj Kevin Gaudette, USAF and Kevin Sweeney
A number of environmental uncertainties in system acquisition can lead to
situations where exact requirements are difficult to estimate. The uncertainties
are often disregarded in favor of a simple point estimate. This point estimate, or
Best Estimated Quantity (BEQ), is then used in the Request for Proposals
(RFP) and subsequent source selection, with the evaluation conducted at the
BEQ price using deterministic techniques. An alternative approach is presented
for BEQ and deterministic price evaluation. The approach involves the solicitation
of a range of bid prices for all potential quantities in lieu of a BEQ, with a
complementary probabilistic analysis technique in lieu of a deterministic price
evaluation. The probabilistic approach is then used to evaluate the range of bid
prices and make an award decision. The methodology is presented in the context
of an actual case study in which it was implemented in 1996.
M
any acquisition programs face a source selection, proposals are evaluated
challenging and ever-changing at the quantity that is certain at the time
environment. Uncertain, dy- of the Request for Proposal (RFP), regard-
namic requirements make program deci- less of how certain that quantity is.
sions and cost estimates extremely diffi- This quantity, often called the Best
cult. This difficulty is magnified at the Estimated Quantity (BEQ), is therefore
time of a source selection, when the re- simply a crude point estimate made at a
quirements must be frozen for communi- distinct point in time. When quantities are
cation to potential bidders. In a typical certain and static, as is sometimes the case,
DISCLAIMER
The views expressed in this article are those of the authors and do not reflect the official
policy or position of the United States Air Force, Department of Defense, or the U.S.
government.
261
Acquisition Review Quarterly — Fall 2002
the BEQ is a perfectly acceptable meth- be significantly higher. This stands to rea-
odology. When program characteristics son, given that the bidders will seek to re-
make requirements uncertain and dy- duce their own risk and that price is nor-
namic, however, the use of a BEQ and mally evaluated only at the BEQ. Finally,
deterministic evaluation techniques drives the buyer may be forced to buy excess
cost risk to unnecessary levels. In fact, the items to satisfy the terms of the contract
only thing certain about a forecast made and avoid default penalties.
in an uncertain environment is that it will The second possible outcome is that the
be wrong. This is an important concern, quantity needed increases. As was the case
because inaccurate quantity estimates with the overestimated quantity described
result in one of only two possible out- above, if the higher quantity is priced in
comes, illustrated by the “bath tub” curve the contract, it is likely to be a higher unit
in Figure 1. price. If not priced, the buyer must either
In the first outcome, the quantity re- conduct a new source selection for the
quired is lower than the estimate used and additional units or must renegotiate the
evaluated in the source selection. Three existing contract. Either option will result
things can happen in this case. If the lesser in a higher price than if the higher quan-
quantity was not priced in the contract, tity had been considered in the original
then the buyer is forced to renegotiate and procurement.
will likely pay a significantly higher price. The increased cost risk described above
If the lesser quantity was priced in the con- is often reduced, albeit to a limited extent,
tract, it is still likely that the unit price will by the use of such techniques as Variable
Traditional BEQ
Unit Approach
Price
Probabilistic
Approach
BEQ Quantity
Figure 1. Notional Example of the Relationship Between Unit Price
and Actual Quantity Purchased for BEQ and Probabilistic Approaches
262
Price Enumeration and Probabilistic Evaluation in System Acquisition
Indefinite Quantity (VIQ) contract options detail using supporting calculations,
and limited liability termination clauses. examples from the case study, and a brief
Unfortunately, these techniques are partial discussion of its advantages. The Discus-
solutions. The latter is a “safety net” sion section offers a discussion of the gen-
designed to limit the government’s liabil- eral program conditions that make proba-
ity, while the bid prices in the former are bilistic methods attractive. Strengths and
usually not incorporated effectively into potential pitfalls are included to aid
the proposal evaluation. These problems acquisition professionals in implementa-
can be more rigorously addressed using a tion efforts. Finally, the Results section
simple probabilistic approach. This paper reports the outcome of the case study
describes the use of one such approach in source selection.
system acquisition and presents a case
study in which it was developed and
CASE STUDY
applied. The method, which abandons the
point estimate methodology altogether,
was successfully used in a $200 million PROGRAM BACKGROUND
source selection for the Joint Tactical PE2 was implemented during a 1996
Information Distribution System (JTIDS) source selection for JTIDS, an Acquisi-
in 1996. tion Category ID joint program. The pri-
For simplicity, the approach is hereafter mary function of JTIDS is to distribute
referred to as PE2. The acronym, although tactical information in digital form. Spe-
admittedly contrived for convenience, cifically, JTIDS terminals provide jam-
nevertheless captures both primary com- resistant digital commu-
ponents of the methodology described in nication of data and
“The acronym
this paper. The first “PE” is the request voice for command and
[PE2], although
for an all-inclusive set of proposal prices control, navigation, rela-
admittedly
covering the entire range of possibilities, tive positioning, and
contrived for
called price enumeration. The second is identification. JTIDS
convenience,
the use of probabilistic techniques to technology also locates
nevertheless
evaluate that set of proposal prices, called and identifies subscrib- captures both
probabilistic evaluation. When used to- ers with respect to other primary compo-
gether (PE2), the efficacy of these two users and is capable of nents of the
techniques is maximized for reasons that transmission rates far methodology
will become clear. above those of most ex- described in
this paper.”
The remainder of the paper is organized isting communication
into four major sections: Case Study, systems. In effect, the
Methodology, Discussion, and Results. system gives airmen, sailors, and soldiers
The Case Study section provides an over- a real-time “God’s-eye view” of the battle-
view of the program in which the meth- field (Scott, 1996). JTIDS platforms range
odology was implemented and describes from Marine ground-based trucks, to Navy
some of the unique program characteris- ships, to Air Force fighters, Airborne
tics that made it necessary. The Method- Warning and Control Systems (AWACS),
ology section describes the method in and Joint Surveillance Target Attack Radar
263
Acquisition Review Quarterly — Fall 2002
Systems (JSTARS). The 1996 source were reluctant to commit due to budget
selection was held to award the final uncertainties in the federal Planning, Pro-
production contract of JTIDS Class 2/2H gramming, and Budgeting System
terminals. Two contractors had been in- (PPBS).
volved in the research and development, In combination, the diversity of the
low-rate initial production, and first full- customer base, budget uncertainties, and
rate production contracts, and they were the time span of the contract made the
the only companies realistically capable quantities nearly impossible to forecast.
of performance. This highly uncertain and dynamic envi-
ronment rendered the traditional BEQ
PROGRAM CHARACTERISTICS approach to estimating quantities for a
The JTIDS program has many charac- long-term contract inadequate. A proba-
teristics that illuminate the advantages of bilistic approach was therefore used to
using a probabilistic methodology. All reduce program cost risk, while ensuring
four U.S. military services had expressed a reasonable price for all JTIDS users.
requirements for 10 different variants of
the terminals and had personnel working
METHODOLOGY
in the Joint Program Office (JPO) at
Hanscom Air Force
Base. In addition, there THE DISCRETE PROBABILITY DISTRIBUTION
“The JTIDS pro-
were several NATO cus- As discussed previously, PE2 actually
gram has many
tomers, one of which consists of two techniques used in tandem
characteristics
was physically repre- — price enumeration and probabilistic
that illuminate
the advantages sented on the JPO staff. evaluation of the complete set of prices.
of using a The diverse customer The two are technically independent, but
probabilistic base, each with its own their symbiotic relationship dictates that
methodology.” independent line of they be used together for maximum ef-
funding, made require- fectiveness. A complete range of prices
ments forecasting ex- can certainly be solicited without using
tremely difficult. Projections would often probabilistic techniques to evaluate them,
change on a daily basis, depending on the although it makes little sense to do so in
funding posture of each service and NATO practice. Likewise, any probabilistic tech-
customer. nique could be employed in an evaluation
Since JTIDS is a subsystem installed even if a limited number of bid prices are
on various land, sea, and airborne plat- solicited in the RFP, but this limits the
forms, the requirements were also subject efficacy of the technique since the full
to fluctuations in projected platform range of potential prices will not be
end-strengths. In addition, the final full- explicitly included in the contract.
rate production (FRP) contract spanned It should be noted that the evaluation
five years, making forecasts even more technique described in this section,
difficult. Firm requirements were a rela- although chosen for several reasons that
tively simple matter for the first year, but will be discussed, is only one of many
for the four subsequent years the services techniques that are available. Alternative
264
Price Enumeration and Probabilistic Evaluation in System Acquisition
techniques like Monte Carlo simulation, Where
stochastic math programming, and simple q = jth possible value of q for line
ij
sensitivity analysis, used individually or item i
in combination, can all perform equally c = bid unit price (cost) when
ij
well. They can also be used to enhance buying j units of line item i
PE2. p(q )= probability of buying j units
ij
The probabilistic evaluation used in the of line item i
JTIDS program involves a technique em- M = number of different line items
ploying a discrete probability distribution on contract
and expectation values. The primary rea- N = number of possible quantities
i
son that this technique was chosen over for line item i
competing probabilistic techniques for the E(C) = expected cost of the contract
JTIDS source selection is simplicity. Tra-
ditional methods like those mentioned For each item, it is improtant to note
above are excellent tools for analysis of a that the sum of all probabilities must
set of possible outcomes, but they can be equal one.
unnecessarily complex for the problem at
hand. Additionally, the results can be dif-
ficult to interpret and explain to decision
makers. Since the solicitation of prices for
all possible quantities is a straightforward Although the equation represents one
concept, it is omitted from the method- of the most basic concepts in probability
ological discussion for brevity. Imple- theory, its use is predominantly absent
mentation issues associated with bid price from military acquisition. This simple
enumeration are discussed in the Discus- equation, however, can significantly re-
sion section, but for now the focus is di- duce the cost risk in a program if used
rected toward the probabilistic evaluation properly. By explicitly incorporating
technique. prices for all potential quantities in the
In the context of a source selection for price evaluation, direct incentive is pro-
uncertain quantities, a discrete probabil- vided to bidders to offer reasonable prices
ity distribution is composed of a finite set across the entire range of quantities. In the
N of quantities q, for each of M line items JTIDS source selection, the equation was
— each with an associated probability of incorporated into a spreadsheet that was
occurrence p(q). The expected contract distributed on a floppy disk with the
cost E(C) is simply the sum of the prod- Request for Proposal (RFP) (See Figure
ucts of all possible outcomes and their as- 2). The formulas and probabilities, there-
sociated probabilities of occurrence, fore, were fully visible to the bidders. Al-
summed over all line items in the contract though hundreds of bid prices were
(Ross, 2000). The equation illustrates this required, the bidders could easily fill in
mathematically. the blanks and conduct internal “what if”
analyses to arrive at their desired bottom
lines.
265
Acquisition Review Quarterly — Fall 2002
Weight Total Avg Unit Weighted
Qty % Price Price (AUP) AUP
0 45.00% $200,000 $0 $0
1 25.00% $200,000 $200,000 $50,000
2 20.00% $350,000 $175,000 $35,000
3 10.00% $450,000 $150,000 $15,000
Figure 2. Sample Spreadsheet Entry for JTIDS Class 2 Terminals,
Configuration 1 (F-15/MAOC), Fiscal Year 1997
In Figure 2, for example, the hypotheti- quantities, since each will ultimately
cal bids show a nonincreasing unit price impact the price evaluation.
for increasing quantities. This example Second, the bidders explicitly submit
illustrates the quantity discount one would bids for each possible quantity. This has
expect. Since the BEQ (in bold border) is the obvious benefit of eliminating nearly
zero units, the F-15 terminal would likely all cost risk associated with quantity un-
not have been included in a traditional certainty, since the unit prices for all pos-
BEQ-based solicitation. Its inclusion here sible quantities are contained in the con-
avoids costly changes later should the user tract. Lloyd (2000) was a proponent of this
decide to exercise an option to buy termi- approach whenever possible contract
nals. This example is purposely chosen changes threaten future prices.
here, since an option was in fact later Third, it allows program offices the
exercised for F-15 aircraft at Mountain ability to proceed with a source selection,
Home Air Force Base participating in a if it makes sense to do so, even if the final
Theater Missile Defense (TMD) Advanced funded quantities are still uncertain
Technology Concept Demonstration (Lloyd, 2000). In the case of JTIDS, if
(ATCD) (Scott, 1996). the JPO had waited until quantities were
certain to conduct the source selection, it
ADVANTAGES would have been delayed indefinitely.
A probabilistic approach offers four pri- Finally, any attempt to take advantage
mary advantages over the traditional BEQ of the system using complex bidding
method. First, the evaluation price that strategies is immediately evident to the
ultimately drives the cost evaluation is a source selection evaluation team. Should
probabilistically-weighted average of all some of the bid prices be unreasonable, it
possible quantities. This leads to a more is a simple matter to translate that unrea-
defendable forecast of the expected cost sonableness into technical risk. This ad-
of the contract than does a simple point vantage is consistent with the recent push
estimate (Anderson & Cherwonik, 1997). toward Best Value source selections, in
It also offers incentive to bid fairly for all which awards to a bidder other than the
one with the lowest cost are often deemed
266
Price Enumeration and Probabilistic Evaluation in System Acquisition
advantageous to the government. Any requisites, for reasons that will become
proposed prices that are considered apparent.
unreasonable can be directly and quan- Quantity Uncertainty – Quantity un-
titatively penalized in the technical evalu- certainty is an obvious prerequisite con-
ation (O’Connor, Faris, & Lovelace, dition for the use of probabilistic tech-
1997). This alleviates the problem of niques, since there is no need for these
subjective technical evaluations that fail techniques if the quantity is known. The
to stand up to protests by the losing bid- degree of uncertainty, however, must be
ders — a perennial problem in source evaluated to determine whether it is high
selections (Lloyd, 2000; Raymond, 1999). enough to warrant application of PE2. In
the JTIDS source selection, the degree of
uncertainty was clear. The firm require-
DISCUSSION ments totaled just 40 terminals, represent-
ing the minimum buy in the first contract
Certain conditions tend to favor the use year. Over the course of three years of ter-
of PE2 in source selections. As alluded to minal buy options, how-
throughout this paper, the method has ever, the maximum
many advantages over traditional point quantity was 230 termi- “Certain conditions
estimate evaluation techniques. Unfortu- nals. The high degree of tend to favor the
nately, there are also several pitfalls in- uncertainty and range of use of PE2 in source
herent in the methodology that must be potential quantities war- selections.”
carefully avoided. Program managers and ranted the use of proba-
cost analysts need to recognize the condi- bilistic techniques. Un-
tions under which the use of PE2 is both certain procurement quantities in the
appropriate and preferred, and also must JTIDS case resulted from a diverse cus-
have a firm grasp of the strengths and tomer base, budget uncertainties, multiple
potential pitfalls. The conditions favoring platforms, and multi-year contracts; but
application of PE2 are discussed first. other programs may experience other
causal factors leading to quantity uncer-
CONDITIONS FAVORING PE2 tainty.
Although the list of four conditions dis- Current and Accurate Cost Data –
cussed here is certainly not intended to The second prerequisite condition pre-
be all-inclusive, it contains four of the sented here is that of current and accurate
most important conditions in terms of any cost data. Because the proposals will con-
probabilistic methodology. It is not neces- tain a large number of bid prices, each
sary that all conditions be met for PE2 to must be assessed for reasonableness. Any
be used, but obviously the number met and bid prices that are deemed unreasonable
the degrees to which they are met are fac- can then be translated to the technical
tors that must be considered. The four con- evaluation in the form of risk. Without
ditions are quantity uncertainty, current current and accurate cost data, it is very
and accurate cost data, large quantities, difficult in practice to quantify that risk
and high unit cost. The first two can be and thereby discourage bidders from taking
distinguished from the latter two as pre- advantage of the system. In the case of
267
Acquisition Review Quarterly — Fall 2002
JTIDS, the maturity of the program meant Risk Reduction – Risk reduction is the
that the JPO had a great deal of historical most obvious advantage, and in fact the
cost data to use in the evaluation. one that drove the use of probabilistic
Large Quantities – A large procure- techniques in the JTIDS program. In any
ment in terms of quantity can also war- program where quantity uncertainty is
rant the use of PE2 or similar techniques. present, program managers run the risk of
For such contracts, even a small percent paying higher prices when quantities
deviation from the estimate can represent change (Lloyd, 2000). PE2 can reduce or
a significant increase or decrease. Again, even eliminate that risk by requiring prices
in the case of JTIDS the range of quanti- for all possible quantities at the time of
ties for terminals was 190, with a firm proposal, rather than adjusting those
(minimum) requirement of 40 and a maxi- quantities later. The possibilities are ex-
mum of 230. If only 10 percent of the plicitly enumerated and probabilistically
uncertain quantities were subsequently evaluated from the beginning.
purchased, the total buy would increase Visibility – Because the complete set
by 19 terminals. This represents nearly a of prices is required in the proposal, any
50 percent increase in units from the firm attempt to take advantage of the system is
BEQ originally contemplated for the immediately visible to source selection
source selection. evaluation team members. Where under-
High Unit Cost – A related but distinct lying bidding strategies are masked to the
condition is the unit cost team using the BEQ method, they are
of the items in the con- uncovered using enumerative pricing. This
“PE2 has many
tract. Just as large quan- makes the quantitative assessment of risk
advantages…
tities can be significant straightforward and defendable, avoiding
risk reduction,
even at low unit costs, the traps that a subjective technical evalu-
visibility, ease of
implementation, large unit costs can be ation can bring (Lloyd, 2000).
and flexibility.” significant even at low Ease of Implementation – PE2 is rela-
quantities. JTIDS termi- tively straightforward to implement. Al-
nals cost between $50 though its application to source selection
thousand and $180 thousand each, de- is unique, the equation is a mathematically
pending on the type, so even a deviation simple concept. The evaluation price can
of one terminal can translate to a substan- easily be described to decision authorities
tial increase in cost. When combined with as a probabilistically-weighted average,
large quantities, the sensitivity to small and therefore should be readily accepted
deviations becomes even more significant. by program managers, cost analysts, and
contracting officers alike. As demon-
STRENGTHS OF PE2 strated in Figure 2, the equation can be
PE2 has many advantages. The four incorporated into a spreadsheet and dis-
discussed here are, by nature of the un- tributed to potential bidders electronically
derlying conditions, the most important. with the RFP.
These are risk reduction, visibility, ease Flexibility – Finally, PE2 is flexible. In
of implementation, and flexibility. some programs, quantity uncertainty may
exist only for a subset of contract line
268