Table Of Content

Identifying FactoLrEs SthSaOt CNoSnt rLibEuAteR tNo EPrDogram Success IDENTIFYING FACTORS THAT CONTRIBUTE TO PROGRAM SUCCESS Maj. Kenneth J. Delano, USAF Department of Defense acquisition programs and projects frequently experience cost overruns, performance deficiencies, schedule delays, or cancellation. Often, a good program manager using proven program management practices can mean the difference between success and failure. By surveying program managers, examining successful programs within DoD and relevant literature on program management and defense acquisition, we have identified factors that contribute to program success. D epartment of Defense (DoD) acqui- DoD acquisition system. The real impact sition programs and projects fre- (beyond the negative publicity) is on de- quently experience cost overruns, fense readiness, performance, and cost performance deficiencies, schedule de- effectiveness. Since World War II, six lays, or cancellation. U.S. defense acqui- blue-ribbon commissions have studied sition is arguably the largest “business” DoD acquisition and recommended rem- in the world. Annual purchases by DoD edies. Adoption of some of these recom- of approximately $178 billion exceed the mendations, new regulations, and laws has combined purchases of General Motors, failed to alter the paradigm (“Rx for Ail- Exxon, and IBM. Defense acquisition in- ing Procurement System,” 1990). volves almost 15 million contract actions This study sought to identify factors that annually and employs more than 165,000 contribute to program success. The fac- civilian and military workers who man- tors identified can effectively improve the age research and development, procure- current acquisition system, vice the mul- ment, logistics, and support activities tiple attempts to reform the system itself. (Sammet and Green, 1990). We used two techniques to determine With such a large system, errors and these factors. First was a survey in which inefficiencies are bound to occur. Ex- program managers were asked to identify amples are frequently reported in news- factors they believed were key to their papers and magazines, which use these programs’ success. We also conducted a examples to illustrate the poor state of the “factor analysis” of acquisition literature. 35 Report Documentation Page Form Approved OMB No. 0704-0188 Public reporting burden for the collection of information is estimated to average 1 hour per response, including the time for reviewing instructions, searching existing data sources, gathering and maintaining the data needed, and completing and reviewing the collection of information. Send comments regarding this burden estimate or any other aspect of this collection of information, including suggestions for reducing this burden, to Washington Headquarters Services, Directorate for Information Operations and Reports, 1215 Jefferson Davis Highway, Suite 1204, Arlington VA 22202-4302. Respondents should be aware that notwithstanding any other provision of law, no person shall be subject to a penalty for failing to comply with a collection of information if it does not display a currently valid OMB control number. 1. REPORT DATE 3. DATES COVERED 1998 2. REPORT TYPE 00-00-1998 to 00-00-1998 4. TITLE AND SUBTITLE 5a. CONTRACT NUMBER Identifying Factors that Contribute to Program Success 5b. GRANT NUMBER 5c. PROGRAM ELEMENT NUMBER 6. AUTHOR(S) 5d. PROJECT NUMBER 5e. TASK NUMBER 5f. WORK UNIT NUMBER 7. PERFORMING ORGANIZATION NAME(S) AND ADDRESS(ES) 8. PERFORMING ORGANIZATION Air University,Air Command & Staff College,Maxwell AFB,AL,36112 REPORT NUMBER 9. SPONSORING/MONITORING AGENCY NAME(S) AND ADDRESS(ES) 10. SPONSOR/MONITOR’S ACRONYM(S) 11. SPONSOR/MONITOR’S REPORT NUMBER(S) 12. DISTRIBUTION/AVAILABILITY STATEMENT Approved for public release; distribution unlimited 13. SUPPLEMENTARY NOTES Acquisition Review Quarterly, Winter 1998 14. ABSTRACT 15. SUBJECT TERMS 16. SECURITY CLASSIFICATION OF: 17. LIMITATION OF 18. NUMBER 19a. NAME OF ABSTRACT OF PAGES RESPONSIBLE PERSON a. REPORT b. ABSTRACT c. THIS PAGE Same as 16 unclassified unclassified unclassified Report (SAR) Standard Form 298 (Rev. 8-98) Prescribed by ANSI Std Z39-18 Acquisition Review Quarterly—Winter 1998 APPROACH AND METHODS were key factors of program success. Ar- eas addressed include program manage- The approach used is based on work by ment, personnel, resources, and user re- Emory and Cooper (1991), and can be quirements. These factors were used in a summarized thus: literature analysis to identify those that contribute the most to program success. • Define the management question. (In The survey results were analyzed to this case it is “What are the most sig- select key factors considered important by nificant factors contributing to success- the respondents. These factors were ful DoD acquisition programs?”) ranked in order of importance and catego- rized into common subject areas. • Identify research population and With data from the survey results, we sample to be questioned. applied McFarland’s (1992) factor analy- • Develop questionnaire surveys. sis technique. This technique measures the occurrence of key factors in a review of • Collect data. relevant literature. The occurrence of a key • Analyze data collected. factor in each article is noted. In a representative sample of literature, one can de- • Determine factors that contribute to termine the relative importance of each program success. key factor to the subject by noting and comparing the number of occurrences. We POPULATION, QUESTIONNAIRE, ranked those occurrences in order of fre- DATA COLLECTION quency and by subject area. Thirty-two program managers partici- With these results in hand, we com- pated in the survey, the text of which is in piled a list of factors that contribute to Appendix A. Survey responses were un- program success. Those mentioned most signed and untraceable as to the respon- frequently (as determined by factor analy- dent. Each respondent was provided with sis) were the ones examined for identifi- a blank copy of the survey and a self-ad- cation as factors of success. Qualitative dressed stamped envelope. indicators such as the degree of applica- The questions were developed to help bility and history of success of each fac- the respondents identify what they felt tor were considered. Maj. Kenneth J. Delano, USAF, is a student at Air Command and Staff College. Previously, he was Chief, C4I Training Requirements at Headquarters Air Education and Training Command. He has more than 13 years acquisition experience, including 8 years working in program of- fices, and is a graduate of DSMC’s APMC 98-1. He received his M.S. degree in technology management from the University of Maryland and his B.S. degree in electrical engineering from Texas A&M University. He is Acquisition Corps certified in program management, systems planning research development and engineering, communications and computers, and test. 36 Identifying Factors that Contribute to Program Success RESULTS objectives. Answers to the second survey question SURVEY RESULTS AND ANALYSIS (“List the factors you believe contribute to the success of a program or are indica- Eighteen surveys were returned—a 56 tive of program success”) fell into eight percent rate of return. This response rate categories: is judged sufficient to validate the survey results. (One program manager cautioned • total team concept; against expecting a high response. “We are all surveyed out,” he said, explaining that • program manager skills; program managers are frequent targets of • program manager responsibility and official and unofficial surveys.) authority; The responses to the first survey question “Please rank the following in order • well-defined requirements; of their importance as indicators of pro- • stability; gram success: meets initial operational capability date, meets technical perfor- • quality people; mance objectives, meets logistics support- • adequate staffing; and ability objectives, works well when fielded, meets cost objectives,” are pre- • acquisition strategy. sented in Table 1. Two factors, technical performance and Program managers were next asked actual performance (works well when whether they were helped or hindered by fielded) were deemed most important. By the user, support agencies, higher com- averaging the ranking from all responses, mands, Congress, and the General Ac- technical performance was first. Judged counting Office (GAO). They responded least important were meeting logistics sup- that user involvement and input helped portability objectives and meeting cost their program, and that for the most part, Table 1. Survey Results Rank of Program Success Indicators Most Least Average Program Success Indicator Important Important Rank* 12.5% 31.3% 3.1 (3) Meets Initial Operational Capability Date 37.5% 6.2% 2.2 (1) Meets Technical Performance Objectives 0% 31.3% 4.2 (5) Meets Logistics Supportability Objectives 37.5% 18.7% 2.4 (2) Works Well When Fielded 12.5% 12.5% 3.1 (4) Meets Cost Objectives *(1 = highest) 37 Acquisition Review Quarterly—Winter 1998 they viewed involvement from support not considered to be very important to pro- agencies and higher commands as a hin- gram success. drance. All those who responded felt that involvement of Congress and the GAO in LITERATURE FACTOR ANALYSIS specific programs was a detriment to pro- The factor analysis technique gram success. Table 2 presents their re- (MacFarland, 1992) measures the occur- sponses. rence of key factors in a survey of litera- Finally, program managers were asked ture. The occurrence of a key factor in each to rank a list of program success factors article is noted. In a representative litera- identified from a preliminary literature re- ture sample, the relative importance of view using a Likert scale (1, not very im- each key factor to the subject area can be portant; 2, somewhat important; 3, impor- inferred. tant; 4, very important; 5, critical). The fac- By comparing the occurrences of a key tors and results are presented in Table 3. factor in a number of articles against oc- The respondents felt that program man- currences of other key factors in the same ager communication and leadership skills articles, one can calculate the relative im- were important. They also felt that the portance of each factor . For example, if a type and quality of people assigned to sup- key factor is mentioned in 5 out of 10 ar- port the program was important, as was a ticles, it has an importance of 50 percent good relationship with the user organiza- for comparison purposes. This figure can tion. Adequate resources and stability (re- be compared to those calculated for other quirement, design, and funding) were factors and conclusions drawn as to the judged to be next in level of importance. importance or emphasis the literature They did not believe that the degree of places on each factor. technical difficulty (low or high) of the The results from the first, second, and program affected program success. The fourth survey questions were used in results also indicated that the program developing factors used in the literature manager’s technical ability or use of a review. The factors were divided into total quality management program were two main areas: acquisition factors and Table 2. Survey Results Has Program Success Been Helped or Hindered? Factor Helped Hindered User 71% 29% Support Agencies 20% 80% Higher Commands 20% 80% Congress 0% 100% General Accounting Office 0% 100% 38 Identifying Factors that Contribute to Program Success Table 3. Survey Results Program Success Factors Rank By Importance Average Score Program success factor 4.42 Program manager’s ability to communicate 4.25 Type and quality of people associated with program 4.25 Program manager’s ability to lead 4.25 Good relationship with the user organization 4.17 Resources: People, facilities, money 4.08 Product requirements and design stability 3.91 Funding stability 3.83 Good relationship with the prime contractor 3.58 Program’s acquisition strategy 3.58 Program manager’s acquisition experience 3.25 Program personnel continuity 3.00 Program manager continuity 3.00 High degree of technical difficulty 2.92 Program manager’s field experience 2.67 Program manager’s technical ability 2.33 Total quality management program 2.25 Low degree of technical difficulty 1 = Not very important 2 = Somewhat important 3 = Important 4 = Very important 5 = Critical resource factors. Tables 4 and 5 give the and program management. The highest results of the literature survey, with each correlation between a factor and the lit- source denoted by a letter. Table 6 lists erature reviewed was 47 percent. A tie the correspondence to the actual source occurred between three factors: quality in the bibliography. people, well-defined requirements, and The literature factor analysis reveals that acquisition strategy. All had a correlation there is a broad range of subject matter of 47 percent. This reflects the fact that within the general topic of acquisition literature articles frequently focus on these 39 Acquisition Review Quarterly—Winter 1998 e g a nt % % % % % % % % % e 7 7 2 2 1 6 1 5 0 c 4 4 3 3 2 1 1 r e P al ot 9 9 6 6 4 3 2 1 0 T S X R X X Q X P X X X s i s y O X l a n N X A r M X o t c a L F n ce K X X X o r i u sit So J i u q I X X c A H X X X X X . 4 e G X X l b a F T E X X X X D X X X X X C X X B X A X X X X X X e p c hi an Factor Well definedrequirements Acquisitionstrategy Works wellwhen fielded Stability Good relationswith contractor Total qualitymanagementprogram Meets performobjectives Meets costobjectives Meets initialoperationalcapability date 40 Identifying Factors that Contribute to Program Success e g a nt % % % % % % % % % % % % e 7 2 7 7 6 0 0 5 5 5 5 0 c 4 4 3 3 1 1 1 r e P al ot 9 8 7 7 3 2 2 1 1 1 1 0 T S R X Q P X s si O X X X y l a N X X X X X n A M X X r o ct L X X X X a F e e rc K X c u ur So J o s e I R . H X X X 5 e G X X X X X l b a T F E X X X X X D X X X X C X X X B A X X X X X s s g Factor Quality people Program managerresponsibilityand authority Total team concept Program managerskills Congressionalinvolvement User involvement Adequate resource Adequate staffing Support agencyinvolvement Higher commandinvolvement Program manager’technical ability General AccountinOffice involvement 41 Acquisition Review Quarterly—Winter 1998 Table 6. Correspondence Between Codes and Sources A: Baumgartner, Brown and Kelley B: Beltramo C: Clay D: Gansler E: Gregory F: Heberling and Graham G: Hicks, Rich, Wertheim and Meyer H: Hirsch and Waelchli I: Kish J: Lesser K: Nelson L: Price and Valentine M: “Rx for Ailing Procurement System” N: Sammet and Green O: Settlemyer P: Snoderly and Acker Q: Total Quality Management Master Plan R: Weiss S: Zairi Please consult References for complete citation. important aspects of acquisition and pro- DISCUSSION gram management. Because the highest correlation was 47 Elements of program success were percent, the degree of significance was identified by surveying DoD program calculated by using 47 percent as the maxi- managers. A factor analysis was per- mum. Factors with correlations between formed using relevant literature. Those 32 percent and 47 percent were consid- factors with the highest correlation be- ered to be the most significant. Factors tween survey and literature content were with correlations between 17 percent and identified as those contributing to program 31 percent were judged moderately sig- success. This two-step method provides a nificant. Factors with less than 17 percent means of cross-checking survey results correlation were considered to be least with current literature works and focus- significant. ing on factors that are considered important to program success by both. The most significant elements contributing to program success, as identified in 42 Identifying Factors that Contribute to Program Success the factor analysis of the literature, fall into with the users to ensure that the require- two categories. The first is acquisition fac- ments are understood and well defined. tors, which consist of: Hicks, Rich, Wertheim, and Meyer (1991) believe that not enough time and • well-defined requirements; attention are paid to successful programs • the acquisition strategy; that could serve as possible models for the future. They note that the GAO, which sel- • a program product that works well dom compliments the defense acquisition when fielded; and process, identified the Navy’s Fleet Bal- • stability in the program. listic Missile program as one such highly successful program spanning 15 years. The second category we define as re- The GAO identified open dialogue be- source factors: tween the program manager and the prime contractor and continuous communica- • quality people; tions with the ultimate users as reasons for this success. • program manager responsibility and authority; • total team concept; and AQUISITION STRATEGY Like any business strategy or strategic • program manager skills. plan, the acquisition strategy is situation and resource dependent. The program These factors also ranked high in the manager should examine the internal and survey of program managers. This high external environment to gauge resources degree of correlation with the survey adds and support validity to MacFarland’s analysis tech- available. The “Like any business nique. From these results comes the fol- acquisition strategy or strategic lowing list of factors that contribute to pro- strategy should plan, the acquisition gram success. be crafted to strategy is situation help further the and resource WELL-DEFINED REQUIREMENTS program objec- dependent.” A requirement is a formal description tives, while of a desired operational capability. Prod- meeting constraints placed upon the uct stability depends on realistic require- program by external regulators and ments and minimizing changes. As regulations. Baumgartner, Brown, and Kelley (1984) Snoderly and Acker (1981) cite one state, systems that have problems are usu- strategy used to reduce acquisition time ally those that have many changes during and costs. The Defense Support Program, design and production, especially changes which produces ballistic missile early driven by the user. Thus users should take warning satellites, had a requirement to care to avoid overstated requirements that purchase four satellites from their sole- delay production and lead to higher costs. source contractor, TRW, over a five-year And producers must carefully coordinate period. Normally, the four satellites would 43