Review of NASA's Aerospace Technology Enterprise: An Assessment of Nasa's Pioneering Revolutionary Technology Program

136 Pages·2003·4.47 MB·English

by  National Research Council

Review of NASA’s
Aerospace Technology Enterprise
An Assessment of NASA’s
Pioneering Revolutionary Technology Program
Committee for the Review of NASA’s
Pioneering Revolutionary Technology (PRT) Program
Aeronautics and Space Engineering Board
Division on Engineering and Physical Sciences
THE NATIONAL ACADEMIES PRESS 500 Fifth Street, N.W. Washington, DC 20001
NOTICE: The project that is the subject of this report was approved by the Governing Board of the
National Research Council, whose members are drawn from the councils of the National Academy of
Sciences, the National Academy of Engineering, and the Institute of Medicine. The members of the com-
mittee responsible for the report were chosen for their special competences and with regard for appropriate
balance.
This study was supported by Contract No. NASW 99037 between the National Academy of Sciences
and the National Aeronautics and Space Administration. Any opinions, findings, conclusions, or recom-
mendations expressed in this publication are those of the author(s) and do not necessarily reflect the views
of the agency that provided support for the project.
International Standard Book Number 0-309-09080-6 (Book)
International Standard Book Number 0-309-52759-7 (PDF)
Cover images courtesy of the National Aeronautics and Space Administration.
Available in limited supply from: Additional copies are available from:
Aeronautics and Space Engineering Board National Academy Press
500 Fifth Street, N.W. Box 285
Washington, DC 20001 500 Fifth Street, N.W.
[phone] Washington, DC 20055
[phone]
[phone] (in the Washington
metropolitan area)
http://www.nas.edu
Copyright 2003 by the National Academy of Sciences. All rights reserved.
Printed in the United States of America
The National Academy of Sciences is a private, nonprofit, self-perpetuating society of distinguished schol-
ars engaged in scientific and engineering research, dedicated to the furtherance of science and technology
and to their use for the general welfare. Upon the authority of the charter granted to it by the Congress in
1863, the Academy has a mandate that requires it to advise the federal government on scientific and tech-
nical matters. Dr. Bruce M. Alberts is president of the National Academy of Sciences.
The National Academy of Engineering was established in 1964, under the charter of the National Acad-
emy of Sciences, as a parallel organization of outstanding engineers. It is autonomous in its administration
and in the selection of its members, sharing with the National Academy of Sciences the responsibility for
advising the federal government. The National Academy of Engineering also sponsors engineering pro-
grams aimed at meeting national needs, encourages education and research, and recognizes the superior
achievements of engineers. Dr. Wm. A. Wulf is president of the National Academy of Engineering.
The Institute of Medicine was established in 1970 by the National Academy of Sciences to secure the
services of eminent members of appropriate professions in the examination of policy matters pertaining to
the health of the public. The Institute acts under the responsibility given to the National Academy of
Sciences by its congressional charter to be an adviser to the federal government and, upon its own initiative,
to identify issues of medical care, research, and education. Dr. Harvey V. Fineberg is president of the
Institute of Medicine.
The National Research Council was organized by the National Academy of Sciences in 1916 to associate
the broad community of science and technology with the Academy’s purposes of furthering knowledge and
advising the federal government. Functioning in accordance with general policies determined by the Acad-
emy, the Council has become the principal operating agency of both the National Academy of Sciences and
the National Academy of Engineering in providing services to the government, the public, and the scientific
and engineering communities. The Council is administered jointly by both Academies and the Institute of
Medicine. Dr. Bruce M. Alberts and Dr. Wm. A. Wulf are chair and vice chair, respectively, of the National
Research Council.
www.national-academies.org
COMMITTEE FOR THE REVIEW OF NASA’S PIONEERING REVOLUTIONARY
TECHNOLOGY (PRT) PROGRAM
RAYMOND S. COLLADAY, Chair, Consultant and President (retired), Lockheed Martin
Astronautics, Denver
BENJAMIN BUCHBINDER, NASA (retired), Bonaire, Netherlands Antilles
LEONARD H. CAVENY, Ballistic Missile Defense Organization (retired), Fort Washington,
Maryland
SERGIO GUARRO, Aerospace Corporation, El Segundo, California (from June 2002 until
April 2003)
DAVID J. KASIK, The Boeing Company, Seattle
DIMITRI MAVRIS, Georgia Institute of Technology, Atlanta
DENNIS K. McBRIDE, Potomac Institute for Policy Studies, Arlington, Virginia
TODD J. MOSHER, Utah State University, Logan
JAMES ODOM, Science Applications International Corporation, Huntsville, Alabama
LEE D. PETERSON, University of Colorado, Boulder
JOSEPH B. REAGAN (NAE), Lockheed Martin Missiles and Space (retired), Saratoga,
California
CYNTHIA R. SAMUELSON, Logistics Management Institute, McLean, Virginia
MARC SNIR, University of Illinois, Urbana-Champaign
MICHAEL J. ZYDA, Naval Postgraduate School, Monterey, California
PANEL ON COMPUTING, INFORMATION, AND COMMUNICATIONS TECHNOLOGIES (CICT)
MICHAEL J. ZYDA, Chair, Naval Postgraduate School, Monterey, California
WILLIAM COHEN, Consultant, Pittsburgh (from June 2002 until June 2003)
DELORES M. ETTER (NAE), United States Naval Academy, Annapolis, Maryland
MARY JEAN HARROLD, Georgia Institute of Technology, Atlanta
CHANDRIKA KAMATH, Lawrence Livermore National Laboratory, Livermore, California
DAVID J. KASIK, The Boeing Company, Seattle
ALFRED U. MacRAE (NAE), MacRae Technologies, Berkeley Heights, New Jersey
DUANE T. McRUER (NAE), Systems Technology, Inc., Manhattan Beach, California
RICHARD MULLER (NAE), University of California, Berkeley
CYNTHIA R. SAMUELSON, Logistics Management Institute, McLean, Virginia
JUDE SHAVLIK, University of Wisconsin, Madison
SANDEEP SINGHAL, ReefEdge, Inc., Fort Lee, New Jersey
MARC SNIR, University of Illinois, Urbana-Champaign
PANEL ON ENGINEERING FOR COMPLEX SYSTEMS (ECS)
DENNIS K. McBRIDE, Chair, Potomac Institute for Policy Studies, Arlington, Virginia
TORA K. BIKSON, RAND Corporation, Santa Monica, California
BENJAMIN BUCHBINDER, NASA (retired), Bonaire, Netherlands Antilles
PHILIP R. COHEN, Oregon Health and Science University, Beaverton
SERGIO GUARRO, Aerospace Corporation, El Segundo, California (from June 2002 until
April 2003)
MYRON HECHT, SoHaR Incorporated, Beverly Hills, California
JIM LARUS, Microsoft Research, Redmond, Washington
DIMITRI MAVRIS, Georgia Institute of Technology, Atlanta
RONALD WESTRUM, Consultant, Ann Arbor, Michigan (from June 2002 until February 2003)
F. GORDON WILLIS, Vulcan Works, LLC, Ann Arbor, Michigan
iv
PANEL ON ENABLING CONCEPTS AND TECHNOLOGIES (ECT)
LEE D. PETERSON, Chair, University of Colorado, Boulder
CLINTON A. BOYE, Sandia National Laboratories, Albuquerque, New Mexico
LEONARD H. CAVENY, Ballistic Missile Defense Organization (retired), Fort
Washington, Maryland
STANLEY V. GUNN, Rocketdyne (retired), Chatsworth, California
ANTHONY K. HYDER, University of Notre Dame, South Bend, Indiana
DIMITRIS C. LAGOUDAS, Texas A&M University, College Station
TODD J. MOSHER, Utah State University, Logan
JAY S. PEARLMAN, The Boeing Company, Seattle
JOSEPH B. REAGAN (NAE), Lockheed Martin Missiles and Space (retired), Saratoga,
California
NANCY R. SOTTOS, University of Illinois, Urbana-Champaign
GREGORY G. SPANJERS, Air Force Research Laboratory, Albuquerque, New Mexico
MICHAEL J. STALLARD, Aerospace Corporation, Albuquerque, New Mexico
COMMITTEE AND PANELS STAFF
KAREN E. HARWELL, Study Director
DOUGLAS H. BENNETT, Program Officer
GEORGE M. LEVIN, Director, Aeronautics and Space Engineering Board
BRIDGET R. EDMONDS, Senior Project Assistant
JENNIFER D. PINKERMAN, Research Associate
ANNA L. FARRAR, Financial Associate
v
AERONAUTICS AND SPACE ENGINEERING BOARD
WILLIAM W. HOOVER, Chair, U.S. Air Force (retired), Williamsburg, Virginia
A. DWIGHT ABBOTT, Aerospace Corporation (retired), Palos Verdes Estates, California
RUZENA K. BAJCSY (NAE/IOM), University of California, Berkeley
JAMES (MICKY) BLACKWELL, Lockheed Martin (retired), Marietta, Georgia
ANTHONY J. BRODERICK, Aviation Safety Consultant, Catlett, Virginia
SUSAN M. COUGHLIN, Aviation Safety Alliance, Washington, D.C.
ROBERT L. CRIPPEN, Thiokol Propulsion (retired), Palm Beach Gardens, Florida
DONALD L. CROMER, USAF (retired) and Hughes Space and Communications (retired),
Fallbrook, California
JOSEPH FULLER, Jr., Futron Corporation, Bethesda, Maryland
RICHARD GOLASZEWSKI, GRA Incorporated, Jenkintown, Pennsylvania
JAMES M. GUYETTE, Rolls-Royce North America, Chantilly, Virginia
JOHN L. JUNKINS (NAE), Texas A&M University, College Station
JOHN M. KLINEBERG, Space Systems/Loral (retired), Redwood City, California
ILAN M. KROO, Stanford University, Stanford, California
JOHN K. LAUBER, Airbus North America, Inc., Washington, D.C.
GEORGE K. MUELLNER, The Boeing Company, Seal Beach, California
DAVA J. NEWMAN, Massachusetts Institute of Technology, Cambridge
JAMES G. O’CONNOR (NAE), Pratt & Whitney (retired), Coventry, Connecticut
MALCOLM O’NEILL, Lockheed Martin Corporation, Bethesda, Maryland
CYNTHIA R. SAMUELSON, Logistics Management Institute, McLean, Virginia
KATHRYN C. THORNTON, University of Virginia, Charlottesville
HANSEL E. TOOKES II, Raytheon International (retired), Falls Church, Virginia
DIANNE S. (WILEY) PALMER, The Boeing Company, Washington, D.C.
THOMAS L. WILLIAMS, Northrop Grumman, Bethpage, New York
GEORGE M. LEVIN, Director
vi
Preface
The Committee for the Review of NASA’s Pio- The committee and panels did not assess other pro-
neering Revolutionary Technology (PRT) Program of grams within NASA on which the PRT program and its
the National Research Council (NRC) and its three sup- portfolio depend or other programs within NASA that
porting panels have completed an approximately 20- research similar technology areas. The committee and
month-long study evaluating the technical quality of panels did recommend when these programs should be
the National Aeronautics and Space Administration’s integrally connected and the PRT portfolios managed
(NASA’s) PRT program. The statement of task for this with the global NASA investment in mind.
study is given in Appendix A. The study was spon- I wish to take this opportunity to thank the chairs
sored by NASA and conducted by a committee and and members of the three supporting panels for their
three supporting panels appointed by the NRC (see leadership, detailed assessments, and commitment of
Appendix B for biographies of committee and panel time to the review. Their input has been vital to the
members). The Office of Management and Budget quality of the entire review. On behalf of the commit-
(OMB) requested the review and assisted in the formu- tee and panels, I would also like to thank the various
lation of the statement of task. NASA program managers and technical staff for their
This report provides a technical assessment of the cooperation in providing the information necessary to
quality of the PRT program and its components and complete the review and in hosting our panel members
offers recommendations for improving the program. at various site visits and for their open discussion dur-
The committee and panels note that they refrained from ing these opportunities. We also thank those who took
drawing any conclusions on matters of budget or rec- the time to participate in committee and panel meet-
ommending increases in budget levels. While some ings and provide background materials. Finally, this
areas may suffer from a lack of critical mass, recom- study and the final report would not have been possible
mendations for increased resources to address the prob- without the expert support of the NRC staff. Their dedi-
lem are of little value to management and have been cation to keeping the review on track deserves special
avoided. The committee and panels also refrained, as recognition and thanks. Thanks go especially to Karen
much as possible, from commenting on matters related E. Harwell, study director, for her professional steering
to programmatics and program organization unless a of the overall committee effort as well as her support to
link could be established between these concerns and the ECT panel, and to Douglas H. Bennett for his sup-
technical quality, portfolio management, or interaction port to the CICT and ECS panels.
within NASA and with the external technical commu-
nity. NASA’s Aerospace Technology Advisory Com- Raymond S. Colladay, Chair
mittee (ATAC) and its PRT subcommittee hold an an- Committee for the Review of NASA’s
nual relevance and programmatic review for the PRT Pioneering Revolutionary Technology
program. (PRT) Program
vii
Acknowledgment of Reviewers
This report has been reviewed in draft form by individuals chosen for their diverse perspectives and technical
expertise, in accordance with procedures approved by the National Research Council’s (NRC’s) Report Review
Committee. The purpose of this independent review is to provide candid and critical comments that will assist the
institution in making its published report as sound as possible and to ensure that the report meets institutional
standards for objectivity, evidence, and responsiveness to the study charge. The review comments and draft manu-
script remain confidential to protect the integrity of the deliberative process. We wish to thank the following
individuals for their review of this report:
Dwight Abbott, Aerospace Corporation (retired),
Douglas Allen, Schafer Corporation,
George Apostolakis, Massachusetts Institute of Technology,
Daniel Baker, University of Colorado,
Vicki Bier, University of Wisconsin,
John Evans, COMSAT (retired),
Michael Frank, Safety Factor Associates, Inc.,
Henry Helvajian, Aerospace Corporation,
William Howard, Consultant,
James McGroddy, IBM Corporation,
Phil Papadopoulos, University of California, San Diego,
Suraj Rawal, Lockheed Martin Corporation,
Walter Robb, Vantage Management, Inc.,
Richard Schwartz, Purdue University,
Norman Sleep, Stanford University,
Patrick Stadter, Johns Hopkins University Applied Physics Laboratory,
David Waltz, NEC Research Institute, Inc. (retired), and
Mary Young, HRL Laboratories.
Although the reviewers listed above have provided many constructive comments and suggestions, they were
not asked to endorse the conclusions or recommendations, nor did they see the final draft of the report before its
release. The review of this report was overseen by Alexander H. Flax, Consultant. Appointed by the National
Research Council, he was responsible for making certain than an independent examination of this report was carried
out in accordance with institutional procedures and that all review comments were carefully considered. Respon-
sibility for the final content of this report rests entirely with the authoring committee and the institution.
viii
Contents
EXECUTIVE SUMMARY 1
1 INTRODUCTION 11
Background, 11
Approach to Assessment, 11
Report Organization and Development, 12
Reference, 12
2 OVERALL ASSESSMENT OF THE PIONEERING REVOLUTIONARY 13
TECHNOLOGY PROGRAM
Overall Assessment, 13
Common Themes, 14
Systems Analysis, 14
Benchmarking and Metrics, 15
External Peer Review and Competition, 15
Stability and Continuity, 17
Research Portfolio Balance, 17
Technology Transition, 19
References, 19
Briefings, 19
3 REPORT OF THE PANEL ON COMPUTING, INFORMATION, AND 20
COMMUNICATIONS TECHNOLOGY
Introduction, 20
Review Process, 21
Overall Observations on the CICT Program, 21
General Observations, 22
Research Program Architecture, 23
ix
x CONTENTS
Service-Oriented Tasks, 23
Final Research Applications, 26
Final Products and Research Benchmarks, 26
Research Portfolio, 27
Detailed Assessment of Research Portfolio, 27
Overlap with Other PRT Programs, 28
Expanding Existing Research Areas, 28
Critical Computing Expertise That May Be Missing, 30
Research Plans and Methodology, 30
Task Deliverables and Their Fit to NASA Goals, 31
Maturing a Technology, 31
Reviewing and Selecting Proposals, 32
Technology Readiness Level, 33
Reorganization of Projects and Management Structure, 33
Technical Community Connections, 33
Awareness of Relevant Research, 35
Use of Talent Inside and Outside NASA, 36
Benchmark Datasets and Problem Sets, 36
Facilities, Personnel, and Equipment, 36
References, 37
Briefings, 37
4 REPORT OF THE PANEL ON ENGINEERING FOR COMPLEX SYSTEMS 38
Introduction, 38
Review Process, 38
General Observations, 39
Programmatic Risk Management, 39
Technical Quality, 40
Challenge Areas, 41
Specific Task Discussions, 41
System Reasoning for Risk Management, 42
Connections to the External Community, 43
Research Portfolio, 44
People and Facilities, 44
Methodology, 45
Quality of Work, 45
Observations on Specific SRRM Tasks, 45
Knowledge Engineering for Safety and Success, 45
Human and Organizational Risk Management, 45
Knowledge Management, 47
Observations on Specific KESS Tasks, 47
Resilient Systems and Operations, 49
Intelligent and Adaptive Operations and Control, 49
Resilient Software Engineering, 49
Observations on Specific RSO Tasks, 51
References, 52
Briefings, 52