NASA Aviation Safety Program Systems Analysis/Program Assessment Metrics Review

60 Pages·2003·41.1 MB·English

by  Louis, Garrick E.

I . ;
c
I
UNIVERSITY OF VIRGINIA
SCHOOL OF ENGINEERING AND APPLIED SCIENCE
DEPARTMENOTF SYSTEMS AND mORMATION ENGINEERING
NASA AVIATIONS AFETYPR OGRAM SYSTEMS
ANALYSIS/PROGRAM ASSESSMENMT ETRICS REVIEW
For
NASA-LANGLEYRE SEARCHC ENTER/
AVIATIONS AFETYP ROGRAM
-
SHARONM . JONES PROGRAOMF FICER
BY
GARRICKE . LOUISP, FUNCIPAILN VESTIGATOR
KATHERINE ANDERSON
TISANA HMAD
ALI BOUABID
M-4Y.A SIRNVL4RDANA
PATRICGKU ILBAUD
FINAL
-
TABLEO F CONTENTS i
..
LISTO F FIGURES 11
LISTO F TABLES 1..1..1 .
ABBREVIATIONS AND ACRONYMS iv
EXECUTIVSE~ ~ ~ M A R Y PAGE1
BACKGROUND PAGE2
GOAL PAGE2
OBJECTIVES PAGE2
PROJECT SCHEDULE AND DELIVERABLES PAGE3
OBJECTIVEOSF NASA AND AvSP PAGE4
NASA’s THREEM AINO BJECTIVES IN IWROVING AVIATIONSA FETY PAGE4
SOURCES PAGE4
NASA/ AvSP GOALSA ND OBJEC~VEFSL OWCHART PAGE6
SUMMARY DISCUSSION OF THE AVSP OBJECTIVES AND THEIR PAGE 7
RELATIONSHIP TO NASA’s AVIATIONSA FETY GOAL
CURRENmT mcs USEDW IT” AvSP PAGE9
PROGRAMA SSESSMENOT BJECTIVEASN D METRIcs PAGE9
LISTO F CURRENT METRICS PAGE1 0
SURVEY OF OTHER METRICS PAGE2 4
CONCLUSIOANNSD RECOMMENDATIONS PAGE5 0
REFERENCES PAGE5 1
i
LISTO F FIGURES
Figure 1: Goals and Objectives Flowchart .......................................................................... 6
Figure 2: PAT Objectives and Metrics .............................................................................. 10
Figure 3: Implementation Risk Assessment Process ......................................................... 11
.Figure 4: Implementation Risk Categorization ................................................................. 12
Figure 5: Technical Development Assessment Process .................................................... 15
Figure 6: Technical Development Risk Categorization .................................................... 15
Figure 7: Safety Benefit Assessment Process ................................................................... 18
Figure 8: Safety Benefit Risk Categorization ................................................................... 19
Figure 9: Return on Investment Assessment Process ....................................................... 20
Figure 10: Return On Investment Categorization ............................................................. 21
Figure 11: Cost Analysis Assessment Process .................................................................. 23
Figure 12: Cost Analysis Categorization .......................................................................... 23
Figure 13: Early AvSP Metrics ......................................................................................... 25
Figure 15: Relation between ASIST, OAT and the AvSP ................................................ 36
Figure 16: ASIST focus areas ........................................................................................... 38
Figure 17: Metrics forecast for transport aircrafts ............................................................ 39
Figure 18: Metrics forecast for commuters aircrafts ......................................................... 40
Figure 19: Accident risk assessment process !NLR-ARlsA>. ......................................... 41
Figure 20: risk regions (NLR ARIBA) ............................................................................. 42
Figure 21: Accident risk tolerability matrix (NLR ARIBA) ............................................. 43
Figure 22: Tolerability matrix (NLR ARIBA) .................................................................. 44
Figure 23: Adapted version of the Bow-Tie (NLR ARIBA) ............................................ 46
..
11
LISTO F TABLES
Table 1: Implementation Risk Assessment Criteria .......................................................... 13
Table 2: Translating Risk Rating to Risk Score ................................................................ 13
Table 3: Estimating the Probability of Failure (Pf = R1 + R2 + R3 + R4 + R5 ) ......... 16
Table 4: Estimating the Severity of Impact (Cf j .............................................................. 16
Table 5: Overall Risk Rating = (Pf + Cf)/2 ................................................................. 17
Table 6:R OI Stakeholder Matrix ...................................................................................... 22
Table 7: Comparison Matrix ............................................................................................. 26
Table 8: Rank Assignments .............................................................................................. 27
Table 9: JSIT Feasibility Scoring. ..................................................................................... 30
Table 10: JlMDAT Effectiveness Rating Table ................................................................ 34
...
111
ABBREVIATIONANSD ACRONYMS
AHP Analytic Hierarchy Process
AM Accident Mitigation
ATM Air Traffic Management
ARlBA ATM system safety criticality Raises Issues in Balancing Actors
responsibility
ASAFE Aviation Safety Analysis and Functionality Evaluation
ASIST Aviation Safety Investment Strategy Team
ASMM Aviation System Modeling and Monitoring System
AvSP Aviation Safety Program
BBN Bayesian Belief Network
CAST Commercial Aviation Safety Team
CFIT Center Flight Into Terrain
CI Consistency Index
CR Consistency Ratin
DoD Department of Defense
FAA Federal Aviation Administration
FOOQA Flight Operational Quality Assurance
GA General Aviation
GAIN Global Aviation Information Network
n u
Implementation Risk Level
JSAT Joint Safety Analysis Team
JSIT Joint Safety Implementation Teams
LaRC Langley Research Center
LMI Logistics Management Institute
MMWFH Maintenance Man HourslFlight Hours
M&T Method and Tools
NAS National Aerospace System
NASA National Aeronautics and Space Administration
OAST Office of Aero-Space Technology
OAT Office of Aerospace Technology
iv
PAT Program Assessment Team
PCA Program Commitment Agreement
PRA Probabilistic Risk Assessment
ROI Return-on-Investment
R&D Research and Development
R&T Research and Technology
SAAP Single Aircraft Accident Prevention
sv
Synthetic Vision
SWAP System Wide Accident Prevention
WG B Working Group B
WXAP Weather Accident Prevention
V
vi
EXECUTIVSUEM MARY
The goal of this project is to evaluate the metrics and processes used by NASA’s
Aviation Safety Program in assessing technologies that contribute to NASA’s aviation
safety goals.
There were three objectives for reaching this goal. First, NASA’s main objectives
for aviation safety were documented and their consistency was checked against the main
objectives of the Aviation Safety Program. Next, the metrics used for technology
investment by the Program Assessment function of AvSP were evaluated. Finally, other
metrics that could be used by the Program Assessment Team (PAT) were identified and
evaluated.
This investigation revealed that the objectives are in fact consistent across
organizational levels at NASA and with the FAA.
Some of the major issues discussed in this study which should be further
investigated, are the removal of the Cost and Return-on-Investment rnetrics, the lack of
the metrics to measure the balance of investment and technology, the interdependencies
between some of the metric risk driver categories, and the conflict between “fatal
accident rate” and “accident rate” in the language of the Aviation Safety goal as stated in
different sources.
BACKGROUND
The goal of the NASA Aviation Safety Program (AvSP) is to develop and
demonstrate technologies that contribute to a reduction in the aviation fatal accident rate
by a factor of 5 by year 2007 and by a factor of 10 by year 2022.’ The program is a
partnership that includes NASA, the Federal Aviation Abnistration (FAA), the aviation
indusky and the Department of Defense. NASA’s role is to develop technology and
research needed to help the FAA and industry partners to achieve the President’s
challenge to improve aviation safety. The DoD’s main role is to provide access to useful
data and certain technologies. The NASA Aviation Safety Program has defined products
that will possibly modify airline and/or air traffic control (ATC) operations, enhance
aircraft systems, and improve the identification of potential hazardous situations within
the National Aerospace System (NAS).
GOAL
The goal of this project is to evaluate the metrics and processes used by NASA’s
Aviation Safety Program in assessing technologies that contribute to NASA’s aviation
safety goals.
OBJECTIVES
There are three primary objectives in fulfilling this goal:
1. To document NASA’s three main objectives for aviation safety and check their
consistency with the three main objectives of the Aviation Safety Program.
2. To evaluate the metrics used for technology investment by the Program
Assessment function of AvSP.
3. To identify and evaluate other metrics that could be used by the Program
Assessment Team (PAT).
2
PROJECT SCHEDULE AND DELIVERABLES
Below is a table outlining our project schedule.
I
Deliverable Status
1st Draft of Written Report Due 60 mp leted
LaRC comments Completed
Oral Presentation Completed
7/26 LaRC comments Completed
Final Written Report Due Completed
3