Methodological Issues in the Study of Airplane Accident Rates by Pilot Age

28 Pages·2004·0.46 MB·English

DOT/FAA/AM-04/8 Methodological Issues in the
Study of Airplane Accident
Office of Aerospace Medicine
Rates by Pilot Age: Effects of
Washington, DC 20591
Accident and Pilot Inclusion
Criteria and Analytic Strategy
Dana Broach
Civil Aerospace Medical Institute
Federal Aviation Administration
Oklahoma City, OK 73125
May 2004
Final Report
This document is available to the public
through the National Technical Information
Service, Springfield, Virginia 22161.
NOTICE
This document is disseminated under the sponsorship of
the U.S. Department of Transportation in the interest of
information exchange. The United States Government
assumes no liability for the contents thereof.
The views expressed in this report are those of the author
and do not necessarily reflect those of the Federal Aviation
Administration or the Department of Transportation.
Technical Documentation Page
1. Report No. 2. Government Accession No. 3. Recipient's Catalog No.
DOT/FAA/AM-04/8
4. Title and Subtitle 5. Report Date
Methodological Issues in the Study of Airplane accident Rates by Pilot May 2004
Age: Effects of Accident and Pilot Inclusion Criteria and Analytic 6. Performing Organization Code
Strategy
7. Author(s) 8. Performing Organization Report No.
Broach D
9. Performing Organization Name and Address 10. Work Unit No. (TRAIS)
Aerospace Human Factors Research Division (AAM-500)
Civil Aerospace Medical Institute 11. Contract or Grant No.
P.O. Box 25082
Oklahoma City, OK 73125
12. Sponsoring Agency Name and Address 13. Type of Report and Period Covered
Office of Aerospace Medicine
Federal Aviation Administration
800 Independence Avenue, S.W.
Washington, DC 20591
15. Supplementary Notes
This work was performed under approved research task AAM-00-A-HRR-520.
16. Abstract
At the direction of the U.S. Senate, Broach, Schroeder, and Joseph (2000a, b) examined accident rates by age
for professional air transport and commercial pilots using an analysis of variance (ANOVA) approach. As an
extension of that work, this report focuses on methodological issues requiring careful consideration and
definition in any analysis of aviation accident rates by pilot age. Three methodological issues are considered:
(a) accident inclusion criteria; (b) pilot inclusion criteria; and (c) analytic strategy. Previous studies are
interpreted with respect to these issues, and an additional analysis is presented to illustrate the impact of
methodological choices on study outcomes. Overall, the comparisons and additional analysis indicate that
accident and pilot inclusion criteria and analytic strategy have substantial impact on study outcomes.
Recommendations are presented for future studies of the relationship of pilot age to aviation accidents.
17. Key Words 18. Distribution Statement
Pilot Age, Aviation Accidents,Accident Rates, Document is available to the public through the
Age 60 Rule, Mandatory Pilot Retirement National Technical Information Service,
Springfield, VA 22161
19. Security Classif. (of this report) 20. Security Classif. (of this page) 21. No. of Pages 22. Price
Unclassified Unclassified 26
Form DOT F 1700.7 (8-72) Reproduction of completed page authorized
i

M I S A A R
ETHODOLOGICAL SSUES IN THE TUDY OF IRPLANE CCIDENT ATES
P A : E A P
BY ILOT GE FFECTS OF CCIDENT AND ILOT
I C A S
NCLUSION RITERIA AND NALYTIC TRATEGY
INTRODUCTION is, accident rates were higher at younger ages (less than
40), lower through the forties and early fifties, then rose
In 1959, the Federal Aviation Administration (FAA) again in the late fifties. A linear trend also fit the age-ac-
adopted what has come to be known as the “Age 60 Rule” cident rate data, but less well than the quadratic trend.
(24 Fed. Reg. 9,767, December 5, 1959). This regulation On the other hand, the a priori planned comparison of
prohibits any air carrier from using the services of any the accident rates for pilots age 55-59 with that for pilots
person as a pilot or co-pilot, and prohibits any person from 60-63 was significant in one study but not the other.
serving as a pilot or co-pilot, on an airplane engaged in When the study population was defined as professional
operations under Part 121 of the Federal Aviation Regu- pilots holding ATP and Class 1 medical certificates in
lations (FARs) if that person has reached his or her 60th Study 3, no significant difference was found between
birthday [14 C.F.R. §121.383(c)]. The “Age 60 Rule” has accident rates for pilots in the 55-59 and 60-63 year old
been, and continues to be controversial. The rule has been age groups. But when the study population was defined
the subject of commentary, research, and legal challenge as professional pilots holding ATP or CP and Class 1 or
since its inception. Most recently, the rule has been the 2 medical certificates in Study 4, the accident rate for
subject of Congressional interest. For example, three bills pilots in the 60-63 group (0.52 accidents per 100,000
(Senate 361 and House Resolutions 481 and 1063) were annualized flight hours) was significantly greater than
introduced in the 106th Congress to change the rule. The the rate for pilots in the 55-59 age group (0.27 accidents
U.S. Senate directed the FAA in 1999 to conduct a study per 100,000 annualized flight hours; unequal variances,
of pilot age and accident rates (U.S. Senate, 1999). The t(11.18) = 2.34, p < .05).
Senate language provided very specific directions as to
how to conduct the desired study. In response, the FAA METHODOLOGICAL ISSUES
completed four studies. The first study (“Study 1”) was
an annotated bibliography of the research literature from The difference between the results reported in Studies
1991 to 1999 (Schroeder, Harris, and Broach, 1999). 3 and 4 suggest that methodological choices may have
The second study (“Study 2”) presented a re-analysis substantial influence on study outcomes. As noted by
of data included in a study of pilot age, accidents, and Li in a 1994 review, two key choices in the study of
incidents reported by the Chicago Tribune in 1999 and a aviation accident rates are (a) the selection of events
discussion of methodological issues in the study of pilot to include in the numerator and (b) the estimate of
age and accident rates (Broach, 1999). exposure in the denominator. Each of these choices
The third and fourth reports documented two empirical entails decisions such as selection of a source database
studies of pilot age and accident rate. Study 3 (Broach, and record inclusion and exclusion criteria. However,
Schroeder, & Joseph, 2000a) examined accident rates by another key choice is analytic strategy, as discussed in
age for professional pilots holding Air Transport Pilot the previous empirical studies. Particular considerations
(ATP) and Class 1 medical certificates. The fourth and are data aggregation, data grouping, rates versus counts
final study (“Study 4”) in the set investigated accident of events as the dependent variable, and the analytic
rates by age for professional pilots holding ATP or Com- technique used. The purpose of the present study was
mercial Pilot (CP) and Class1 or 2 medical certificates to assess the effects of these methodological choices on
(Broach, Schroeder, & Joseph, 2000b). The results of the study outcomes. The previous studies of age and ac-
analyses of accident rate, with pre-planned comparisons cident rates are compared in terms of events included,
between age groups on either side of the “Age 60 Rule” exposure estimates, and analytic strategies. An additional
in the two empirical studies were mixed. On one hand, analysis of the data set developed by Broach, Schroeder
a significant relationship between age and accident rate and Joseph in 2000 (“FAA Age 60 data set”) is presented
was observed in both studies. Specifically, a statistically to illustrate the sensitivity of results to the methodology.
significant quadratic trend between age and accident Implications for future studies of pilot age and accident
rate was reported in both of the empirical studies. That rates are discussed in closing.
1
Accident Inclusion Criteria for on-demand/air taxi operations only in Study 3, com-
The first methodological choice is the selection of pared with over half (768, or 57.5%) in Study 4.
events to analyze. Studies 3 and 4 were constrained by There were differences also in the aircraft characteristics
language in the U.S. Senate (1999) report directing those between the two studies. For example, 91 (or 13%) of
efforts. The critical passage reads: the accidents included in Study 3 involved fixed-wing,
The Committee directs the FAA to conduct a survey of all single-engine aircraft, compared with 439 (33%) in Study
available non-scheduled commercial (and non-commercial,
4. Aircraft with 9 or fewer seats were involved in over a
if available) data concerning the relative accident data cor-
third (258, or 38%) of the accidents in Study 3, com-
related with the amount of flying by pilots as a function of
pared with two-thirds (864, or 65%) of the accidents in
their age for pilots of age 60–63 and comparing it with all
four year groupings of scheduled commercial pilots (and Study 4. Aircraft with reciprocating engines accounted
non-commercial pilots, if available) declining from age 60, for a third (34% or 229) of the accidents included in
i.e., 56–59, 55–58, 54–57, * * * to 21–24. etc. In addition, Study 3, compared with over half (53%, or 707) of the
compare the discernable groups in their entirety and track
Study 4 accidents.
accident frequency as a function of age. (p. 80).
This review of the previous studies by Broach et al.
The Senate language focused on accidents. The Na- indicated that the numerator in Study 4 was dominated by
tional Transportation Safety Board (NTSB) maintains the accidents involving smaller aircraft operating under Part
official system of records for aviation accidents. Therefore, 135. Historically, operations with smaller aircraft under
the first methodological choice made in both Studies 3 Part 135 have had higher accident rates than operations
and 4 was to select the NTSB system of accident records with larger aircraft under Part 121 (Federal Aviation
(“NTSB database”) as the source for event data. The Administration, 1999; National Transportation Safety
next step was to define event inclusion criteria. Given Board, 2002). It is unclear, therefore, if the analysis of
the Senate language and the fact that the rule applied accident rate by pilot age reflected the influence of age,
only to pilots of aircraft engaged in common carriage the inherent risk of small aircraft operations character-
(e.g. commercial operations), Broach, Schroeder and izing Part 135 operations, or both.
Joseph focused on accidents in commercial operations.
As shown in Figure 1, criteria for extraction of accident Pilot Inclusion Criteria
records from the NTSB database were: (a) the regula- The next methodological choice focuses on pilot in-
tion under which the flight was conducted (Parts 121 clusion criteria. As noted by Li (1994), the denominator
or 135); (b) the date of the accident (January 1, 1988 in epidemiological studies of aviation accidents typically
through December 31, 1997); and (c) availability of a consists of some measure of exposure (to the risks of
final report. Using these criteria, the NTSB extracted flight). Sources for exposure estimates include (a) flight
1,359 aviation accident records for analysis by the FAA. hours documented by NTSB in the accident investiga-
Records with incomplete or missing pilot identifiers (24 tion and (b) self-reported flight hours from the medical
records) or for events caused by terrorism (one record) examination. Estimates of flight hours based on NTSB
were excluded from the data set as shown in Figure 1, accident investigations are, by definition, restricted to
resulting in a pool of 1,334 aviation accident records for accident pilots only, and thus do not provide a proper
the studies. The records were further reduced in Study 3 estimate for the population at risk (e.g., all pilots or some
to 680 by selecting those records for accidents where the subset, including those not involved in accidents). Self-
pilot held an ATP and Class 1 medical certificate at the reported flight hours at the time of the medical examina-
time of the accident. In Study 4, all 1,334 records were tion have been used in several investigations, including
included in the analysis. Studies 3 and 4. The official system of records for these
Differences in the characteristics of the accidents in- data is the FAA’s Comprehensive Airman Information
cluded in each study were examined as the first step in System (CAIS).
the investigation of methodological influences on study The researcher again faces the problem of defining
results. One difference between Studies 3 and 4 was in criteria by which to extract records for a subset of air-
the proportion of flights conducted under Part 135. man of interest from a large, complex database. The pilot
Study 3 included 393 flights conducted under Part 135 inclusion criteria for Studies 3 and 4 are illustrated in
(58% of accidents), while Study 4 included 1,047 flights Figure 2. The inclusion criteria were: date of medical
conducted under Part 135 (78% of included accidents). examination; self-reported occupation; age at the time
Similarly, 391, or just over half (57%) of the flights in of the medical examination; type of pilot certificate; and
Study 3 were scheduled flights, compared with 424, or class of medical certificate issued. Employer type was used
just under a third (32%), in Study 4. One-third (221, or as an exclusion criterion in both studies after matching
32.5%) of operators of accident aircraft were certificated the exposure records to NTSB accident records by pilot
2 3
Extraction NTSB Accidents
(FAR = 121, 135; (N=1,359) Review for
NTSB Accidents Dates = 01/01/88 Missing Data
Database to 12/31/97)
Usable NTSB NTSB Accident
Records 2nd Review for Records
(N=1,334) Missing Data (N=1,332)
Study 3 Fixed-wing,
Study 4 land airplane?
Yes
ATP ATP or CP
Multi-engine?
Certificate? Certificate?
Yes Yes
Yes
Class 1 Class 1 or 2
Medical? Medical?
Seats > 9?
Yes Yes
Yes
Study 3 Accidents Study 4 Accidents
(n=680) (n=1,334)
Passenger
operation?
Yes
NOT air taxi or
cargo only?
Yes
ATP
Certificate?
Yes
Study 5 Accidents
(n=213) Yes Class 1
Medical?
Figure 1
Accident inclusion logic for Study 5 compared with Studies 3 and 4
2 3
FAA CAIS
Study 3 Study 5
Study 4
ExamDate = ExamDate = ExamDate =
(1988,1997) (1988,1997) (1988,1997)
Occupation= Occupation= Occupation=
Prof Pilot Prof Pilot Prof Pilot
Medical = Medical = Medical =
Class 1 Class 1 or 2 Class 1
Certificate = Certificate = Certificate =
ATP ATP or COM ATP
Employer =
Employer NOT Employer NOT
Same as
GOVT, MIL GOVT, MIL
Accident pilots
Study 3 Records Study 4 Records
Recent Hours
(n=891,183) (n=939,769)
>= 200
Total Hours >=
1,500
Study 5 Records
(n=753,842)
Figure 2
Pilot inclusion logic for Study 5 compared with Studies 3 and 4
4 5
identifiers and year. Matched exposure-accident records serving as the pilot-in-command or co-pilot of aircraft
for pilots age 64 or older were also excluded in view of engaged in operations under Part 121; the rule does not
the Senate direction to compare accident rates for pilots require mandatory retirement of a pilot or termination
age 60-63 with accident rates for younger pilots. of the pilot’s employment upon reaching age 60. Thus, it
In both studies, records for examinations conducted is possible that the older pilots continued working for an
between January 1, 1988 and December 31, 1997, were air carrier after reaching age 60 in roles other than pilot
extracted from the CAIS medical database. Only records for flights operated under Part 121. Second, many airline
for pilots reporting “professional pilot” as the occupa- operators are certified for multiple classes of operations.
tion were retained in both studies. In Study 3, records It may be the case that older pilots remained with the
for pilots age 23 or greater at the time of their medical reported carrier and served as the pilot for flights con-
and holding ATP certificates were included. In Study 4, ducted under regulations other than Part 121.
records for pilots age 18 or greater at the time of their While some pilots stayed on, it appears that most had
medical examination and with either ATP or CP certifi- sought other opportunities as evidenced by the ten-fold
cates were included. Study 3 included pilots with Class reduction in the number of records for pilots age 60-63
1 medical certificates, while Study 4 included pilots with compared with age 55-59. As shown in Table 1, pilots age
either Class 1 or 2 medical certificates. The inclusion 60 to 63 at the time of medical examination represented
criteria resulted in a pool of 891,183 records of Class 1 1.2% of matched accident-exposure records in both stud-
medical examinations for ATPs in Study 3. There were ies. As a result, estimates of exposure hours for this age
939,767 records of Class 1 or 2 medical examinations group are based on far fewer records than for pilots in
for ATPs or Commercial Pilots available for Study 4. other age ranges. It is also worth noting that the number
The CAIS medical records were aggregated by year and of matched accident-exposure records declines with age
pilot identifiers and exposure hours annualized and then starting with the 40-44 age group in both studies. Ad-
matched to NTSB accident records as described in the ditional research into the “career history” of professional
previous reports. Matched exposure-accident records for pilots might be warranted. Such research might investigate
pilots employed by the government or military and pilots the reasons for attrition, including health and retirement,
age 64 or older were excluded in both analyses in view from the occupation at different ages.
of the specific Senate language focusing on pilots age Inspection of Table 1 also indicates that the distribu-
60 to 63. The characteristics of the resulting samples of tion of pilots across employer types changes with pilot
pilots by age group and type of employer in each study age in both studies. For example, 86% of the 61,663
are presented in Table 1. accident-exposure records for pilots in the 55-59 age
Records for pilots indicating employment by an identi- group indicated employment by an identifiable Part
fiable Domestic, Flag, or Supplemental (Part 121) carrier 121-certificated operator in Study 3, compared with
dominated both matched accident-exposure data sets. In just 71% of the 7,258 accident-exposure records for
Study 3, records from pilots reporting employment by pilots in the 60-63 age group who indicated employ-
an identifiable Part 121 carrier accounted for 75% of all ment by Part 121 operators. The proportion of pilots age
matched accident-exposure records from which accident 55-59 employed by Part 121 operators was significantly
rates were calculated. Even among the oldest pilots (ages 60 greater than the proportion of Part 121 employee pilots
to 63), identifiable Part 121 carriers dominated, account- age 60-63 in Study 3 (Z = 27.27, p < .001). Moreover,
ing for 71% of those 7,258 matched accident-exposure the proportion of matched accident-exposure records
records. In Study 4, records for pilots reporting employ- indicating employment by “Other” for pilots in the 60-
ment by Part 121 carriers accounted for 73% of the total 63 age group was nearly double the proportion of pilots
matched accident-exposure records. Even among the old- in the 55-59 age group. A similar pattern of change in
est pilots age 60 to 63, 67% of the records were for pilots employer types with age was also observed in Study 4. It
reporting employment by identifiable Part 121 carriers. is unclear from review of the CAIS data dictionaries what
In other words, the FAA Age 60 data set was dominated types of operators are included in the “Other” employer
by records from pilots of all ages reporting employment category. Moreover, as shown in Table 1, records for pilots
by identifiable Part 121 operators, that is, the popula- employed by entities other than Part 121 and 135 carriers
tion most likely impacted by the “Age 60 Rule.” At first, were included in the matched accident-exposure records
this seemed puzzling given the common description of in Studies 3 and 4.
the rule as forcing retirement at age 60 on airline pilots. The effect of this sampling error (e.g., inclusion
There are two possible explanations. First, the “Age 60 of pilots employed by entities other than Part 121 or
Rule” prohibits a person who has reached age 60 from 135 carriers) was to inflate the exposure estimate in the
4 5
557140383 9737769615
256412654 6327423530
otal 44,693,314,409,405,691,661,67,228,0 5,946,700,821,815,211,196,464,58,070,8
T 111 6 1111 6

141975568 6014680465
g 578204605 0893353701
n 465554215 2577766329
ssi 3, 4,
Mi
4
and elf 5275716477465824467 266411210685105797539691
3 S
s
e 726413463 3875848384
di r 958071616495107453 81488356168869100258
age group for Stu Industry Othe 8522,7127,7623,10020,10117,11411,617,311,639133, 343,8623,9030,9826,12623,12319,13413,768,332,800151, e 64 or older. e 64 or older.
mployer type and Employer Type ment Military 1Study 3 2Study 4 ecords for pilots agecords for pilots ag
e rn e re r
s by Gove posurposur
d xx
ent recor oreign 3341,1571,6561,2491,0399157591497,258 4491,5262,4092,2311,8071,5671,4941,37527913,137 ccident-eccident-e
ure-accid rt 121 F 19,09861,41586,54985,48884,99477,46653,0545,15573,219 1,24219,40664,33990,25887,88187,49479,68954,1755,39689,880 ed 1,396 aed 1,803 a
s a 4 4 dd
o P uu
p clcl
ex 5 486118975581624769 99874585449515785098 exex
atched Part 13 1,62,11,81,31,263 9,3 11,52,21,81,41,273 9,7 y 3 total y 4 total
1 er of m Group Total Total 1Stud2Stud
able umb Age E 29 0-34 5-39 0-44 5-49 0-54 5-59 0-63 E 24 5-29 0-34 5-39 0-44 5-49 0-54 5-59 0-63 otes:
TN L3344556 L23344556 N
6 7