AR-2007-053 Analysis, Causality and Proof in Safety Investigations

121 Pages·2008·3.92 MB·English

ATSB TRANSPORT SAFETY RESEARCH REPORT
Aviation Research and Analysis Report – AR-2007-053
Analysis, Causality and
Proof in Safety Investigations

ATSB TRANSPORT SAFETY REPORT
Aviation Research and Analysis Report

AR-2007-053

Analysis, Causality

and Proof in Safety Investigations

- i -
Published by: Australian Transport Safety Bureau
Postal address: PO Box 967, Civic Square ACT 2608
Office location: 15 Mort Street, Canberra City, Australian Capital Territory
Telephone: [phone]; from overseas + 61 2 6274 6440
Accident and incident notification: 1800 011 034 (24 hours)
Facsimile: 02 6247 3117; from overseas + 61 2 6247 3117
E-mail: [email protected]
Internet: www.atsb.gov.au
© Commonwealth of Australia 2008.
This work is copyright. In the interests of enhancing the value of the information contained in this
publication you may copy, download, display, print, reproduce and distribute this material in
unaltered form (retaining this notice). However, copyright in the material obtained from other
agencies, private individuals or organisations, belongs to those agencies, individuals or
organisations. Where you want to use their material you will need to contact them directly.
Subject to the provisions of the Copyright Act 1968, you must not make any other use of the
material in this publication unless you have the permission of the Australian Transport Safety
Bureau.
Please direct requests for further information or authorisation to:
Commonwealth Copyright Administration, Copyright Law Branch
Attorney-General’s Department, Robert Garran Offices, National Circuit, Barton ACT 2600
www.ag.gov.au/cca

ISBN and formal report title: see ‘Document retrieval information’ on page vii.
- ii -
CONTENTS

THE AUSTRALIAN TRANSPORT SAFETY BUREAU............................... VIII

EXECUTIVE SUMMARY...................................................................................... IX

1
INTRODUCTION.................................................................................................1

1.1 Purpose of this report...................................................................................1

1.2 Overview of the report.................................................................................1

1.3 Purpose of safety investigations..................................................................2

1.3.1 Relevant international and national standards............................2

1.3.2 Australian legislation...................................................................3

1.3.3 Means by which safety investigations can enhance safety .......4

1.4 The analysis phase of an investigation .......................................................5

1.5 Need for an enhanced analysis framework.................................................6

1.6 Overview of the ATSB analysis framework ..............................................8

1.7 Overview of concerns that have been expressed........................................9

2 KEY TERMS USED IN THE ATSB ANALYSIS FRAMEWORK............13

2.1 Safety factor................................................................................................13

2.1.1 ATSB definition.........................................................................13

2.1.2 Key points to note......................................................................13

2.1.3 Types of safety factor ................................................................14

2.2 Contributing safety factor..........................................................................15

2.2.1 ATSB definition.........................................................................15

2.2.2 Key points to note......................................................................15

2.2.3 Comparison with previous BASI / ATSB approaches ............17

2.2.4 Comparison with some other definitions..................................18

2.3 Safety issue.................................................................................................19

2.3.1 ATSB definition.........................................................................19

2.3.2 Key points to note......................................................................19

2.3.3 Levels of risk associated with safety issues .............................20

2.4 Contributing safety factors versus safety issues.......................................20

2.5 Arguments, premises and findings............................................................22

2.5.1 Definitions..................................................................................22

2.5.2 Types of findings in safety investigation reports.....................22

2.5.3 Key findings ...............................................................................22

2.5.4 Intermediate findings.................................................................23

2.5.5 Types of arguments....................................................................24

- iii -
2.6 Uncertainty, probability and likelihood expressions ...............................24

2.6.1 Probability and likelihood .........................................................24

2.6.2 Verbal probability expressions..................................................24

2.6.3 IPCC definitions.........................................................................25

2.6.4 ATSB approach to using verbal probability expressions........26

2.6.5 The need for evidence................................................................28

3 ACCIDENT DEVELOPMENT MODELS .....................................................31

3.1 General comments on accident development...........................................31

3.2 The role of accident development models in safety investigation..........32

3.3 The Reason model......................................................................................33

3.3.1 Overview of the model ..............................................................33

3.3.2 Some general points...................................................................34

3.4 The ATSB investigation analysis model ..................................................35

3.4.1 Introduction ................................................................................35

3.4.2 Occurrence events......................................................................38

3.4.3 Individual actions.......................................................................40

3.4.4 Local conditions.........................................................................41

3.4.5 Risk controls...............................................................................41

3.4.6 Organisational influences ..........................................................42

3.5 How far should an investigation go? ........................................................42

3.6 Use of charts to show relationships ..........................................................44

3.7 Comparison of the ATSB model with other models................................47

4 ATSB ANALYSIS PROCESS ...........................................................................49

4.1 General description ....................................................................................49

4.2 Overview of safety factors analysis ..........................................................50

4.3 Testing potential factors ............................................................................51

4.4 Evidence tables...........................................................................................52

4.4.1 Description of evidence tables ..................................................52

4.4.2 Process for developing an evidence table.................................53

4.5 Criteria for evaluating existence, influence and importance...................55

5 STANDARDS OF PROOF AND RELATED CONCEPTS .........................57

5.1 Legal proceedings ......................................................................................57

5.1.1 Causation in legal proceedings .................................................57

5.1.2 Standards of proof......................................................................60

5.1.3 The Briginshaw scale.................................................................62

5.1.4 Relationship between probability and standards in legal

settings ..................................................................................65

- iv -
5.2 Coronial inquests........................................................................................67

5.2.1 General aspects of coronial inquests.........................................67

5.2.2 Determination of causes in coronial inquests...........................69

5.2.3 Standard of proof and standard of evidence in coronial

inquests..................................................................................70

5.3 Safety investigations..................................................................................71

5.3.1 Contributing factors versus causal factors................................71

5.3.2 Determining contribution or causation.....................................72

5.3.3 Two approaches to making judgements about factors.............72

5.3.4 Relative-to-occurrence approach ..............................................73

5.3.5 Link-by-link approach ...............................................................76

5.3.6 Standards of proof used in safety investigations......................77

5.4 Revisiting the ATSB analysis framework ................................................79

5.4.1 Standard of proof .......................................................................79

5.4.2 Standard of evidence..................................................................81

5.5 Summary comparison between the ATSB framework and other

approaches ...............................................................................................83

6 CONCERNS WITH THE ATSB AND SIMILAR ANALYSIS

APPROACHES ..................................................................................................87

6.1 Should the ATSB use the same approach as Coronial inquests for

determining findings?.............................................................................87

6.2 Is the ATSB standard of proof too precise? .............................................88

6.3 Is the ATSB standard of proof too low?...................................................88

6.4 Should the ATSB use the Briginshaw scale? ...........................................89

6.5 Is the ATSB analysis model biased towards finding that

organisational factors contributed to accidents?...................................91

6.6 Has the pendulum swung too far towards searching for

organisational factors?............................................................................92

6.7 Should safety investigations identify the ‘most important’

contributing factors?...............................................................................93

6.8 Should safety investigations identify who is (most) responsible?..........95

6.9 Are Reason model analysis methods too prescriptive? ...........................96

6.10 The Reason model does not appropriately reflect the nature or

complexity of modern accidents ............................................................97

- v -
7 CONCLUDING COMMENTS.........................................................................99

APPENDIX A – ATSB CRITERIA FOR THE TEST FOR EXISTENCE ....101

APPENDIX B – ATSB CRITERIA FOR THE TEST FOR INFLUENCE....103

APPENDIX C – ATSB CRITERIA FOR THE TEST FOR

IMPORTANCE................................................................................................107

APPENDIX D – SUBMISSIONS...........................................................................109

- vi -
DOCUMENT RETRIEVAL INFORMATION

Report No. Publication date No. of pages ISBN
AR-2007-053 11 March 2008 106 978-1-921165-97-9
Publication title
Analysis, Causality and Proof in Safety Investigations
Author(s)
Walker, Michael B., and Bills, Kym M.
Prepared by Reference No.
Australian Transport Safety Bureau Mar2008/Infrastructure 8060
PO Box 967, Civic Square ACT 2608 Australia
www.atsb.gov.au
Abstract
The quality of a safety investigation’s analysis activities plays a critical role in
determining whether the investigation is successful in enhancing safety. However,
safety investigations require analysis of complex sets of data and situations where
the available data can be vague, incomplete and misleading. Despite its importance,
complexity, and reliance on investigators’ judgements, analysis has been a neglected
area in terms of standards, guidance and training of investigators in most
organisations that conduct safety investigations.
To address this situation, the Australian Transport Safety Bureau (ATSB) developed
a comprehensive investigation analysis framework. The present report provides an
overview of the ATSB investigation analysis framework and concepts such as the
determination of contribution and standard of proof. The report concludes by
examining the nature of concerns that have been raised regarding the ATSB analysis
framework and the ATSB’s consideration of these concerns.
The ATSB believes that its investigation analysis framework is well suited to its role
as an independent, no-blame safety investigation body. It is hoped and expected that
ongoing development and provision of information about the framework can help
the safety investigation field as a whole consider some important issues and help
develop the best means of conducting safety investigations to enhance future safety.

- vii -
THE AUSTRALIAN TRANSPORT SAFETY BUREAU

The Australian Transport Safety Bureau (ATSB) is an operationally independent
multi-modal bureau within the Australian Government Department of
Infrastructure, Transport, Regional Development and Local Government. ATSB
investigations are independent of regulatory, operator or other external
organisations.
The ATSB is responsible for investigating accidents and other transport safety
matters involving civil aviation, marine and rail operations in Australia that fall
within Commonwealth jurisdiction, as well as participating in overseas
investigations involving Australian registered aircraft and ships. A primary concern
is the safety of commercial transport, with particular regard to fare-paying
passenger operations.
The ATSB performs its functions in accordance with the provisions of the
Transport Safety Investigation Act 2003 and Regulations and, where applicable,
relevant international agreements.
Purpose of safety investigations
The object of a safety investigation is to enhance safety. To reduce safety-related
risk, ATSB investigations determine and communicate the safety factors related to
the transport safety matter being investigated.
It is not the object of an investigation to determine blame or liability. However, an
investigation report must include factual material of sufficient weight to support the
analysis and findings. At all times the ATSB endeavours to balance the use of
material that could imply adverse comment with the need to properly explain what
happened, and why, in a fair and unbiased manner.
Developing safety action
Central to the ATSB’s investigation of transport safety matters is the early
identification of safety issues in the transport environment. The ATSB prefers to
encourage the relevant organisation(s) to proactively initiate safety action rather
than release formal recommendations. However, depending on the level of risk
associated with a safety issue and the extent of corrective action undertaken by the
relevant organisation, a recommendation may be issued either during or at the end
of an investigation.
The ATSB has decided that when safety recommendations are issued, they will
focus on clearly describing the safety issue of concern, rather than providing
instructions or opinions on the method of corrective action. As with equivalent
overseas organisations, the ATSB has no power to implement its recommendations.
It is a matter for the body to which an ATSB recommendation is directed (for
example the relevant regulator in consultation with industry) to assess the costs and
benefits of any particular means of addressing a safety issue.
About ATSB investigation reports: How investigation reports are organised and
definitions of terms used in ATSB reports, such as safety factor, contributing safety
factor and safety issue, are provided on the ATSB web site www.atsb.gov.au.
- viii -