Aviation Rulemaking Advisory Committee

512 Pages·2013·4.87 MB·English

A viation Rulemaking Advisory Committee

Airman Testing Standards + Training Working Group

A  REPORT FROM THE

AIRMAN TESTING STANDARDS + TRAINING WORKING GROUP

TO THE AVIATION RULEMAKING ADVISORY COMMITTEE

September 4, 2013
Aviation Rulemaking Advisory Committee
Airman Testing Standards and Training Working Group

LETTER TO AVIATION RULEMAKING ADVISORY COMMITTEE CHAIR

Page ii Letter to ARAC Chair
Aviation Rulemaking Advisory Committee
Airman Testing Standards and Training Working Group

TABLE OF CONTENTS
LETTER TO AVIATION RULEMAKING ADVISORY COMMITTEE CHAIR ........................... ii
TABLE OF CONTENTS ......................................................................................................iii
EXECUTIVE SUMMARY.................................................................................................. vii
1.0 AIRMAN TESTING STANDARDS + TRAINING WORKING GROUP ..................... 1
1.1 Airman Testing Standards and Training Aviation Rulemaking Committee ............ 1
1.2 Formation of ARAC ATST WG ....................................................................................... 2
2.0 BACKGROUND ................................................................................................. 3
2.1 ARAC ATST WG Tasking ................................................................................................. 3
2.2 Justification for Change ................................................................................................ 4
2.2.1 ACS Philosophy ........................................................................................................ 4
2.2.2 Sound Educational Principles ................................................................................ 6
2.2.3 Industry Standards and Best Practices ................................................................. 7
2.2.4 Safety Management System Methodology ........................................................ 7
2.3 Constituencies Represented ........................................................................................ 8
2.3.1 Certificated Flight Instructors/Designated Pilot Examiners ................................ 9
2.3.2 Aviation Academic Community ........................................................................... 9
2.3.3 Industry Advocacy Associations ........................................................................... 9
2.3.4 Training and Test Preparation Providers ............................................................... 9
2.3.5 FAA Subject Matter Experts .................................................................................. 10
2.4 Methodology ................................................................................................................ 10
2.4.1 Work Plan ................................................................................................................ 10
2.4.2 Meetings and Teleconferences .......................................................................... 11
2.4.3 ACS Development................................................................................................. 11
2.4.4 Aligning Handbooks and Guidance Material ................................................... 12
2.4.5 Developing Appropriate Test Questions and Structures .................................. 12
3.0 CURRENT AIRMAN TRAINING + TESTING FRAMEWORK ................................ 13
3.1 Practical Test Standards .............................................................................................. 13
3.2 Airman Knowledge Testing Program ........................................................................ 13
3.2.1 Airman Knowledge Test Question Bank ............................................................. 14
3.2.2 Knowledge Test Guides ........................................................................................ 14
3.2.3 Learning Statement Reference Guide ............................................................... 14
3.2.4 Computer Testing Supplements .......................................................................... 15
Table of Contents Page iii
Aviation Rulemaking Advisory Committee
Airman Testing Standards and Training Working Group

3.3 Handbooks.................................................................................................................... 15
4.0 DEVELOPMENT OF ACS DOCUMENTS ............................................................ 16
4.1 Development of an ACS (Approach to Future Documents) ................................ 16
4.1.1 ACS Structure ......................................................................................................... 16
4.1.2 ACS Development Process .................................................................................. 18
4.1.3 Coding .................................................................................................................... 19
4.2 Conversion Approach and Priorities ......................................................................... 23
5.0 GUIDANCE MATERIAL .................................................................................... 24
5.1 Updates ......................................................................................................................... 25
5.2 Coordination ................................................................................................................ 25
5.3 Distribution..................................................................................................................... 26
5.4 Communication ........................................................................................................... 26
6.0 QUESTION DEVELOPMENT + TESTING ............................................................. 27
6.1 Test Development ........................................................................................................ 27
6.1.1 Content ................................................................................................................... 27
6.1.2 Coding .................................................................................................................... 28
6.1.3 Test Mapping .......................................................................................................... 29
6.2 Test Evaluation .............................................................................................................. 29
6.3 Test Management ....................................................................................................... 30
6.3.1 Quality Management System (QMS) .................................................................. 31
6.3.2 Sample Questions .................................................................................................. 31
6.3.3 Professional Test Management ........................................................................... 32
7.0 RECOMMENDATIONS ..................................................................................... 34
7.1 Recommendations on Adoption and Implementation of Airman Certification
System ........................................................................................................................... 34
7.1.1 Standards ................................................................................................................ 34
7.1.2 Guidance ............................................................................................................... 34
7.1.3 Testing ..................................................................................................................... 35
7.2 Recommendations on Effectively Managing the Integrated Airman
Certification System..................................................................................................... 35
7.2.1 Stakeholder Participation ..................................................................................... 35
7.2.2 QMS ......................................................................................................................... 35
8.0 CONCLUSION ................................................................................................. 36
Page iv Table of Contents
Aviation Rulemaking Advisory Committee
Airman Testing Standards and Training Working Group

APPENDICES ............................................................................................................. A-i
APPENDIX A: DRAFT PRIVATE PILOT ACS + TRACKING MATRIX ............................. A-1
APPENDIX B: DRAFT INSTRUMENT RATING ACS + TRACKING MATRIX ...................B-1
APPENDIX C: FEDERAL REGISTER NOTICE + COMMENTS ON PRIVATE PILOT +
INSTRUMENT RATING ACS DOCUMENTS ........................................... C-1
APPENDIX D: DRAFT AUTHORIZED INSTRUCTOR ACS + TRACKING MATRIX .......... D-1
APPENDIX E: FEDERAL REGISTER NOTICE + COMMENTS ON AUTHORIZED
INSTRUCTOR, PRIVATE PILOT + INSTRUMENT RATING ACS
DOCUMENTS ....................................................................................... E-1
APPENDIX F: DRAFT COMMERCIAL PILOT ACS + TRACKING MATRIX ................... F-1
APPENDIX G: HANDBOOKS/COMPUTER TESTING SUPPLEMENTS +
RECOMMENDED CHANGES .............................................................. G-1
APPENDIX H: REFERENCE DOCUMENTS + RECOMMENDED CHANGES ................. H-1
APPENDIX I: SAMPLE TEST MAPS .............................................................................. I-1
APPENDIX J: TEST QUESTION DEVELOPMENT GUIDELINES ...................................... J-1
APPENDIX K: SAMPLE TEST QUESTIONS ................................................................... K-1
APPENDIX L: OBSOLETE QUESTIONS ........................................................................ L-1
APPENDIX M: FREQUENTLY ASKED QUESTIONS ...................................................... M-1
APPENDIX N: PROPOSED PRIORITIES FOR ACS CONVERSION + DEPLOYMENT .... N-1
APPENDIX O: AIRMAN CERTIFICATION SYSTEM QMS ............................................. O-1
APPENDIX P: JOB AIDS FOR ACS TRANSITION ........................................................ P-1
APPENDIX Q: PTS-TO-ACS REFERENCES MATRIX .................................................... Q-1
APPENDIX R: ACS CODE SYSTEM .............................................................................R-1
APPENDIX S: AIRMAN CERTIFICATION SYSTEM WORKING GROUP GUIDELINES ... S-1
APPENDIX T: ESTABLISHMENT OF ARAC ATST WG .................................................. T-1

Table of Contents Page v
Aviation Rulemaking Advisory Committee
Airman Testing Standards and Training Working Group

APPENDIX U: ARAC ATST WG MEMBERS + FAA PARTICIPANTS ............................. U-1
Co-Chair Biographies ............................................................................................................ U-2
Industry Member Biographies ............................................................................................... U-3
FAA Participant Biographies ................................................................................................. U-9
APPENDIX V: ABBREVIATIONS + ACRONYMS ......................................................... V-1
Page vi Table of Contents
Aviation Rulemaking Advisory Committee
Airman Testing Standards and Training Working Group

EXECUTIVE SUMMARY
In accordance with its tasking from the Federal Aviation Administration (FAA) Aviation
Rulemaking Advisory Committee (ARAC), the Airman Testing Standards and Training Working
Group (ATST WG) has reviewed the existing standards and guidance material for airman
training and testing and developed the following framework and recommendations.
1. The ATST WG drafted Airman Certification Standards (ACS) documents for the private
pilot, authorized instructor, and commercial pilot certificates and the instrument rating.
By aligning, defining, and integrating the aeronautical knowledge topics required by Title
14 of the Code of Federal Regulations (14 CFR) Part 61 with the Part 61 flight
proficiency metrics enumerated in the existing Practical Test Standards (PTS), the
recommended ACS approach:
• Creates a clear standard for FAA knowledge testing and the industry-developed
training it drives;
• Provides both the conceptual foundation and the tools (e.g., ACS-based coding
system) needed to align testing standards with FAA guidance materials and test
questions, and to maintain that alignment; and
• Enhances safety by listing specific risk management behaviors appropriate to each
Area of Operation.
Three of the four draft ACS documents provided in the appendices to this report (Private
Pilot – Airplane ACS in Appendix A, Instrument Rating ACS in Appendix B, and
Authorized Instructor ACS in Appendix D) reflect comments and suggestions received
from the public through the formal notice and comment process as published in the
Federal Register, and the documents were made available in public dockets that the FAA
established on behalf of the ATST WG (FAA Docket Numbers: FAA-2013-0316 and
FAA-2013-0649).
2. The ATST WG formulated recommendations to align and, where appropriate, consolidate
FAA handbooks (FAA-H-8083-XX series) and computer testing supplements (FAA-CT-
8080-XX series) with the ACS, as well as recommendations for updating these materials
and coordinating, distributing, and communicating changes with/to stakeholders in a
timely fashion.
3. The ATST WG also discussed the airman knowledge testing framework and devised
recommendations for knowledge test development, to include both test question
development and allocation of test question topics; knowledge test evaluation, to include
review ("boarding") of proposed questions; and overall knowledge test management
consistent with professionally-accepted standards and best practices.
Executive Summary Page vii
Aviation Rulemaking Advisory Committee
Airman Testing Standards and Training Working Group

The ATST WG's specific recommendations, as set forth in Chapter 7.0 of this report, fall under
two categories, as described below.
Adoption and Implementation of the Airman Certification System: The first set of
recommendations describes the steps needed for the FAA to adopt and implement the
three components of a fully integrated airman certification system: testing standards in
the form of the ACS, associated guidance material, and the knowledge test itself. These
recommendations include consideration for a carefully constructed beta test of the ACS
approach with representative segments of the stakeholder community.
Management of the Integrated Airman Certification System: The second set of
recommendations lists the steps needed for the FAA to effectively manage the integrated
airman certification system. These recommendations include establishment of a
government/industry Airman Certification System Working Group (ACSWG) and
development of a comprehensive internal Quality Management System (QMS) process
that incorporates all three components of the airman certification system. The
recommended QMS process should also define channels and procedures for input from,
and feedback to, both internal and external stakeholders.
The ATST WG strongly believes that comprehensive adoption of its recommendations will
enable the FAA to provide an integrated airman certification system that enhances aviation
safety and better serves the aviation community.

Page viii Executive Summary
Aviation Rulemaking Advisory Committee
Airman Testing Standards and Training Working Group

1.0 AIRMAN TESTING STANDARDS + TRAINING WORKING GROUP
For a number of years, the general aviation (GA) training community has sharply criticized
Federal Aviation Administration (FAA) pilot/instructor certification standards, training, and
testing materials as being outdated, irrelevant, and out of touch with current technology and
education/training methods. Industry has also faulted the agency for its piecemeal and unilateral
efforts to make revisions.
These long-simmering concerns reached a boiling point in early 2011, when changes to the flight
instructor knowledge (written) test on fundamentals of instructing caused the failure rate to
skyrocket. In fielding questions on this topic at a May 2011 pilot training reform symposium
sponsored by the Society of Aviation and Flight Educators (SAFE), then-FAA Administrator
Randy Babbitt indicated support for a government/industry group to address these concerns.
1.1 Airman Testing Standards and Training Aviation Rulemaking Committee
On September 21, 2011, the FAA chartered the Airman Testing Standards and Training Aviation
Rulemaking Committee (ATST ARC) to provide a forum for the U.S. aviation community to
share its experience and expertise on more effective training and testing in the areas of
aeronautical knowledge and flight proficiency required for safer operation in the evolving
National Airspace System (NAS).
The FAA’s charge to the ARC was to help ensure that the FAA’s technical information related to
existing standards for airman knowledge and skill tests, computer testing supplements,
knowledge test guides, practical test standards, and associated training handbooks incorporates
the most current, relevant, and effective approaches to training and testing. The FAA specifically
tasked the ARC to provide recommendations on a process for ongoing stakeholder participation
in developing the content of these materials, as well as methodologies for developing better
knowledge test item bank questions. The FAA also asked the ARC to develop a prioritized list of
certificates and ratings for update using the new process/methodology.
The ARC submitted its report and nine recommendations to the FAA on April 13, 2012.1 The
ARC’s key recommendation on content called for the FAA to integrate knowledge, skills, and
risk management elements for each major task in the current Practical Test Standards (PTS) into
a single Airman Certification Standards (ACS) document for each airman certificate and rating.
The ARC believed that this approach would support the FAA’s goal of reducing the fatal GA
accident rate, because clearly aligning the aeronautical knowledge testing standards required by
Title 14 of the Code of Federal Regulations (14 CFR) part 61 with the flight proficiency
standards in the existing PTS would significantly improve the relevance, quality, usability, and
effectiveness of aeronautical knowledge testing and training materials for all stakeholders.

1 A Report from the Airman Testing Standards and Training Aviation Rulemaking Committee to the Federal
Aviation Administration (April 13, 2012). (www.faa.gov/aircraft/draft_docs/arc)
Final Report Page 1
Aviation Rulemaking Advisory Committee
Airman Testing Standards and Training Working Group

1.2 Formation of ARAC ATST WG
To accomplish this task and certain other ARC recommendations, the FAA accepted the ARC’s
process and methodology recommendations to establish a stakeholder body of industry subject
matter experts (SME). In August 2012, the FAA assigned this task to the Aviation Rulemaking
Advisory Committee (ARAC), a formal standing committee comprised of industry and aviation
association representatives. The ARAC provides industry input in the form of information,
advice and recommendations to be considered in the full range of FAA rulemaking activities,
including regulatory support.
The FAA announced the ARAC’s acceptance of this task through a Notice in the Federal
Register published on September 12, 2012.2 This Notice described the undertaking and solicited
participants for the ARAC Airman Testing Standards and Training Working Group (ATST WG).
The ARAC tasked the ATST WG to address content and methodologies for the private pilot
certificate, flight instructor certificate, and instrument rating testing and training materials by
developing an integrated ACS document for each certificate/rating. The FAA also tasked the
ATST WG to develop a detailed proposal to realign and, as appropriate, streamline and
consolidate existing FAA guidance material (e.g., FAA-H-8083-XX series handbooks) with the
integrated ACS documents, and to propose knowledge test item bank questions whose content
and structure are consistent with both the ACS documents and the test question development
principles set forth in the ATST ARC's recommendations.
The ATST WG was established and, under the leadership of industry co-chairs representing the
Aircraft Owners and Pilots Association (AOPA) and the National Association of Flight
Instructors (NAFI), began its work in November 2012. ATST WG members and FAA
participants are listed in Appendix U to this report.

2 Notice—Aviation Rulemaking Advisory Committee (ARAC); New Task Assignment for the ARAC:
Establishment of Airman Testing Standards and Training Working Group, 77 FR 56251 (September 12, 2012).
Page 2 Final Report