Standardizing Test Scores - RAND Corporation

169 Pages·2008·0.72 MB·English

THE ARTS This PDF document was made available
CHILD POLICY from www.rand.org as a public service of
CIVIL JUSTICE
the RAND Corporation.
EDUCATION
ENERGY AND ENVIRONMENT Jump down to document6
HEALTH AND HEALTH CARE
INTERNATIONAL AFFAIRS The RAND Corporation is a nonprofit
NATIONAL SECURITY
research organization providing
POPULATION AND AGING
objective analysis and effective
PUBLIC SAFETY
solutions that address the challenges
SCIENCE AND TECHNOLOGY
SUBSTANCE ABUSE facing the public and private sectors
TERRORISM AND
around the world.
HOMELAND SECURITY
TRANSPORTATION AND
INFRASTRUCTURE Support RAND
WORKFORCE AND WORKPLACE
Purchase this document
Browse Books & Publications
Make a charitable contribution
For More Information
Visit RAND at www.rand.org
Explore RAND Project AIR FORCE
View document details
Limited Electronic Distribution Rights
This document and trademark(s) contained herein are protected by law as indicated
in a notice appearing later in this work. This electronic representation of RAND
intellectual property is provided for non-commercial use only. Unauthorized
posting of RAND PDFs to a non-RAND Web site is prohibited. RAND PDFs are
protected under copyright law. Permission is required from RAND to reproduce,
or reuse in another form, any of our research documents for commercial use. For
information on reprint and linking permissions, please see RAND Permissions.
This product is part of the RAND Corporation monograph series.
RAND monographs present major research findings that address the
challenges facing the public and private sectors. All RAND mono-
graphs undergo rigorous peer review to ensure high standards for
research quality and objectivity.
The Weighted Airman
Promotion System
Standardizing Test Scores
Michael Schiefer, Albert A. Robbert, John S. Crown,
Thomas Manacapilli, Carolyn Wong
Prepared for the United States Air Force
Approved for public release; distribution unlimited
PROJECT AIR FORCE
The research described in this report was sponsored by the United States
Air Force under Contract FA7014-06-C-0001. Further information may
be obtained from the Strategic Planning Division, Directorate of Plans,
Hq USAF.
Library of Congress Cataloging-in-Publication Data
Is Available
The RAND Corporation is a nonprofit research organization providing
objective analysis and effective solutions that address the challenges
facing the public and private sectors around the world. RAND’s
publications do not necessarily reflect the opinions of its research clients
and sponsors.
R
® is a registered trademark.
© Copyright 2008 RAND Corporation
All rights reserved. No part of this book may be reproduced in any
form by any electronic or mechanical means (including photocopying,
recording, or information storage and retrieval) without permission in
writing from RAND.
Published 2008 by the RAND Corporation
1776 Main Street, P.O. Box 2138, Santa Monica, CA 90407-2138
1200 South Hayes Street, Arlington, VA 22202-5050
4570 Fifth Avenue, Suite 600, Pittsburgh, PA 15213-2665
RAND URL: http://www.rand.org
To order RAND documents or to obtain additional information, contact
Distribution Services: Telephone: [phone];
Fax: [phone]; Email: [email protected]
Preface
This study, conducted in the Manpower, Personnel, and Training Pro-
gram of RAND Project AIR FORCE (PAF), is a follow-on to Air Force
Enlisted Force Management: System Interactions and Synchronization
Strategies (Schiefer et al., 2007). This monograph explores a potential
modification to the enlisted promotion system, one of the primary sys-
tems that affect the enlisted force.
Brig Gen Glenn Spears sponsored this work in fiscal year 2006
as the Director of Force Management Policy, Deputy Chief of Staff
for Personnel, Headquarters U.S. Air Force. The research was accom-
plished as part of a project entitled Enlisted Force Management. This
monograph should be of interest to those responsible for Air Force
enlisted testing and promotion policies, to those who develop enlisted
promotion tests, to strength managers, to the Chief’s Group at the Air
Force Personnel Center, and to enlisted career field managers.
We appreciate that the findings in this monograph will not gener-
ate change without the support of key leaders within the Air Force. The
document was prepared with that audience in mind.
RAND Project Air Force
RAND Project AIR FORCE, a division of the RAND Corporation, is
the U.S. Air Force’s federally funded research and development center
for studies and analyses. PAF provides the Air Force with independent
analyses of policy alternatives affecting the development, employment,
combat readiness, and support of current and future aerospace forces.
iii
iv The Weighted Airman Promotion System: Standardizing Test Scores
Research is conducted in four programs: Aerospace Force Develop-
ment; Manpower, Personnel, and Training; Resource Management;
and Strategy and Doctrine.
Additional information about PAF is available on our Web site:
http://www.rand.org/paf/
Contents
Preface ............................................................................. iii
Figures ............................................................................. ix
Tables .............................................................................xiii
Summary ..........................................................................xv
Acknowledgments .............................................................. xix
Abbreviations .................................................................... xxi
CHAPTER ONE
Introduction ....................................................................... 1
The Air Force Cannot Be Achieving Its Primary Enlisted Promotion
Objective ....................................................................... 2
The Motivation for Enlisted Promotion Equity ................................ 4
The Air Force Has Not Achieved All TOPCAP and Secondary
Promotion Objectives ......................................................... 7
Organization of the Monograph ................................................10
CHAPTER TWO
The Weighted Airman Promotion System: Motivation, Mechanics,
Reality, and Theory .........................................................13
A Fundamental Relationship ....................................................17
Reality ..............................................................................18
Theoretical Impacts of Differences in Variation .............................. 24
Measuring Variation ............................................................. 26
E5 WAPS Component Impacts .............................................. 26
E6 WAPS Component Impacts ...............................................31
E7 WAPS Component Impacts ...............................................33
v
vi The Weighted Airman Promotion System: Standardizing Test Scores
E8 WAPS Component Impacts .............................................. 36
E9 WAPS Component Impacts ...............................................37
Chapter Summary ................................................................39
CHAPTER THREE
Standardizing Test Scores ......................................................41
What Is Test Standardization? .................................................. 42
Why Standardize? ................................................................ 42
Approaches to Standardizing PFE/SKT Scores ............................... 43
Standardization Mechanics ..................................................... 44
An Alternative Approach to Standardization ................................. 46
Disclaimer ..........................................................................47
CHAPTER FOUR
Testing Impact and Selection Timing .......................................49
Selections to E2–E4 ..............................................................49
Selections to E5 ................................................................... 50
A Univariate Perspective of Selections to E5 .................................52
A Multivariate Perspective of Selections to E5 ............................. 54
Selections to E6 .................................................................. 56
A Univariate Perspective of Selections to E6 ................................57
A Multivariate Perspective of Selections to E6 ..............................58
Selections to E7 ................................................................... 60
A Univariate Perspective of Selections to E7 ................................61
A Multivariate Perspective of Selections to E7 ..............................63
Selections to E8 .................................................................. 64
E8 Selection Factors ............................................................65
Selections to E9 ....................................................................67
Chapter Summary ............................................................... 68
CHAPTER FIVE
Effects ..............................................................................71
Inconsistent and Random Selection Standards ................................71
Senior NCO (E7–E9) Manning ................................................72
Unequal Opportunities to Make E8 and E9 ...................................73
Contents vii
Disproportionate Selectivity for E9 Nominative and Commander-
Involvement Positions ........................................................74
Standardization Strategies ........................................................76
Transition Issues ...................................................................79
Standardization Costs ........................................................... 80
CHAPTER SIX
Conclusions and Recommendation ..........................................83
Conclusions ........................................................................83
Recommendation .................................................................83
APPENDIXES
A. The Impact of WAPS Factors by Grade and AFSC .................. 87
B. AFSC Titles ................................................................. 97
C. WAPS Changes over Time .............................................. 101
D. Periodic WAPS Revalidation ........................................... 105
E. Four Approaches to Measuring the Impacts of WAPS Factors ... 107
F. Multivariate Models to Predict Selection Rates to E5 ............. 113
G. Multivariate Models to Predict Selection Rates to E6 ............. 125
H. Multivariate Models to Predict Selection Rates to E7 ............. 131
I. ACT, SAT, and ASVAB Approaches to Standardization ........... 137
References ....................................................................... 143