DTIC ADA495761: Metrics for Assessing Cognitive Skills in the Maneuver Captains Career Course

3.2 MB·English

by  Defense Technical Information Center

Research Product 2009-04

Metrics for Assessing Cognitive Skills in the
Maneuver Captains Career Course

Bruce C. Leibrecht
Northrop Grumman Corporation

Jennifer S. Tucker
U.S. Army Research Institute

R. Bruce Haverty, Paul N. Blankenbeckler
and D. Chris Green
Northrop Grumman Corporation

January 2009

ARI-Fort Benning Research Unit

United States Army Research Institute
for the Behavioral and Social Sciences

Approved for public release; distribution is unlimited.
U.S Army Research Institute

for the Behavioral and Social Sciences

A Directorate of the Department of the Army
Deputy Chief of Staff, G1

Authorized and approved for distribution:

BARBARA A. BLACK, Ph.D. MICHELLE SAMS, PhD.
Research Program Manager Director
Training and Leader Development
Division

Research accomplished under contract
for the Department of the Army

Northrop Grumman Corporation

Technical Review by

Scot t B. Shadrick, U.S. Army Research Institute
Kenneth L. Evans, U.S. Army Research Institute

NOTICES

DISTRIBUTION: Primary distribution of this Research Product has been made by ARI.
Please address correspondence concerning distribution of reports to: U.S. Army
Research Institute for the Behavioral and Social Sciences, Attn: DAPE-ARI-ZXM,
2511 Jefferson Davis Highway, Arlington, Virginia 22202-3926.

FINAL DISPOSITION: This Research Product may be destroyed when it is no longer
needed. Please do not return it to the U.S. Army Research Institute for the Behavioral
and Social Sciences.

NOTE: The findings in this Research Product are not to be construed as an official
Department of the Army position, unless so designated by other authorized documents.
REPORT DOCUMENTATION PAGE

1. REPORT DATE (dd-mm-yy) 2. REPORT TYPE 3. DATES COVERED (from. . . to)
March 2009 Final July 2007 - November 2008
4. TITLE AND SUBTITLE 5a. CONTRACT OR GRANT NUMBER
Metrics for Assessing Cognitive Skills in the Maneuver Captains W74V8H-04-D-0045 (DO 0021)
Career Course 5b. PROGRAM ELEMENT NUMBER
633007
6. AUTHOR(S) 5c. PROJECT NUMBER
Bruce C. Leibrecht (Northrop Grumman); Jennifer S. Tucker (U.S. A792
Army Research Institute); R. Bruce Haverty, Paul N. Blankenbeckler,
5d. TASK NUMBER
and D. Chris Green (Northrop Grumman)
359

5e. WORK UNIT NUMBER

7. PERFORMING ORGANIZATION NAME(S) AND ADDRESS(ES) 8. PERFORMING ORGANIZATION REPORT
NUMBER
Northrop Grumman U.S. Army Research Institute for the
Bldg. 1467 Behavioral and Social Sciences
Fort Knox, KY 40121 Fort Benning Research Unit
PO Box 52086
Fort Benning, GA 31995-2086
9. SPONSORING/MONITORING AGENCY NAME(S) AND ADDRESS(ES) 10. MONITOR ACRONYM
U. S. Army Research Institute for the Behavioral & Social Sciences ARI
ATTN: DAPE-ARI-IJ
11. MONITOR REPORT NUMBER
2511 Jefferson Davis Highway
Research Product 2009-04
Arlington, VA 22202-3926
12. DISTRIBUTION/AVAILABILITY STATEMENT

Approved for public release; distribution is unlimited.
13. SUPPLEMENTARY NOTES
Contracting Officer’s Representative: Jennifer S. Tucker Subject Matter Expert POC: Bruce Haverty
14. ABSTRACT (Maximum 200 words):
This report presents the products of research conducted to create student assessment metrics for the Maneuver
Captains Career Course–Reserve Component (MC3-RC). Two Student Assessment Packages (SAPs) were
developed, one for company command competencies and one for battalion staff competencies. This report
describes the contents, organization, and special features of both SAPs. It also discusses guidelines for using the
metrics to meet research, instructional, and programmatic goals and offers recommendations for disseminating
and leveraging the products. Both SAPs are included in their entirety as appendixes. They support MC3-RC
instructors in developing testing activities, tracking student progress, and providing grounded feedback to
augment the learning process. The SAPs are currently being used to provide feedback to course proponents at
Ft. Knox, KY, regarding the cognitive skill proficiency levels of the Captains in the courses.

15. SUBJECT TERMS
Army Training Officer Education System Student Assessment Knowledge Testing Cognitive Skills
Situational Judgment Test Test Methodology Captain Competencies
SECURITY CLASSIFICATION OF
19.LIMITATIONOF 20. NUMBER 21.RESPONSIBLEPERSON
ABSTRACT OF PAGES
16. REPORT 17. ABSTRACT 18. THIS PAGE Diane Hadjiosif, Technical
Unclassified Unclassified Unclassified Unlimited 266 Publication Specialist
703/602-8047
i
ii
Research Product 2009-04

Metrics for Assessing Cognitive Skills in the
Maneuver Captains Career Course

Bruce C. Leibrecht
Northrop Grumman Corporation

Jennifer S. Tucker
U.S. Army Research Institute

R. Bruce Haverty, Paul N. Blankenbeckler, and D. Chris Green
Northrop Grumman Corporation

ARI-Fort Benning Research Unit
Scott E. Graham, Chief

U.S. Army Research Institute for the Behavioral and Social Sciences
2511 Jefferson Davis Highway, Arlington, Virginia 22202-3926

January 2009

Army Project Number Personnel Performance
633007A792 and Training

Approved for public release; distribution is unlimited.
iii
ACKNOWLEDGMENT

We would like to acknowledge individuals for the initial metric development (this report)
and for continued support for an ongoing data collection effort using the metrics (forthcoming
report).

For the initial product development, the authors are indebted to the following individuals
who provided guidance and support to the research team and granted access to the Maneuver
Captains Career Course (MC3) classes at Fort Knox:
 Dr. Robert Bauer, Deputy Director for Training, Doctrine, and Combat Development,
U.S. Army Armor Center.
 MAJ Mark Bailey, Chief, Armor Professional Development Division, 3d Squadron,
16th Cavalry Regiment.
 LTC Scott King, Commander, 3d Squadron, 16th Cavalry Regiment.
 Mr. James Harrison, 3d Squadron, 16th Cavalry Regiment, MC3-RC Instructor.

Special thanks also go to the following individuals for reviewing the metrics:
 Observer/controllers and Leader Training Program coaches at the Joint Readiness
Training Center (JRTC), especially John Burch and Steve Seybert.
 Observer/controllers at the National Training Center (NTC), especially Rick Travis.
 MAJ Bobby Toon, Combined Arms and Tactics Directorate (CATD), Team Chief,
MC3, Ft. Benning, GA.
 CPT Matthew Giblin, CATD, MC3-RC Course Manager, Ft. Benning, GA.

For a follow-on project, examining the psychometric characteristics and utility of the
assessment metrics, which will be discussed in a forthcoming report after completion of the data
collection and analyses, we also are indebted to and acknowledge individuals at Fort Benning
who are granting access to classes and providing valuable support for the follow-on data
collection effort.
 LTC Lee Sikes, Chief of Tactics, CATD.
 MAJ Roy Bolar, CATD, Team Chief, MC3.
 MAJ Matthew Bunch, CATD, Small Group Instructor, S3, MC3.

At Fort Knox, we also acknowledge other individuals who are supporting the follow-on data
collection effort and granting access to classes.
 MAJ Joseph Albright, 3d Squadron, 16th Cavalry Regiment, Course Manager, MC3.
 MAJ Ethan Harding, 3d Squadron, 16th Cavalry Regiment, Team Chief, MC3.
 MAJ James Embry, 3d Squadron, 16th Cavalry Regiment, Team Chief, MC3.

We are grateful to all of the MC3 students who are currently participating in the follow-on data
collection effort. We appreciate their insights and contributions to the refinement of the metrics.

iv
Finally, we thank Rich Wampler and William Cooper of Northrop Grumman Corporation
for their expert input to the project. We also thank Randi Drake of Northrop Grumman for her
assistance in integrating the materials into user-friendly packages.
v
vi
METRICS FOR ASSESSING COGNITIVE SKILLS IN THE MANEUVER CAPTAINS
CAREER COURSE

CONTENTS

Page

INTRODUCTION...........................................................................................................................1
Background................................................................................................................................1
Metrics Design Parameters........................................................................................................3
Development Method.................................................................................................................5

COMPANY-LEVEL METRICS.....................................................................................................8
Overview....................................................................................................................................8
Family of Modules.....................................................................................................................8
Contents of the Modules..........................................................................................................10
Instructor’s Guide....................................................................................................................12
Special Features.......................................................................................................................12

BATTALION-LEVEL METRICS................................................................................................14
Overview..................................................................................................................................14
Family of Modules...................................................................................................................14
Contents of the Modules..........................................................................................................15
Instructor’s Guide....................................................................................................................16
Special Features.......................................................................................................................17

GUIDELINES FOR LEVERAGING THE COMPANY- AND BATTALION-LEVEL SAPs....18
Assessment Strategy................................................................................................................18
Creating Tailored Tests............................................................................................................20
Test Conditions........................................................................................................................21
Scoring.....................................................................................................................................22

CONCLUSIONS AND RECOMMENDATIONS........................................................................23
Conclusions..............................................................................................................................23
Recommendations....................................................................................................................23
Utilization and Dissemination of Products..............................................................................24

REFERENCES..............................................................................................................................25

APPENDIX A COMPANY-LEVEL STUDENT ASSESSMENT PACKAGE.......................A-i

APPENDIX B BATTALION-LEVEL STUDENT ASSESSMENT PACKAGE.....................B-i

APPENDIX C LIST OF DOCTRINAL SOURCES.................................................................C-1

vii
CONTENTS (continued)

Page

APPENDIX D JRTC DATA COLLECTION INSTRUMENTS.............................................D-1

APPENDIX E COMPANY-LEVEL DATA AND RESULTS.................................................E-1

APPENDIX F BATTALION-LEVEL DATA AND RESULTS...............................................F-1

APPENDIX G ACRONYMS AND ABBREVIATIONS........................................................G-1

LIST OF TABLES

TABLE 1. DESIGN GOALS FOR MC3-RC STUDENT METRICS..........................................4

TABLE 2. OBSERVER/CONTROLLER DEMOGRAPHICS.....................................................6

TABLE 3. DESCRIPTION OF THE TWENTY COMPANY-LEVEL MODULES....................9

TABLE 4. FIVE STANDARD ELEMENTS OF A METRICS MODULE................................10

TABLE 5. REPRESENTATIVE SCENARIO FROM A COMPANY-LEVEL METRICS
MODULE...................................................................................................................10

TABLE 6. THREE TYPES OF QUESTIONS USED IN COMPANY-LEVEL METRICS,
WITH EXAMPLE STEMS........................................................................................11

TABLE 7. DESCRIPTION OF THE FIFTEEN BATTALION-LEVEL MODULES................15

TABLE 8. REPRESENTATIVE SCENARIO FROM A BATTALION-LEVEL METRICS
MODULE...................................................................................................................16

TABLE 9. THREE TYPES OF QUESTIONS USED IN BATTALION-LEVEL METRICS,
WITH EXAMPLE STEMS........................................................................................17

viii