The development of a performance rating scale for the evaluation of shipboard performance of enlisted naval personnel

159 Pages·04.981 MB·English

by  Mackie, Robert Ramsay

THE DEVELOPMENT OF A PERFORMANCE RATING SCALE
FOR THE EVALUATION OF SHIPBOARD PERFORMANCE
OF ENLISTED NAVAL PERSONNEL
A Dissertation
Presented to
the Faculty of the Department of Psychology
The University of Southern California
In Partial Fulfillment
of the Requirements for the Degree
Doctor of Philosophy
by
Robert Ramsay Mackie
August 1950
UMI Number: DP30406
All rights reserved
INFORMATION TO ALL USERS
The quality of this reproduction is dependent upon the quality of the copy submitted.
In the unlikely event that the author did not send a complete manuscript
and there are missing pages, these will be noted. Also, if material had to be removed,
a note will indicate the deletion.
Oissertatior Publ sh*ng
UMI DP30406
Published by ProQuest LLC (2014). Copyright in the Dissertation held by the Author.
Microform Edition © ProQuest LLC.
All rights reserved. This work is protected against
unauthorized copying under Title 17, United States Code
ProQuest LLC.
789 East Eisenhower Parkway
P.O. Box 1346
Ann Arbor, Ml 48106- 1346
Ps
pu. o. /'/' / o'- F
This dissertation, written by
...............E.Qt>.e.r.t.„Raxns.ay..jyia-C-ki.e...................
under the guidance of h.ls... Faculty Committee
on Studies, and approved by all its members, has
been presented to and accepted by the Council
on Graduate Study and Research, in partial fu l­
fillm ent of requirements for the degree of
DO CTO R OF P H ILO S O P H Y
Committee on Studies
Chawman
ACKNOWLEDGMENT
This study was made possible through a grant from
the Office of Naval Research. It was a part of a
more general project directed by Dr. Clark L. Wil­
son, and designed to develop shipboard performance
criteria. The writer is indebted to Dr. Wilson for
guidance throughout the study.
TABLE OF CONTENTS
CHAPTER PAGE
I. THE PROBLEM AND DEFINITION OF T E R M S .......... 1
Introduction . . . . ........................ 1
The problem................................ 2
Statement of the problem ................. 2
Importance of the study ........ 3
Definition of terms use d.......... 5
Pay grade or r a t e ....................... 5
Reliability .............................. 5
Objectivity .............................. 5
STEN-score ................................ 6
Organization of remainder of the disserta­
tion ...................................... 6
II. REVIEW OF THE LITERATURE..................... 8
Graphic devices ......................... 10
Ranking and comparison devices .......... 13
Check-list devices ....................... 18
The critical-incident technique ........ 19
Forced-choice technique ................. 21
Comparisons of techniques ............... 23
III. DESCRIPTION OF THE RATING SCALE AND THE NATURE
OF THE SAMPLES.............................. 31
The format................................ 31
Selection of the t r a i t s ................. 31
iii
CHAPTER PAGE
Wording of the traits .......... 33
The samples.............................. 34
The raters .............................. 34
Instructions to the raters............... 35
Scoring of the s c a l e s ................... 36
IV. MEANS AND DISPERSIONS OF RATINGS............. 37
V. INTERCORRELATIONS OF TRAITS AND RESULTS OF THE
FACTOR ANALYSES.............. 4l
Intercorrelation of traits, Procedure I . 42
Intercorrelation of traits, Procedure II
(variance within pay grade) .......... 48
Intercorrelation of traits, Procedure III
(pay grade partialled out)............. 54
Results of the factor analyses .......... 60
Identification of factors ............... 69
VI. AGREEMENT OF THE RATERS . . '................. 77
VII. THE RELIABILITY OF THE RATING SCALE AND SOME
INDICATIONS OF VALIDITY................... 83
Reliability' of the scale ........ 83
Relationship to other measures of perform­
ance ....................................... 84
VIII. SUMMARY AND CONCLUSIONS..................... 88
S ummary.................................... 88
Conclusions................................ 95
iv
CHAPTER PAGE
SELECTED BIBLIOGRAPHY................................ 100
APPENDIX A. SAMPLE OF RATING SCALES RF 101 AND
RF 1 0 5 ......................................... 105
APPENDIX B. SAMPLE PROCEDURE FOR REDUCING THE
EFFECTS OF PAY GRADE VARIANCE............... 145
LIST OF TABLES
TABLE PAGE
I. Mean and Middle 80 Per Cent Range of Rat­
ings, Mean and Middle 80 Per Cent Range
of Standard Deviations for Each Trait . . . 38
II. Intercorrelations of Rating Scale Traits,
Ability Check List, Navy G.C.T. and Bio­
graphical Information Including Pay Grade
(First Sample) . . ..................... 43
III. Intercorrelations of Rating Scale Traits,
Navy G.C.T. and Biographical Information
Including Pay Grade (Second Sample) . . . . 45
IV. Intercorrelations of Rating Scale Traits,
Ability Check List, Navy G.C.T. and Bio­
graphical Information Based on Variance
Within Pay Grade (First Sample) ............ 49
V. Intercorrelations of Rating Scale Traits,
Navy G.C.T. and Biographical Information
-Based on Variance Within Pay Grade (Second
Sample).................................... 51
VI. Intercorrelations of Rating Scale Traits,
Navy G.C.T. and Biographical Information,
Pay Grade Partialled Out Statistically
(First Sample) ........................... 55
vi
TABLE PAGE
VII. Intercorrelations of Rating Scale Traits,
Navy G.C.T. and Biographical Information,
Pay Grade Partialled out Statistically
(Second Sample) ........................... 57
VIII. Factor Loadings After Rotation and Communali-
ties Before and After Rotation (First
Sample, N = 187> Pay Grade Acting as a
Variable).................................. 63
IX. Factor Loadings After Rotation and Com-
munalities Before and After Rotation (First
Sample, N = l87> Based on Variance Within
Pay Grade).................................. 64
X. Factor Loadings After Rotation and Communali-
ties Before and After Rotation (First
Sample, N = l87> Pay Grade Partialled Out). 65
XI. Factor Loadings After Rotation and Communali-
ties Before and After Rotation (Second
Sample, N = 286, Pay Grade Acting as a
Variable).................................. 66
XII. Factor Loadings After Rotation and Communali-
ties Before and After Rotation (Second
Sample, N = 286, Based on Variance Within
Pay Grade).................................. 67
Vll
TABLE PAGE
XIII. Factor Loadings After Rotation and Communali-
ties Before and After Rotation (Second
Sample* N = 286, Pay Grade Partialled Out). 68
XIV. Inter-Rater Agreement on the Various Rating
Scale Traits and on Average Score, for
Both Samples, with Pay Grade in as a Vari­
able and with Pay Grade Held Constant . . . 80
XV. Relationship Between Performance Check List
(Form RF 107) Total Scores and Scores on
Individual Traits of the Rating Scale
(Form RF 101), General Classification
Test Scores, and Selected Biographical
Information, with the Influence of Pay
Grade Held Constant....................... 86