AN EXPLORATORY STUDY OF HALO EFFECT IN RATING

212 Pages·13.995 MB·English

by  GRANT, DONALD LINDSAY

AN EXPLORATORY STUDY OP HALO EFFECT
IN RATING
DISSERTATION
Presented in Partial Fulfillment of the Requirements
for the Degree Doctor of Philosophy in the
Graduate School of The Ohio State
University
By
Y-jdiSAH
•.
Donald L. Grant, A«B., M.A.
The Ohio State University
1952
Approved by:
ACKNOWLEDGMENTS
For initially making the study possible, and for
continued assistance and encouragement:
Dr. Robert Selover, Director,
Dr. Reign Bittner, Assistant Director,
Personnel Research Division,
The Prudential Insurance Company.
For their enthusiastic participation in the pro­
ject to select men to fill vacancies in the Personnel
Division, and for their subsequent wholehearted co­
operation in furnishing additional information and
assistance, the following members of the staff of the
Personnel Division, The Prudential Insurance Company:
Mr. Herbert Abrams, Assistant Director
Mr. William Huelsenbeck, Assistant Director
Mr. Peter Belmont, Personnel Assistant
Mr. Wallace Dusenberry, Personnel Assistant
Mr. Kenneth Franet, Personnel Assistant
Mr. Robert Searle, Personnel Assistant
Mr. Bernard Spelman, Personnel Assistant
Mr. Samuel Simmons, Personnel Assistant
For quickly and accurately computing correlations
for the follow-up study of the project to select men to
fill vacancies in the Personnel Division, the following
members of the staff of the Personnel Research Division,
The Prudential Insurance Company:
(Mrs.) Kathryn Feeney
(Mrs.) Ann Groninger
(Mrs.) Barbara Pondish
(Mrs.) Mary Ann Rlpca
8 2 S 7 4 0
il
For reviewing the draft of the dissertation, and for
offering many helpful comments and suggestions:
Dr* Herbert A. Toops,
Department of Psychology,
The Ohio State University.
For invaluable guidance and assistance in initially
helping to structure the problem, in clarifying concepts,
and in suggesting methods of analysis. For reviewing re­
sults and pointing to ways for improving analyses. For
continually offering encouragement and support, and in
many other ways generously contributing his time, effort,
knowledge, and Ingenuity to the study:
Dr. Robert J. Wherry,
Department of Psychology,
The Ohio State University.
For his continued guidance, as adviser to the
author. For many helpful comments and suggestions, for
quickly and carefully reviewing reports of results, for
facilitating essential administrative matters, and for
very thoroughly reviewing the final version of the dis­
sertation. Finally, for doing everything possible to
assist the author, not only to complete the dissertation,
k
but throughout his years as a graduate student;
Dr. Harold E. Burtt,
Department of Psychology,
The Ohio State University.
ill
TABLE OP CONTENTS
Chapter Page
X Introduction. .................. 1
II Data Studied. ..................... 19
III Classification of Comments.............. 38
IV Factor Analysis of the Rating
Variables . . . . . . . . . . . . . . 46
V Development of a Measure of
Halo Effect........................... 77
VI Tests of Hypotheses Regarding
Relationships of Selected Variables
to Halo Effect. ........... .. . 90
VII Effect of Halo on Prediction........... 128
VIII Summary...................... 151
References............... 168
Appendix
I Evaluation Report.......... 175
II Evaluation Report - Examples. . . . . . 181
III Classification of Comments by Content . 187
1
AM EXPLORATORY STUDY OF HALO EFFECT IN RATING
Chapter I
Introduction
In 1907 Wells noted with regard to ratings of lit­
erary merit that (76, p. 21),
There Is a possibility of one rather dis­
turbing constant error in measures of this
nature, whose extent it Is never possible
to know accurately. There is noted in-
trospectively a tendency to grade for
general merit at the same time as for the
qualities, and to allow an individual's
general position to Influence his position
in the qualities. This would be the case
especially in the case of those qualities
that were ill-defined in the minds of the
subjects, and tended to be interpreted
rather in terms of general merit.... This
would make the correspondence of such
qualities appear closer than they were.
In 1915 Webb, in his classical rating study (73),
observed the same phenomenon described by Wells. A few
years later (1920) Thorndike (61) cited evidence from
ratings made In Industry, the armed services, and educa­
tional Institutions to the effect that correlations
between ratings of diverse traits are (61, p. 27), ”...
too high and too even.” He interpreted the empirical
findings as indicating (61, p. 25),
..•that those gfving the ratings were un­
° O
able to analyze out these different aspects
o
of the person*s nature and achievement and
rate each'in ©independence of the others.
Their rat^n^a^.ehe apparently affected by o
O ; c
a marlced'^'b'snfihncy °to think of the person In
o o
general a% rather.good or rather inferior
and to coU'6'r° thh qualities by this general
feeling.- ’ °'o °' °
© © • :
■
Thorndike furtl^r#trig's" (61, p. 28), • o o, . o o •.o'
°©©©
■ . ■ : ••v.v!' o Y« - ' . •
In the cause's ivSp'.f'ar »the correlations are a" (§>c
resultant■’ the real facts, (2) ttys 0 , 0
const an’t’:/e'iri;;^r^qf ° the * halo* , as we may . ° © o ©©
call i t ’the reverse error of / o
attenusfrlo^%^/'‘tb chance inaccuracies °in o ?
the ra^shatf*o ' ” ■ ■ o
v* o O
He concludes ( . 6 * ^ •• J2. 8)V■ >°>- .•• • ’ I Q .
° . V©'©©''
The®wrii|ervh4ds®;b'ecome convinced that even a Oo
very capable;, foreman*; employer, teacher, or © ^
department/.^ unable to treat an in-©
. :: ..°o
dlvldua^'s^a.ikC’£^POund #of separate qualities r @ -
and” td ‘^sig» a0magnitude . to each of these •;.
in indej«.n^e.n:ce ^f the others* The magni- o © ©
tude of ^h^jj;c9h^ant error of the. halo, as °°°o 0 ® .
we have0 ca:ii^d.:lt;, also seems surprisingly ;; °
large, lack objective criteria ° © o
by whichv'*1;pv^^ermflie its exact size. ° °
' •°°oV © • . ■ ■ o' » ’o ©
As a result;:.-q^'£fie'Thorndike article *halo* became0 o °&
• »“ ©• o o °o °
the term for the° ratirfg, phenomenon initially described
,©
by Wells* It was ^prmsilly incorporated in psychological
vocabulary by #ai?reht72)S. who expanded the term to
; o
'.'V.' .
*Halo Effect** and defined it as follows (72, p* 120), °
®
t o' * • 9
...a tendency, whdn one is estimating or
rating si person .with respect to a certain © 0 o
trait,, to be Influenced by an.estimate of
some other t^salt or by one's general ®
O
©
impression of the person.
3
The concept has remained unchanged since the
Thorndike article. It has received frequent mention
In the literature on rating since 1920. Halo effect
has been Investigated to some extent, though difficul­
ties in operationally defining the concept have re­
stricted the number of such studies.
Bingham (2) sought to modify the concept some­
what. He made a distinction between what he termed
* valid1 and invalid1 halo. He failed* however, to set
forth how the two kinds of halo can be operationally
distinguished.
Dudyeha (19) suggested renaming the phenomenon.
He proposed that the term perseveration error1 be sub­
stituted for ,halot. There is no evidence In the more
recent literature on rating that this suggested modi­
fication In terminology has met with any response.
Determination of Halo Effect in Rating
The existence of halo effect is generally recog­
nised from the presence of high positive intercorrela-
tlons between traits or items In a rating scale. Tiffin
(66) and Driver (18) have suggested that its presence can
also, be determined from item by Item analyses of in­
dividual ratings.
Attempts have been made to determine the magnitude
of halo effect for particular samples of ratings.
4
Symonds (57) used partial correlation methods to par­
tial halo effect out of trait ratings by two Independent
raters* The difference between the two sets of corre­
lations determined the magnitude of halo effect for each
trait in the rating scale.
Ohi (15) criticised Symond»a method and developed
a method for obtaining two sets of trait intercorrela-
tIons Involving the ratings of three Independent raters,
namely* (1) Intaroorrelatlons caused by halo effect, and
(S3) interoorre latl one with halo effect part tale d out.
The difference between the two sets of Intercorrelations
represented the magnitude of halo effect for the sample
studied* Ohl ooncluded (13, p. 242), "Raters probably
differ to some extent In making this error of judgment
and traits for estimation are subject to the halo effect
in different degrees.”
Stookford and Bissell (56, p. 109) report the use
of length of acquaintance between the raters and ratees
as a measure of 'halo* or * bias' in rating.
The measures of the magnitude of halo effect des­
cribed above have not appeared In other than the studies
cited. As far as this author can determine no measure
of the magnitude of halo for each rating made by each
rater in a particular situation has been reported.
1
5
Correlates of Halo Effect
Difficulties in measuring halo effect have
placed limitations on attempts to determine the corre­
lates of this rating error, Stockford and BIssel
(56) correlated several variables with measures of
reliability, lienency, and halo. The variables stu­
died were: (1) Intelligence of the rater, (2) in­
terests of the rater, (3) age of the rater, (4) educa­
tion of the rater, (5) supervisory experience of the
rater, (6) training of the rater In proper rating
procedures, and (7) size of the group rated.
Because the measure of halo effect employed, length
of acquaintance between the rater and the ratee,
appears to be a possible correlate of halo rather
than a measure of the phenomenon itself, the results
obtained are of questionable value from the stand­
point of gaining a better understanding of halo
effect In rating,
Denlency of rating has been investigated more
extensively than has halo effect, probably because
the measure of leniency, simply the total rating score,
Is readily obtainable. As has been pointed out (see -
above) Stockford and BIssell (56) related several vari­
ables to leniency In rating. The relationship of length
1
6
of acquaintance between the rater and the ratee to
leniency has been reported in several sources (7,
23, 56). The relationships of situational factors, the
conditions under which the ratings were obtained, to
leniency has also been investigated (56, 58).
The relationships of various variables to the re­
liability of rating have also been the subject of some
study (23, 56).
Possible correlates of halo effect have been
suggested by several Investigators. Hollingworth (32,
pp. 94-95), for example, points to the possible influ­
ence of irrelevant factors on human judgments (e.g.,
influence of the student’s personality upon the grading
'of his essays and papers by his teachers). Guilford
(29, p. 275) quotes Knight to the effect that, ”...ac­
quaintance increases the halo effect.” Burtt (7, p.
393) states,
Other factors besides long acquaintance
may produce bias. Many of us have in­
dividual prejudices against certain
types of physiognomy, voice, or race,
or against one sex.
It may be fairly stated that no correlates of halo
effect have been clearly established. The available re­
search suggests hypotheses only.
Control of Halo Effect
Many suggestions have been made with regard to