DIFFERENTIAL ITEM FUNCTIONING AND ADVERSE IMPACT

148 Pages·2015·4.35 MB·English

DIFFERENTIAL ITEM FUNCTIONING AND ADVERSE IMPACT: A
COMPARISON OF MANTEL-HAENZSEL AND LOGISTIC REGRESSION
Heather JoAn Whiteman
B.A., University of California, Davis 2006
THESIS
Submitted in partial satisfaction of
the requirements for the degree of
MASTER OF ARTS
m
PSYCHOLOGY
(Industrial/Organizational Psychology)
at
CALIFORNIA STATE UNNERSITY, SACRAMENTO
SPRING
2011
DIFFERENTIAL ITEM FUNCTIONING AND ADVERSE IMPA CT: A
COMPARISON OF MANTEL-HAENZSEL AND LOGISTIC REGRESSION
A Thesis
by
Heather JoAn Whiteman
Approved by:
ittee
Chair
/ S. Meyers, P
,SecondR~
Phli
Luerrlian,
11
Student: Heather JoAn Whiteman
I certify that this student has met the requirements for format contained in the University
format manual, and that this thesis is suitable for shelving in the Library and credit is to
be awarded for the thesis.
(ij'r:>/~1.;
· I Date
Department of Psychology
iii
Abstract
of
DIFFERENTIAL ITEM FUNCTIONING AND ADVERSE IMPA CT: A
COMPARISON OF MANTEL-HAENZSEL AND LOGISTIC REGRESSION
by
Heather JoAn Whiteman
This study serves as a comparative analysis of two measures for detecting differential
item functioning (DIF) in item responses of 29,171 applicants on a 49 item selection test.
The methods compared in this study were two of the more commonly used DIF detection
· procedures in the testing arena: the Mantel-Haenszel chi-square and the logistic
regression procedure. The study focused on the overall effect each method had on
adverse impact when used for the removal of items from a test. The study found that the
presence of adverse impact findings were decreased by the removal of items displaying
DIF, and that the effect on adverse impact differed by method ofDIF detection. The
study does not however, provide e~ough evidence to support the use of one DIF detection
method over the -other in applied settings where considerations such as cost and test
reliability are of concern.
rittee Cbak
P .
Lawrence S. Meyers,
ACKNOWLEDGMENTS
I would first like to thank Biddle Consulting Group, Inc. the equal employment
opportunity, affirmative action, and employee selection firm in the western United States
who allowed me to use their data.
I would like to thank all of the professors in the Industrial/Organizational Psychology
program who have.influenced me. I would particularly like to thank Dr. Meyers, Dr. Kuthy,
and Dr. Berrigan who served on my committee and who assisted in guiding me through the
thesis process. I would like to especially appreciate and thank Dr. Meyers for the time he
took in giving me advice on the thesis and the enthusiasm and exceptional teaching he
offered in his courses. I attribute the bulk of my learning in Industrial/Organizational
Psychology to Dr. Meyers and the courses that he taught. I would like to thank Dr. Kuthy for
serving as a professional role model and instilling in me a knowledge and respect for the
applied Industrial/Organizational field. I would like to thank Dr. Berrigan for his dedication
to the students and his willingness to contribute to my thesis.
I would also like to thank my parents, Randy and Carrie, who have always been
supportive of my goals. I would also like to thank my friends and colleagues who have
provided support, input and camaraderie.
v
TABLE OF CONTENTS
Page
Acknowledgments ................................................................................... v
List of Tables ........................................................................................ ix
List of Figures .......................................... :. ........................................... xi
Chapter
1. INTRODUCTION ................................................................................ 1
Early History of Selection Testing ........................................................ 1
Setting Legal Precedent for Fairness in Selection Testing ............................ 4
Setting the Standards for Fairness in Selection Procedures ........................... 7
Adverse Impact .............................................................................. 9
Validity in Selection Procedures ......................................................... 13
Evidence Based on Test Content ............................................... 14
Evidence Based on Relation of a Test to Other Variables ................... 16
Evidence Based on Response Processes ....................................... 19
Evidence Based on Internal Structure of a Test ............................... 20
Evidence Based on Consequences of Testing ................................ 22
Validity and Reliability ................................................................... 22
Differential Item Functioning ... : ........................................................ 2.6
Differential Item Functioning and Item Bias ................................. 26
Measuring Differential Item Functioning .................................... 29
Ability Measures in DIF Analyses .............................................. 30
vi
Factors Affecting DIF Detection .............................................. 33
Uniform/N"on-Uniform DIF .................................................... 34
DIF Detection Methods .................................................................. 36
Mantel-Haenszel ................................................................. 36
Logistic Regression .............................................................. 40
DIF Detection Method Comparisons ........................................... 44
Purpose of the Study ........................................................................ 46
2. METHOD ......................................................................................... 48
Sample Description ........................................................................ 48
Instrument .................................................................................. 48
Procedure ................................................................................... 50
DIF Analysis for Item Removal ................................................ 50
Adverse Impact Analyses ....................................................... 54
3. RESULTS ........................................................................................ 56
DIF and Item Removal .................................................................... 56
Mantel-Haenszel Analyses ...................................................... 56
Logistic Regression Analyses .................................................. 61
Comparison of the MH and LR Methods for DIF Detection and
Item Removal ............................................. , ....................... 64
Adverse Impact Analyses ................................................................ 70
Original Test 80% Rule Adverse Impact Analyses ......................... 72
MH Test 80% Rule Adverse Impact Analyses .............................. 74
LR Test 80% Rule Adverse Impact Analyses ................................ 77
Vil
Comparison of 80% Rule Adverse Impact Analyses ........................ 78
4. DISCUSSION .................................................................................... 86
Findings & Conclusions .................................................................. 86
Limitations ................................................................................. 89
Implications for Future Studies ......................................................... 91
Appendices .............................................................................................. 93
Appendix A. Item Means and Standard Deviations ............................................ 94
Appendix B. MH DIF Values and Classification Level by Item ............................. 96
Appendix C. Nagelkerke R2 Values and DIF Classification Category by Item
and Comparison Group ................................................................................................ ·l 06
Appendix D. Number of Applicants Passing at Cut-off Score Level by Test
and Comparison Group ........................................................................... 120
Appendix E. Fisher's Exact Statistical Significance Results of Adverse Impact
by Test & Comparison Group .................................................................... 125
References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . 130
Vlll
LIST OF TABLES
Page
1. Table 1 Demographic Characteristics of Examinees ................................. 49
2. Table 2 Descriptive Statistics ofExaminee Test Scores ............................. 50
3. Table 3 MH Method DIF Classifications by Reference Group ..................... 57
4. Table 4 MH Method DIF Classifications by Item Number .......................... 59 ·
5. Table 5 Item Numbers Displaying Small or No DIF with the MH Method ...... 60
6. Table 6 Descriptive Statistics of the MH Test Scores ............................... 60
7. Table 7 LR Method DIF Classifications by Reference Group ...................... 62
8. Table 8 LR Method DIF Classifications by Item Number ........................... 62
9. Table 9 Item Numbers Displaying Small or No DIF with the LR Method ....... 63
10. Table 10 Descriptive Statistics of LR Test Scores .................................... 64
11. Table 11 MH & LR DIF Classifications by Item Number .......................... 69
12. Table 12 Descriptive Statistics of the Original, MH and LR Test Scores ......... 70
13. Table 13 Number of 80% Rule Violations and Practically Significant 80%
Rule Violations in the Original Test by Comparison Groups ....................... 73
14. Table 14 Number of80% Rule Violations and Practically Significant 80%
Rule Violations in the Original Test by Cut-off Score Levels ....................... 75
15. Table 15 Number of80% Rule Violations and Practically Significant 80%
Rule Violations in the MH Test by Comparison Groups ............................ 76
16. Table 16 Number of 80% Rule Violations and Practically Significant 80%
Rule Violations in the MH Test by Cut-off Score Levels ........................... 77
17. Table 17 Number of 80% Rule Violations and Practically Significant 80%
Rule Violations in the LR Test by Comparison Groups ............................. 79
ix
18. Table 18 Number of 80% Rule Violations and Practically Significant 80%
Rule Violations in the LR Test by Cut-off Score Levels ............................ 80
x