Development and evaluation of a computer adaptive test of personality: the basic traits inventory

242 Pages·2016·2.83 MB·English

by  Vorster, Paul

COPYRIGHT AND CITATION CONSIDERATIONS FOR THIS THESIS/ DISSERTATION
o Attribution — You must give appropriate credit, provide a link to the license, and indicate if
changes were made. You may do so in any reasonable manner, but not in any way that
suggests the licensor endorses you or your use.
o NonCommercial — You may not use the material for commercial purposes.
o ShareAlike — If you remix, transform, or build upon the material, you must distribute your
contributions under the same license as the original.
How to cite this thesis
Surname, Initial(s). (2012) Title of the thesis or dissertation. PhD. (Chemistry)/ M.Sc. (Physics)/
M.A. (Philosophy)/M.Com. (Finance) etc. [Unpublished]: University of Johannesburg. Retrieved
from: https://ujcontent.uj.ac.za/vital/access/manager/Index?site_name=Research%20Output (Accessed:
Date).
DEVELOPMENT AND EVALUATION OF A COMPUTER ADAPTIVE
TEST OF PERSONALITY: THE BASIC TRAITS INVENTORY

by

PAUL P. VORSTER
200603252

Thesis submitted in fulfilment
of the requirements for the degree

DOCTORATE IN PHILOSOPHY Industrial Psychology

in the
FACULTY OF MANAGEMENT
at the
Department of Industrial Psychology and People Management

Supervisor: Professor Gideon P. de Bruin
ABSTRACT
Background: Recent developments in technology have made the creation of computer adaptive
tests of personality a possibility. Despite the advances promised by computer adaptive testing,
personality testing has lagged behind the ability testing domain regarding computer adaptive
testing and adaptation. A principal reason why personality tests have not enjoyed computer
adaptive adaptation is because few working computer adaptive tests are available for study or
comparison to their original fixed form counterparts. In addition, personality tests tend to be
predominantly based on classical test theory, whereas item response theory is required for the
development of a computer adaptive test. Despite these impediments, numerous attitudinal
measures have been adapted to function as computer adaptive tests and have demonstrated good
psychometric properties and equivalence to their fixed form counterparts. As computer adaptive
testing holds numerous advantages both psychometrically and practically, the development of a
computer adaptive personality test may further advance psychometr ic testing of personality.
Research Purpose: This study aimed to address the lack of progress made in the field of computer
adaptive personality testing through the evaluation and simulated testing of a hierarchical
personality inventory, namely the Ba sic Traits Inventory (BTI), within a computer adaptive test
framework. The research aimed to demonstrate the process of computer adaptive test preparation
and evaluation (study 1 and study 2); as well as the simulation of the scales of the BTI as computer
adaptive tests (study 3). This was conducted to determine whether the BTI scales could be used as
computer adaptive tests, and to determine how the BTI computer adaptive scales compare to their
fixed form counterparts.
Research Design: A sample of 1962 South African adults completed the BTI for selection,
development and career counselling purposes. The instrument was investigated on a scale by scale
basis with specific emphasis placed on scale dimensionality (study 1) and scale fit to the one-
dimensional Rasch item response theory model (study 2). These factor analytic and item response

2
theory evaluations were necessary to determine the suitability of the BTI scales for computer
adaptive testing as well as prepare the BTI for computer adaptive test simulation. Poor performing
items were removed and a set of ‘core’ items selected for computer adaptive testing. Finally, the
efficiency, precision, and equivalence of the person parameters generated by the computer
adaptive core scales, as simulated in a computer adaptive framework, were compared to their non-
adaptive fixed form counterparts to determine their metric equivalence and functioning (study 3).
Main Findings:
Study 1: The initial evaluation of dimensionality of the BTI scales indicated that the orthogonal
bifactor model was the best fitting dimensional model for the BTI scales. The scales of the BTI
was therefore not strictly unidimensional, but rather was composed of a dominant general factor
with some group factors (the facets) accounting for unique variance beyond t he general factor.
Except for Extraversion, all other scales of the BTI evidenced general factor dominance, which
indicated that a total score could be interpreted for at least four of t he five BTI scales. This total
score interpretation at the scale level allows the BTI to be used computer adaptively on the scale
(general factor) level. Although Excitement Seeking accounted for unique variance beyond the
general factor, the facet was sti ll included when the scale was fit to the Rasch model.
Study 2: A total of 59 items were flagged for removal following fit to the one -dimensional Rasch
model. These items were flagged for removal because they did not fit the Rasch rating scale model
effectively or because the items demonstrated either uniform or non -uniform DIF by ethnicity
and/or gender. Item parameters were also generated for the shortened and optimised BTI scales
(core scales) for computer adaptive adaptation. In general, all the scales of the BTI fit the Rasch
model well after flagged items were removed which justified the inclusion of the Excitement
Seeking facet in the Extraversion scale for computer adaptive testing.
Study 3: The optimised computer adaptive ‘core’ BTI scales used on average 50 – 67% fewer
items when compared to their fixed form non -computer adaptive counterparts during computer

3
adaptive test simulation. Person parameter estimates were estimated at or below the standard error
criterion of .33 which indicate rigorous measurement precision. The BTI scales also demonstrated
strong correlations to their non-adaptive full form counterparts with correlations ranging between
.89 (Extraversion) to .94 (Neuroticism).
Summary and Implications: It is possible for a standard non-computer adaptive test of
personality to be converted into a computer adaptive test without compromising the psychometric
properties of the instrument. Study 1 and 2 were evaluative and helped to prepare the BTI scales
for computer adaptive test application. The final study indicated that good scale preparation results
in better equivalence between the computer adaptive and fixed form non-computer adaptive tests.
Additionally, a lower standard error of person parameter estimation as well as greater item
administration efficiency was attained by the ‘prepared’ item banks. Although future research
should take into consideration test-mode differences and content balancing of subscales, the
research demonstrates that a computer adaptive test of personality can be as precise, reliable, and
accurate, while being more efficient, than their fixed form non-computer adaptive counterparts.

4
TABLE OF CONTENTS

ABSTRACT ............................................................................................................................... 2
ACKNOWLEDGEMENTS ....................................................................................................... 9
LIST OF TABLES ................................................................................................................... 10
LIST OF FIGURES ................................................................................................................. 12
CHAPTER 1: INTRODUCTION AND ORIENTATION TO THE STUDY ......................... 14
1.1. Introduction ....................................................................................................................14
1.2. The progress made in the computer adaptive testing of personality ..............................15
1.3. Overview of the present study ........................................................................................18
CHAPTER 2: BACKGROUND OF COMPUTER ADAPTIVE TESTING .......................... 20
2.1. Key terms in computer adaptive testing .........................................................................20
2.2. Equivalence of computer adaptive tests to non-computer adaptive tests .......................23
2.3. Classical test theory and item response theory ...............................................................27
2.3.1. The problem of mean error estimation in classical test theory .......................................30
2.3.2. The impact of the number of test items administered ....................................................32
2.3.3. Local independence, person-free items estimation, and item-free person estimation ....33
2.3.4. Measurement invariance .................................................................................................38
2.4. The advantages of computerized adaptive testing ..........................................................39
2.4.1. Increased relevance for test-takers .................................................................................40
2.4.2. Reduction of testing time................................................................................................41
2.4.3. Reducing the burden of testing .......................................................................................42
2.4.4. Immediate feedback after testing....................................................................................42
2.4.5. Testing is not limited to one setting ...............................................................................43
2.4.6. Greater test security ........................................................................................................43
2.4.7. Invariant measurement ...................................................................................................44
2.4.8. Error estimates for each item ..........................................................................................44
2.4.9. Advancement of psychometric testing in general ..........................................................44
2.5. The requirements for the development of computer adaptive tests ................................45
2.6. Preview of the contents of the following chapters .........................................................46
CHAPTER 3: THE DIMENSIONALITY OF THE BTI SCALES ......................................... 48
3.1. Introduction ....................................................................................................................48

5
3.1.1. Testing the dimensionality of hierarchical personality scales ........................................49
3.1.2. Evaluation of the dimensionality of hierarchical personality scales ..............................52
3.1.3. The Basic Traits Inventory (BTI) ...................................................................................54
3.2. Method ............................................................................................................................55
3.2.1. Participants…………………………………………………………………………….55
3.2.2. Instrument……………………………………………………………………………...56
3.2.3. Data Analysis..................................................................................................................56
3.2.4. Ethical Considerations ....................................................................................................58
3.3. Results ............................................................................................................................58
3.3.1. Fit indices for the bifactor model (Model3) ...................................................................60
3.3.2. Reliability of the BTI scales ...........................................................................................61
3.3.3. Reliability of the BTI subscales .....................................................................................62
3.3.4. The bifactor pattern matrix .............................................................................................64
3.4. Discussion.......................................................................................................................66
3.4.1. The fit of the bifactor model ...........................................................................................66
3.4.2. The dimensionality of the BTI scales .............................................................................67
3.4.3. Implications for fit to one-dimensional item response theory models ...........................68
3.5. Overview of Chapter 3 and a preview of Chapter 4 ..................................................69
CHAPTER 4: FITTING THE BTI SCALES TO THE RASCH MODEL ............................ 70
4.1. Introduction ....................................................................................................................70
4.1.1. The use of the Rasch model for computer adaptive test development ...........................71
4.1.2. The application of Rasch diagnostic criteria for psychometric evaluation ....................73
4.2. Method ............................................................................................................................75
4.2.1. Participants …………………………………………………………………………….75
4.2.2. Instrument……………………………………………………………………………...75
4.2.3. Data Analysis..................................................................................................................76
4.2.4. Ethical Considerations ....................................................................................................78
4.3. Results ............................................................................................................................82
4.3.1. BTI scale infit and outfit statistics ..................................................................................82
4.3.2. Person separation and reliability indices ........................................................................86
4.3.3. Rating scale performance ...............................................................................................87
4.3.4. Differential item functioning ..........................................................................................92
4.3.5. Criteria for item exclusion from the core item bank ......................................................98

6
4.3.6. Functioning of the ‘core’ BTI scales ..............................................................................99
4.3.7. Cross-plotting person parameters of the full-test and the reduced test scales ..............113
4.4. Discussion.....................................................................................................................118
4.4.1. Rasch rating scale model fit .........................................................................................119
4.4.2. Item spread and reliability ............................................................................................119
4.4.3. Rating scale performance .............................................................................................120
4.4.4. DIF by ethnicity and gender .........................................................................................121
4.4.5. Conclusion…………………………………………………………………………….121
4.5. Overview of the current chapter and preview of the forthcoming chapter ...................123
CHAPTER 5: AN EVALUATION OF THE COMPUTER ADAPTIVE BTI ..................... 125
5.1. Introduction ..................................................................................................................125
5.1.1. Computer adaptive test simulation ...............................................................................126
5.1.2. Item banks used in computer adaptive testing ..............................................................128
5.1.3. Computer adaptive testing ............................................................................................129
5.2. Method ..........................................................................................................................142
5.2.1. Participants……………………………………………………………………………142
5.2.2. Instrument…………………………………………………………………………….142
5.2.3. Data Analysis................................................................................................................143
5.2.4. Ethical Considerations ..................................................................................................150
5.3. Results ..........................................................................................................................150
5.3.1. Comparing person parameter estimates of the different BTI scales .............................151
5.3.2. Computer adaptive core test performance indices........................................................167
5.4. Discussion.....................................................................................................................179
5.4.1. Correlations between person parameter estimates of the various adaptive and non-adaptive
test forms……………………………………………………………………………...180
5.4.2. Adaptive core and adaptive full performance indices ..................................................183
5.4.3. Item usage statistics ......................................................................................................185
5.4.4. Implications for computer adaptive testing of personality ...........................................185
5.4.5. Recommendations for future research ..........................................................................186
5.4.6. Conclusion and final comments ...................................................................................189
CHAPTER 6: DISCUSSION AND CONCLUSION ............................................................ 191
6.1. Introduction ..................................................................................................................191
6.1.1. Aims and objectives of the three studies ......................................................................191

7
6.1.2. Study 1 objectives: The dimensionality of the BTI scales ...........................................192
6.1.3. Study 2 objectives: Fitting the BTI scales to the Rasch model: Evaluation and selection of a
core item bank for computer adaptive testing ..............................................................193
6.1.4. Study 3 objectives: An evaluation of the simulated Basic Traits Inventory computer adaptive
test…………………………………………………………………………………….193
6.2. Discussion of Results for the Three Studies .................................................................194
6.2.1. Study 1 results: The dimensionality of the BTI scales .................................................194
6.2.2. Study 2 results: Fitting the BTI scales to the Rasch model ..........................................197
6.2.3. Study 3 results: An evaluation of the computer adaptive BTI .....................................198
6.3. Limitations and suggestions for future research ...........................................................199
6.4. Implications for practice ...............................................................................................201
6.5. Conclusion ....................................................................................................................202
REFERENCES ...................................................................................................................... 203
APPENDIX A: ITEM USAGE STATISTICS FOR THE ADAPTIVE FULL AND
ADAPTIVE CORE TEST VERSIONS ................................................................................. 227
APPENDIX B: MAXIMUM ATTAINABLE INFORMATION WITH SUCCESSIVE ITEM
ADMINISTRATION ............................................................................................................. 232
APPENDIX C: NUMBER OF ITEMS ADMINISTERED ACROSS THE TRAIT
CONTINUUM ....................................................................................................................... 237

8
ACKNOWLEDGEMENTS

I am deeply indebted to the following people and institutions:
Professor G. P. de Bruin, thank you Professor for all the kind, and hard, words and for
all the support you have given me. I am greatly indebted to you for making this possible. I have
learned so much from you and I hope that I can add to our field and make you proud.
I dedicate this doctoral thesis to two people who have walked the hard miles with me. Firstly,
to my mother Maxine Vorster, without your help, guidance, support and love I would never
have found the courage to complete this endeavour. Thank you for standing by me through
this adventure called ‘life’.
Secondly, but by no means second, thank you Marié Minnaar for standing by me and
giving up our quality time so that I could complete this work. You are truly the love of my life
and without that love I would have been lost. Thank you from the deepest part of my heart.
You are my touchstone.
I would also like to specially thank Professor Freddie Crous. Thank you Professor for
the words of encouragement and the hope you have instilled in me to be the best I can be.
Thank you for always being interested and encouraging. It has meant more to me than you will
ever know.
Finally, I would like to thank Dr. Nicola Taylor and Dr. Brandon Morgan for their
constant assistance and support. Without the two of you I would not have had the motivation
to embark and complete this endeavour. Thank you both for being not only fantastic colleagues,
but good friends.
I would also like to give a final thanks to the Centre for Work Performance and the
Department of Industrial Psychology and People Management. Thank you for your support,
both academically and financially.

9