Test Taking Behavior in Multiple Choice Exams

40 Pages·2015·0.83 MB·English

Hit or Miss? Test Taking Behavior in Multiple
Choice Exams
S¸. Pelin Akyol 1 James Key 2 Kala Krishna 3
1BilkentUniversity
2UniversityofWesternAustralia
3PennsylvaniaStateUniversity
October 12, 2015
Akyol,KeyandKrishna ÖSSExams
Motivation
Multiple choice tests are widely used
Universityentranceexams(Turkey,Greece,Japan,Korea,
China,...)
TheSATandGRE
Disadvantage: Random guessing is possible
Applypenaltyforincorrectanswerstopreventrandom
guessing
Decisiontoguess/notdependsonknowledgeandrisk
aversion.
Does the exam format grant certain groups an advantage?
Fair?
Akyol,KeyandKrishna ÖSSExams
Literature
Literature: Women skip more often
Reducedform: Ben-ShakharandSinai(1991)
Experimental: Baldiga(2013),EspinosaandGardeazabal
(2010)
Propergradingrules: Bernardo(1998),Burgos(2004),
EspinosaandGardeazabal(2005)
SemiStructural: Pekkarinen(2014)(Raschmodel),
Tannenbaum(2012)
Akyol,KeyandKrishna ÖSSExams
Education System
ÖSS Exam - held annually
Paperbasedmultiplechoiceexam
Mostimportantdeterminantofuniversityadmission weights
Four sections: math, science, social science and Turkish
45questionsineachpart
Expectation of 0 if guess randomly
5answers
+1pointforcorrect,-0.25forincorrect
Students can skip the question, giving 0 points
Attitudes to risk will impact outcomes
Akyol,KeyandKrishna ÖSSExams
The Data
Sample of students taking 2002 University Entrance Exam
Scoresineachsection
Backgroundinformation
Focus on social science track, 1st time takers (8917
students)
Two sections of interest: social science and Turkish
Akyol,KeyandKrishna ÖSSExams
Gender Gap
There is a gender gap Scores
Only 9% of these students gain university entrance
Males are over-represented in the top 9%
9.4%ofmalesareinthistopgroup
Compareto8.5%offemales
A model where students form beliefs regarding the chance
of success when answering a question
Akyol,KeyandKrishna ÖSSExams
The Model
Students generate beliefs regarding answers
The questions are attempted independently
For each answer n ∈ {1,...,5}, the student draws a signal
x
n
The correct answer draws from a Pareto distribution with
shape parameter α and scale parameter A
Incorrect answers draw from a Pareto distribution with
shape parameter β and scale parameter B
Know parameters, but not which distribution they are
drawing signal from
Based on signals, they form beliefs regarding which
answer is correct answer
Akyol,KeyandKrishna ÖSSExams
The Distributions
Assumption
The scale parameters of the distributions are equal: A = B > 0.
That is, the minimum signal with positive support is the same
for both the incorrect answers and the correct answer.
Student can never be absolutely certain of the answer
(either correct or incorrect)
Simplifies the state space of student types
Interpretation of the parameters more intuitive
Proposition
The outcome of the model is independent of the size of A
Proposition
The outcome of the model depends only on the ratio β/α
Akyol,KeyandKrishna ÖSSExams
Student Ability
Without loss of generality, A = 1, α = 1, so that β is ability.
Distributions of signals for a student with β = 3,
approximately median
Distributions of Signals
1
0.9
0.8
0.7
0.6
P 0.5 Correct
Incorrect
0.4
0.3
0.2
0.1
0
1 2 3 4 5 6 7
Signal
Akyol,KeyandKrishna ÖSSExams
To Answer or Not
Students draw signals, {x ,x ,x ,x ,x }, for each answer.
1 2 3 4 5
Form beliefs
Student knows which answer is most likely to be correct
and the probability
But should the student choose that answer? Or should
they skip it?
Risk preferences: cutoff c
Ifchanceofsuccessisgreaterthanc,attempt
Otherwise,skip
Akyol,KeyandKrishna ÖSSExams