ERIC ED608843: An Application of a Random Mixture Nominal Item Response Model for Investigating Instruction Effects

2016·0.2 MB·English

by  ERIC

An Application of a Random Mixture Nominal
Item Response Model for Investigating
Instruction Effects
Hye-JeongChoi,AllanS.Cohen,andBrianA.Bottge
Abstract Thepurposeofthisstudywastoapplyarandomitemmixturenominal
itemresponsemodel(RIM-MixNRM)forinvestigatinginstructioneffects.Thehost
study design was a pre-test-and-post-test, school-based cluster randomized trial.
A RIM-MixNRMwasusedtoidentifystudents’errorpatternsinmathematicsatthe
pre-testandthepost-test.Instructioneffectswereinvestigatedintermsofstudents’
transitioninginerrorpatterns.Thatis,we comparedstudents’errorpatternsinthe
EnhancedAnchoredInstruction (EAI) condition with students’ error patterns in a
business-as-usual(BAU)instructionalconditionfollowingeachinstruction.Wealso
comparederrorpatternsofstudentswithmathdisabilitiesandstudentswithoutmath
disabilitiesfollowingthetwotypesofinstruction.
Keywords Randomitemmodel • MixtureIRTmodel • Nominalresponsesmodel
1 Introduction
Mixture item response theory (MixIRT) models have been used for modeling
populationheterogeneity(e.g.,Mislevy&Verhelst1990;Rost1990).Mostmixture
models consider only persons random but items fixed. Random item IRT models
(De Boeck 2008),however,consider bothitem and personparametersas random.
This is more appealing as (1) both items and personsare typically assumed to be
randomsamples fromsome populationand (2)treating bothitems and personsas
randompermitsinclusionofcovariatesonbothitemandpersonparameterstohelp
explaindifferencesinbothitemandexamineeparameters(VandenNoortgate,De
Boeck,&Meulders,2003;Wang2011).
Also, to date, most MixIRT models have primarily focused on dichotomously
orpolytomouslyscoreditems. MixIRTmodelscanbeusefullyappliedtonominal
H.-J.Choi((cid:2))•A.S.Cohen
UniversityofGeorgia,AderholdHall125,Athens,GA30601,USA
e-mail:[email protected]
B.A.Bottge
UniversityofKentucky,222TaylorEducationBuilding,Lexington,KY,USA
©SpringerInternationalPublishingSwitzerland2016 39
L.A.vanderArketal.(eds.),QuantitativePsychologyResearch,Springer
ProceedingsinMathematics&Statistics167,DOI10.1007/978-3-319-38759-8_4
40 H.-J.Choietal.
responses items. For example, MixIRT models can effectively model to capture
information about specific error patterns which individual distractors for multiple
choiceitemsmaycontain.
Thepurposeofthisstudywastoapplyarandomitemmixturenominalresponse
modeltoanempiricaldatasetforinvestigatinginstructionaleffects.Animportant
benefitofsuchamodelisthatitispossibletoexplicitlymodelrandomnessofitem
and ability parameters as well as specific aspects of students’ response patterns.
We providea brief descriptionof the randomitem mixture nominalmodel(RIM-
MixNRM)andasimulationstudytoevaluatethequalityoftheestimationmethod.
Then, we provide an empirical example in which a RIM-MixNRM is applied
to mathematic test data to investigate effects of an experimental instruction on
students’errorpatternsonfractionscomputation.
2 ARandomItem Mixture NominalResponseModel
The probability of selecting individual categories in an item with two or more
nominalcategoriescanbewrittenasalinearfunctionofitemcategoryandperson
parameters. Bock (1972), for instance, introduces a nominal model in which the
probability of selecting category k of item i, P .(cid:2)/, is defined as a multinomial
ik j
logisticfunction:
exp.(cid:9) (cid:2) C(cid:12) /
P .(cid:2)/D P ik j ik ; (1)
ik j K exp.(cid:9) (cid:2) C(cid:12) /
kD1 ik j ik
where
iD1;:::;nitems,
k D1;:::;K responsecategories,
jD1;:::;N examinees,
(cid:12) denotestheinterceptforcategorykofitemi,
ik
(cid:9) denotestheslopeforcategorykofitemi,and
ik
(cid:2) denotesthepersonparameterofpersonj.
j
Bolt, Cohen, and Wollack (2001) extended this model to a mixture nominal
IRT model as a way of detecting heterogeneityin a population.In doing so, Bolt
et al. (2001) included a class-specific category intercept parameter to specify the
propensityofselectingagivencategoryofitemiformembersoflatentclassg.The
class-specificprobabilityofaresponseisgivenby
exp.(cid:9) (cid:2) C(cid:12) /
P .(cid:2)/D P ik j gik ; (2)
gik j K exp.(cid:9) (cid:2) C(cid:12) /
kD1 ik j gik
withmarginalprobability
XG
P .(cid:2)/D (cid:11) P .(cid:2)/; (3)
ik j g gik j
g
AnApplicationofaRandomMixtureNominalItemResponseModel 41
where(cid:12) denotestheclass-specificcategoryintercept,g D 1:::;Glatentclasses,
gik P
and(cid:11) mixingproportion( G(cid:11) D1).Forresolvinganindeterminacyfortheitem
g g g P P
categoryparameters,constraintsof K(cid:9) D 0 and K(cid:12) D 0 were set for all
k ik k gik
itemsandallclasses.
Bolt et al. (2001) applied Markov chain Monte Carlo (MCMC) algorithm to
estimate the model in a general hierarchical framework and a fully Bayesian
approachasimplementedinthecomputersoftwarecomputersoftwareWinBUGS.
Theyusedfollowingconjugatepriors:
cj (cid:3)Multinomial.1j(cid:11)1;:::;(cid:11)G/
(cid:11) D.(cid:11)1;:::;(cid:11)G/(cid:3)Dirichlet.˛1;:::;˛G/
(cid:2)jjcj Dg(cid:3)N.(cid:10)(cid:2)g;(cid:5)(cid:2)2g/
(cid:9)ik (cid:3)N.(cid:10)(cid:9);(cid:5)(cid:9)2/
(cid:12)gik (cid:3)N.(cid:10)(cid:12)g;(cid:5)(cid:12)2g/:
In their model, however, item parameters were treated as fixed as in the
conventionalmixtureitemresponsemodels.Inthecurrentstudy,weextendedtheir
modeltoamodelwherebothitemandpersonparametersaretreatedasrandom.
3 SimulationStudy
Thesimulationstudydescribedbelowwasdesignedtoexaminethebehaviorofthe
RIM-MixNRMunderpracticaltestingconditions.
3.1 SimulationDesign
Hundred sets of 20 four-choice item responses were simulated from a standard
normaldistributions,N.0;1/. Six hundredexamineesfor three latent classes were
simulated and mixing proportionswere 0.33 and ability was generated as N.0;1/
foreachclass.ItemparameterestimatesadoptedfromBoltetal.(2001)wereused
to select item generating parameters. Generating values for model parameters are
giveninTable1.AscanbeseeninEq.(2),inthisparticularRIM-MixNRM,(cid:12) is
gik
theparametertodistinguishlatentclasses.
Theparametersforhyperpriorswereused:˛1 D:::D˛G D0:5;(cid:10)(cid:2)g (cid:3)N.0;1/;
1=(cid:5)(cid:2)2g (cid:3) Gamma.2;4/; (cid:10)(cid:9) (cid:3) N.0;1/; 1=(cid:5)(cid:9)2 (cid:3) Gamma.2;4/, (cid:10)(cid:12)g (cid:3) N.0;1/;
1=(cid:5)2 (cid:3)Gamma.2;4/.TheseparametersweresimilarwithonesusedbyBoltetal.
(cid:12)
g
(2001)andonlyprovidedminimuminformationforeachparameter.Inadditionto
42 H.-J.Choietal.
8 0 0 7 0 7 8 4 7 4 1 3 7 4 0 2 3 4 2 2
9 1 1 8 4 6 7 1 5 2 5 7 6 8 1 7 2 5 6 5
0. 0. 0. 0. 1. 0. 1. 1. 1. 0. 0. 0. 0. 0. 0. 0. 1. 0. 0. 0.
(cid:12)4 (cid:2) (cid:2) (cid:2) (cid:2) (cid:2) (cid:2) (cid:2) (cid:2) (cid:2) (cid:2) (cid:2) (cid:2) (cid:2) (cid:2) (cid:2) (cid:2)
4 4 3 1 3 1 9 1 4 5 2 2 8 2 9 2 9 2 8 3
0 8 3 3 9 5 1 9 8 1 9 7 3 2 6 1 0 4 4 2
0. 0. 0. 0. 0. 2. 5. 3. 3. 2. 1. 2. 2. 3. 2. 0. 1. 1. 0. 1.
(cid:12)3 (cid:2) (cid:2)
0 0 4 8 0 2 0 9 2 6 6 5 4 4 4 4 4 1 2 2
2 3 3 7 5 0 2 6 0 3 1 3 8 6 5 5 1 2 3 1
0. 0. 0. 0. 0. 0. 0. 0. 1. 1. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.
(cid:12)2 (cid:2) (cid:2) (cid:2) (cid:2) (cid:2) (cid:2) (cid:2)
3
4 4 7 2 7 6 1 2 5 7 5 6 3 0 5 6 0 7 8 3
Class(cid:12)1 0.7 0.4 (cid:2)0.5 (cid:2)0.2 (cid:2)0.9 (cid:2)1.8 (cid:2)1.1 0.4 1.2 (cid:2)0.7 1.2 0.1 1.6 (cid:2)2.2 0.4 0.0 0.0 (cid:2)0.6 (cid:2)0.7 (cid:2)0.8
8 0 0 7 0 7 8 4 7 4 1 3 7 4 0 2 3 4 2 2
9 1 1 8 4 6 7 1 5 2 5 7 6 8 1 7 2 5 6 5
0. 0. 0. 0. 1. 0. 1. 1. 1. 0. 0. 0. 0. 0. 0. 0. 1. 0. 0. 0.
(cid:12)4 (cid:2) (cid:2) (cid:2) (cid:2) (cid:2) (cid:2) (cid:2) (cid:2) (cid:2) (cid:2) (cid:2) (cid:2) (cid:2) (cid:2) (cid:2) (cid:2)
4 4 3 1 3 1 9 1 4 5 8 2 2 2 9 2 9 2 8 3
0 8 3 3 9 5 6 4 3 3 5 2 1 7 1 1 0 4 4 2
0. 0. 0. 0. 0. 2. 2. 1. 1. 0. 0. 0. 0. 0. 0. 0. 1. 1. 0. 1.
(cid:12)3 (cid:2) (cid:2) (cid:2) (cid:2) (cid:2)
0 0 4 8 0 2 0 9 2 6 6 5 4 4 4 4 4 1 2 2
7 8 8 2 0 5 2 6 0 3 1 3 8 6 5 5 1 2 3 1
2. 2. 2. 3. 3. 2. 0. 0. 1. 1. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.
(cid:12)2 (cid:2) (cid:2) (cid:2) (cid:2) (cid:2) (cid:2) (cid:2)
2
4 4 7 2 7 6 1 2 5 7 5 6 3 0 5 6 0 7 8 3
Class(cid:12)1 0.7 0.4 (cid:2)0.5 (cid:2)0.2 (cid:2)0.9 (cid:2)1.8 (cid:2)1.1 0.4 1.2 (cid:2)0.7 1.2 0.1 1.6 (cid:2)2.2 0.4 0.0 0.0 (cid:2)0.6 (cid:2)0.7 (cid:2)0.8
8 0 0 7 0 7 8 4 7 4 1 3 7 4 0 2 3 4 2 2
9 1 1 8 4 6 7 1 5 2 5 7 6 8 1 7 2 5 6 5
0. 0. 0. 0. 1. 0. 1. 1. 1. 0. 0. 0. 0. 0. 0. 0. 1. 0. 0. 0.
dy (cid:12)4 (cid:2) (cid:2) (cid:2) (cid:2) (cid:2) (cid:2) (cid:2) (cid:2) (cid:2) (cid:2) (cid:2) (cid:2) (cid:2) (cid:2) (cid:2) (cid:2)
u
st 4 4 3 1 3 1 9 1 4 5 8 2 2 2 9 2 9 2 8 3
on 0.0 0.8 0.3 0.3 0.9 2.5 2.6 1.4 1.3 0.3 0.5 0.2 0.1 0.7 0.1 0.1 1.0 1.4 0.4 1.2
ati (cid:12)3 (cid:2) (cid:2) (cid:2) (cid:2) (cid:2)
ul
m 0 0 4 8 0 2 0 9 2 6 6 5 4 4 4 4 4 1 2 2
datasetsforsi Threshold Class1(cid:12)(cid:12)12 0.740.2 0.440.3 (cid:2)0.570.3 (cid:2)0.220.7 (cid:2)0.970.5 (cid:2)1.860.0 (cid:2)1.110.2 (cid:2)0.420.6 (cid:2)1.251.0 (cid:2)0.771.3 (cid:2)1.250.1 0.160.3 (cid:2)1.630.8 (cid:2)2.200.6 (cid:2)0.450.5 0.060.5 0.000.1 (cid:2)(cid:2)0.670.2 (cid:2)(cid:2)0.780.3 (cid:2)0.830.1
g
neratin (cid:9)4 (cid:2)1.17 (cid:2)1.07 (cid:2)0.89 (cid:2)0.80 (cid:2)0.62 (cid:2)0.55 (cid:2)1.62 (cid:2)0.95 (cid:2)1.08 (cid:2)1.81 (cid:2)0.94 (cid:2)1.79 (cid:2)0.54 (cid:2)1.31 (cid:2)0.96 (cid:2)0.73 (cid:2)0.94 (cid:2)1.01 (cid:2)0.80 (cid:2)0.42
e
g
or 14 39 51 13 31 54 38 50 27 78 43 34 32 36 40 32 11 84 62 35
erf (cid:9)3 (cid:2)0. 0. (cid:2)0. 0. 0. (cid:2)0. 0. 0. 0. 0. (cid:2)0. 0. (cid:2)0. 0. (cid:2)0. 0. (cid:2)0. (cid:2)0. (cid:2)0. (cid:2)0.
et
m
mpara (cid:9)2 0.30 (cid:2)0.21 (cid:2)0.29 (cid:2)0.32 (cid:2)0.42 (cid:2)0.53 0.29 (cid:2)0.61 (cid:2)0.39 0.12 0.46 0.03 (cid:2)0.23 (cid:2)0.24 0.46 (cid:2)0.52 (cid:2)0.29 0.21 0.12 (cid:2)0.29
Ite pe 1 9 9 9 3 2 5 6 0 1 1 2 9 9 0 3 4 4 0 6
1 Slo (cid:9)1 1.0 0.8 1.6 0.9 0.7 1.6 0.9 1.0 1.2 0.9 0.9 1.4 1.0 1.1 0.9 0.9 1.3 1.6 1.3 1.0
e
Tabl 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20
AnApplicationofaRandomMixtureNominalItemResponseModel 43
P P P
Kk (cid:9)ik D 0, Kk (cid:12)gik D 0, and Gg (cid:11)g D 1, for identification, (cid:10)(cid:2) and (cid:5)(cid:2) set to
zeroandoneforthefirstclass.Thesepriors,hyperpriors,andconstraintswerealso
usedforanalyzingtheempiricaldatasetinthelatersection.Thecomputersoftware
computer software WinBUGS (Spiegelhalter, Thomas, Best, & Lunn, 2007) was
usedforbothsimulationandempiricalstudies.
Convergence of the MCMC algorithm was examined using the Geweke test
(1992)with a single chain as implementedin the computerprogramConvergence
Diagnosis and Output Analysis for MCMC (CODA: Plummer, Best, Cowles, &
Vines, 2006). Based on the Geweke test (1992) with a single chain and plots of
autocorrelations, kernel density estimates of the marginal posterior distributions,
andhistoryplotsofdrawsfromposteriors,aburn-inof3000iterationswassufficient
to achieve stationarity for all parameter estimates. Following this burn-in, an
additional5000iterationsweredrawntoobtainestimatesforeachoftheposterior
distributionsofmodelparameterestimates.
3.2 SimulationStudyResults
3.2.0.1 ModelSelection To investigatewhethermodelfit indicescouldidentify
thecorrectnumberoflatentclasses, one-tofour-classRIM-MixNRMswerefitto
thedatasets.TheBayesianinformationcriterion(BIC:Schwarz1978)andAkaike’s
informationcriterion(AIC:Akaike1974)werechosenasmodelfitindicesbecause
oftheirpopularityamongresearchers.Bothindiceswereabletoidentifythecorrect
model,thatis,bothindicatedthethree-classmodelwasthebestfitmodelfor93of
100replications.Fortheremainingreplications,BIC suggesteda two-classmodel
andAICafour-classmodel.
Label Switching Label switching is a well-known problem in finite mixture
modeling. Two types of label switching can occur with mixture modeling (Cho,
Cohen,&Kim,2013).ThefirstoccursoverasingleMCMCchain:thelabelsofthe
latentclassesswitchduringestimation.Thesecondtypeoflabelswitchingmaybe
observedwhenlabelsswitchbetweenmultipledatasetsormultipleanalysesinboth
Bayesianandmaximumlikelihoodestimation.Inthecontextofasimulationstudy,
oneneedstobeawareofthepossibilityoflabelswitching,aswhenlabelsswitchon
differentreplicatedatasets,thismaycauseconfusionwheninterpretingresults.In
thecurrentstudy,thepossibilityofoccurrenceoflabelswitchingwasinvestigated
byinspectingprofilesofitemestimatesacrosslatentclasses.Whenlabelswitching
was detected, latent classes were renamed by matching the profiles of parameter
estimates across replications before calculating bias, mean squared error (MSE),
andclassificationaccuracyrates.
Recovery of Parameters A recovery analysis was done to determine whether
the MCMC algorithm accurately recovered the model parameters of the
RIM-MixNRM. In addition to inspection of label switching, parameter estimates
hadbeenplacedonthemetricofthegeneratingparametersandthenbiasandMSEs
44 H.-J.Choietal.
ofparameterestimateswerecalculated.Correlationsbetweengenerationvaluesand
estimatesalsousedfortherecoveryanalysis.
ResultsshowedthatmostoftheparametersoftheRIM-MixNRMwererecovered
well: bias of item slopes, personability and mixing proportionwere aboutor less
than0.05;MSEwereaboutorlessthan0.16;andcorrelationswereaboutorhigher
than 0.93. Correlations between item threshold parameters and estimates, (cid:12), was
0.94,butbiasandMSEwere(cid:2)0:16and0.16,respectively,andappearedtodepart
slightlyfromgeneratedvaluesthanotherparameters.TheRIM-MixNRMcorrectly
classifiedexamineesintotheirtrue(i.e.,simulated)classes87.85%ofthetime.
4 Empirical Study:Instruction Effects on Students’
Fractions Computation
In thissection, we illustrate howa RIM-MixNRMcan be used to help investigate
effects of instruction on students’ learning process. In this example, students’
error patterns were examined on a test of fractions computation in a multi-year
clusterrandomizedinstructionalintervention.Themainpurposeofthestudywasto
investigateanexperimentalinstructionalconditioneffectsonstudents’errorpatterns
incomputingfractions.
4.1 Data Description
Study Design The host study designwas a school-basedcluster randomizedtrial.
Participantsincluded446middleschoolstudentsinGrades6–8in25generaleduca-
tionmathclassroomsin12middleschoolsinandaroundalargemetropolitanarea
intheSoutheast.Studentswererandomlyassignedtoanexperimentalinstructional
condition (N D 214) or to a business-as-usual (BAU) condition (N D 232).
Therewere123studentswithlearningdisabilitiesand323studentswithoutlearning
disabilitiesinthestudy.
The experimental condition implemented Enhanced Anchored Instruction
(EAI; Bottge, Ma, Toland, Gassaway, & Butler, 2012). EAI was designed for
use in helping to improvecomputationand problemsolving skills of adolescents,
includinglow-performingstudentswithlearningdisabilitiesbyincludingpractical,
hands-on applications to help students visualize the abstract concepts present in
the problem. Teachers ask probing questions and offer instructional guidance to
studentsastheyviewthevideoandhelpthemidentifyrelevantinformationtosolve
theproblem.Thiseliminatestheneedforreading,askillmanylow-achievingmath
studentsalsolack.
FractionsComputationTestAFractionsComputationTest(FCT)consistingof20
partial credit items (14-additionand 6-subtraction items) was designed by Bottge
AnApplicationofaRandomMixtureNominalItemResponseModel 45
et al. (2012)to measure students’ ability to add and subtract simple fractions and
mixed numbers with like and unlike denominators. The FCT was administered
for the pre-test and the post-test. Math education experts identified 11 types of
errors from students’ incorrect responses to these items. The most common were
Combining (C) and Selecting Denominator (SD). The remaining nine other types
of errors occurred less frequently and were combined into a single Other (O) for
thisstudy.Inthecurrentstudy,thefocuswasonthesethreetypesoferrorsasthey
reflect students’ misunderstandingsabout computingwith fractions as well as the
correctresponse(i.e.,No errors).CombiningandSelectingDenominatorerrorsare
describedbelow:
• Combining (C): Student combines numerators and combines denominators,
consistentlyapplyingthesameoperationtonumeratoranddenominator.
• SelectDenominator(SD):Studentselects oneofthe denominatorslisted in the
problemandmakesnoattempttomakeequivalentfraction.Denominatorgiven
intheanswermustbepresentintheproblem.
4.2 Model Estimation
To investigate EAI effects on students’ error patterns in fractions computation,
we applied RIM-MixNRMs to take into account randomness in students and
itemsparameters.Theinstructionalmethod(i.e.,EAIvsBAU)andstudents’math
learningdisabilitystatus(MD)wereincludedinthemodelascovariatestopredict
the latent class membership as this could reflect of EAI effects on students’ error
patterns.Thiswasdonebysubstituting(cid:11) inEq.(3)with(cid:11) asgivenby
g gjX
(cid:11) D Pexp.ˇg0Cˇ0gX/ (4)
gjX GgD1exp.ˇg0Cˇ0gX/
whereˇg0 denotesaninterceptforclassg,andˇg D .ˇg1;ˇg2;:::;ˇgp/isavector
of logistic regression coefficients of covariates in the model. For this study, those
covariatesweretheinstructionalmethodandstudents’mathlearningdisability.
Forthoseˇ0s,normaldistributionswithmeanofzeroandstandarddeviationof
10wereusedasconjugatepriorsandforrestparameters,thesamepriorswereused
asinthesimulationstudy.Foridentification,thefirstclasswasusedasareference
group.
The Geweke test, for convergence diagnosis, indicated that a burn-in of 4000
iterations was sufficient to achieve stationarity for all parameters. A subsequent
6000 iterations were used for estimating the model parameters. An exploratory
analysiswasappliedtodeterminethenumberoflatentclassesinthedata.Thatis,
RIM-MixNRMswithfromone-tofive-classwerefittothepre-andpost-testdata.
BasedonBICresults,afour-classandathree-classRIM-MixNRMsforpre-testand
post-testdata,respectivelywerechosenforthisstudy.
46 H.-J.Choietal.
Combining Error Select Denominator Error Other Error Mixed Error
0.81.0 1 4 0.81.0 1 2 4 0.81.0 3 4 0.81.0 1 3 4
Probability0.6 Probability0.6 Probability0.6 1 Probability0.6
0.4 0.4 0.4 0.4
3
0.2 0.2 3 0.2 0.2 2
0.0 2 0.0 0.0 2 0.0
−3 −2 −1 0 1 2 3 −3 −2 −1 0 1 2 3 −3 −2 −1 0 1 2 3 −3 −2 −1 0 1 2 3
Theta Theta Theta Theta
Combining Error Select Denominator Error Other Error
1.0 1 4 1.0 4 1.0 4
2 3
0.8 0.8 0.8 Error Type
Probability0.40.6 Probability0.40.6 Probability0.40.6 1234====CSONeoothlm eEecbrrtri onDrinegnominator
0.2 23 0.2 1 3 0.2 12
0.0 0.0 0.0
−3 −2 −1 0 1 2 3 −3 −2 −1 0 1 2 3 −3 −2 −1 0 1 2 3
Theta Theta Theta
Fig.1 Itemcategorycharacteristiccurvesforitem16showinglatentclassdifferencesinstudents’
errorpatternsonfractionscomputation
4.3 Results
4.3.1 CharacteristicsofLatentClasses
Itemcategorycharacteristiccurves(ICCC’s)forItem16areshowninFig.1foreach
latent class. These plots illustrate differences in types of errors made by students
of the individual latent classes. The plots in the upper panel are for the pre-test
and the plots in the lower panel are for the post-test. Students in all classes had
a greaterprobabilityof notmakinganyerrorsastheypossessedmoreability(i.e.,
Category4).However,thereweredistincterrorpatternswhichstudentsinmiddleor
lowerabilityinindividuallatentclassestendedtomake.Somestudentsinmiddleor
lowerabilitytendedtomistakenlycombineeachnumeratorandeachdenominator
(i.e., Category 1), some had a greater probability of making an error of selecting
denominator(i.e.,Category2),somehadagreaterprobabilityofmakingothererrors
(i.e.,Category3),andothershadagreaterprobabilityofmakingeithercombining
or selecting denominators (i.e., Category 1 or 2). Based on these patterns, each
classislabeledasCombining,SD,Other,orMixed showninFig.1.Thesedistinct
differencescanbeinterpretedasreflectingstudents’errorpatternontheFCT.
4.3.2 InstructionEffects
Table2presentsacross-tabulationofthefrequenciesofstudentsineachlatentclass
onthepre-andthepost-test.Thisshowsageneralpatternofstudents’transitioning
in class membership from the pre-test to the post-test. To investigate effects of
students’ learning disability status and instructional type on such transitioning,
those were included as covariates. On the pre-test, neither types of instruc-
AnApplicationofaRandomMixtureNominalItemResponseModel 47
Table2 Transitioning
Post-test
pre-testtopost-testlatent
Pre-test Combine SD Other Total
classes
Combine 57 27 82 166
SD 3 25 33 61
Other 12 1 96 109
Mixed 19 16 75 110
Total 91 69 286 446
tion nor students’ learning disability status did significantly impact on students
membership in latent classes except that students with learning disability had
significantlyloweroddsofbelongingtoOthererrorclassthanCombiningerrorclass
(i.e.,ˇ20 D(cid:2)1:11,0.33times).Aftertheintervention,however,EAIhadsignificant
impact on students’ error patterns on fraction computation. Students in EAI had
significantlyhigheroddsofbelongingtoOtherorSDerrorclassesthanCombining
errorclass(i.e.,ˇ22 D 1:06;2:89timesandˇ32 D 0:85,2.34times,respectively).
After the instruction, students might better understand about denominators and
could distinguish them from numeratorsbut still not fully understand the concept
ofcommondenominatorinfractions.
5 ConclusionandDiscussions
Itisnotuncommonthatresearchersorpractitionersdesignaninstrumenttorequire
nominalresponseswithaspecificpurposeineducationalandpsychologicalresearch
area.Forinstance,increatingmultiplechoiceitems,itemwriterstypicallyconstruct
distractorsin orderto representspecific errorsstudentsmightmake.NominalIRT
modelscanbeusedtoobtaininformationregardingtheseerrors.Further,amixture
nominal IRT model can be used to take into account population heterogeneity;
however, it does not consider randomness in items. In this study, we used a
RIM-MixNRM in which both items and person parameters were considered a
randomsamplefromapopulationandtakenintoaccounttheirrandomness.Results
froma simulationstudy suggestedthatthe modelparameterswere well recovered
and both AIC and BIC provided useful information for model selection. Results
fromthe middleschoolfractionscomputationdata revealedtherewere fourlatent
classesandthreelatentclassesonthepre-testandthepost-test,respectively,which
could reflect students’ error pattern on fractions computation. The results also
show that instructionaltype had an significant impact on transitioning these error
patterns subsequent to an instructional intervention. It is also possible to include
item covariates. Inclusion of a Q-matrix (e.g., Tatsuoka 1983), for instance, as a
covariate for individual categories of an item could be implemented to describe
componentsofknowledgerequiredforcorrectlyansweringagivenquestion.
48 H.-J.Choietal.
Acknowledgements Thedatausedinthearticlewerecollectedwiththefollowingsupport:the
U.S.DepartmentofEducation,InstituteofEducationSciences,PRNumberH324A090179.
References
Akaike, H. (1974). A new look at the statistical model identification. IEEE Transactions on
AutomaticControl,19,716–723.
Bock,R.D.(1972). Estimatingitemparameters andlatentabilitywhenresponses arescored in
twoormorenominalcategories.Psychometrika,37,29–51.
Bolt,D.M.,Cohen, A.S.,&Wollack,J.A.(2001). Amixturemodelformultiplechoicedata.
JournalofEducationalandBehavioralStatistics,26,381–409.
Bottge,B.,Ma,X.,Toland,M.,Gassaway,L.,&Butler,M.(2012).EffectsofEnhancedAnchored
Instructiononmiddleschoolstudentswithdisabilitiesinmath,DepartmentofEarlyChildhood,
SpecialEducation,andRehabilitationCounseling,UniversityofKentucky,Lexington,KY.
Cho,S.-J.,Cohen,A.S.,&Kim,S.-H.(2013).MarkovChainMonteCarloestimationofamixture
itemresponsetheorymodel.JournalofStatisticalComputationandSimulation,83,278–306.
DeBoeck,P.(2008).RandomitemIRTmodels.Psychometrika,73,533–559.
Geweke,J.(1992).Evaluatingtheaccuracyofsampling-basedapproachestocalculatingposterior
moments. InJ.M.Bernardo, J. O.Berger, A.P.Dawid,&A.F.M.Smith(Eds.), Bayesian
Statistics4:ProceedingsoftheFourthValenciaInternationalMeeting(pp.169–194).Oxford:
OxfordUniversityPress.
Mislevy,R.J.,&Verhelst,N.(1990). Modelingitemresponses whendifferent subjects employ
differentsolutionstrategies.Psychometrika,55,195–215.
Plummer,M.,Best,N.,Cowles,K.,&Vines,K.(2006).CODA:Convergencediagnosisandoutput
analysisforMCMC.RNews,6(1),7–11.
Rost,J.(1990).Raschmodelsinlatentclasses:Anintegrationoftwoapproachestoitemanalysis.
AppliedPsychologicalMeasurement,14,271–282.
Schwarz,G.(1978).Estimatingthedimensionofamodel.TheAnnalsofStatistics,6,461–464.
Spiegelhalter,D.,Thomas,A.,Best,N.,&Lunn,D.(2007).WinBUGS,1.4[Computerprogram].
Tatsuoka,K.K.(1983).Rulespace:Anapproachfordealingwithmisconceptionsbasedonitem
responsetheory.JournalofEducationalMeasurement,20(4),345–354.
Van den Noortgate, W., De Boeck, P., & Meulders, M. (2003). Cross-classification multilevel
logistic models in psychometrics. Journal of Educational and Behavioral Statistics, 28(4),
369–386.
Wang, A. (2011). A mixture cross-classification IRT model for test speededness. Unpublished
doctoraldissertation,UniversityofGeorgia.