Predicting High-Stakes Tests of Math Achievement using a Group-Administered RTI Instrument

176 Pages·2017·2.47 MB·English

UUnniivveerrssiittyy ooff TTeennnneesssseeee,, KKnnooxxvviillllee
TTRRAACCEE:: TTeennnneesssseeee RReesseeaarrcchh aanndd CCrreeaattiivvee
EExxcchhaannggee
Doctoral Dissertations Graduate School
8-2014
PPrreeddiiccttiinngg HHiigghh--SSttaakkeess TTeessttss ooff MMaatthh AAcchhiieevveemmeenntt uussiinngg aa GGrroouupp--
AAddmmiinniisstteerreedd RRTTII IInnssttrruummeenntt:: VVaalliiddaattiinngg SSkkiillllss MMeeaassuurreedd bbyy tthhee
MMoonniittoorriinngg IInnssttrruuccttiioonnaall RReessppoonnssiivveenneessss:: MMaatthh
Jeremy Thomas Coles
University of Tennessee - Knoxville, [email protected]
Follow this and additional works at: https://trace.tennessee.edu/utk_graddiss
Part of the Applied Statistics Commons, Educational Assessment, Evaluation, and Research
Commons, Longitudinal Data Analysis and Time Series Commons, Quantitative Psychology Commons,
and the School Psychology Commons
RReeccoommmmeennddeedd CCiittaattiioonn
Coles, Jeremy Thomas, "Predicting High-Stakes Tests of Math Achievement using a Group-Administered
RTI Instrument: Validating Skills Measured by the Monitoring Instructional Responsiveness: Math. " PhD
diss., University of Tennessee, 2014.
https://trace.tennessee.edu/utk_graddiss/2811
This Dissertation is brought to you for free and open access by the Graduate School at TRACE: Tennessee
Research and Creative Exchange. It has been accepted for inclusion in Doctoral Dissertations by an authorized
administrator of TRACE: Tennessee Research and Creative Exchange. For more information, please contact
[email protected].
To the Graduate Council:
I am submitting herewith a dissertation written by Jeremy Thomas Coles entitled "Predicting
High-Stakes Tests of Math Achievement using a Group-Administered RTI Instrument: Validating
Skills Measured by the Monitoring Instructional Responsiveness: Math." I have examined the
final electronic copy of this dissertation for form and content and recommend that it be
accepted in partial fulfillment of the requirements for the degree of Doctor of Philosophy, with a
major in School Psychology.
R. Steve McCallum, Major Professor
We have read this dissertation and recommend its acceptance:
Sherry M. Bell, William L. Seaver, Jennifer A. Morrow, Brian E. Wilhoit
Accepted for the Council:
Carolyn R. Hodges
Vice Provost and Dean of the Graduate School
(Original signatures are on file with official student records.)
Predicting High-Stakes Tests of Math Achievement using a Group-Administered RTI Instrument:
Validating Skills Measured by the Monitoring Instructional Responsiveness: Math

A Dissertation Presented for the
Doctor of Philosophy
Degree
The University of Tennessee, Knoxville

Jeremy Thomas Coles

August 2014
i
Copyright 2014© by Jeremy Thomas Coles

All rights reserved.

ii
Dedication
To my fiancée, Mackenzie, I cannot express how important your love and support was
throughout graduate school; you gave me purpose and clarity when I needed it most. To Steve,
Susan, and Ashley, you are already family to me. To my parents, Brian and Valerie, you believed
in me before I believed in myself. To my best friend and little brother, Braden, now it’s your turn
to follow your dreams.

iii
Acknowledgements
This dissertation would not have been possible without the help of Dr. Steve McCallum.
His critical analysis and suggestions ensured that each idea was carefully evaluated and focused.
I would like to thank Dr. Sherry Bell. Her extensive knowledge of education and psychology
helped me bridge ideas across disciplines. I would like to thank Dr. Morrow and Dr. Seaver for
their knowledge and assistance in statistical methodology. I would like to thank Dr. Wilhoit for
his ability to put everything into perspective.
I would like to thank Dr. Michael Hopkins and Dr. Angela Hilton-Prillhart who, along
with Drs. McCallum and Bell, helped created the MIR probes. I would like to thank Dr. Scott
Graves, Dr. Christopher Skinner, and Dr. Jerry Morton. I would not be where I am today without
their guidance and support as I navigated graduate school and internship. Most importantly, their
reassurance provided clarity during the most uncertain and difficult times along this path.
Finally,I would like to thank the members of the research group who helped organize the probes
and enter data. Specifically, I would like to thank Elizabeth Hayes, Lakmal Walpitage, Kelly
Smyth, Brooke Browarnik, and Emily Taylor. I cannot express how grateful I am for the time
you spent assisting me with this project.

iv
Abstract

Three universal screeners and nine progress monitoring probes from the Monitoring
Instructional Responsiveness: Math (MIR:M), a silent, group-administered math assessment
designed for implementation with an RTI Model, were administered to 223 fifth-grade students.
The growth parameters of the overall MIR:M composite and two global composites (math
calculation and math reasoning) identified significant variation in student growth, within
significant linear and quadratic trajectories. However, there were significant differences in the
nature of the growth trajectories that have applied educational implications. In addition, growth
parameters across the three composites provided significant predictive potential when using the
Tennessee Comprehensive Assessment Program (TCAP) Achievement Test, a high-stakes, end
of the year assessment of academic achievement, as the criterion measures (p < .001).
Furthermore, these parameters were predictive at the classroom and student level. Differential
predictive potential of the parameters and the composites provide additional information about
the nature of the MIR:M data. Altogether, the growth modeling and the predictive modeling
provide evidence to support two practical uses of the MIR:M.

v
Table of Contents
CHAPTER I LITERATURE REVIEW .......................................................................................1
High Stakes Assessment ..................................................................................................2
Curriculum-Based Measurement ......................................................................................9
Student Growth with CBMs ........................................................................................... 23
Growth as a Predictor of High-Stakes Assessment ......................................................... 33
Growth and High-Stakes Predictive Potential of a Multidimensional M-CBM ............... 37
Statement of the Problem ............................................................................................... 39
Research Questions ....................................................................................................... 41
CHAPTER II METHOD ........................................................................................................... 42
Participants .................................................................................................................... 42
Instruments .................................................................................................................... 42
Procedures ..................................................................................................................... 48
Data Cleaning ................................................................................................................ 50
Data Analyses ................................................................................................................ 51
CHAPTER III RESULTS.......................................................................................................... 58
Descriptive Statistics for MIR:M Scores ........................................................................ 58
Modeling Parameters of MIR:M .................................................................................... 59
Predicting TCAP Scores ................................................................................................ 65
CHAPTER IV DISCUSSION ................................................................................................... 76
Modeling of Initial Skills and Growth ............................................................................ 77
Global Composite Differences in Growth ...................................................................... 81
Variables Impacting Common Applied Modeling Procedures ........................................ 87
Predictive Validity ......................................................................................................... 90
vi
Summary ..................................................................................................................... 103
Limitations .................................................................................................................. 107
Future Research ........................................................................................................... 110
REFERENCES ....................................................................................................................... 112
APPENDIX ............................................................................................................................ 136
MIR: M EXAMPLE PROBE ....................................................................................... 137
TCAP MATH PROCESSES ITEM SAMPLE ............................................................. 139
TCAP NUMBERS AND OPERATIONS ITEM SAMPLES ........................................ 140
TCAP ALGEBRA ITEM SAMPLES .......................................................................... 144
TCAP MEASUREMENT AND GEOMETRY ITEM SAMPLES ................................ 146
TCAP DATA ANALYSIS, STATISTICS AND PROBABILITY ITEM SAMPLES ... 148
TABLES ................................................................................................................................. 149
VITA ...................................................................................................................................... 166

vii
List of Tables
Table 1 Descriptive Statistics of MIR:M Total by Probe.......................................................... 149
Table 2 Descriptive Statistics of MIR:M Math Calculation by Probe ....................................... 150
Table 3 Descriptive Statistics of MIR:M Math Reasoning by Probe ........................................ 151
Table 4 Correlations of MIR:M Total by Probe ....................................................................... 152
Table 5 Correlations of MIR:M Math Calculation by Probe .................................................... 153
Table 6 Correlations of MIR:M Math Reasoning by Probe ...................................................... 154
Table 7 MIR:M Total Growth Models ..................................................................................... 155
Table 8 MIR:M Math Calculation Growth Models .................................................................. 156
Table 10 Descriptive Statistics of TCAP Scales ....................................................................... 158
Table 11 Correlations of TCAP Scales .................................................................................... 159
Table 12 Correlations of TCAP Scales by MIR:M Components .............................................. 160
Table 13 Correlations of TCAP Scales by MIR:M Predicted Values at TCAP Administration . 161
Table 14 Percentage of Variance by MIR:M Components, MIR:M Scale, and Hierarchical Levels
........................................................................................................................................ 162
Table 15 Best Predictive Model for Total Composite, MR Global, and MC Global Scores ...... 163
Table 16 Comparison of Common Modeling Procedures of Growth ........................................ 164
Figure 1 Mean Total Score by Probe Administration ............................................................... 165
viii