ERIC ED602023: College Entrance Exams: How Does Test Preparation Affect Retest Scores? Research Report 2019-2

2019·1 MB·English

by  ERIC

College Entrance Exams:
How Does Test Preparation
Affect Retest Scores?
Raeal Moore, PhD, Edgar Sanchez, PhD, and Sweet San Pedro, PhD
Research Report
2019 - 2
ABOUT THE AUTHORS SUMMARY
Raeal Moore, PhD A quasi-experimental method and test-retest design were used to
explore the impact of participating in test preparation, whether the
Raeal Moore is a senior research scientist
impact of test preparation depends on the first ACT® score, and the
specializing in survey methodological research
impact of specific test preparation activities. Test preparation
and research on education best practices in
improved students’ retest scores, and this effect did not differ
P-12 schools.
depending on students’ first ACT score. Among specific test prep
activities, only the number of hours using a private tutor resulted in
Edgar Sanchez, PhD
increased score gains above the overall effect of test prep.
Students who reported feeling inadequately prepared for the
Edgar Sanchez, a senior research scientist in
second test had ACT Composite scores that were lower than those
the Validity and Efficacy Research department
students who felt adequately prepared.
at ACT, works on predictive modeling of student
educational outcomes. He is currently focusing
on the efficacy of test preparation programs.
SO WHAT?
Sweet San Pedro, PhD
Performance on high-stakes, standardized college entrance exams
Maria Ofelia “Sweet” San Pedro is a research
(e.g., SAT, ACT) provides important information that is used by
scientist in the Learning, Assessment and
postsecondary institutions in their admission process. Higher exam
Navigation Experiences department at ACT
scores increase the chances of a student getting admitted in more
specializing in learning science and learning
selective schools and eligibility for merit-based scholarships. This
analytics. She researches K-12 learning
said, students and their families are spending time and resources to
experiences, assessment and instruction within
prepare for these assessments. It is important to provide research-
digital learning environments.
based evidence as to which types of test preparation activities work.
NOW WHAT?
Given the financial burden test preparation can place on families, it is
imperative to understand what kind of impact is reasonable to expect.
We would like to continue this work by investigating what types of
activities students engage in, and among those activities, which are
most efficacious.
® 2019 by ACT, Inc. All rights reserved | R1743 Rev 1
Table of Contents

INTRODUCTION ................................................................................................................................ 1
METHOD .......................................................................................................................................... 3
DATA ............................................................................................................................................... 3
MEASURES ...................................................................................................................................... 4
DATA ANALYSIS ............................................................................................................................... 5
RESULTS ......................................................................................................................................... 7
PROPENSITY SCORE MATCHING .................................................................................................... 7
DESCRIPTIVE STATISTICS ............................................................................................................... 9
DISCUSSION .................................................................................................................................. 14
College Entrance Exams: How Does Test
Preparation Affect Retest Scores?
Raeal Moore, PhD, Edgar Sanchez, PhD, and Sweet San Pedro, PhD
Introduction
Performance on high-stakes, standardized college standardized college entrance exams. A study by
entrance exams (e.g., SAT, ACT®) provides Briggs (2001) showed test preparation solutions
important information that is used by such as commercial preparation classes and
postsecondary institutions in their admission tutoring had very small effects on ACT subject test
process (Clinedinst, 2014). Higher exam scores scores, in which score increases did not exceed a
increase the chances of a student getting full score point. The National Association for
admitted in more selective schools and eligibility College Admissions Counseling (NACAC)
for merit-based scholarships (Carnevale & Rose, reviewed efficacy studies on test preparation and
2003; Doyle, 2006). Given this context, test also found minimal effects on test scores—an
preparation programs have emerged to address average gain of approximately 30 points on the
students’ desire to improve their test scores SAT (Briggs, 2009). Meta-analyses on test
whether by “improving the skills measured by the preparation for the SAT or similar achievement
test or by improving the skills for taking the test, tests show a modest test score gain of 0.25
or both” (Messick, 1982). Test developers and standard deviations (Bangert-Drowns, Kulik, &
other service providers have developed a number Kulik, 1983). Assuming similar gains for the ACT
of test preparation solutions, including test, this equates to a gain of about 1.3 to 1.7
workbooks, practice tests, coaching, class points for the subject tests and the Composite
curricula, as well as in-person or online tutoring score (Montgomery & Lily, 2012; Powers, 1993).
packages. These solutions can be characterized
by instruction aimed at developing skills and Irrespective of engaging in test preparation
abilities, practice and exposure to practice activities, research has found that students who
problems that resemble the actual text, as well as retest tend to increase their score (Appelrouth,
the awareness and practice of test-taking Zabrucky, & Moore, 2015). For example, a
strategies. Additionally, test preparation activities meta-analysis found increases in test scores of
vary by duration and intensity. approximately 0.25 standard deviations for
students who retook a cognitive ability test when
Despite the growth of the test preparation industry assessed between the first and second
(Barnes Reports, 2017), the existing research on administration (Hausknecht et al., 2007). For the
the efficacy of test preparation has shown mixed ACT test, retesting has shown a 1 to 2 point
results depending on the type of test preparation scale score increase, on average (Andrews &
activity, population tested, or study Ziomek, 1998). The exact gain associated with
design/methodology (Lane, Raymond, & retesting is dependent upon a number of factors,
Haladyna, 2015). On average, research shows including the number of times a student has
test preparation has small to moderate positive tested, in what grade a student takes the ACT
effects on actual test scores, with average test test, and how much time has elapsed between
score gains falling within the margin of error for test administrations (ACT, 2017).
ACT Research & Policy | ACT Research Report | R1743 2
Aside from efficacy studies, research on test tests. They found students’ perceptions of negative
preparation has also looked into practice pressure about testing negatively influenced their
opportunities and motivational factors that may test performance, while students’ perceptions of
result in test score gains. Appelrouth, Zabrucky, their ability in math and reading were positively
and Moore (2015) used regression analyses to associated with their performance in math and
examine the relationship between SAT score gains reading assessments. Hong, Sas, and Sas (2006)
and several factors of test preparation sessions. looked into awareness of students' own level of
Their findings showed that time spent on individual preparedness and whether there were differences
tutoring was positively associated with an increase between high- and low-achieving high school
in the SAT Total score, with each additional hour students for an algebra test. Their findings show
spent on individual tutoring increasing the final SAT that high-achieving students were more aware of
score by 2.34 points.1 In addition, the number of their test preparedness in cognitive areas
months students use test preparation was also (competence, understanding of the material, and
positively associated with SAT score gains: study behavior). Peng, Hong, and Mason (2014)
Students who start test preparation before June of modeled the relationships among motivational
their junior year would stand to gain 4.3 points for variables in test-taking and test preparation
every proceeding month. As such, a student who strategies of 10th graders and found students with
would start preparing for the SAT in February of high self-efficacy applied more effort in test-taking
their junior year would score 34.4 points higher and performed better than students with low self-
than a student who would start preparing in efficacy. Findings in these studies are consistent
September of their senior year (Appelrouth, et al., with self-efficacy having a positive influence on
2015). student achievement (Bandura, 1993).
For the ACT test, research also shows that test Although numerous and extensive, existing studies
preparation is only associated with modest gains in that have looked at effects of test preparation on
ACT test scores, with higher increases related to SAT or ACT test performance have some
length of preparation. Students who took and limitations. According to the previously mentioned
prepared for a second ACT test gained an average NACAC report (Briggs, 2009), more than 30
of 1.4 points on their second ACT Composite score studies have been conducted since 1953
(Schiel & Valiga, 2014a). Students who prepared evaluating the effect of coaching on the SAT. Only
for the second test for over 20 hours attained two such studies had examined the effect of
Composite scores 0.7 point higher, on average, coaching on the ACT, and few were conducted on
compared to those who reported three to six hours students taking the tests since 2000. Many of the
of preparation (Schiel & Valiga, 2014b). Short-term previous studies also used small samples that
preparation activities (e.g. commercial workbooks) were not necessarily representative of high school
were found to be associated with 1.2 to 1.5 point students taking the SAT or ACT (Briggs, 2009) or
increases in ACT Composite score, while longer- used research designs that raised concerns about
term learning activities (e.g. high school either internal or external validity (Lane et al.,
coursework) were associated with 2.5 to 5.8 point 2015). Likewise, recent work (i.e., Appelrouth et
increases in ACT Composite score (ACT, 2005). al., 2015) focused on students who paid for
services at select SAT test preparation centers. As
In addition to factors related to practice test preparation becomes widely accessible
opportunities (i.e., length of test preparation), through different delivery systems, large-scale
studies have also looked into the relationship studies of test preparation efficacy that involve a
between motivational factors and test performance variety of test preparation activities become more
in students’ test preparation and test-taking, such important to understanding the value and impact of
as awareness of their test preparedness or their test preparation activities on both the ACT and
self-efficacy. Mulvenon, Stegman, and Ritter (2005) SAT.
evaluated the impact of anxiety, pressure, and self-
efficacy on student performance on standardized
ACT Research & Policy | ACT Research Report | R1743 3
In this paper, we examine the impact of result in higher test scores. Issues of the fairness
participating in test preparation prior to retaking of ACT score use in college admissions will also
the ACT test. The present study aims to address be explored in light of our findings. We focus on
the call by Briggs (2009) for additional and more the effects of test preparation on ACT Composite
robust research on the efficacy of test score as it is most commonly used and
preparation programs for the ACT. This study referenced by students and admissions offices
used a fairly large sample of ACT-tested alike.
students. For the first research question, we
used a quasi-experimental deign that takes Method
advantage of propensity score matching
techniques to make causal claims related to test Data
preparation effectiveness. The second research
question used traditional regression techniques The data for this study were taken from a
with covariates used to account for group previously conducted study on test preparation
differences. (Schiel & Valiga, 2014a; Schiel & Valiga, 2014b).
In that study, an online survey was administered
to a random sample of 76,000 ACT test-takers
Using a pretest-posttest design, do
who had completed the ACT for the first time in
students who participate in test
April or June 2012 and retested in October 2012
preparation have larger score gains
or completed the ACT for the first time in
relative to students who did not
September, October, or December 2012 and
participate in test preparation; does the
retested in April 2013. These students were
test preparation effect depend on
invited via email to participate in a survey about
students’ pretest scores?
their test preparation activities. Of these
students, 9,654 students responded to the
Among students who participated in test
survey (12.7% response rate).
preparation, is the number of hours spent
participating in each of 10 test preparation
We were interested in students that had retested
activities related to retest scores?
for the ACT, had not used test preparation aids
prior to their first test, but had used test
Among students who participated in test
preparation aids prior to their second test. These
preparation, do their own beliefs that
selection criteria and the use of propensity score
they might have been ill-prepared to
matching (PSM; detailed in the Data Analysis
take the test, regardless of the test
section) resulted in a total number of 2,660
preparation activities they engaged in,
students in the analysis sample. Of these, 1,330
impact retest scores?
students indicated that they did prepare for their
second test, and another 1,330 students
Unlike prior observational studies about test indicated that they did not prepare for their
preparation for the ACT, the present study second test and thus were selected as a
utilizes a quasi-experimental method, applying a comparison group. Student self-reported
more rigorous design that will allow us to make demographic and background information (e.g.,
causal inferences about efficacy of test race, parental income) as well as academic
preparation and significant factors that may performance indicators (e.g., high school GPA)
were provided by students at the time of ACT
test registration.
ACT Research & Policy | ACT Research Report | R1743 4
Participants test administration. This survey question was
written such that we asked students about test
Of the 2,660 students used in the analysis for
preparation activities “outside of normal
this study, most were female (62%), in their
classroom participation.”
junior year of high school (85%), and White
(66%) (See Table 4). Approximately 47% of
Lack of Preparation
study participants were middle income
($36,000–$100,000) with fewer percentages of This measure consisted of students’ answers to
students coming from lower- and higher-income three survey items: “I have not yet taken the
families. The study sample closely resembled class(es) necessary for doing well on one or
the population of 2013 ACT test-takers (58% more areas of the test”; “Some areas of the test
White, 13% African-American, 15% had not been covered at all or had not been
Hispanic/Latino, 4% Asian, and 4% other/multi- covered adequately in my high school classes”;
race) but differed somewhat on gender (58% and “I realized I had not done anything to
female, 42% male; ACT, 2015). Survey prepare myself for taking this type of test.” For
respondents also had a slightly higher ACT each item, students were instructed to answer
Composite test score (M = 22.5, SD = 4.7) than yes or no. If a student said yes to any one of the
the national average (M = 20.9, SD = 4.8). three questions, they were coded as perceiving
themselves to have a lack of preparation for the
Measures ACT test.
Survey of Test Preparation Activities Amount of Test Preparation
for the ACT
The survey also asked students about their
The ACT test preparation survey consisted of exposure to 10 specific test preparation
three sections that measured test preparation, activities. These activities ranged from
how well students thought they were prepared, interaction with online test materials to self-
and amount of time allocated to 10 preparation directed workbooks to one-on-one test
activities. From this survey, data on test preparation (Table 1). Students were asked to
preparation activity, lack of academic preparation indicate the number of hours they spent on each
for the ACT, and amount of test preparation the of the activities in preparation for the second
student participated in were collected. test, using a four-point scale (0 = 0 hours spent
on activity; 1= 1-5 hours; 2 = 6-10 hours; 3 = 11
Test Preparation or more hours spent on activity). Example items
include: “Worked with a private tutor or
Students were asked to indicate whether or not
consultant” and “Took a commercial test-
they prepared for the first and/or second ACT
preparation course(s).”
Table 1. Test Preparation Activities Investigated in the Survey
Test Preparation Activities
Practice Test in ACT’s Free Preparing for the ACT
ACT Online Prep
Another Web-Based Test-Preparation Program
The Real ACT Prep Guide
Another Test Prep Workbook
Test Prep Workshops/Courses Offered by the High School
Commercial Test Prep Course
Private Tutor or Consultant
One-on-One with a High School Teacher
Other Test Prep Software
ACT Research & Policy | ACT Research Report | R1743 5
ACT Composite Score and prior student achievement. However, random
assignment in educational settings raises ethical
The ACT is a curriculum-based battery of four
and logistical concerns (Walker, Hoggart, &
multiple-choice tests of educational
Hamilton, 2008). Propensity score matching
achievement—English, mathematics, reading,
presents an alternative to randomization that
and science—and an optional writing test. The
models the assignment of students to the
ACT, typically taken in the 11th or 12th grade,
treatment conditions: in this case, the assignment
measures students’ academic readiness for
of students to either receive test preparation or
college in key content areas. The ACT
not, using an estimated propensity score
Composite score is the arithmetic mean of the
(Rosenbaum & Rubin, 1983; see Nagengast,
four subject test scores rounded to the nearest
Marsh, & Hau, 2013, for a discussion of how
whole number and reported on a scaled score
propensity score matching is a better alternative
from 1 to 36.
to covariate adjustment). This estimated
propensity score measures the probability of
Statistical Controls
being assigned to the test preparation group
In order to better isolate the effect of test given the covariates entered into the propensity
preparation, we included additional student score matching analysis. Through this process,
characteristics in our analysis. Each student we created control and treatment groups that are
characteristic was included because of the comparable on the propensity score which
theoretical relationship it has with ACT incorporates all included covariates.
performance (Appenrouth et al., 2015; Radunzel
In this study, we used the SAS software macro
& Noble, 2012; Sanchez, 2013). In this study,
“OneToManyMTCH” (Parsons, 2004). This macro
we statistically controlled for the impact of
uses a greedy, nearest-neighbor matching
whether students commonly experience test
algorithm to identify a matched sample for the
anxiety; prior ACT Composite score; whether the
control group, using an eight to one-digit match.
student took at least trigonometry in high school;
We used one-to-one matching between treatment
whether the student took at least physics in high
and control groups. The calculation of the
school; whether advanced placement courses in
propensity score was based on a logistic
mathematics, science, and English were taken
regression model that included 13 related
in high school; high school GPA; whether
covariates. Of those, 10 covariates were retained
students felt they needed assistance in core
using stepwise selection and forced inclusion
content area(s); whether high school attended
(Table 2).2
was public or private; whether a fee waiver was
used to take the ACT; highest parental
Linear Regression
education level; student’s grade level; student’s
gender and race; and parents’ income. Most of
Two linear regression models were estimated to
these student academic and demographic
address the research questions. The first
characteristics were collected at the time of
regression analysis sought to determine if
registration while information about test anxiety
students who participate in test preparation
was collected from the survey.
activities have a higher ACT Composite retest
score than those who did not prepare for the test,
Data Analysis controlling for academic and demographic
characteristics. These academic and
Propensity Score Matching (PSM)
demographic characteristics were included in the
In studying the effect of test preparation, the propensity score model and balanced by
interest is in estimating the causal effects of matching. We include them here in an effort to
engaging in test preparation while controlling for better capture the relationship between
confounding variables, such as parents’ income potentially confounding variables, activities of
ACT Research & Policy | ACT Research Report | R1743 6
interest, and our outcome of ACT test score. Results
Interactions between preparation status and
demographic information, and between the Propensity Score Matching
students’ first ACT score and test preparation
status, were tested. This allowed us to test Table 3 illustrates the balancing that was
whether there is a differential impact of achieved in the control and treatment groups by
preparation for student subgroups (i.e., race, using propensity score matching. This table
gender, and parent income) and across the range shows the standardized mean difference between
of prior ACT Composite scores. groups on each of the covariates investigated.
Prior to matching, large group differences were
The second regression model, including only observed for minority group membership,
those students who participated in test parental income, highest parental education, test
preparation, sought to determine whether the test anxiety, using a fee waiver, and high school GPA.
preparation effect identified in the first research After matching, the standardized mean
question varied by the number of hours of differences between the treatment and control
participation. In addition to the variables entered groups for all covariates were well below 10%.
into the model for the first research question, we
included in this second regression model the For continuous variables, the standardized mean
number of hours spent on 10 test preparation difference can be estimated as , where
activities and an indicator for whether students 𝑋𝑋���1�−�𝑋𝑋��2�
thought they were ill prepared to take the second and denote the treatment 𝑑𝑑an=d �c𝑠𝑠o12+n𝑠𝑠t22rol
2
test, regardless of the preparation activities sample means, and and denote the
𝑋𝑋1 𝑋𝑋2
engaged in. The 10 test preparation activities corresponding sample2 varian2ces. For binary
𝑠𝑠1 𝑠𝑠2
were treated as continuous measures in the variables, the standardized mean difference is
regression model since dummy coding each
activity for four categories would have resulted in
, where and denote the
estimating many coefficients relative to the (𝑃𝑃�1−𝑃𝑃�2)
reduced sample size used for research question 𝑑𝑑 = �𝑃𝑃�1�1−𝑃𝑃�1�+𝑃𝑃�2�1−𝑃𝑃�2�� 𝑃𝑃�1 𝑃𝑃�2
prop�ortions of the treatment and control groups.
2
two. For those test preparation activities found to
An extension of this binary case is available,
be statistically significant in the linear model, pair-
which makes use of multivariate Mahalanobis
wise comparisons across the number of hours
distance to handle multinomial variables (Yang &
participating in the test preparation activity were
Dalton, 2012).
conducted.
In addition to this empirical estimation of
balance, the distributions of each covariate were
examined. For illustrative purposes, the
distribution of the propensity score for the
treatment and control groups after matching are
displayed in Figure 1. The distributions for both
groups are very similar. Similar distributions
were found when examining other continuous
variables and similar proportions were found
across groups for categorical variables.
ACT Research & Policy | ACT Research Report | R1743 7
Table 2. Student Characteristics Considered and Retained in the Propensity Score Model
Retained in
Propensity
Score Model Student Characteristic
 Prior ACT Composite score
 High school GPA
 Interaction between prior ACT Composite and high school GPA
 Gender (Female)
 Minority membership (African American, Hispanic, American Indian, or Native Hawaiian)
 Family Income
 Highest parental Education
 Self-reported experience of anxiety during testing
 Fee waiver status indicator
Taken math coursework beyond Algebra II
Taken Science coursework beyond Chemistry
Taken advanced mathematics coursework
 Taken advanced science coursework
Taken advanced English coursework
Note: To qualify for an ACT fee waiver, a student must be currently enrolled in the 11th or 12th grade; a US citizen or testing in the
US, US territories, or Puerto Rico; and must meet one or more indicators of economic need listed on the ACT Fee Waiver form.
These students are eligible to receive a fee waiver to cover the cost of taking the ACT.