ERIC ED563073: The SAT® as a Predictor of Different Levels of College Performance. Research Report No. 2006-3

2006·0.55 MB·English

by  ERIC

Research Report
No. 2006-3
The SAT® As
a Predictor of
Different Levels
of College
Performance
Jennifer L. Kobrin and Rochelle S. Michel
www.collegeboard.com
College Board Research Report No. 2006-3
The SAT As a
®
Predictor of Different
Levels of College
Performance
Jennifer L. Kobrin and Rochelle S. Michel
The College Board, New York, 2006
Jennifer L. Kobrin is a research scientist at the College
Board.
Rochelle S. Michel is a doctoral student at Fordham
University.
Researchers are encouraged to freely express their
professional judgment. Therefore, points of view or
opinions stated in College Board Reports do not necessarily
represent official College Board position or policy.
The College Board: Connecting Students to College Success
The College Board is a not-for-profit membership
association whose mission is to connect students to
college success and opportunity. Founded in 1900,
the association is composed of more than 5,000
schools, colleges, universities, and other educational
organizations. Each year, the College Board serves
seven million students and their parents, 23,000 high
schools, and 3,500 colleges through major programs and
services in college admissions, guidance, assessment,
financial aid, enrollment, and teaching and learning.
Among its best-known programs are the SAT®, the
PSAT/NMSQT®, and the Advanced Placement Program®
(AP®). The College Board is committed to the principles of
excellence and equity, and that commitment is embodied
in all of its programs, services, activities, and concerns.
For further information, visit www.collegeboard.com.
Additional copies of this report (item #060481783) may
be obtained from College Board Publications, Box 886,
New York, NY 10101-0886, 800 323-7155. The price is $15.
Please include $4 for postage and handling.
© 2006 The College Board. All rights reserved.
College Board, Advanced Placement Program, AP,
SAT, and the acorn logo are registered trademarks of
the College Board. connect to college success and SAT
Reasoning Test are trademarks owned by the College
Board. PSAT/NMSQT is a registered trademark of
the College Board and National Merit Scholarship
Corporation. All other products and services may be
trademarks of their respective owners. Visit the College
Board on the Web: www.collegeboard.com.
Printed in the United States of America.
Figures
Contents
1. Overall accuracy rates for total sample ....... 2
2. Overall accuracy rates for successful
Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .1 students .................................. 3
3. Overall accuracy rates for unsuccessful
Method . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .1 students .................................. 3
4. Accuracy rates for successful females......... 4
Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .2
5. Accuracy rates for successful males .......... 4
Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .6 6. Accuracy rates for successful Asian
American students......................... 4
References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .7 7. Accuracy rates for successful African
American students......................... 5
Appendix . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .8 8. Accuracy rates for successful white
students .................................. 5
Tables 9. Accuracy rates for successful Hispanic
1. Percentage of Students Achieving Various students .................................. 5
Levels of Success........................... 2
10. Accuracy rates for successful students
2. Summary of Results for Predicting attending very selective colleges ............. 5
Successful Students ........................ 3
11. Accuracy rates for successful students
A1. Accuracy Rates for Logistic Regression attending moderately selective colleges ....... 5
Models for Total Sample.................... 8
12. Accuracy rates for successful students
A2. Accuracy Rates for Logistic Regression attending least selective colleges ............. 5
Models by Gender ......................... 8
A3. Accuracy Rates for Logistic Regression
Models by Racial/Ethnic Group ............. 9
A4. Accuracy Rates for Logistic Regression
Models by College Selectivity............... 10

Introduction
of predicting FGPAs of 3.5 or higher. The study also
found that both ACT and high school grades are effective
The SAT Reasoning Test™ (SAT®) is the most widely used predictors when success is defined as a FGPA of C (2.0)
standardized test for college admissions. Research on the or better, but that in this case, high school grades were
validity of SAT scores for predicting college performance actually a better predictor.
has been conducted over several decades, going back Based on the same hypothesis offered by Noble and
to the 1950s and continuing to this day. Fishman and Sawyer (2002), this study examined the validity of the
Pasanella (1960) reviewed some of the earliest SAT SAT for predicting various levels of student success in
predictive validity studies, and Young (2001) provides college, as measured by FGPAs. Logistic regression was
the most recent review. Hezlett et al. (2001) performed used to predict the probability that a student would be
a comprehensive meta-analysis of approximately 3,000 successful or unsuccessful in achieving a FGPA at various
validity studies with more than 1 million students, and levels, based on that student’s SAT scores and high
found that the SAT is a valid predictor of first-year college school grade point average (HSGPA). Although most
grade point average (FGPA), with multiple correlations predictive validity studies employ multiple regression
ranging from .44 to .62. as the main analytic procedure, this study employed
Colleges typically use both high school grades and logistic regression. In multiple regression, the focus is
scores on standardized tests such as the SAT to predict a on prediction of a value on the dependent variable. One
student’s probability of success in college. One of the most prediction equation is produced for the sample used in
persistent questions regarding the validity of the SAT is the analysis and prediction occurs on the full range of
whether the SAT adds substantially to the prediction of values of the dependent variable. In this study, however,
college success after high school grades are taken into the primary research question focused on whether the
account. To address this question, the majority of SAT SAT or HSGPA is a better predictor of FGPA for students
predictive validity studies examine the relationship of with high FGPAs as compared to students with lower
SAT scores and high school grades to first-year college FGPAs. Therefore, it was necessary to dichotomize the
grades. Many of these studies find that SAT scores make dependent variable to make the distinction between low
a substantial contribution to the prediction of FGPA, and high levels of FGPA, and use logistic regression to
and that using admissions test scores such as the SAT predict the probability of success at each level.
in combination with a measure of high school grades
produces higher validity coefficients than using either
measure alone (Camara, 2005). Method
Although high school grades and SAT scores are
highly correlated, these two indicators measure slightly
different constructs and are expected to be related to Analyses were conducted using data from 30 colleges
college achievement in different ways. The SAT measures that agreed to participate in a multiyear validity study
reasoning ability and educational achievement related to sponsored by the College Board. The sample consisted
successful performance in college. High school grades of approximately 34,000 students who entered college
measure educational achievement, but also measure in the fall of 1995. The sample was 53 percent female
noncognitive factors such as effort, attendance, conformity, and 47 percent male, and approximately 75 percent
and motivation (Stiggins, Frisbie, and Griswold, 1989). Based white/Caucasian, 10 percent Asian American, 5 percent
on the research of Goldman and others (e.g., Goldman and African American, 5 percent Hispanic, 1 percent Native
Hewitt, 1975; Goldman, Schmidt, Hewitt, and Fisher, 1974; American, and 4 percent other or unknown. The gender
Goldman and Widawski, 1976), Noble and Sawyer (2002) distribution in the sample was very similar to the 1995
hypothesized that while average college grades are more population of college-bound seniors; however, the ethnic
likely to reflect noncognitive components, high college distribution differed somewhat. The sample had fewer
grades are more likely to reflect cognitive achievement and African American and Hispanic students and more white
less likely to reflect noncognitive factors. Based on this students than the 1995 population.
hypothesis, Noble and Sawyer expected that predictions The sample was of higher ability than the population
of moderate first-year college grades would be better of college-bound seniors, with a mean SAT verbal score
when based on high school grades, while predictions of 577 (compared to 504 in the population) and a mean
of high FGPAs would be better when based on college SAT mathematics score of 585 (compared to 506 in the
admissions test scores (in their case, the ACT). They used population). The standard deviations of SAT verbal and
logistic regression to study how well the ACT and high mathematics scores were slightly smaller in the sample
school grades predicted various levels of FGPAs (e.g., 2.0 (91 for SAT-V and 93 for SAT-M) than in the population
or higher, 3.5 or higher). Their results indicated that ACT (113 and 112 for SAT-V and SAT-M, respectively). The
composite scores did a better job than high school grades mean self-reported high school grade point average for the

Table 1
sample was 3.5 with a standard deviation of .53 (compared
Percentage of Students Achieving Various Levels of
to a mean of 3.19 and standard deviation of .66 in the
Success
population). The mean FGPA for the sample was 2.8 with
a standard deviation of .72 (comparison means for the FGPA Greater than or Equal to:
population were not available). Subgroup N 2.00 2.50 3.00 3.25 3.50 3.75
Females 8,09 89.7 73.4 47.3 3.4 7.8 6.7
Logistic regression analyses were performed six times
Males 5,860 84. 65. 39.5 6.4 5.5 6.6
on each subgroup using different levels of FGPA as the
Native
criterion variable. The criterion variable was whether the 65 8.5 55.8 30.6 8.5 9.4 .6
American
student earned a FGPA greater than or equal to 2.0, 2.5,
Asian
3.0, 3.25, 3.5, or 3.75. The regression models included the 3,36 89. 73.3 49.4 34.4 0.3 8.0
American
following predictors: 1) HSGPA only; 2) SAT verbal and African
,776 74.5 48.9 .8 .8 5. .7
math scores only (SAT); and 3) SAT and HSGPA. The American
logistic regression analyses were performed on the group Hispanic ,60 8.0 6.3 34. 9.7 .0 3.4
as a whole, and separate analyses were conducted on each White 5,60 88. 7. 45. 30. 7.5 7.
gender and racial/ethnic group. Other 74 87.6 7.8 47.4 34.3 9.7 6.6
Because selective colleges may place more weight on TOTAL N 34,069 9,67 3,689 4,870 9,894 5,706 ,67
cognitive components in the grading of students, separate TOTAL % ****** 87. 69.5 43.6 9.0 6.7 6.7
analyses were also performed on subgroups based on the
selectivity of the college they attended. The information example, Asian American students were more likely than
used to calculate an index of selectivity for the colleges other subgroups to obtain FGPAs at or above all of the
and universities in the sample was taken from the College criterion levels, while African American students were
Board’s 1995 College Handbook. The selectivity index is least likely to obtain each of the six criterion levels.
the ratio of the number of students who were accepted to Figure 1 and Appendix Table A1 show the accuracy
the college/university compared to the number of students rates for the various logistic regression models for the
who applied to that college/university. Three subgroups of total sample. The accuracy rate represents the proportion
students based on their colleges’ selectivity were created: of test-takers who were correctly classified, either as
Level 1 included students attending the most selective successful or unsuccessful.2 An overall summary of the
colleges (those accepting less than half of their applicants). results is presented in Table 2. This table shows whether
Level 2 included students attending colleges with average the SAT or HSGPA was a better predictor of successful
selectivity (those accepting between 50 and 75 percent of
their applicants), and Level 3 included students attending 100
the least selective colleges (those accepting more than 75
HSGPA SAT Both
percent of their applicants). There were 4 Level 1 schools,
11 Level 2 schools, and 11 Level 3 schools in the sample.1
80
Results
60
The results focused on the accuracy of the logistic
regression models to correctly classify the individuals as 40
successful or unsuccessful in their first year of college.
Table 1 gives the percentage of students in the sample by
gender and racial/ethnic group achieving the various levels
20
of success as defined by their FGPA. At the lowest level of
success, approximately 87 percent had FGPAs at or above
2.0; while at the highest level of success, about 7 percent
had FGPAs at or above 3.75. The percentages varied for 0
2.00 2.50 3.00 3.25 3.50 3.75
the different gender and racial/ethnic subgroups. For
Figure 1. Overall accuracy rates for total sample.
1 The 1995 College Handbook did not have the information necessary to compute the selectivity index for 4 of the 30 institutions in the sample.
The average selectivity is about .68, meaning about 68 percent of those who applied were admitted. The most selective school had an index of
.22, admitting only 22 percent of those who applied, and the least selective school in the sample had an index of .95, admitting 95 percent of
those who applied.
2 The accuracy rates for the total sample and each subgroup examined in this report can be found in the Appendix to this report.

Table 2
Summary of Results for Predicting Successful Students
FGPA Criterion Level
Subgroup 2.0 2.5 3.0 3.25 3.5 3.75
Total Sample SAT HSGPA SAT SAT SAT ---
Females SAT HSGPA HSGPA SAT SAT ---
Males SAT SAT HSGPA SAT SAT ---
Native American = HSGPA SAT --- --- ---
Asian American = SAT HSGPA SAT SAT ---
African American SAT SAT SAT SAT --- ---
Hispanic SAT SAT HSGPA SAT --- ---
White SAT HSGPA SAT SAT SAT ---
Other SAT SAT HSGPA HSGPA HSGPA ---
Attending Very Selective Colleges = HSGPA SAT SAT SAT ---
Attending Moderately Selective Colleges SAT HSGPA SAT HSGPA SAT SAT
Attending Least Selective Colleges SAT HSGPA HSGPA HSGPA HSGPA SAT
Note: An “=” indicates that the SAT and HSGPA were equally effective predictors; an “---” indicates that neither measure was an effective
predictor, or that there were not enough cases to evaluate the models.
students within each subgroup and for the total sample at success-criterion levels, with the exception of the 2.5
each of the six FGPA criterion levels. and 3.75 levels, the SAT is slightly more accurate than
At two of the highest levels of success (i.e., FGPAs HSGPA in predicting the successful group, but less
greater than or equal to 3.25 and 3.5), the SAT was found accurate than HSGPA in predicting the unsuccessful
to have slightly greater predictive power than HSGPAs. group. Overall we see that as the success-criterion level
Interestingly, at both the lowest and highest level of success, increases, the percentage of correct classifications for
the accuracy rate of either the SAT or HSGPA alone is the the unsuccessful group increases and the percentage of
same as the accuracy rate of the SAT and HSGPA combined. correct classifications for the successful group decreases.
At two levels of criterion success (FGPAs at or above 2.5 This finding is directly related to the decrease in the
to 3.0), HSGPA is better than the SAT when used as the number of cases as the success-criterion level increases,
sole predictor of FGPA. Neither the SAT nor HSGPA was as well as a decrease in the variability of HSGPA and
successful at predicting the highest criterion level, 3.75. SAT scores as the successful group becomes more
Although overall the model is most accurate at the elite. At the highest criterion level (3.75 FGPA), the
3.75 level, there is a difference in the models’ accuracy accuracy rate for both the SAT and HSGPA was zero
in classifying the successful and the unsuccessful for predicting successful students and 100 percent for
individuals. As shown in Figures 2 and 3, at most predicting unsuccessful students.
100 100
HSGPA SAT Both HSGPA SAT Both
90 90
80 80
70 70
60 60
50 50
40 40
30 30
20
20
10
10
0
0
2.00 2.50 3.00 3.25 3.50 3.75 2.00 2.50 3.00 3.25 3.50 3.75
Figure 2. Overall accuracy rates for successful students. Figure 3. Overall accuracy rates for unsuccessful students.
3
Figures 4 and 5 show the accuracy rates for the three d100.0
e
logistic regression models for successful females and males, ssifi90.0 HSGPA
rmesapleesc tmivaeyly b. eT fhoue nrde siun ltAs pfpoern duinx sTuacbclees sAfu2l. Ifne malal lelos giasntidc ctly Cla7800..00 SBAotTh
regression models, females are correctly classified at higher orre60.0
rates than males for the lowest two FGPA criterion levels C
ul 50.0
(2.0 and 2.5), while males are correctly classified at higher sf
s
rates than females for FGPAs of 3.0 and higher. There is ce40.0
c
u
a sizable difference in the accuracy rates for females and e S30.0
males who were successful and unsuccessful. With only ag20.0
nt
one exception, in all models and all FGPA criterion levels, e
successful females were correctly classified at higher rates Perc10.0
0.0
than males. The logistic regression model with HSGPA as 2.0 2.5 3.0 3.25 3.5 3.75
First-Year College Grade Point Average
the sole predictor (Model 1) results in higher accuracy rates
Figure 4. Accuracy rates for successful females.
for successful females at the 2.5 and 3.0 criterion levels,
while the model with the SAT as the sole predictor (Model
2) results in higher accuracy rates for successful females at
the 2.0, 3.25, and 3.5 criterion levels. Conversely, the highest
accuracy rates for unsuccessful females are produced by
Model 2 (SAT only) at the 2.5 and 3.0 levels and Model
100.0
d
1 (HSGPA only) at the 2.0, 3.25, and 3.5 levels. Model 2 e
produces higher accuracy rates than Model 1 for successful ssifi90.0 HSGPA
males at all but two criterion levels (3.0 and 3.75). y Cla80.0 SBAotTh
The classification rate for females using the SAT alone ctl70.0
e
is equal to or slightly higher than using HSGPA alone at all orr60.0
C
FGPA criterion levels. For males, the use of the SAT alone ul 50.0
produces an accuracy rate that is equal to or slightly higher ssf
ce40.0
than that of HSGPA alone for all but two FGPA criterion uc
levels (2.5 and 3.0). The combination of HSGPA and the SAT e S30.0
g
a20.0
(Model 3) increases the accuracy rate over either predictor nt
e
used alone in many but not all cases. Overall for both females erc10.0
P
and males, Model 3 increases the accuracy rate at all FGPA 0.0
2.0 2.5 3.0 3.25 3.5 3.75
criterion levels except for the lowest (2.0) and highest (3.75). First-Year College Grade Point Average
For successful students, Model 3 increases the accuracy rate Figure 5. Accuracy rates for successful males.
at the 3.0 through 3.5 levels; for unsuccessful students, the
two-predictor model increases the accuracy rate over either
of the single-predictor models only at the 2.5 criterion level
for females, and at the 2.0 level for males.
Figures 6–9 show the accuracy rates for successful
students by racial/ethnic group (Native American students
100.0
are not included due to the small number of students in this ed
fi
study). The full results for both successful and unsuccessful ssi90.0 HSGPA
a
students are displayed in Appendix Table A3. For Native Cl80.0 SAT
y Both
American students, the SAT model has a higher accuracy ctl70.0
e
rate than the HSGPA model at the 2.5 and 3.0 levels, while orr60.0
C
both models are equally effective for the 2.0 level. For Asian ul 50.0
American students, both models are equally effective across ssf
ce40.0
all FGPA levels with no difference larger than 0.5 percent. For c
u
African American students, the HSGPA model has a higher e S30.0
g
a20.0
accuracy rate than the SAT model at the 2.5 FGPA level, but nt
e
the SAT model has a much higher accuracy rate for predicting erc10.0
P
success at FGPA levels between and including 3.0 and 3.5. At 0.0
2.0 2.5 3.0 3.25 3.5 3.75
the highest level of success (3.75 and higher), both models First-Year College Grade Point Average
have the same accuracy rate. The HSGPA model is slightly Figure 6. Accuracy rates for successful Asian American
better than the SAT model in predicting Hispanic and white students.
4
100.0
d 100.0
e d
Correctly Classifi67890000....0000 HSBAoStTGhPA orrectly Classifie67890000....0000 HSBAoStTGhPA
uccessful 4500..00 ccessful C4500..00
ge S30.0 e Su30.0
a20.0 g
ent nta20.0
Perc100..00 Perce10.0
2.0 2.5 3.0 3.25 3.5 3.75 0.0
First-Year College Grade Point Average 2.0 2.5 3.0 3.25 3.5 3.75
First-Year College Grade Point Average
Figure 7. Accuracy rates for successful African American Figure 10. Accuracy rates for successful students attending
students. very selective colleges.
100.0 100.0
d d
e e
ssifi90.0 HSGPA ssifi90.0 HSGPA
y Cla80.0 SBAotTh y Cla80.0 SBAotTh
ctl70.0 ctl70.0
e e
orr60.0 orr60.0
C C
ul 50.0 ul 50.0
sf sf
s s
ce40.0 ce40.0
c c
u u
e S30.0 e S30.0
ag20.0 ag20.0
nt nt
e e
erc10.0 erc10.0
P 0.0 P 0.0
2.0 2.5 3.0 3.25 3.5 3.75 2.0 2.5 3.0 3.25 3.5 3.75
First-Year College Grade Point Average First-Year College Grade Point Average
Figure 8. Accuracy rates for successful white students. Figure 11. Accuracy rates for successful students attending
moderately selective colleges.
100.0 100.0
Correctly Classified67890000....0000 HSBAoStTGhPA Correctly Classified67890000....0000 HSBAoStTGhPA
ntage Successful 23450000....0000 ntage Successful 23450000....0000
Perce100..00 Perce100..00
2.0 2.5 3.0 3.25 3.5 3.75 2.0 2.5 3.0 3.25 3.5 3.75
First-Year College Grade Point Average First-Year College Grade Point Average
Figure 9. Accuracy rates for successful Hispanic students. Figure 12. Accuracy rates for successful students attending
least selective colleges.
5