ERIC ED505850: The Impacts of Regular Upward Bound on Postsecondary Outcomes Seven to Nine Years after Scheduled High School Graduation. Final Report

1.2 MB·English

by  ERIC

The Impacts of Regular Upward Bound on Postsecondary
Outcomes Seven to Nine Years After Scheduled High

School Graduation

Final Report
Prepared by:
Neil S. Seftor
Arif Mamun
Allen Schirm

Mathematica Policy Research, Inc.
Princeton, N.J.

Prepared for:
U.S. Department of Education
Office of Planning, Evaluation and Policy Development
Policy and Program Studies Service

2009
This report was prepared for the U.S. Department of Education under Contract No. EA97030001. The project
monitor was Margaret Cahalan in the Policy and Program Studies Service. The views expressed herein are those of
the contractor. The content of this report does not necessarily reflect the views or policies of the U.S. Department of
Education, nor does the mention of commercial products or organizations imply endorsement by the U.S.
government. The inclusion of such information is for the reader's convenience and is not intended to endorse any
views expressed, or products, programs, models or services offered.

U.S. Department of Education
Margaret Spellings
Secretary

Office of Planning, Evaluation and Policy Development
Bill Evers
Assistant Secretary

Policy and Program Studies Service
Alan L. Ginsburg
Director

Program and Analytic Studies Division
David Goodwin
Director

January 2009
______________________________________________________________________________________

This report is in the public domain. Authorization to produce the report in whole or in part is granted. Although
permission to reprint this publication is not necessary, the citation should be: U.S. Department of Education, Office
of Planning, Evaluation and Policy Development, Policy and Program Studies Service, The Impacts of Regular
Upward Bound on Postsecondary Outcomes Seven to Nine Years After Scheduled High School Graduation,
Washington, D.C., 2009.
ACKNOWLEDGMENTS
This report on the effects of Upward Bound on students’ postsecondary outcomes reflects
the contributions of many individuals. The authors would like to thank David Goodwin and
Margaret Cahalan of the Policy and Program Studies Service at the U.S. Department of
Education. We are grateful to David for his continued support, substantive guidance, and
encouragement throughout the study and to Maggie for her many very valuable comments and
other guidance. Marisol Cunnington and Jay Noell of the U.S. Department of Education also
provided helpful suggestions. In addition, the report has been improved by comments from three
anonymous external reviewers selected by the Institute of Education Sciences.

David Myers, who directed the national evaluation of Upward Bound for many years,
played a key role in shaping the study design and led previous impact analyses. Mary Moore, a
principal investigator for the national evaluation, was also instrumental in shaping the study
design. Mark Dynarski and Peter Schochet provided insightful suggestions throughout the
analysis and drafts of the report.

The authors would also like to thank the individuals who contributed to the data collection
and analysis and the production of the report. David DesRoches oversaw the data collection, and
Zhanyun Zhao created the survey weights. Karin Zeller constructed the analysis variables and
computed the impact estimates, and Mary Grider provided technical assistance with data and
programming issues. Jennifer Baskwell produced the document.
iv

CONTENTS
Chapter Page
EXECUTIVE SUMMARY.........................................................................................xiii

I INTRODUCTION..........................................................................................................1
A. CONTEXT AND PURPOSE OF UPWARD BOUND...........................................1
B. CONTEXT FOR INTERPRETING PROGRAM EFFECTS..................................7
1. Value-added of Upward Bound.......................................................................8
2. Upward Bound Applicants.............................................................................11
3. Effects for Students Who Participated During the Mid-1990s......................12
4. Variation in Exposure to Upward Bound.......................................................14

C. PREVIOUS FINDINGS........................................................................................16

II RESEARCH DESIGN AND ANALYTIC ISSUES.....................................................19
A. RESEARCH DESIGN...........................................................................................19
1. Selection of Upward Bound Projects and Random Assignment...................19
2. Outcome Measures.........................................................................................22
3. Data Sources..................................................................................................24
4. Construction of the Outcome Measures.........................................................26

B. ANALYTIC ISSUES.............................................................................................28
1. Estimation of Program Impacts.....................................................................28
2. Subgroup Analysis.........................................................................................31
3. Use of Weights to Account for Sample Design and Survey Nonresponse....35
4. Potential Threats to the Study Design............................................................37

III THE EFFECT OF UPWARD BOUND ON POSTSECONDARY OUTCOMES.......39
A. THE EFFECT OF THE OPPORTUNITY TO PARTICIPATE IN UPWARD
BOUND (ITT).......................................................................................................40
1. The Effect of Upward Bound on Postsecondary Enrollment and College
Selectivity......................................................................................................43
2. The Effect of Upward Bound on Financial Aid Application and Pell
Grant Receipt.................................................................................................43
3. The Effect of Upward Bound on Postsecondary Completion........................44

vi
Chapter Page
III B. THE EFFECT OF PARTICIPATION IN UPWARD BOUND (CACE)..............44
(continued)
C. SUMMARY OF SENSITIVITY ANALYSES.....................................................44
1. Sensitivity Analyses Pertaining to the Measurement of Outcomes...............47
2. Sensitivity Analyses Pertaining to Sample Weighting..................................48

IV THE EFFECT OF UPWARD BOUND ON POSTSECONDARY OUTCOMES
FOR SELECTED SUBGROUPS.................................................................................55
A. THE EFFECT OF THE OPPORTUNITY TO PARTICIPATE IN UPWARD
BOUND FOR SELECTED SUBGROUPS (ITT).................................................56
1. Grade at Application to Upward Bound........................................................56
2. Applicants’ Educational Expectations at the Time of Application
to Upward Bound...........................................................................................57
3. Level of Ninth-Grade Mathematics Class......................................................60
4. Grade Point Average in Ninth Grade.............................................................63

B. THE EFFECT OF PARTICIPATION IN UPWARD BOUND ON
SELECTED SUBGROUPS (CACE).....................................................................65

V THE EFFECT OF ADDITIONAL UPWARD BOUND PARTICIPATION ON
POSTSECONDARY OUTCOMES.............................................................................67
A. RESEARCH QUESTIONS...................................................................................67
B. RESEARCH METHODS......................................................................................68
C. THE EFFECT OF ADDITIONAL UPWARD BOUND PARTICIPATION.......71
1. Postsecondary Enrollment.............................................................................71
2. Financial Aid..................................................................................................73
3. Postsecondary Completion.............................................................................74

D. INTERPRETATION OF THE FINDINGS...........................................................74

REFERENCES..............................................................................................................77

APPENDIX A: SAMPLE DESIGN, UNIT NONRESPONSE, AND WEIGHTS....A.1

APPENDIX B: DATA COLLECTION AND OUTCOME MEASURES................B.1

vii
Chapter Page
APPENDIX C: SENSITIVITY ANALYSES PERTAINING TO THE
MEASUREMENT OF OUTCOMES...............................................C.1

APPENDIX D: SAMPLE SIZES AND WEIGHTED STANDARD
DEVIATIONS FOR ALL OUTCOME VARIABLES.....................D.1

APPENDIX E: ESTIMATION OF IMPACTS AND STANDARD ERRORS........E.1

APPENDIX F: METHODS USED TO ESTIMATE THE EFFECTS
OF ADDITIONAL UPWARD BOUND PARTICIPATION............F.1

APPENDIX G: SENSITIVITY ANALYSES PERTAINING TO SAMPLE
WEIGHTING....................................................................................G.1

APPENDIX H: OTHER SUPPLEMENTAL SERVICE PROGRAMS....................H.1

APPENDIX I: ADDITIONAL SUBGROUP TABLES.............................................I.1

viii

TABLES
Table Page
I.1 Receipt of Supplemental Services...................................................................................9
I.2 Receipt of Upward Bound and Other Supplemental Services......................................10
I.3 Unweighted Rates of Participation in Upward Bound Among Treatment
Group Members............................................................................................................15
II.1 Response Rates.............................................................................................................25
II.2 Control Variables Included in the Regression Models.................................................31
II.3 Sample Size in Applicant Characteristic Subgroups by Treatment Status...................32
II.4 Sample Size in Project Characteristic Subgroups by Treatment Status........................34
III.1 Impact of Upward Bound on Postsecondary Outcomes (ITT)......................................41
III.2 Impact of Upward Bound on Postsecondary Outcomes (CACE).................................45
III.3 Illustrative Impact Estimates from Sensitivity Analyses..............................................52
IV.1 Impact of Upward Bound on Postsecondary Outcomes by Students’ Grade
at Application (ITT)......................................................................................................58
IV.2 Impact of Upward Bound on Postsecondary Outcomes by Students’
Educational Expectations (ITT)....................................................................................59
IV.3 Impact of Upward Bound on Postsecondary Outcomes by Students’
Ninth-Grade Math Class (ITT).....................................................................................62
IV.4 Impact of Upward Bound on Postsecondary Outcomes by Students’
Ninth-Grade GPA (ITT)................................................................................................64
V.1 Duration of Upward Bound Participation and Completion Rates,
Excluding No-Shows....................................................................................................69
V.2 Quasi-Experimental Impact of an Additional Year of Upward Bound
on Postsecondary Outcomes.........................................................................................72
V.3 Quasi-Experimental Impact of Upward Bound Completion on
Postsecondary Outcomes..............................................................................................73

x