Evaluation of the Effectiveness of the Alabama Math, Science

314 Pages·2012·2.65 MB·English

NCEE 2012-4008 U. S. DEPARTMENT OF EDUCATION
Evaluation of the Effectiveness
of the Alabama Math, Science,
and Technology Initiative
(AMSTI)
Evaluation of the Effectiveness of the Alabama Math, Science,

and Technology Initiative (AMSTI)

Final Report

February 2012

Authors:

Denis Newman, Empirical Education Inc., Co-Principal Investigator

Pamela B. Finney, Research Management Leader, Regional Educational Laboratory
Southeast, Co-Principal Investigator

Steve Bell, Abt Associates Inc., Task 2 Methodology Leader

Herb Turner, ANALYTICA, Technical Consultant

Andrew P. Jaciw, Empirical Education Inc., Director of Experimental Design and
Analysis

Jenna L. Zacamy, Empirical Education Inc., Senior Research Manager

Laura Feagans Gould, Academy for Educational Development, Senior Research
Scientist

Project Officer:
Sandra Garcia, Institute of Education Sciences

NCEE 2012-4008

Richard Sawyer, Academy for Educational Development, was the original co-principal investigator.
U.S. Department of Education
Arne Duncan
Secretary

Institute of Education Sciences
John Q. Easton
Director

National Center for Education Evaluation and Regional Assistance
Rebecca A. Maynard
Commissioner

February 2012

This report was prepared for the National Center for Education Evaluation and Regional
Assistance, Institute of Education Sciences, under contract ED–06-CO–0028.
IES evaluation reports present objective information on the conditions of implementation and
impacts of the programs being evaluated. IES evaluation reports do not include conclusions or
recommendations or views with regard to actions policymakers or practitioners should take in
light of the findings in the report.
This report is in the public domain. Authorization to reproduce it in whole or in part is granted.
While permission to reprint this publication is not necessary, the citation should read: Newman,
D., Finney, P.B., Bell, S., Turner, H., Jaciw, A.P., Zacamy, J.L., & Feagans Gould, L. (2012).
Evaluation of the Effectiveness of the Alabama Math, Science, and Technology Initiative
(AMSTI). (NCEE 2012–4008). Washington, DC: National Center for Education Evaluation and
Regional Assistance, Institute of Education Sciences, U.S. Department of Education.

This report is available on the Institute of Education Sciences website at http://ncee.ed.gov and
the Regional Educational Laboratory Program website at http://edlabs.ed.gov.
Alternate Formats Upon request, this report is available in alternate formats, such as Braille,
large print, audiotape, or computer diskette. For more information, please contact the
Department’s Alternate Format Center at 202-260-9895 or 202-205-8113.
Disclosure of potential conflict of interest
None of the authors or other staff involved in the study from Abt Associates, Academy for
Educational Development, ANALYTICA, Empirical Education Inc., or the Regional Educational
Laboratory-Southeast at SERVE Center at the University of North Carolina at Greensboro has
financial interests that could be affected by the content of this report.1 No one on the nine-
member Technical Working Group, convened twice annually by the research team to provide
advice and guidance, has financial interests that could be affected by the study findings.

1 Contractors carrying out research and evaluation projects for IES frequently need to obtain expert advice
and technical assistance from individuals and entities whose other professional work may not be entirely
independent of or separable from the tasks they are carrying out for the IES contractor. Contractors
endeavor not to put such individuals or entities in positions in which they could bias the analysis and
reporting of results, and their potential conflicts of interest are disclosed.
iii
Acknowledgments

This study was the collaborative effort of a team that included professional staff at the
Regional Educational Laboratory-Southeast at SERVE Center at the University of North
Carolina at Greensboro; Empirical Education Inc.; Abt Associates Inc.; and the Academy for
Educational Development (AED). From the Regional Educational Laboratory-Southeast
(SERVE), the authors acknowledge Jean Scott, senior education policy research analyst, for
assistance in maintaining a strong relationship with the Alabama State Department of Education
(ALSDE); David Schaefer, technical editor and advisor; Melissa Williams, evaluation specialist,
and De Fuller, administrative support.

From Empirical Education Inc., the authors acknowledge Laurel Sterling, Margit
Zsolnay, and Ruthie Chang for their contributions to data collection, project management, and
report writing, and Boya Ma, Qingfeng (Breeze) Zhao, Akiko Lipton, and Xiaohui Zheng, for
planning and conducting the statistical analyses.

From Abt, the authors acknowledge Rob Olsen, for his valuable advice on statistical
modeling, and Cay Bradley, for her collaboration on the analysis of the two-year impact.

From AED, the authors acknowledge Harold Wenglensky, Richard Sawyer, Amy Detgen,
Robert Bozick, and Jason Snipes, for their leadership in earlier phases of the project.

Finally, the authors express their deep gratitude to the people at ALSDE and
administrators and teachers in the participating school systems for their assistance and
cooperation in conducting this research.
iv
CONTENTS
DISCLOSURE OF POTENTIAL CONFLICT OF INTEREST .......................................................................................... III
ACKNOWLEDGMENTS ....................................................................................................................................... IV
SUMMARY ....................................................................................................................................................... XX
1: INTRODUCTION AND STUDY OVERVIEW ......................................................................................................... 1
STRENGTHENING SKILLS OF MATHEMATICS AND SCIENCE TEACHERS NATIONWIDE......................................................................... 1
THE ALABAMA MATH, SCIENCE, AND TECHNOLOGY INITIATIVE (AMSTI): A STATE-LEVEL STRATEGY TO STRENGTHEN SKILLS OF
MATHEMATICS AND SCIENCE TEACHERS ............................................................................................................................... 2
ALABAMA MATH, SCIENCE, AND TECHNOLOGY INITIATIVE (AMSTI) THEORY OF ACTION .............................................................. 3
ALABAMA MATH, SCIENCE, AND TECHNOLOGY INITIATIVE (AMSTI) PROGRAM COMPONENTS AND ELEMENTS ................................. 5
PRIOR RESEARCH ON THE EFFECTIVENESS OF THE ALABAMA MATH, SCIENCE, AND TECHNOLOGY INITIATIVE (AMSTI) ON STUDENT
ACHIEVEMENT ................................................................................................................................................................ 6
OVERVIEW OF STUDY DESIGN ............................................................................................................................................ 7
RESEARCH QUESTIONS ..................................................................................................................................................... 9
STRUCTURE OF THE REPORT ............................................................................................................................................ 12
2: STUDY DESIGN AND METHODOLOGY ............................................................................................................. 13
RATIONALE FOR EXPERIMENTAL DESIGN ............................................................................................................................. 13
RECRUITMENT, SELECTION, AND RANDOM ASSIGNMENT OF SCHOOLS ...................................................................................... 14
INCENTIVES TO PARTICIPATE ............................................................................................................................................ 20
DEFINING THE TERM “ALABAMA MATH, SCIENCE, AND TECHNOLOGY INITIATIVE (AMSTI) TEACHER” ........................................... 20
DATA SOURCES, COLLECTION, AND CLEANING ..................................................................................................................... 21
THREATS TO INTERNAL VALIDITY ....................................................................................................................................... 30
DATA ANALYSIS METHODS .............................................................................................................................................. 35
3: ANALYSES OF ALABAMA MATH, SCIENCE, AND TECHNOLOGY INITIATIVE (AMSTI) IMPLEMENTATION ............ 65
ALABAMA MATH, SCIENCE, AND TECHNOLOGY INITIATIVE (AMSTI) IMPLEMENTATION COMPONENTS .......................................... 65
ALABAMA MATH, SCIENCE, AND TECHNOLOGY INITIATIVE (AMSTI) IMPLEMENTATION RESULTS .................................................. 66
CONCLUSIONS .............................................................................................................................................................. 79
4: EFFECT OF THE ALABAMA MATH, SCIENCE, AND TECHNOLOGY INITIATIVE (AMSTI) ON MATHEMATICS
PROBLEM SOLVING ACHIEVEMENT, SCIENCE ACHIEVEMENT, AND CLASSROOM INSTRUCTIONAL STRATEGIES
AFTER ONE YEAR ............................................................................................................................................... 81
EFFECTS OF THE ALABAMA MATH, SCIENCE, AND TECHNOLOGY INITIATIVE (AMSTI) ON STUDENT ACHIEVEMENT AFTER ONE YEAR .... 81
v
EFFECTS OF THE ALABAMA MATH, SCIENCE, AND TECHNOLOGY INITIATIVE (AMSTI) ON CLASSROOM PRACTICES AFTER ONE YEAR ..... 86
5: EFFECT ON MATHEMATICS PROBLEM SOLVING AND SCIENCE ACHIEVEMENT AFTER TWO YEARS ................... 91
EFFECT ON STANFORD ACHIEVEMENT TEST TENTH EDITION (SAT 10) MATHEMATICS PROBLEM SOLVING ACHIEVEMENT .................. 92
EFFECT ON STANFORD ACHIEVEMENT TEST TENTH EDITION (SAT 10) SCIENCE ACHIEVEMENT ..................................................... 94
6: EFFECTS OF THE ALABAMA MATH, SCIENCE, AND TECHNOLOGY INITIATIVE (AMSTI) ON READING
ACHIEVEMENT, TEACHER CONTENT KNOWLEDGE, STUDENT ENGAGEMENT, AND VARIATIONS IN EFFECTS
FOR STUDENT SUBGROUPS AFTER ONE YEAR .................................................................................................... 96
EFFECTS ON STUDENT ACHIEVEMENT IN READING ................................................................................................................ 97
EFFECTS ON TEACHER CONTENT KNOWLEDGE AND STUDENT ENGAGEMENT .............................................................................. 99
SUBGROUP DIFFERENCES IN AVERAGE EFFECTS OF THE ALABAMA MATH, SCIENCE, AND TECHNOLOGY INITIATIVE (AMSTI) ON
STUDENT ACHIEVEMENT AFTER ONE YEAR ........................................................................................................................ 103
7: SUMMARY OF FINDINGS AND STUDY LIMITATIONS ..................................................................................... 106
STUDY DESIGN ............................................................................................................................................................ 106
SUMMARY OF CONFIRMATORY FINDINGS ......................................................................................................................... 106
SUMMARY OF EXPLORATORY FINDINGS ............................................................................................................................ 107
STUDY LIMITATIONS .................................................................................................................................................... 109
APPENDIX A. EXPLANATION OF PRIMARY AND SECONDARY CONFIRMATORY OUTCOME MEASURES ............... A-1
APPENDIX B. EXPLANATION OF EXPLORATORY RESEARCH QUESTIONS ............................................................ B-1
EFFECTS OF TWO YEARS OF EXPOSURE TO THE ALABAMA MATH, SCIENCE, AND TECHNOLOGY INITIATIVE (AMSTI) ....................... B-1
EFFECT ON STUDENT ACHIEVEMENT IN READING ................................................................................................................ B-1
EFFECT ON TEACHER CONTENT KNOWLEDGE ..................................................................................................................... B-2
EFFECT ON STUDENT ENGAGEMENT ................................................................................................................................ B-3
EFFECTS ON SUBGROUPS OF STUDENTS............................................................................................................................ B-3
APPENDIX C. SELECTION AND RANDOM ASSIGNMENT OF SCHOOLS ................................................................ C-1
SUBEXPERIMENTS ....................................................................................................................................................... C-1
LIMITING THE POOL OF SCHOOLS FOR SELECTION ............................................................................................................... C-1
SELECTING AND ASSIGNING SCHOOLS .............................................................................................................................. C-1
THE REAL-TIME AND INTERACTIVE NATURE OF PAIR SELECTION ............................................................................................. C-3
APPENDIX D. STATISTICAL POWER ANALYSIS ................................................................................................... D-1
APPENDIX E. DATA COLLECTION PROCEDURES AND TIMELINE ..........................................................................E-1
APPENDIX F. DESCRIPTION OF PROGRAM IMPLEMENTATION DATA COLLECTED BUT NOT USED IN REPORT ...... F-1
vi
CLASSROOM OBSERVATIONS .......................................................................................................................................... F-1
PROFESSIONAL DEVELOPMENT OBSERVATIONS ................................................................................................................... F-3
PRINCIPAL SURVEYS ...................................................................................................................................................... F-4
APPENDIX G. ALABAMA MATH, SCIENCE, AND TECHNOLOGY INITIATIVE (AMSTI) TEACHER SURVEY #3 ........... G-1
APPENDIX H. DATA CLEANING AND DATA FILE CONSTRUCTION ....................................................................... H-1
STUDENT-LEVEL DATA .................................................................................................................................................. H-1
PROGRAM IMPLEMENTATION DATA ................................................................................................................................ H-1
APPENDIX I. ATTRITION THROUGH STUDY STAGES FOR SAMPLES USED IN THE CONFIRMATORY ANALYSIS ....... I-1
SELECTION OF THE STANFORD ACHIEVEMENT TEST TENTH EDITION (SAT 10) MATHEMATICS PROBLEM SOLVING OUTCOME
SAMPLE ..................................................................................................................................................................... I-1
SELECTION OF THE STANFORD ACHIEVEMENT TEST TENTH EDITION (SAT 10) ......................................................................... I-3
SCIENCE OUTCOME SAMPLE ........................................................................................................................................... I-3
SELECTION OF THE SAMPLE FOR THE ACTIVE LEARNING SCORE OUTCOME ................................................................................. I-6
APPENDIX J. DESCRIPTION OF DEGREE RANK .................................................................................................... J-1
APPENDIX K. EQUIVALENCE OF YEAR 1 BASELINE AND ANALYZED SAMPLES FOR CONFIRMATORY STUDENT-
LEVEL AND CLASSROOM PRACTICE OUTCOMES ............................................................................................... K-1
APPENDIX L. INTERNAL CONSISTENCY AND VALIDITY OF ACTIVE LEARNING MEASURES .................................... L-1
APPENDIX M. NUMBER OF STUDENTS AND TEACHERS IN SCHOOLS IN ANALYTIC SAMPLES USED TO ANALYZE
YEAR 1 CONFIRMATORY QUESTIONS .............................................................................................................. M-1
APPENDIX N. ATTRITION THROUGH STUDY STAGES FOR SAMPLES USED IN YEAR 1 EXPLORATORY ANALYSIS .. N-1
STANFORD ACHIEVEMENT TEST TENTH EDITION (SAT 10) READING OUTCOME ...................................................................... N-1
LEVEL OF TEACHER CONTENT KNOWLEDGE IN MATHEMATICS ............................................................................................... N-3
LEVEL OF STUDENT ENGAGEMENT IN MATHEMATICS .......................................................................................................... N-6
LEVEL OF STUDENT ENGAGEMENT IN SCIENCE ATTRITION ..................................................................................................... N-7
APPENDIX O. TESTS OF EQUIVALENCE FOR BASELINE AND ANALYTIC SAMPLES FOR YEAR 1 EXPLORATORY
OUTCOMES ..................................................................................................................................................... O-1
APPENDIX P. STATISTICAL POWER ANALYSES FOR MODERATOR ANALYSES ..................................................... P-1
APPENDIX Q. DERIVATION AND MOTIVATION OF THE BELL-BRADLEY ESTIMATOR WHEN MEASURING
ESTIMATED TWO-YEAR EFFECT OF THE ALABAMA MATH, SCIENCE, AND TECHNOLOGY INITIATIVE (AMSTI) ..... Q-1
vii
APPENDIX R. ATTRITION THROUGH STUDY STAGES FOR SAMPLES CONTRIBUTING TO ESTIMATION OF TWO-
YEAR EFFECTS ................................................................................................................................................. R-1
SELECTION OF THE STANFORD ACHIEVEMENT TEST TENTH EDITION (SAT 10) MATHEMATICS PROBLEM SOLVING SAMPLE ............... R-1
SELECTION OF THE STANFORD ACHIEVEMENT TEST TENTH EDITION (SAT 10) SCIENCE SAMPLE ................................................. R-5
APPENDIX S. EXAMINATION OF EQUIVALENCE IN BASELINE AND ANALYTIC SAMPLES USED IN THE
ESTIMATION OF TWO-YEAR EFFECTS ................................................................................................................ S-1
APPENDIX T. ESTIMATION MODEL FOR TWO-YEAR EFFECTS OF THE ALABAMA MATH, SCIENCE, AND
TECHNOLOGY INITIATIVE (AMSTI) ................................................................................................................... T-1
APPENDIX U. TOPICS AND INSTRUCTIONAL METHODS USED AT THE ALABAMA MATH, SCIENCE, AND
TECHNOLOGY INITIATIVE (AMSTI) SUMMER INSTITUTE ................................................................................... U-1
APPENDIX V. PARAMETER ESTIMATES ON PROBABILITY SCALE FOR ODDS-RATIO TESTS OF DIFFERENCES
BETWEEN ALABAMA MATH, SCIENCE, AND TECHNOLOGY INITIATIVE (AMSTI) AND CONTROL CONDITIONS IN
YEAR 1 (ASSOCIATED WITH SUMMER PROFESSIONAL DEVELOPMENT AND IN-SCHOOL SUPPORT OUTCOMES) V-1
APPENDIX W. DESCRIPTIVE STATISTICS FOR VARIABLES THAT CHANGE TO A BINARY SCALE USED IN THE
ALABAMA MATH, SCIENCE, AND TECHNOLOGY INITIATIVE (AMSTI) AND CONTROL CONDITIONS IN YEAR 1 .... W-1
APPENDIX X. COMPARISON OF ASSUMED PARAMETER VALUES AND OBSERVED SAMPLE STATISTICS FOR
STATISTICAL POWER ANALYSIS AFTER ONE YEAR ............................................................................................. X-1
APPENDIX Y. PARAMETER ESTIMATES FOR STANFORD ACHIEVEMENT TEST TENTH EDITION (SAT 10)
MATHEMATICS PROBLEM SOLVING AFTER ONE YEAR ...................................................................................... Y-1
APPENDIX Z. PARAMETER ESTIMATES FOR STANFORD ACHIEVEMENT TEST TENTH EDITION (SAT 10) SCIENCE
AFTER ONE YEAR .............................................................................................................................................. Z-1
APPENDIX AA. PARAMETER ESTIMATES FOR ACTIVE LEARNING IN MATHEMATICS AFTER ONE YEAR ............. AA-1
APPENDIX AB. PARAMETER ESTIMATES FOR ACTIVE LEARNING IN SCIENCE AFTER ONE YEAR ........................ AB-1
APPENDIX AC. SENSITIVITY ANALYSES OF EFFECT OF THE ALABAMA MATH, SCIENCE, AND TECHNOLOGY
INITIATIVE (AMSTI) ON STANFORD ACHIEVEMENT TEST TENTH EDITION (SAT 10) MATHEMATICS PROBLEM
SOLVING ACHIEVEMENT AFTER ONE YEAR ..................................................................................................... AC-1
APPENDIX AD. SENSITIVITY ANALYSES OF EFFECT OF THE ALABAMA MATH, SCIENCE, AND TECHNOLOGY
INITIATIVE (AMSTI) ON STANFORD ACHIEVEMENT TEST TENTH EDITION (SAT 10) SCIENCE ACHIEVEMENT
AFTER ONE YEAR ........................................................................................................................................... AD-1
viii
APPENDIX AE. SENSITIVITY ANALYSES OF EFFECT OF THE ALABAMA MATH, SCIENCE, AND TECHNOLOGY
INITIATIVE (AMSTI) ON ACTIVE LEARNING INSTRUCTIONAL STRATEGIES IN MATHEMATICS CLASSROOMS
AFTER ONE YEAR ........................................................................................................................................... AE-1
APPENDIX AF. SENSITIVITY ANALYSES OF EFFECT OF THE ALABAMA MATH, SCIENCE, AND TECHNOLOGY
INITIATIVE (AMSTI) ON ACTIVE LEARNING INSTRUCTIONAL STRATEGIES IN SCIENCE CLASSROOMS AFTER
ONE YEAR ..................................................................................................................................................... AF-1
APPENDIX AG. TESTS FOR VIOLATIONS OF FACTORS ASSOCIATED WITH ASSUMPTION OF EQUAL FIRST YEAR
EFFECTS ON STUDENTS IN ALABAMA MATH, SCIENCE, AND TECHNOLOGY INITIATIVE (AMSTI) AND CONTROL
SCHOOLS ...................................................................................................................................................... AG-1
SPONSOR’S GUIDELINES FOR THE PARAMETERS OF THE INTERVENTION ................................................................................. AG-2
DISTRICT SUPPORT .................................................................................................................................................... AG-3
COMPOSITION OF SCHOOLS THAT DID NOT IMPLEMENT ALABAMA MATH, SCIENCE, AND TECHNOLOGY INITIATIVE (AMSTI) ......... AG-3
CHARACTERISTICS OF TEACHERS WHO DID NOT PARTICIPATE IN ALABAMA MATH, SCIENCE, AND TECHNOLOGY INITIATIVE
(AMSTI) ................................................................................................................................................................ AG-9
EFFORTS BY SCHOOLS TO IMPLEMENT ALABAMA MATH, SCIENCE, AND TECHNOLOGY INITIATIVE (AMSTI) ................................ AG-9
CHARACTERISTICS OF STUDENT COHORTS ..................................................................................................................... AG-12
APPENDIX AH. POST HOC ADJUSTMENT TO STANDARD ERROR FOR ESTIMATE OF TWO-YEAR EFFECT OF THE
ALABAMA MATH, SCIENCE, AND TECHNOLOGY INITIATIVE (AMSTI) ON MATHEMATICS ACHIEVEMENT AFTER
TWO YEARS................................................................................................................................................... AH-1
APPENDIX AI. PARAMETER ESTIMATES FOR EFFECT OF THE ALABAMA MATH, SCIENCE, AND TECHNOLOGY
INITIATIVE (AMSTI) AFTER TWO YEARS ........................................................................................................... AI-1
APPENDIX AJ. PARAMETER ESTIMATES FOR EFFECT OF THE ALABAMA MATH, SCIENCE, AND TECHNOLOGY
INITIATIVE (AMSTI) ON STUDENT READING ACHIEVEMENT AFTER ONE YEAR .................................................. AJ-1
APPENDIX AK. PARAMETER ESTIMATES FOR TEACHER CONTENT AND STUDENT ENGAGEMENT AFTER ONE
YEAR ............................................................................................................................................................. AK-1
APPENDIX AL. ESTIMATES OF EFFECTS FOR TERMS INVOLVING THE INDICATOR OF TREATMENT STATUS IN
THE ANALYSIS OF THE MODERATING EFFECT OF THE THREE-LEVEL PRETEST VARIABLE ...................................AL-1
APPENDIX AM. PARAMETER ESTIMATES FOR THE ANALYSIS OF THE MODERATING EFFECT OF
RACIAL/ETHNIC MINORITY STATUS ON THE IMPACT OF THE ALABAMA MATH, SCIENCE, AND TECHNOLOGY
INITIATIVE (AMSTI) ON READING AFTER ONE YEAR ...................................................................................... AM-1
ix