ebook img

Evaluation of the Effectiveness of the Alabama Math, Science PDF

314 Pages·2012·2.65 MB·English
by  
Save to my drive
Quick download
Download
Most books are stored in the elastic cloud where traffic is expensive. For this reason, we have a limit on daily download.

Preview Evaluation of the Effectiveness of the Alabama Math, Science

NCEE 2012-4008 U. S. DEPARTMENT OF EDUCATION Evaluation of the Effectiveness of the Alabama Math, Science, and Technology Initiative (AMSTI) Evaluation of the Effectiveness of the Alabama Math, Science, and Technology Initiative (AMSTI) Final Report February 2012 Authors: Denis Newman, Empirical Education Inc., Co-Principal Investigator Pamela B. Finney, Research Management Leader, Regional Educational Laboratory Southeast, Co-Principal Investigator Steve Bell, Abt Associates Inc., Task 2 Methodology Leader Herb Turner, ANALYTICA, Technical Consultant Andrew P. Jaciw, Empirical Education Inc., Director of Experimental Design and Analysis Jenna L. Zacamy, Empirical Education Inc., Senior Research Manager Laura Feagans Gould, Academy for Educational Development, Senior Research Scientist Project Officer: Sandra Garcia, Institute of Education Sciences NCEE 2012-4008 Richard Sawyer, Academy for Educational Development, was the original co-principal investigator. U.S. Department of Education Arne Duncan Secretary Institute of Education Sciences John Q. Easton Director National Center for Education Evaluation and Regional Assistance Rebecca A. Maynard Commissioner February 2012 This report was prepared for the National Center for Education Evaluation and Regional Assistance, Institute of Education Sciences, under contract ED–06-CO–0028. IES evaluation reports present objective information on the conditions of implementation and impacts of the programs being evaluated. IES evaluation reports do not include conclusions or recommendations or views with regard to actions policymakers or practitioners should take in light of the findings in the report. This report is in the public domain. Authorization to reproduce it in whole or in part is granted. While permission to reprint this publication is not necessary, the citation should read: Newman, D., Finney, P.B., Bell, S., Turner, H., Jaciw, A.P., Zacamy, J.L., & Feagans Gould, L. (2012). Evaluation of the Effectiveness of the Alabama Math, Science, and Technology Initiative (AMSTI). (NCEE 2012–4008). Washington, DC: National Center for Education Evaluation and Regional Assistance, Institute of Education Sciences, U.S. Department of Education. This report is available on the Institute of Education Sciences website at http://ncee.ed.gov and the Regional Educational Laboratory Program website at http://edlabs.ed.gov. Alternate Formats Upon request, this report is available in alternate formats, such as Braille, large print, audiotape, or computer diskette. For more information, please contact the Department’s Alternate Format Center at 202-260-9895 or 202-205-8113. Disclosure of potential conflict of interest None of the authors or other staff involved in the study from Abt Associates, Academy for Educational Development, ANALYTICA, Empirical Education Inc., or the Regional Educational Laboratory-Southeast at SERVE Center at the University of North Carolina at Greensboro has financial interests that could be affected by the content of this report.1 No one on the nine- member Technical Working Group, convened twice annually by the research team to provide advice and guidance, has financial interests that could be affected by the study findings. 1 Contractors carrying out research and evaluation projects for IES frequently need to obtain expert advice and technical assistance from individuals and entities whose other professional work may not be entirely independent of or separable from the tasks they are carrying out for the IES contractor. Contractors endeavor not to put such individuals or entities in positions in which they could bias the analysis and reporting of results, and their potential conflicts of interest are disclosed. iii Acknowledgments This study was the collaborative effort of a team that included professional staff at the Regional Educational Laboratory-Southeast at SERVE Center at the University of North Carolina at Greensboro; Empirical Education Inc.; Abt Associates Inc.; and the Academy for Educational Development (AED). From the Regional Educational Laboratory-Southeast (SERVE), the authors acknowledge Jean Scott, senior education policy research analyst, for assistance in maintaining a strong relationship with the Alabama State Department of Education (ALSDE); David Schaefer, technical editor and advisor; Melissa Williams, evaluation specialist, and De Fuller, administrative support. From Empirical Education Inc., the authors acknowledge Laurel Sterling, Margit Zsolnay, and Ruthie Chang for their contributions to data collection, project management, and report writing, and Boya Ma, Qingfeng (Breeze) Zhao, Akiko Lipton, and Xiaohui Zheng, for planning and conducting the statistical analyses. From Abt, the authors acknowledge Rob Olsen, for his valuable advice on statistical modeling, and Cay Bradley, for her collaboration on the analysis of the two-year impact. From AED, the authors acknowledge Harold Wenglensky, Richard Sawyer, Amy Detgen, Robert Bozick, and Jason Snipes, for their leadership in earlier phases of the project. Finally, the authors express their deep gratitude to the people at ALSDE and administrators and teachers in the participating school systems for their assistance and cooperation in conducting this research. iv CONTENTS DISCLOSURE OF POTENTIAL CONFLICT OF INTEREST .......................................................................................... III ACKNOWLEDGMENTS ....................................................................................................................................... IV SUMMARY ....................................................................................................................................................... XX 1: INTRODUCTION AND STUDY OVERVIEW ......................................................................................................... 1 STRENGTHENING SKILLS OF MATHEMATICS AND SCIENCE TEACHERS NATIONWIDE......................................................................... 1 THE ALABAMA MATH, SCIENCE, AND TECHNOLOGY INITIATIVE (AMSTI): A STATE-LEVEL STRATEGY TO STRENGTHEN SKILLS OF MATHEMATICS AND SCIENCE TEACHERS ............................................................................................................................... 2 ALABAMA MATH, SCIENCE, AND TECHNOLOGY INITIATIVE (AMSTI) THEORY OF ACTION .............................................................. 3 ALABAMA MATH, SCIENCE, AND TECHNOLOGY INITIATIVE (AMSTI) PROGRAM COMPONENTS AND ELEMENTS ................................. 5 PRIOR RESEARCH ON THE EFFECTIVENESS OF THE ALABAMA MATH, SCIENCE, AND TECHNOLOGY INITIATIVE (AMSTI) ON STUDENT ACHIEVEMENT ................................................................................................................................................................ 6 OVERVIEW OF STUDY DESIGN ............................................................................................................................................ 7 RESEARCH QUESTIONS ..................................................................................................................................................... 9 STRUCTURE OF THE REPORT ............................................................................................................................................ 12 2: STUDY DESIGN AND METHODOLOGY ............................................................................................................. 13 RATIONALE FOR EXPERIMENTAL DESIGN ............................................................................................................................. 13 RECRUITMENT, SELECTION, AND RANDOM ASSIGNMENT OF SCHOOLS ...................................................................................... 14 INCENTIVES TO PARTICIPATE ............................................................................................................................................ 20 DEFINING THE TERM “ALABAMA MATH, SCIENCE, AND TECHNOLOGY INITIATIVE (AMSTI) TEACHER” ........................................... 20 DATA SOURCES, COLLECTION, AND CLEANING ..................................................................................................................... 21 THREATS TO INTERNAL VALIDITY ....................................................................................................................................... 30 DATA ANALYSIS METHODS .............................................................................................................................................. 35 3: ANALYSES OF ALABAMA MATH, SCIENCE, AND TECHNOLOGY INITIATIVE (AMSTI) IMPLEMENTATION ............ 65 ALABAMA MATH, SCIENCE, AND TECHNOLOGY INITIATIVE (AMSTI) IMPLEMENTATION COMPONENTS .......................................... 65 ALABAMA MATH, SCIENCE, AND TECHNOLOGY INITIATIVE (AMSTI) IMPLEMENTATION RESULTS .................................................. 66 CONCLUSIONS .............................................................................................................................................................. 79 4: EFFECT OF THE ALABAMA MATH, SCIENCE, AND TECHNOLOGY INITIATIVE (AMSTI) ON MATHEMATICS PROBLEM SOLVING ACHIEVEMENT, SCIENCE ACHIEVEMENT, AND CLASSROOM INSTRUCTIONAL STRATEGIES AFTER ONE YEAR ............................................................................................................................................... 81 EFFECTS OF THE ALABAMA MATH, SCIENCE, AND TECHNOLOGY INITIATIVE (AMSTI) ON STUDENT ACHIEVEMENT AFTER ONE YEAR .... 81 v EFFECTS OF THE ALABAMA MATH, SCIENCE, AND TECHNOLOGY INITIATIVE (AMSTI) ON CLASSROOM PRACTICES AFTER ONE YEAR ..... 86 5: EFFECT ON MATHEMATICS PROBLEM SOLVING AND SCIENCE ACHIEVEMENT AFTER TWO YEARS ................... 91 EFFECT ON STANFORD ACHIEVEMENT TEST TENTH EDITION (SAT 10) MATHEMATICS PROBLEM SOLVING ACHIEVEMENT .................. 92 EFFECT ON STANFORD ACHIEVEMENT TEST TENTH EDITION (SAT 10) SCIENCE ACHIEVEMENT ..................................................... 94 6: EFFECTS OF THE ALABAMA MATH, SCIENCE, AND TECHNOLOGY INITIATIVE (AMSTI) ON READING ACHIEVEMENT, TEACHER CONTENT KNOWLEDGE, STUDENT ENGAGEMENT, AND VARIATIONS IN EFFECTS FOR STUDENT SUBGROUPS AFTER ONE YEAR .................................................................................................... 96 EFFECTS ON STUDENT ACHIEVEMENT IN READING ................................................................................................................ 97 EFFECTS ON TEACHER CONTENT KNOWLEDGE AND STUDENT ENGAGEMENT .............................................................................. 99 SUBGROUP DIFFERENCES IN AVERAGE EFFECTS OF THE ALABAMA MATH, SCIENCE, AND TECHNOLOGY INITIATIVE (AMSTI) ON STUDENT ACHIEVEMENT AFTER ONE YEAR ........................................................................................................................ 103 7: SUMMARY OF FINDINGS AND STUDY LIMITATIONS ..................................................................................... 106 STUDY DESIGN ............................................................................................................................................................ 106 SUMMARY OF CONFIRMATORY FINDINGS ......................................................................................................................... 106 SUMMARY OF EXPLORATORY FINDINGS ............................................................................................................................ 107 STUDY LIMITATIONS .................................................................................................................................................... 109 APPENDIX A. EXPLANATION OF PRIMARY AND SECONDARY CONFIRMATORY OUTCOME MEASURES ............... A-1 APPENDIX B. EXPLANATION OF EXPLORATORY RESEARCH QUESTIONS ............................................................ B-1 EFFECTS OF TWO YEARS OF EXPOSURE TO THE ALABAMA MATH, SCIENCE, AND TECHNOLOGY INITIATIVE (AMSTI) ....................... B-1 EFFECT ON STUDENT ACHIEVEMENT IN READING ................................................................................................................ B-1 EFFECT ON TEACHER CONTENT KNOWLEDGE ..................................................................................................................... B-2 EFFECT ON STUDENT ENGAGEMENT ................................................................................................................................ B-3 EFFECTS ON SUBGROUPS OF STUDENTS............................................................................................................................ B-3 APPENDIX C. SELECTION AND RANDOM ASSIGNMENT OF SCHOOLS ................................................................ C-1 SUBEXPERIMENTS ....................................................................................................................................................... C-1 LIMITING THE POOL OF SCHOOLS FOR SELECTION ............................................................................................................... C-1 SELECTING AND ASSIGNING SCHOOLS .............................................................................................................................. C-1 THE REAL-TIME AND INTERACTIVE NATURE OF PAIR SELECTION ............................................................................................. C-3 APPENDIX D. STATISTICAL POWER ANALYSIS ................................................................................................... D-1 APPENDIX E. DATA COLLECTION PROCEDURES AND TIMELINE ..........................................................................E-1 APPENDIX F. DESCRIPTION OF PROGRAM IMPLEMENTATION DATA COLLECTED BUT NOT USED IN REPORT ...... F-1 vi CLASSROOM OBSERVATIONS .......................................................................................................................................... F-1 PROFESSIONAL DEVELOPMENT OBSERVATIONS ................................................................................................................... F-3 PRINCIPAL SURVEYS ...................................................................................................................................................... F-4 APPENDIX G. ALABAMA MATH, SCIENCE, AND TECHNOLOGY INITIATIVE (AMSTI) TEACHER SURVEY #3 ........... G-1 APPENDIX H. DATA CLEANING AND DATA FILE CONSTRUCTION ....................................................................... H-1 STUDENT-LEVEL DATA .................................................................................................................................................. H-1 PROGRAM IMPLEMENTATION DATA ................................................................................................................................ H-1 APPENDIX I. ATTRITION THROUGH STUDY STAGES FOR SAMPLES USED IN THE CONFIRMATORY ANALYSIS ....... I-1 SELECTION OF THE STANFORD ACHIEVEMENT TEST TENTH EDITION (SAT 10) MATHEMATICS PROBLEM SOLVING OUTCOME SAMPLE ..................................................................................................................................................................... I-1 SELECTION OF THE STANFORD ACHIEVEMENT TEST TENTH EDITION (SAT 10) ......................................................................... I-3 SCIENCE OUTCOME SAMPLE ........................................................................................................................................... I-3 SELECTION OF THE SAMPLE FOR THE ACTIVE LEARNING SCORE OUTCOME ................................................................................. I-6 APPENDIX J. DESCRIPTION OF DEGREE RANK .................................................................................................... J-1 APPENDIX K. EQUIVALENCE OF YEAR 1 BASELINE AND ANALYZED SAMPLES FOR CONFIRMATORY STUDENT- LEVEL AND CLASSROOM PRACTICE OUTCOMES ............................................................................................... K-1 APPENDIX L. INTERNAL CONSISTENCY AND VALIDITY OF ACTIVE LEARNING MEASURES .................................... L-1 APPENDIX M. NUMBER OF STUDENTS AND TEACHERS IN SCHOOLS IN ANALYTIC SAMPLES USED TO ANALYZE YEAR 1 CONFIRMATORY QUESTIONS .............................................................................................................. M-1 APPENDIX N. ATTRITION THROUGH STUDY STAGES FOR SAMPLES USED IN YEAR 1 EXPLORATORY ANALYSIS .. N-1 STANFORD ACHIEVEMENT TEST TENTH EDITION (SAT 10) READING OUTCOME ...................................................................... N-1 LEVEL OF TEACHER CONTENT KNOWLEDGE IN MATHEMATICS ............................................................................................... N-3 LEVEL OF STUDENT ENGAGEMENT IN MATHEMATICS .......................................................................................................... N-6 LEVEL OF STUDENT ENGAGEMENT IN SCIENCE ATTRITION ..................................................................................................... N-7 APPENDIX O. TESTS OF EQUIVALENCE FOR BASELINE AND ANALYTIC SAMPLES FOR YEAR 1 EXPLORATORY OUTCOMES ..................................................................................................................................................... O-1 APPENDIX P. STATISTICAL POWER ANALYSES FOR MODERATOR ANALYSES ..................................................... P-1 APPENDIX Q. DERIVATION AND MOTIVATION OF THE BELL-BRADLEY ESTIMATOR WHEN MEASURING ESTIMATED TWO-YEAR EFFECT OF THE ALABAMA MATH, SCIENCE, AND TECHNOLOGY INITIATIVE (AMSTI) ..... Q-1 vii APPENDIX R. ATTRITION THROUGH STUDY STAGES FOR SAMPLES CONTRIBUTING TO ESTIMATION OF TWO- YEAR EFFECTS ................................................................................................................................................. R-1 SELECTION OF THE STANFORD ACHIEVEMENT TEST TENTH EDITION (SAT 10) MATHEMATICS PROBLEM SOLVING SAMPLE ............... R-1 SELECTION OF THE STANFORD ACHIEVEMENT TEST TENTH EDITION (SAT 10) SCIENCE SAMPLE ................................................. R-5 APPENDIX S. EXAMINATION OF EQUIVALENCE IN BASELINE AND ANALYTIC SAMPLES USED IN THE ESTIMATION OF TWO-YEAR EFFECTS ................................................................................................................ S-1 APPENDIX T. ESTIMATION MODEL FOR TWO-YEAR EFFECTS OF THE ALABAMA MATH, SCIENCE, AND TECHNOLOGY INITIATIVE (AMSTI) ................................................................................................................... T-1 APPENDIX U. TOPICS AND INSTRUCTIONAL METHODS USED AT THE ALABAMA MATH, SCIENCE, AND TECHNOLOGY INITIATIVE (AMSTI) SUMMER INSTITUTE ................................................................................... U-1 APPENDIX V. PARAMETER ESTIMATES ON PROBABILITY SCALE FOR ODDS-RATIO TESTS OF DIFFERENCES BETWEEN ALABAMA MATH, SCIENCE, AND TECHNOLOGY INITIATIVE (AMSTI) AND CONTROL CONDITIONS IN YEAR 1 (ASSOCIATED WITH SUMMER PROFESSIONAL DEVELOPMENT AND IN-SCHOOL SUPPORT OUTCOMES) V-1 APPENDIX W. DESCRIPTIVE STATISTICS FOR VARIABLES THAT CHANGE TO A BINARY SCALE USED IN THE ALABAMA MATH, SCIENCE, AND TECHNOLOGY INITIATIVE (AMSTI) AND CONTROL CONDITIONS IN YEAR 1 .... W-1 APPENDIX X. COMPARISON OF ASSUMED PARAMETER VALUES AND OBSERVED SAMPLE STATISTICS FOR STATISTICAL POWER ANALYSIS AFTER ONE YEAR ............................................................................................. X-1 APPENDIX Y. PARAMETER ESTIMATES FOR STANFORD ACHIEVEMENT TEST TENTH EDITION (SAT 10) MATHEMATICS PROBLEM SOLVING AFTER ONE YEAR ...................................................................................... Y-1 APPENDIX Z. PARAMETER ESTIMATES FOR STANFORD ACHIEVEMENT TEST TENTH EDITION (SAT 10) SCIENCE AFTER ONE YEAR .............................................................................................................................................. Z-1 APPENDIX AA. PARAMETER ESTIMATES FOR ACTIVE LEARNING IN MATHEMATICS AFTER ONE YEAR ............. AA-1 APPENDIX AB. PARAMETER ESTIMATES FOR ACTIVE LEARNING IN SCIENCE AFTER ONE YEAR ........................ AB-1 APPENDIX AC. SENSITIVITY ANALYSES OF EFFECT OF THE ALABAMA MATH, SCIENCE, AND TECHNOLOGY INITIATIVE (AMSTI) ON STANFORD ACHIEVEMENT TEST TENTH EDITION (SAT 10) MATHEMATICS PROBLEM SOLVING ACHIEVEMENT AFTER ONE YEAR ..................................................................................................... AC-1 APPENDIX AD. SENSITIVITY ANALYSES OF EFFECT OF THE ALABAMA MATH, SCIENCE, AND TECHNOLOGY INITIATIVE (AMSTI) ON STANFORD ACHIEVEMENT TEST TENTH EDITION (SAT 10) SCIENCE ACHIEVEMENT AFTER ONE YEAR ........................................................................................................................................... AD-1 viii APPENDIX AE. SENSITIVITY ANALYSES OF EFFECT OF THE ALABAMA MATH, SCIENCE, AND TECHNOLOGY INITIATIVE (AMSTI) ON ACTIVE LEARNING INSTRUCTIONAL STRATEGIES IN MATHEMATICS CLASSROOMS AFTER ONE YEAR ........................................................................................................................................... AE-1 APPENDIX AF. SENSITIVITY ANALYSES OF EFFECT OF THE ALABAMA MATH, SCIENCE, AND TECHNOLOGY INITIATIVE (AMSTI) ON ACTIVE LEARNING INSTRUCTIONAL STRATEGIES IN SCIENCE CLASSROOMS AFTER ONE YEAR ..................................................................................................................................................... AF-1 APPENDIX AG. TESTS FOR VIOLATIONS OF FACTORS ASSOCIATED WITH ASSUMPTION OF EQUAL FIRST YEAR EFFECTS ON STUDENTS IN ALABAMA MATH, SCIENCE, AND TECHNOLOGY INITIATIVE (AMSTI) AND CONTROL SCHOOLS ...................................................................................................................................................... AG-1 SPONSOR’S GUIDELINES FOR THE PARAMETERS OF THE INTERVENTION ................................................................................. AG-2 DISTRICT SUPPORT .................................................................................................................................................... AG-3 COMPOSITION OF SCHOOLS THAT DID NOT IMPLEMENT ALABAMA MATH, SCIENCE, AND TECHNOLOGY INITIATIVE (AMSTI) ......... AG-3 CHARACTERISTICS OF TEACHERS WHO DID NOT PARTICIPATE IN ALABAMA MATH, SCIENCE, AND TECHNOLOGY INITIATIVE (AMSTI) ................................................................................................................................................................ AG-9 EFFORTS BY SCHOOLS TO IMPLEMENT ALABAMA MATH, SCIENCE, AND TECHNOLOGY INITIATIVE (AMSTI) ................................ AG-9 CHARACTERISTICS OF STUDENT COHORTS ..................................................................................................................... AG-12 APPENDIX AH. POST HOC ADJUSTMENT TO STANDARD ERROR FOR ESTIMATE OF TWO-YEAR EFFECT OF THE ALABAMA MATH, SCIENCE, AND TECHNOLOGY INITIATIVE (AMSTI) ON MATHEMATICS ACHIEVEMENT AFTER TWO YEARS................................................................................................................................................... AH-1 APPENDIX AI. PARAMETER ESTIMATES FOR EFFECT OF THE ALABAMA MATH, SCIENCE, AND TECHNOLOGY INITIATIVE (AMSTI) AFTER TWO YEARS ........................................................................................................... AI-1 APPENDIX AJ. PARAMETER ESTIMATES FOR EFFECT OF THE ALABAMA MATH, SCIENCE, AND TECHNOLOGY INITIATIVE (AMSTI) ON STUDENT READING ACHIEVEMENT AFTER ONE YEAR .................................................. AJ-1 APPENDIX AK. PARAMETER ESTIMATES FOR TEACHER CONTENT AND STUDENT ENGAGEMENT AFTER ONE YEAR ............................................................................................................................................................. AK-1 APPENDIX AL. ESTIMATES OF EFFECTS FOR TERMS INVOLVING THE INDICATOR OF TREATMENT STATUS IN THE ANALYSIS OF THE MODERATING EFFECT OF THE THREE-LEVEL PRETEST VARIABLE ...................................AL-1 APPENDIX AM. PARAMETER ESTIMATES FOR THE ANALYSIS OF THE MODERATING EFFECT OF RACIAL/ETHNIC MINORITY STATUS ON THE IMPACT OF THE ALABAMA MATH, SCIENCE, AND TECHNOLOGY INITIATIVE (AMSTI) ON READING AFTER ONE YEAR ...................................................................................... AM-1 ix

Description:
NCEE 2012-4008. U. S. DEPARTMENT OF EDUCATION. Evaluation of the Effectiveness of the Alabama Math, Science, and Technology Initiative.
See more

The list of books you might like

Most books are stored in the elastic cloud where traffic is expensive. For this reason, we have a limit on daily download.