How Much Value is Added by Value Added Models?

122 Pages·2017·1.81 MB·English

by  Mariana Ristea

CCiittyy UUnniivveerrssiittyy ooff NNeeww YYoorrkk ((CCUUNNYY))
CCUUNNYY AAccaaddeemmiicc WWoorrkkss
Dissertations, Theses, and Capstone Projects CUNY Graduate Center
2-2014
HHooww MMuucchh VVaalluuee iiss AAddddeedd bbyy VVaalluuee AAddddeedd MMooddeellss?? AAnn AAnnaallyyssiiss
ooff TTeeaacchheerrss’’ PPeerrffoorrmmaannccee OOvveerr TTiimmee UUssiinngg NNeeww YYoorrkk SSttaattee
AAsssseessssmmeenntt DDaattaa
Mariana Ristea
Graduate Center, City University of New York
How does access to this work benefit you? Let us know!
More information about this work at: https://academicworks.cuny.edu/gc_etds/1422
Discover additional works at: https://academicworks.cuny.edu
This work is made publicly available by the City University of New York (CUNY).
Contact: [e-mail]
i

Dissertation

HOW MUCH VALUE IS ADDED BY VALUE-ADDED MODELS?
An Analysis of Teachers’ Performance Over Time Using New York State Assessment Data

by

Mariana Ristea

A dissertation submitted to the Graduate Faculty in Educational Psychology in partial fulfillment
of the requirements for the degree of Doctor of Philosophy, The City University of New York
2014
ii

© 2014
Mariana Ristea
All Rights Reserved
iii

This manuscript has been read and accepted for the
Graduate Faculty in Educational Psychology in satisfaction of the
dissertation requirement for the degree of Doctor of Philosophy.

Dr. David Rindskopf
Date: ___________________ __________________________
Chair of Examining Committee

Alpana Bhattacharya
Date: ___________________ ____________________________
Executive Officer

Dr. David Rindskopf
Dr. Sophia Catsambis
Dr. Howard Everson
Supervisory Committee

THE CITY UNIVERSITY OF NEW YORK
iv

Abstract
How Much Value is Added by Value Added Models
by
Mariana Ristea
Adviser: Professor David Rindskopf
There is a strong movement to evaluate teachers on the basis of students’ performance. To
compare teachers fairly, as each may have a mixture of students with different abilities in a given
subject area, one should account for variables reflective of students’ subject knowledge and
background when entering a course. Most methods of control consist of highly sophisticated
statistical models mostly difficult to explain to educators who are being evaluated using such
methods. This research presents two value-added methods that could be replicated by using in-
house resources and standardized student assessment data which are either continuous or ordinal.
One method is simpler to implement if one’s goal is to evaluate teachers’ performance based on
students’ assessments scores reported as ordinal measures. The second method is similar to a
more typical value-added approach and uses hierarchical linear structures to determine a
classification of teachers’ performance based on their students’ assessment scores reported as
continuous measures. Teachers’ “value-added” in a given academic year is typically calculated
using students’ longitudinal New York State assessment data, reported in both ordinal and
continuous forms. Comparison of results obtained from both methods, along with their
interpretations, are used to examine trade-offs between accuracy of methods and their ease of use
and transparency. The code used is included for practitioners who may wish to replicate this
value-added methodology. Suggestions related to educational policy and feasibility of
implementation of methods are also discussed.
v

Table of Contents

I. Introduction …………………………………………………………………………. … 1

II. Background
II.A. What is value-added? ……………………………………………………...... 4
II.B. Value-added in education ………………………………………………....… 5

III. Instruments, Measurement and Issues in Measurement ………..……………..………. 9
IV. Growth and Value Added Models …………………………………………….............. 15
Limitations of Value-Added Models …………………… ..……………..……..... 43
V. Mathematics Assessment Data: Overview ……………………………………….…….. 45

VI. Questions …………………… ……………………………………………………….. 49

VII. Methodology….. …………………………………………………………………….... 50

VIII. Software for Growth Analysis ……………………………………….……………... . 71

IX. Results ………………………………………………………………………………… 72

X. Findings and Limitations …………………………………………………………….. 89
XI. Practical Significance of the Study and Further Research …………………………. 91
vi

Appendix 1 – SPSS Sorting Feature and Format Change ……………………………….. 95

Appendix 2 – Excel Procedure to Generate Counts for Cross Tables …………………… 96
Appendix 3 - NYS Math Weights used in Method 1 Procedure …………………… …… 96
Appendix 4 - Bootstrap Procedure for 2- and 3- score estimates ………………………… 98
References ….……..………………………………………………………………………. 105
vii

List of Tables

1. Quick Summary of Status and Growth Models ……………………………….... 16
2. Cohort, instruments and rounds for data collection for longitudinal analyses .... 45
3. NYS Performance indicators for standardized yearly math assessments……… 47
4. Variables for Multi-level Analyses …………..………………………………..... 48
5. Relation between Year 0 and Year 1 Scores for a 25-student Class …………… 54
6. Observed Student Score Counts by Year ……………………………………….. 54
7. Weighted Averages by Year …….……………………………………………..... 55
8. Calculated Teacher Performance at the End of Year 1………………………….. 56
9. Adjusted Weighted Teacher Performance at the End of Year 1………………….... 57
10. Descriptive Statistics for Continuous Student Pre and Post Scores …………….… 60
11. Correlation table among student level variable considered for HLM analyses …... 63
12. Summary of Multilevel Models under Consideration ………………………….…. 65
13. 3-Score Category Means for Teachers with 95% and 84% Confidence Intervals in
Ascending Order …………………………………………………………………... 73
14. 2-Score Category Means for Teachers with 95% and 84% Confidence
Intervals in Ascending Order ….……………………..……………………….….… 76
15. Model 1- Final estimation of fixed effects………………………………………... 78
16. Model 1 - Final estimation of variance components ………………………….…. 78
17. Model 2 - Final estimation of fixed effects ………………………………….…... 79
18. Model 2 - Final estimation of variance components ……………………….……. 79
19. Results of Pre-score only model with 84% confidence intervals ……………….... 81
viii

20. Teachers’ 2- and 3- score Average Correlations …………………………….…… 83
21. Side-by-side Mean Comparisons for Teachers with Students in 2- and 3-score
categories ………………………………………………………………………….. 84
22. Side-by-side Mean Comparisons for teachers with students in 2- and 3-score categories
with unadjusted end of year average included ………………………………… 85
ix

List of Diagrams

1. Evaluating New York Teachers, Perhaps the Numbers Do Lie ………………….……… 3
2. Chart of Confidence Intervals with Included Teachers’ Class Averages at the End
of the Year and Overlapping 84% and 95% Confidence Intervals for Continuous
Scores ………………………………………………………………………...……......... 74
3. Q-Q Plot to Check Normality for Students Continuous Scores ………………………. 77
4. Teacher Ranking based on Pre-Score Model only for Continuous Scores ……………… 82
5. Correlation between the adjusted and the unadjusted 3-score teacher averages ……….. 86
6. Correlations between 3-scores, 2-scores and unadjusted averages for teachers’ scores …… 87
7. Raw Means Diagram for Teacher Rankings with Scores in Two Categories …..…………….. 88
8. Bootstrapped Means Diagram for Teacher Rankings with Scores in Two Categories ………... 88