ERIC ED562709: A Bayesian Nonparametric Causal Model for Regression Discontinuity Designs

2013·0.17 MB·English

by  ERIC

Abstract Title Page
A Bayesian Nonparametric Causal Model for Regression Discontinuity Designs1
George Karabatsos
University of Illinois-Chicago
Department of Educational Psychology
e-mail: [email protected]
and
Stephen G. Walker
The University of Texas at Austin
Division of Statistics and Scienti(cid:133)c Computation
September 30, 2013
First preference: Section on Research Methods.
Second preference: Section on Understanding the E⁄ects of Education Policies.
1This research is supported by a National Science Foundation Grant SES-1156372, from the Program in
Methodology, Measurement, and Statistics.
SREE Spring 2014 Conference Abstract
A Bayesian Nonparametric Causal Model for Regression Discontinuity Designs
1 Background, Purpose and Novelty of Study
The regression discontinuity (RD) design (Thistlewaite & Campbell, 1960; Cook, 2008) pro-
videsaframeworktoidentifyandestimatecausale⁄ectsfromanon-randomizeddesign. Each
subject of a RD design is assigned to the treatment (versus assignment to a non-treatment)
whenever her/his observed value of an the assignment variable equals or exceeds a cuto⁄
value. The RD design provides a "locally-randomized experiment" under remarkably mild
conditions, so that the causal e⁄ect of treatment outcomes versus non-treatment outcomes
can be identi(cid:133)ed and estimated at the cuto⁄ (Lee, 2008). Such e⁄ect estimates are similar
to those of a randomized study (Goldberger, 2008/1972). As a result, since 1997, at least 74
RD-based empirical studies have emerged in the (cid:133)elds of education, political science, psy-
chology, economics, statistics, criminology, and health science (see van der Klaauw, 2008;
Lee & Lemieux, 2010; Bloom, 2012; Wong et al. 2013; Li et al., 2013). Polynomial and
local linear models are standard for RD designs (Bloom, 2012; Imbens & Lemieux, 2008).
However, these models can produce biased causal e⁄ect estimates, due to the presence of
outliers of treatment outcomes; and/or due to incorrect choices of the bandwidth parameter
for the local linear model. Currently, the correct choice of bandwidth has only been justi-
(cid:133)ed by large-sample theory (Imbens & Kalyanaraman, 2012), and the local linear model for
quantile regression (Frandsen et al., 2012) su⁄ers from the "quantile crossing" problem.
WeintroduceanovelformulationofourBayesiannonparametricregressionmodel(BLIND,
2012), which provides causal inference for RD designs. It is an in(cid:133)nite-mixture model, that
allows the entire probability density of the outcome variable to change (cid:135)exibly as a function
of the assignment variable. Moreover, our Bayesian model can provide inferences of causal
e⁄ects, in terms of how the treatment variable impacts the mean, variance, a quantile, distri-
bution function, probability density, hazard function, and/or any other chosen functional of
theoutcomevariable. Moreover,theaccuratecausale⁄ectestimationreliesonapredictively-
accurate model for the data. The Bayesian nonparametric regression model attained best
overall predictive performance, over many real data sets, compared to many other regression
models (BLIND, 2012). Finally, we will illustrate our Bayesian model through the causal
analysis of two real educational data sets.
2 Identifying Causal E⁄ects in a RD Research Design
In a RD design, let R be a continuous-valued assignment variable (Berk & Rauma, 1983)
i
havingaknowncuto⁄r , foreachsubjecti. Theninsuchadesign, thetreatmentassignment
0
mechanism is de(cid:133)ned by A(Ri) = 1(R r ), with a realization denoted by a(ri) = 1(r r ),
r0 i (cid:21) 0 r0 i (cid:21) 0
where 1( ) is the indicator function. In the sharp RD design (Thistlewaite & Campbell,
(cid:1)
1960), thetreatmentreceiptprobabilityfunction isde(cid:133)nedbyPr(T = 1 R = r) = 1(r r ),
0
j (cid:21)
and thus it has a discontinuous jump of 1 at r . In a fuzzy RD design (Trochim, 1984), the
0
probability function Pr(T = 1 R = r) has a discontinuous jump that is smaller than 1, at
j
r . This smaller jump is a result of imperfect treatment compliance, which can occur in
0
settings where the assignment variable R measures the eligibility to receive a treatment, and
some ineligible subjects (with R < r ) decided to receive treatment (i.e., T = 1), and some
i 0 i
eligible subjects (with R r ) decided receive the non-treatment (i.e., T = 0).
i 0 i
(cid:21)
For each subject of a given RD study, indexed by i = 1;:::;n, let T (A(Ri)) = 1 indicate
i r0
SREE Spring 2014 Conference Abstract, September 30, 2013 1
A Bayesian Nonparametric Causal Model for Regression Discontinuity Designs
receipt of the treatment, and let T (A(Ri)) = 0 indicates receipt of the non-treatment, when
i r0
assigned treatment A(Ri) t = 0;1 . Also, denote Y (A(Rn);T(A(Rn))) as the 22n potential
r0 2 f g i r0 r0
outcomes to treatments that de(cid:133)ned a common time point, for all R = (R ;:::;R ) ;
n 1 n >
A(Rn) = (A(R1);:::;A(Rn)) , and all T(A(Rn)) = (T (A(R1));:::;T (A(Rn))) (e.g., Angrist,
r0 r0 r0 > r0 1 r0 n r0 >
et al., 1996). Now, suppose that data of a RD design satis(cid:133)es the following (cid:133)ve assump-
tions: RD: the existence of limits lim E(T r) = lim E(T r) (Hahn, et al. 2001); Local
SUTVA (LS): Y (A(Rn0);T(A(Rn0)))r="r0Y (Aj(Ri);6 T (A(rR#ri)0)) forj all n subjects with r near
i r0 r0 i r0 i r0 0 i
r (Cattaneo et al. 2013); Local Exclusion Restriction (LER): Y (A(Rn0);T(A(Rn0))) =
0 i r0 r0
(R ) (R )
Y (A n0 ;T(A n0 )) for all (R ;R ) and for all n subjects with r near r (e.g., Angrist,
i r0 r0 n0 n0 0 i 0
et al., 1996); Local Monotonicity (LM): Ti(A(rr00+(cid:15))) (cid:21) Ti(A(rr00(cid:0)(cid:15))) for some (cid:15) > 0 and for every
subject i with r (r (cid:15);r + (cid:15)) (Hahn, et al. 2001); Local Randomization (LR): Each
i 0 0
2 (cid:0)
subject, indexedbyw, has"imprecisecontrol"overR, i.e., F (r w) = Pr(R r w)iscontin-
R
j (cid:20) j
uous in r at r , with 0 < F (r w) < 1 (Lee, 2008). Then, for the subgroup of subjects with
0 R 0
j
assignment variables r near r , a complier is a subject with (T (1);T (0)) = (0;1); the 22n0
i 0 i i
potential outcomes Y (A(Rn0);T(A(Rn0))) reduce to two potential outcomes Y (t);t = 0;1;
i r0 r0 i
and then Y(1) Y(0) is a causal e⁄ect of T on Y; and for any functional h of Y, an
(cid:0) f(cid:1)g
estimator of the causal e⁄ect of T on h Y ; at the cuto⁄r , is given by:
0
f g
lim E[h Y r] lim E[h Y r]
(cid:28) = E[h Yi(1) h Yi(0) r0 and i is a complier] = r#r0 f gj (cid:0) r"r0 f gj
f g(cid:0) f gj Pr[i is a complier r ]
0
j
(1)
(Imbens & Lemieux, 2008). Depending on the choice of h , the causal e⁄ect estimator
f(cid:1)g
(1) describes how much the treatment T impacts either the mean, variance, distribution
function, a quantile, probability density, or any other feature of the outcome h Y . The
f g
denominator of (1) is identical to lim E[T r] lim E[T r]; and a sharp RD design has
r#r0 j (cid:0) r"r0 j
Pr[i is a complier r ] = 1, and trivially satis(cid:133)es assumptions RD, LER, and LM. Also, the
0
j
two data sets that will analyzed in the the present study, satis(cid:133)es assumption LR, because
arguably for each data set, each subject has imprecise control over the assignment variable.
3 A Statistical (Causal) Model for RD Designs
For the sharp RD design, our Bayesian nonparametric model is given by:
f(y r ;a(ri)) = 1 n(y (cid:22) ;(cid:27)2)! ((cid:17) (r );(cid:27) (r )); i = 1;:::;n, (2a)
ij i r0 ij j j j ! i ! i
j=
(cid:0)1
! ((cid:17) (r);(cid:27) (r)) = (cid:8)P( j (cid:17) (r) =(cid:27) (r)) (cid:8)( j 1 (cid:17) (r) =(cid:27) (r)) (2b)
j ! ! ! ! ! !
f (cid:0) g (cid:0) f (cid:0) (cid:0) g
(cid:17) (r) = (cid:12) +(cid:12) r+(cid:12) a(r) (2c)
! 0! !1 !2 r0
(cid:27) (r) = exp((cid:21) +(cid:21) r+(cid:21) a(r))1=2 (2d)
! !0 !1 !2 r0
((cid:22) ;(cid:27)2) normal((cid:22) (cid:22) ;(cid:27)2)inverseGamma((cid:27)2 1;b ) (2e)
j j (cid:24) jj (cid:22) (cid:22) jj (cid:27)
((cid:22) ;(cid:27)2) normal((cid:22) (cid:22) ;(cid:27)2)uniform((cid:27) 0;b ) (2f)
(cid:22) (cid:22) (cid:24) (cid:22)j 0 0 (cid:22)j (cid:27)(cid:22)
(b ;(cid:12) ;(cid:21) ) gamma(b a ;b )normal((cid:12) ;(cid:21) 0;vI ) (2g)
(cid:27) ! ! (cid:27) 0 0 ! ! p+1
(cid:24) j j
SREE Spring 2014 Conference Abstract, September 30, 2013 2
A Bayesian Nonparametric Causal Model for Regression Discontinuity Designs
where (cid:8)( ) is the Normal( 0;1) c.d.f.; and the mixture weights ! ((cid:17) (r);(cid:27) (r)) sum to 1 at
j ! !
(cid:1) (cid:1)j
every value of r. Our model (2) has in(cid:133)nite-dimensional parameter (cid:0) = (((cid:22);(cid:27)2) ; (cid:22) ;
j 1j= (cid:22)
(cid:27)2; b ; (cid:12) ; (cid:21) ), and the degree of multimodality of f(y r;a(r)) depends on the size o(cid:0)f1(cid:27) (r)
(cid:22) (cid:27) ! ! j r0 !
(BLIND, 2012). Obviously, there is a discontinuity in the regression at r , when either of
0
the parameters ((cid:12) ;(cid:21) ) is non-zero. Moreover, when prior information is limited, we may
!2 !2
specify vague prior hyper-parameters (cid:22) = 0;(cid:27)2 , a 0, b 0, and v = 105, along
0 0 ! 1 0 ! 0 !
with a choice of prior parameter b that re(cid:135)ects prior knowledge about the range of Y.
(cid:27)(cid:22)
A set of data = y ;r ;a(ri) n updates the prior density (cid:25)((cid:0)) to a posterior den-
Dn f i i r0 gi=1
sity, given by (cid:25)((cid:0) ) n f(y r ;a(ri);(cid:0))(cid:25)((cid:0)) up to a proportionality constant. Then
jDn / i=1 ij i r0
E (y r;a(r)) = yf(y r;a(r);(cid:0))dy d(cid:5)((cid:0) ) gives the posterior predictive expectation
n j r0 jQ r0 jDn
ofY conditionalRlynoRn(r;a(rr0)). Ifall(cid:133)voeassumptionsholdforthesharpRDdesign,thenapos-
teriorestimate of the causal e⁄ect of T onY is givenby(cid:28) = E (h y r ;1) E (h y r ;0);
h n 0 n 0
f gj (cid:0) f gj
for any choice of functional h . Existing Markov Chain Monte Carlo (MCMC) methods
f(cid:1)g
can be used to estimate the posteriors (cid:25)((cid:0)jDn) and En(bhfygjr;a(rr0)) (BLIND, 2012).
Our causal model (2) can be extended to a fuzzy RD design, where it is only known that
Pr[i is a complier r ] < 1. This extension involves the estimation of bounds of the causal
0
j
e⁄ects(cid:28) , overaplausiblerangesofPr[i isacomplier r ] andof LMandERviolationmagni-
h 0
j
tudes (Angrist et al., 1996). It is prudent to estimate such bounds, because in the fuzzy RD
design, the identifying LM and LER assumptions are empirically falsi(cid:133)able and unveri(cid:133)able,
and because the estimation of Pr[i is a complier r ] is also empirically unveri(cid:133)able because
0
j
the estimator (1) does not identify the compliers (e.g., Balke & Pearl, 1997).
4 Applicability of Model; Setting; Interventions; Subjects; Data Collection and
Analysis; Results; Conclusions
Two data sets were collected under a partnership between four Chicago university schools of
education, which implemented a new curriculum that aims to train and graduate teachers
to improve Chicago public school education. Using Windows-based software developed by
the (cid:133)rst author, we analyzed each of the two data sets using the Bayesian model, speci(cid:133)ed
by the vague priors mentioned earlier, along with prior speci(cid:133)cation b = 5. All posterior
(cid:27)(cid:22)
estimates, reportedbelow, arebasedon40KMCMCsamples, whichledtoaccurateposterior
estimates according to standard convergence assessments (Geyer, 2011).
For the (cid:133)rst data set, the aim is to estimate the e⁄ect of the new teacher education cur-
riculumonmathteachingability,amongundergraduateteachereducationstudentsattending
one of four Chicago universities. This data set involves a sharp RD design, speci(cid:133)cally an
interruptedtime-seriesdesign(Cook&Campbell, 1979, Ch. 5), withtheassignmentvariable
of time, ranging from fall semester 2007 through spring semester 2013. The new curriculum
(treatment) was instituted in Fall 2010 (the cuto⁄, r ), and the old teacher curriculum (non-
0
treatment) was active before that time. The outcome variable is the number-correct score
on the 25-item Learning Math for Teaching (LMT) test (University of Michigan). The LMT
score was obtained from each student, who had just completed a course on teaching algebra.
A total of n = 347 students completed the LMT test (89:9% female; 135 and 212 students
under the old and new curriculum). Among these students, the Cronbach(cid:146)s alpha reliability
SREE Spring 2014 Conference Abstract, September 30, 2013 3
A Bayesian Nonparametric Causal Model for Regression Discontinuity Designs
of the LMT test score is :63, and the average LMT score was 12.9 (s.d.= 3:44).
Using our Bayesian model, we analyzed the data to estimate the e⁄ect of the new cur-
riculum, versus the oldcurriculum, onstudent abilitytoteachmath(LMTscore), at the Fall
semester 2010 cuto⁄. The model included the LMT test z-score as the outcome (dependent)
variable, and included covariates of the assignment variable TimeF10 = (Year 2010:9)=10
(cid:0)
and of the treatment assignment variable CTPP = A(TimeF10) = 1(TimeF10 0), with time
2010:9 (cid:21)
2010:9 referring to the Fall 2010 cuto⁄. Our model displayed good (cid:133)t to these data. The
standardized residuals ranged from 0:84 to 0:77 over the 347 observations, and R-squared
(cid:0)
was :92. Figure 1 presents the model(cid:146)s posterior predictive density estimate of the LMT
outcome, for the new curriculum (treatment) and for the old curriculum (non-treatment),
at Fall 2010. As shown, the new curriculum, compared to the old curriculum, increased
the LMT scores, in terms of shifting the density of LMT scores to the right. This shift
corresponds to an increase in the mean (from :17 to :20), the 10%ile ( 1:43 to 1:35), the
(cid:0) (cid:0)
median (:07 to :15), and corresponds to a variance decrease (1:78 to 1:69).
The second data set, from another sharp RD design, involves n = 205 undergraduate
teacher education students, each of whom enrolled into one of the four Chicago schools
of education during either the year of 2010, 2011, or 2012 (90% female; mean age=22:5,
s.d.=5:35, n = 203); 47%, 21%, 10%, and 22% attended the four universities; 49%, 41% and
10% enrolled in 2010, 2011, and 2012). It is of interest to investigate the causal e⁄ect of
basicskillsonteacherperformance(e.g., Gitomer&Brown, 2011), becausemostU.S.schools
of education based their undergraduate admissions decisions on the ability of individual
applicants to pass basic skills tests. Here, the assignment variable is de(cid:133)ned by the score on
anIllinoistestofreadingbasicskills, withminimumcuto⁄passingscoreof240. Theoutcome
variable is the total score on the 50-item Haberman (2008) Teacher Pre-screener assessment,
which has a test-retest reliability of .93, and has a 95% accuracy rate in predicting which
teachers will stay and succeed in the teaching profession (Haberman, 2008). A score in the
40-50 range indicates a very e⁄ective teacher, and many schools use the Haberman Pre-
screener to assess applicants of teaching positions. Among all the 205 students of the RD
design, theaveragereadingbasicskillsscoreis204:69(s.d.=33:7); andtheaverageHaberman
Pre-screener score is 29:82 (s.d.=4:32).
Using the Bayesian model, we analyzed the data set to estimate the causal e⁄ect of
passing the reading basic skills exam (treatment), versus not passing (non-treatment), on
students(cid:146)ability to teach in urban schools. The model included the Haberman z-score
as the outcome (dependent) variable, and included covariates of the assignment variable
Rd240 = (Read 240)=10 and the reading (Read) score passing (assignment) indicator
(cid:0)
ReadPass = A(Read) = 1(Read 240). Our model (cid:133)t the data well. The standardized
240 (cid:21)
residuals ranged from 1:7 to 1:22 over the 205 observations, and R-squared was :98. Figure
(cid:0)
2 presents the model(cid:146)s posterior predictive density estimates, for treatment versus non-
treatment. A detailed inspection revealed that passing the basic skills reading test causally
increased the Haberman z-score, in terms of the mean (from :13 to :26), median (:05 to :28),
75%ile (:97 to 1:43), 90%ile (1:66 to 2:21), 95%ile (2:13 to 2:82), and variance (1:60 to 3:36);
and causally decreased the 5%ile ( 1:74 to 2:38) and 10%ile ( 1:30 to 1:70). Also, the
(cid:0) (cid:0) (cid:0) (cid:0)
treatment density and the non-treatment density each has two modes (clusters) of students,
with below-average and above-average Haberman z-scores, respectively.
SREE Spring 2014 Conference Abstract, September 30, 2013 4
A Bayesian Nonparametric Causal Model for Regression Discontinuity Designs
References
Angrist, J., Imbens, G., & Rubin, D. (1996). Identi(cid:133)cation of causal e⁄ects using instru-
mental variables. Journal of the American Statistical Association, 91, 444-455.
Balke, A., & Pearl, J. (1997). Bounds on treatment e⁄ects from studies with imperfect
compliance. Journal of the American Statistical Association, 92, 1171-1176.
Berk, R., & Rauma, D. (1983). Capitalizing on nonrandom assignment to treatments: A
regression-discontinuity evaluation of a crime-control program. Journal of the American
Statistical Association, 78, 21-27.
Bloom, H. (2012). Modern regression discontinuity analysis. Journal of Research on Educa-
tional E⁄ectiveness, 5, 43-82.
Cattaneo,M.,Frandsen,B.,&Titiunik,R. (2013). Randomization inference in the regression
discontinuity design: An application to the study of party advantages in the U.S. senate
(Tech. Rep.). University of Michigan: Department of Statistics.
Cook, T. (2008). Waiting for life to arrive: A history of the regression-discontinuity design
in psychology, statistics and economics. Journal of Econometrics, 142, 636-654.
Cook, T., & Campbell, D. (1979). Quasi-experimentation: Design and analysis issues for
(cid:133)eld settings. Chicago: Rand McNally.
Frandsen, B., Fr(cid:246)lich, M., & Melly, B. (2012). Quantile treatment e⁄ects in the regression
discontinuity design. Journal of Econometrics, 168, 382-395.
Geyer, C. (2011). Introduction to MCMC. In S. Brooks, A. Gelman, G. Jones, & X. Meng
(Eds.), Handbook of Markov Chain Monte Carlo (p. 3-48). Boca Raton, FL: CRC.
Gitomer, D., Brown, T., &Bonett, J. (2011). Usefulsignalorunnecessaryobstacle? Therole
of basic skills tests in teacher preparation. Journal of Teacher Education, 62, 431-445.
Goldberger, A. (2008/1972). Selection bias in evaluating treatment e⁄ects: Some formal
illustrations. In D. Millimet, J. Smith, & E. Vytlacil (Eds.), Modelling and evaluating
treatment e⁄ects in economics (p. 1-31). Amsterdam: JAI Press.
Haberman, M. (2008). The Haberman Star Teacher Pre-Screener. Houston: The Haberman
Educational Foundation.
Hahn, J., Todd, P., & Klaauw, W. V. der. (2001). Identi(cid:133)cation and estimation of treatment
e⁄ects with a regression-discontinuity design. Econometrica, 69, 201-209.
Imbens, G., & Kalyanaraman, K. (2012). Optimal bandwidth choice for the regression
discontinuity estimator. The Review of Economic Studies, 79, 933-959.
SREE Spring 2014 Conference Abstract, September 30, 2013 A(cid:150)1
A Bayesian Nonparametric Causal Model for Regression Discontinuity Designs
Imbens, G. W., &Lemieux, T. (2008). Regressiondiscontinuitydesigns: Aguidetopractice.
Journal of Econometrics, 142, 615-635.
Klaauw, W. V. der. (2008). Regression-discontinuity analysis: A survey of recent develop-
ments in economics. Labour, 22, 219-245.
Lee, D. (2008). Randomized experiments from non-random selection in U.S. house elections.
Journal of Econometrics, 142, 675-697.
Lee, D., & Lemieux, T. (2010). Regression discontinuity designs in economics. The Journal
of Economic Literature, 48, 281-355.
Li, F., Mattei, A., & Mealli, F. (2013). Bayesian inference for regression discontinuity
designs with application to the evaluation of Italian university grants (Tech.Rep.). Duke
University: Department of Statistics.
Thistlewaite, D., & Campbell, D. (1960). Regression-discontinuity analysis: An alternative
to the ex-post facto experiment. Journal of Educational Psychology, 51, 309-317.
Trochim, W. (1984). Research design for program evaluation: The regression-discontinuity
approach. Newbury Park, CA: Sage.
Wong, V., Steiner, P., & Cook, T. (2013). Analyzing regression-discontinuity designs with
multipleassignmentvariables: Acomparativestudyoffourestimationmethods. Journal
of Educational and Behavioral Statistics, 38, 107-141.
SREE Spring 2014 Conference Abstract, September 30, 2013 A(cid:150)2
A Bayesian Nonparametric Causal Model for Regression Discontinuity Designs
Appendix B: Figures
CTPP = 0 (Blue) vs. 1 (Red)
TimeF10 = 0
0.08
0.07
)%0.06
5
9
( .f.d.p00..0045
y
tisne0.03
D
0.02
0.01
0
4 3 2 1 0 1 2 3 4
Z_posttest
Figure 1: For the LMT z-score outcomes, the posterior predictive density estimates of Y(1)
(red) and of Y(0) (blue).
ReadPass = 0 (Blue) vs. 1 (Red)
Rd240 = 0
0.16
0.14
0.12
)%
59 0.1
( .f.d0.08
.p
y
tisn0.06
e
D
0.04
0.02
0
4 3 2 1 0 1 2 3 4 5
Z_haberman
Figure 2: For the Haberman z-score outcomes, the posterior predictive density estimates of
Y(1) (red) and of Y(0) (blue).
SREE Spring 2014 Conference Abstract, September 30, 2013 B(cid:150)1