Ambition Success

30 Pages·2012·0.66 MB·English

Has the "Ambition Success" educational
program achieved its ambition? ∗
Magali Beffy† Laurent Davezies§
Still Preliminary - Do not quote
Abstract
This paper provides an evaluation of the impact of the educational program en-
titled "Ambition Success Networks" (Réseau Ambition Réussite) which have succeed
to the "Educational Priority Zones" (Zones d’éducation prioritaire) in 2006. We ex-
ploit two fuzzy regression discontinuity designs to control the selection bias on two
specific groups of treated junior high schools. We use longitudinal data to improve
accuracyofestimates. Inthetwogroupsoftreatedschoolstheincreaseofteachersby
pupils and the decrease of class size is lower than expected on the basis of Ministry
declaration. For one group of treated schools the program decreases the proportion
of self-employed children entering in sixth grade and increases the proportion of blue
collar children. Simultaneously the characteristics of teachers change: we observe an
increase in proportion of oldest teachers and in proportion of teachers who have not
an usual diploma. For the both groups of treated schools we observe a decrease in
the pupils achievement measured by the score to the final exam in ninth grade. So
the program increases the sorting on pupils ability between schools and/or have a
negative effect on the pupils achievement in the schools.
Keywords: educational policy, school-oriented compensation policy, regression discontinu-
ity.
∗We thank Alexia Stéphanou, Cédric Afsa and the staff of the statistical department of the French
Ministry of Education for their help to build the data. We thank Xavier D’Haultfœuille and Eric Maurin
and the participants at the DEPP seminar of the French Ministry of Education, at the D3E seminar of
the French statistical institut and at the IWAEE 2011 workshop for their valuable comments.
†CREST, magali.beff[email protected]
§CREST, [email protected]
1
JEL Classification : I28, I24, C21.
2
In many democratic countries the main objectives of educational systems are twofold:
to offer equal opportunities to every pupil, and simultaneously to promote success for
each student. In France, as in other countries, these ambitions are thrown back into
doubt since the 60’s and 70’s because some inequalities of achievement related to parental
characteristics persist among students (see for instance Coleman et al. (1966) for the USA
and Plowden & the Central Advisory Council for Education (1967) for the UK). Hence
many countries have set up compensation policies to offer the same opportunities to pupils
whatever the parental characteristics.
In France the principal compensation policy has targeted schools instead of students and
is named Politique d’Education Prioritaire, or Preferential Educational Policy.1 The very
first French school-oriented compensation policy was introduced in 1982 and named Pri-
ority Education Zones (ZEP). Overhauled twice, it was eventually replaced in 2006 by the
Réseaux Ambition Réussite policy (RAR) on which we focus here. The former policy was
criticized for scattering its funds on a far too large number of schools and evaluations of the
ZEP policy conclude that it had no positive effect on student achievement (see Benabou
et al. (2009), and Meuret (1994)) but an adverse effect at a tremendous cost. Hence the
RAR policy, or Ambition Success Network policy, concentrated its funds on a lower number
of schools: 249 junior high schools in 2006-2007 against around 900 with the ZEP policy.2
In this paper we provide a first evaluation of this policy. The evaluation of school-oriented
policies must deal with several methodological issues. First, treated schools are not ran-
domly selected: they greatly differ on many aspects from non selected schools. Using a
fuzzy-regression discontinuity (RD) strategy, we here overcome these issues and we develop
proper strategy to find valid counterfactual schools for two groups of treated schools. A
second class of methodological issues concern the choice of pertinent outcomes used to
evaluate the policy. The actual treatment can differ from the announcement of the Min-
istry, because implementation is delegated to a local level. In order to describe the actual
treatment, we compare the number of teachers in the treated and non-treated schools.
Moreover these policies may have adverse effects on the sorting of teachers and students
1Though pupil- or class-oriented policies are more widespread in developed countries, a few noticeable
educational policies target schools. For instance, in the United States the Title I of the Elementary
and Secondary Education Act in 1965 funded schools and school districts with a high percentage of
disadvantaged students. In the United Kingdom the Education Priority Areas (EPA) was launched in
1967, disappeared at the end of the 70’s but following the victory of the New Labour in the 90’s was
reborn in two new programs Education Action Zones and Excellence in the Cities. They both targeted
schools (see Machin & Vignoles (2005)).
2A network was made up of a junior high-school (from the 6th to the 9th grades) and a few primary
and/or nursery schools (from the 1rst to the 5th grades, or before the 1rst grade).
3
between schools. Labeling the school as a priority school may induce a poorer-quality
teacher structure, as well as a brain drain. So we study the policy consequences on the
pupil-school matching and on the teacher-school matching using the observable parental
characteristics and the diploma of teachers. Finally, we also compare the junior high school
final exam results between treated and non-treated schools. This last comparison is the
resultant of two mixed effects: the change of efficiency of the teaching team for a given
population of students and the change in sorting on unobserved characteristics of pupils
between schools.
Our analysis highlight that this policy is disappointing. For a first group of treated schools,
the policy has basically no effect on the number of teachers by pupils, on the teachers and
pupils characteristics but an adverse effect on the achievement. A second group of treated
schools have benefited from a larger number of teachers in the 6th and 7th grade but not
for the other grades. In these schools, the RAR policy increases the part of older teacher
and the part of teacher with a non-usual diploma. The pupils structure is also modified in
these schools: the part of blue-collar children increase whereas the part of Self-employed
children decrease. And last, for the both groups of treated schools, the results to the
final junior high-school exam are worse than in similar non-treated schools. Finally, the
only way to rationalize these empirical evidence with a positive effect of the policy on
the pupils achievement is an increase in the sorting of pupils in term of cognitive ability
between schools. Hence a positive effect of the policy on the teaching efficiency remains
hypothetical.
The rest of this paper is organized as follow. The next section describes the RAR policy in
details. Section 3 develops the identification strategy, we describe the two groups of treated
schools for which the we build counterfactual, and details estimation strategy. Section 4
presents the results. Section 5 concludes.
1 The Ambition Success Network policy policy (or RAR policy)
1.1 A brief history
In 1982 a new program setting Education Priority Zones (ZEP) was launched. The effects
of social origin and geographical situation on pupils achievement were established and
commonly accepted. One solution that arises was to increase efforts in unsuccessful zones
to reduce inequalities. These new education priority zones were supposed to make up an
educational project, and the underachieving students were to be helped. In 1985 the main
4
focus was reoriented on core learning such as reading and French speaking. In the following
years, successive reforms took place, each one expanding the number of schools concerned,
but none was with increasing means.
Benabou et al. (2004) draw three main conclusion concerning these Education Priority
Zones. Firstly, the rather substantial subsidies were divided into too many schools and
they were mainly given to teachers via supplementary wages. The funds directly affected
to pupils were scarce. Secondly, the junior high schools chosen to be in these education
priority list experienced a decrease in their total number of students and an increase in
the proportion of socially disadvantaged pupils. Teachers also flew from these schools.
Their turnover increased after the classification of these schools into priority zones. Finally,
Benabouetal.(2004)findnosignificanteffectondifferentmeasuresofstudentachievement,
neither on high-school graduation. These disappointing results suggested a restructuration
of the ZEP program, which better targets efforts and funds.
1.2 The Ambition Success Network policy
The last reform introduced in 2006 pursued that goal. The education priority map was
deeply reshaped, and the means were given to a smaller number of schools (249 junior
high schools in 2006-2007 against around 900 with the previous policy). New zones were
defined and named "Réseaux Ambition Réussite" (Ambition Success Networks). These
networks are made up of one junior high-school (6th to 9th grades) and some primary
and/or nursery schools (1rst to 5th grades, or before 1rst grade). These entities share
a common project under the guidance of a committee which is composed of the head of
the junior high-school and some representatives of the primary or nursery schools. At the
beginning of the 2006-2007 school year, 249 networks were created which consisted of 249
junior high-schools and 1,715 elementary schools. It represented 126,000 pupils, and one
junior student over twenty.
The ministry of Education recommended to its regional heads ("recteurs") that each net-
work got three to four supplementary teachers, some teaching assistants and at least one
full-time nurse. On a national level, this amounts to 1,000 supplementary teachers, and
3,000teachingassistants.Butconsiderablediscretionwaslefttotheregionalheads. Though
the list of "RAR" junior high-schools was decided in a concerted way, the list of primary
and nursery schools, and the means devoted to each network were chosen by the regional
heads. Moreover the type of educational services provided is left to the discretion of junior
high schools and regional heads, with no accountability requirement.
5
The number of RAR networks has evolved since their introduction. From 249 in 2006-
2007, it consists of 254 public junior high-schools at the beginning of the 2008-2009 school
year and 118,000 junior students. There are large regional discrepancies: the proportion
of "RAR" junior high-schools ranges from 0.4% for the regional area of Grenoble to 13%
for the regional area of Aix-Marseille. There are also a few private "RAR" junior high
schools, namely 11 in 2008-2009. The networks or zones that were not targeted by the new
policy in 2006 constitute a different network called "network of school success" ("Réseaux
de Réussite Scolaire", RRS). In 2007-2008, over the 253 RAR junior high schools, 238 were
previously "ZEP". The previous denomination "ZEP" disappeared in September 2008.
1.3 How where the RAR schools selected?
Internal notes of the Statistical Service of the Education Ministry inform us on the process
of selection of the secondary schools. The RAR networks were chosen on the basis of
three main criteria evaluated for 2004-2005 school year. First, the proportion of socially
disadvantaged students in 6th grade must be equal to or above 67%. Second, either the
proportion of students who have repeated two grades or more when they enter the 6th
grade is at or above 10% or the school average scores at the entrance evaluation of the 6th
grade are at or below 47%.
Additional criteria were used to define the final list of RAR junior high schools. Local
unemploymentrateandproportionofpersonsbenefitingfromsocialassistancewereusedin
particular. Instead of the 164 initially chosen with the three criteria previously mentioned,
249 were finally kept after an agreement between the French Ministry of Education and its
regional heads.
1.4 Expected results
If we assume that the 1,000 supplementary teachers were uniformly assigned across the
new RAR junior high schools, it would have induced a 8% increase in the total number of
teachers in RAR schools versus non RAR schools for metropolitan junior high schools. But
as mentioned before, the regional head master has a considerable discretion. For instance,
even if its ministerial tutelage give to him a few teachers that must be specifically assigned
in the RAR junior-high-schools, the regional head master can assign the large majority
of teachers to his convenience. In particular teachers already in the RAR schools maybe
instated to non RAR schools resulting in a stability of the global number of teachers both
6
in RAR and non RAR schools. This eviction phenomenon is credible, if regional head
master are reluctant to treat differently schools that do not differ significantly.
The assignement to the treatment can also be interpreted as a negative signal by some
parents, and so the number of pupils in the school can decrease. Benabou et al. (2004) and
Benabou et al. (2009) have shown such a decrease for the ZEP-schools during the 90’s. In
this case, the number of teachers per pupils can increase more than expected.
If the number of teachers by pupils increase sufficiently, one can expect that class size
decrease. The effect of the class-size on the pupils achievement using the seminal method
of Angrist & Lavy (1999) is often reported as negative (see Angrist & Lavy (1999), Leuwen
et al. (2008), and for the French case, Gary-Bobo & Mahjoub (2006) and Piketty & Valde-
naire (2006)). Another group of estimates comes from the Tennesse STAR class size ex-
periment. Word et al., (1990), Finn & Achilles (1990), Krueger (1999), and Krueger &
Whitmore (2001) all find that smaller class sizes have a significant and lasting impact on
academic achievement and educational attainment.
Another set of questions concern the type of teachers in the RAR schools. First, the
supplementary teachers can have some specific characteristics. In the previous studies on
the ZEP policies, it appears that the new teachers affected in schools are often younger
and have less experience. One can suspect that the affectation of the school to the RAR-
program constitute a negative signal for the professor who preferred to be affected in
non-treated schools. The rules of affectation of teachers in secondary schools by regional
head-master implies that the less experienced teachers have the lower bargaining power.
MoreoverinthepreviousZEPprogram, teachershiredinZEPschoolsarepromotedquicker
to encourage them to teach in such schools. But the more promoted you are, the more you
can ask a transfer to another school. So it has been notice that turn-over of teaching team
was often more important in the ZEP schools, and such a phenomenon can be harmful
to the school management. Ly (2010) observe that the reform of the list of ZEP in 1999
seems to have put to flight the oldest teachers in the schools that have been affected to the
program.
Finally the policy can modify the sorting of pupils between schools. Benabou et al. (2004)
and Benabou et al. (2009) find that the social disparities measured by the fraction of pupils
not enrolled in the school’s cafeteria’s plan between treated and untreated schools increase.
Another part of the literature shows that parents take into account the schools when the
choose where to move (Fack & Grenet (2010) for the case of Paris). More generally,
sociologists have largely reported existence of parental strategies concerning the schooling
of their children (François & Poupeau (2004)) and economists have tried to quantify the
7
valuation of schools (Black (1999), Fack & Grenet (2010), Gibbons & Machin (2003)).
All these parental strategies are based on the available information of schools and the
assignment to the RAR-program is a new observable characteristic of schools and can
affect parental choice. Even if one can not a priori exclude that parents give a higher
valuation to the RAR-schools because such schools get higher means, the empirical results
in the literature suggest that schools-based discrimination increase segregation. In case of
such an increase of segregation the effect on average test score at the final exam is muddled:
the policy may increase the efficiency of treated schools but such an increase of efficiency
may be compensated by a sorting effect. Our data only allow us to compare the average
result between schools. This is a clear limit of this paper and the reader must keep in mind
that estimates on the final score exam mix the two effects.
2 Data and some descriptive statistics
To analyze the effectiveness of the RAR program, we use school-level data resulting from
different administrative files of the ministry of education:
• Anexhaustivepupil-levelcross-sectionaldata(Scolarité)foreverystudentinjunioror
secondary high school. We have yearly information about age, nationality, residence
location, main parent’s occupation, class, languages and other options, school lunch
status, and the same variables for the year before.
• An exhaustive teacher-level panel data (Relais) for every teacher in junior or sec-
ondary high school. We have the total number of hours they give in each school, for
each topic they teach, their age...
• An exhaustive pupil-level data with their scores to the final national exam (Brevet
des Colleges) that they take at the end of the 9th grade. This data can not be
merged with the previously pupil-level data since they do not contain any student
identification number.
• An exhaustive panel of French junior and secondary high schools with their status
relativetotheEducationPrioritypolicy,andthethreecriteriausedtobecomeeligible
to the "RAR" status.
We construct a school-level dataset for the school years 2003/2004 to 2008/2009. We re-
strict our attention to public junior high schools in mainland France. Such a restriction is
8
done because the private schools are almost never assigned to the treatment, and because
in the French overseas departments and territories the junior-high schools are almost al-
ways assigned to the treatment. Moreover private schools, which represent around 20% of
pupils, are often specific comparing to public schools. Similarly, in the French overseas de-
partments and territories, pupils have often particular sociocultural and living conditions.
We obtain an exhaustive panel of around 5000 public juniors high schools in metropolitan
France. On these schools, 206 were affected to the RAR program.
Educationally disadvantaged students who have severe and long-running problems with
core learning, are in special classes called "Segpa" (section d’enseignement général et pro-
fessionnel adapté). Special funds are dedicated to them, and do not interfere with means
allocated to the education priority policy. Similarly, when we evaluate the average number
of pupils per class, or the social structure of pupils moving up into the 6th grade, or other
types of criteria, we exclude these students.
Concerning the final exam, we focus our attention on the formal written literacy and maths
tests. Every year, the final exams took place in June for pupils in ninth grade from 2003
to 2009. For each school, we evaluate average and quantiles of distribution of score tests
in the school. For these computations, we exclude disabled pupils3 and pupils following
studies with a professional education curriculum.
Descriptive statistics one year before the treatment for the whole sample of public junior
high schools in mainland France are presented in Table 1. The RAR program was launched
in September 2006. The RAR public junior high schools have a higher proportion of pupils
entering the sixth grade after repeating grade. A junior entering the 6th grade should
be 11 years old. The proportion of pupils who have repeated at least one grade when
entering in September 2005 in RAR schools is 40% against 20% in non RAR schools.
This discrepancy is roughly the same after 2006, even though the proportion of students
having repeated a grade before the junior high school decreases since 2003 due to the
no repeating grade policy in the French educational system. In the sixth and the ninth
grades, the socially disadvantaged pupils are overrepresented in the RAR schools. Only
2.08% of juniors entering the sixth grade in RAR schools belongs to a family whose head4
is a manager. This proportion reaches 15.88% within non RAR schools. Unlike manager,
unemployed and blue collar are over-represented within RAR schools. In the sixth grade
in 2006/2007, more than one pupil over four has an unemployed parent in RAR schools
3The way they take the national exam is different from the non disabled.
4Inthedata, onlytheoccupationofoneofthetwoparentsisavailable. Thisparent, whoisusuallythe
father is called the family head.
9
against less than 9% in non RAR schools.
Table 1 also presents proxies of average expenditures per pupils, namely the average class
size and the average weekly hours of teaching per pupil. One year before the treatment,
the average class size was around 21 in RAR schools and 24 in non RAR schools. Similarly,
the number of hours of teaching dispensed in the school by week divided by the number
of pupils was about 12% higher in the future RAR schools (1.41 versus 1.24 in the future
non-RARschools). Supplementaryanalysisnotreproducedhereshowthatthisresultholds
whatever the grade. This difference can be explained by the overrepresentation of the ZEP
schools among future RAR schools, which benefit from a larger number of teachers by
pupils.
Concerning the characteristics of the teaching staff before the treatment, teachers have
lower diploma and are younger in RAR junior high schools. PE teachers were more nu-
merous in RAR schools.
Table 1 provides a comparison between RAR and non RAR schools in maths and literacy
scores at the final national exam. Each student at the end of his/her ninth grade takes
written tests in maths and in French. The scores to the exam range between 0 and 40.
For each junior high school, we compute the average test score in Math and French. The
comparison of the means highlight the huge discrepancy between RAR and non-RAR
results just before the treatment. For the exam which took place in June 2006 just before
the treatment, the average test score in French for RAR schools is 14.32 against 19.01 for
the non-RAR schools. The same holds in Math. Comparison of the quantiles (not reported
here) shows that this result holds for all the distribution of test scores.
All these descriptive statistics show that using a naive comparison between the treated
and non-treated schools cannot be used to test the efficiency of the RAR program. Con-
ditioning on some important observable covariates may mitigate the bias but even in this
case the interpretation is likely to be misleading because the affectation to the treatment
is probably driven by some unobserved variables. Another strategy is to used the longitu-
dinal dimension of the data to adopt a difference in difference strategy. Such a strategy is
valid only if the temporal trend are the same between the treated and non treated schools.
Simple panel-data regressions reported in Table 2 show that temporal trends between 2003
and 2005 differ significantly at the 5% level concerning a majority of variables.5 During
the years preceding the treatment, the discrepancy concerning in the number of 13 years-
old pupils entering the sixth grade has decreased, contrary to the discrepancy concerning
5To be more precise, the trend of treated and untreated schools differ significantly for at least one
modality for every categorical variable such as parental profession, teachers characteristics, age of pupils
at the beginning of the 6th grade.
10