Data Verification and Quality Assessment Education Management Information System Afghanistan ...

109 Pages·2017·2.38 MB·English

Data Verification and Quality Assessment
Education Management Information System

Afghanistan II

Final Report

June 2017

This document was produced at the request of the United States Agency for International Development. It was
prepared independently under contract with Checchi and Company Consulting, Inc. for USAID’s Afghanistan
“Services under Program and Project Offices for Results Tracking Phase II project.
This report was conducted under Services Under Program and Project Office for Results Tracking Phase
II (SUPPORT-II), USAID Contract Number: AID-306- C-12-00012.

Assignment Title: Data Verification and Quality Assessment Education Management Information
System Afghanistan II

Team Leader: Jehanzaib Khan
Team Members: Patricia Mclaughlin
Abdul Wakeel

Activity Start Date: July 2016
Completion Date: June 2017

Paul DeLucco, Chief of Party
Waheed Ahmadi, Deputy Chief of Party
Checchi and Company Consulting, Inc.
Kabul, Afghanistan

Disclaimer:
The views expressed in this report are those of the author and do not necessarily reflect the views of
USAID, the Government of the Islamic Republic of Afghanistan, or any other organization or person.

i
TABLE OF CONTENTS

Acronyms and Abbreviations………………………………………………………………………….iv
1.0 Executive Summary: Data Verification and Quality Assessment - EMIS 2……………………….1
1.1 Assessment Purpose and Questions……………………………………………………...1
1.2 Methods and Limitations………………………………………………………………….1
1.3 Key Findings……………………………………………………………………………….2
1.4 Conclusions……………………………………………………………………………….4
1.5 Recommendations………………………………………………………………………...5
1.5.1 For USAID……………………………………………………………………...5
1.5.2 For MOE………………………………………………………………………..6
2.0 Final Report: Data Verification and Quality Assessment - EMIS 2…………………….…………..7
2.1 Background…...…………………………………………………….…………………….7
2.2 Purpose...…………………...……………………………………….……………………7
2.3 Methods………………………………...………………………………………………...8
2.3.1 Survey…………………………………………………………………………..8
2.3.2 Qualitative Interviews …………………………………………………………9
2.4 Sampling …………………………………………………………………………………9
2.4.1 Methodology ……………………………………………………………...........9
2.4.2 Limitations ………………………………………………...………….……….11
2.5 Findings & Conclusions ……..…………………………………………………….……11
2.5.1 Qualitative Findings: Data Collection Process………………………………..11
2.5.2 Quantitative Survey Findings …………………………………………………13
2.6 Recommendations ……………...……………...………………………….………….....33
2.6.1 For USAID…………………...………………………………………………..33
2.6.2 For MOE...……………………………………………………………………..33
Annexes…………………………………..……………………………………………………….......35
Annex I: Statement of Work ………………………………………………………………………...35
Annex II: Data Collection Schedule ………………………………………………………………….38
Annex III: School Lists ………………………………………………………………………………..41
Annex IV: Teachers and Students Attendance & Teachers and Students by Province……………....47
Annex V: Data Collection Instruments ……………………………………………………………...50
Annex VI: Geolocalization..…………………………………………………………………………...64
LIST OF TABLES

Table 1: Sample………………………………………………………………………………………….. 10
Table 2: Summary Statistics For School Type, Level, And Building…………………………………..….. 16
Table 3: Summary Statistics for Estimated Total Teachers (Weighted) By Type And Gender…………. 18
Table 4: Summary Statistics for Estimated Total Head Teachers (Weighted) by Type and Gender…… 28
Table 5: Summary Statistics for Estimated Student Enrollment Numbers (Weighted) by School Level
and Gender……………………………………………………………………………………………… 28
Table 6: Data Collection Schedule……………………………………………………………………….38
Table 7: Qualitative Interview Schedule: PED Respondents…………………………………...…………39
Table 8: Qualitative Interview Schedule: DED Respondents.…………………...………...……………....39
Table 9: Qualitative Interview Schedule: MoE-EMIS Respondents………………………………………..40
Table 10: List of Closed Schools (Temporarily or Permanently)…………………………………………41
ii
Table 11: List of Schools with Discordant School Level Designations…………………………………....44
Table 12: List of Schools with Permanently Absent Teachers……………………………………………46
Table 13: Total Number of Teachers Present by Attendance Record Vs Actually Present at School…..47
Table 14: Numbers of Teachers by Province……………………………………………………………..47
Table 15: Number of Present Students According to School Register Vs Actually Present at School….48
Table 16: Numbers of Student Enrollment by Province………………………………………………….47

LIST OF FIGURES

Figure 1: Schools by Level…………………………………………………………………………………11
Figure 2: Replacement Schools…………………………………………………………………………….11
Figure 3: Schools by Gender………………………………………………………………………………14
Figure 4: Long Term/Permanently Closed Schools………………………………………………………..17
Figure 5: Temporarily Closed Schools…………………………………………………………………….17
Figure 6: Total Teachers by Type and Gender……………………………………………………………19
Figure 7: Total Primary School Teachers by Type and Gender…………………………………………...20
Figure 8: Total Secondary School Teachers by Type and Gender………………………………………...20
Figure 9: Total High School Teachers by Type and Gender………………………………………………20
Figure 10: Overall Sample Total Teachers (National & by School Level)…………………………………20
Figure 11: Teachers by Attendance Register Vs. Actually Present by School Level……………………....21
Figure 12: Overall Teachers by Attendance Register Vs. Actually Present at School …………………....21
Figure 13: Head Teachers by Type and Gender…………………………………………………………..23
Figure 14: Overall Total Sampled Head Teachers………………………………………………………...23
Figure 15 A & B: Comparison of Estimated Male and Female Teacher Totals by Province……………...25
Figure 16: Primary School Students by Gender…………………………………………………....………27
Figure 17: Secondary School Students by Gender………………………………………………………...27
Figure 18: High School Students by Gender………………………………………………………………28
Figure 19: Overall Students (MoE vs Survey)……………………………………………………………...28
Figure 20: Attendance Register vs Actually Present Students…………………………………………….29
Figure 21: Overall Students by Attendance Register Vs. Actually Present Students……………………...29
Figure 22: Female and Male Student Totals by Province from the Survey and MoE Data………………..32

iii
ACRONYMS AND ABBREVIATIONS

ADS Automated Directives System
DEC Development Experience Clearinghouse
DED District Education Department
DVQA Data Verification and Quality Assessment
EMIS Education Management Information System
FY Fiscal Year
MOE Ministry of Education
NGO Non-Governmental Organization
PED Provincial Education Department
SOW Statement of Work
Tashkeel Afghan Civil Service
USAID U.S. Agency for International Development

iv
1.0 EXECUTIVE SUMMARY

Universal public education in Afghanistan has gained significant ground since 2002. Overall, school
enrollment increased from less than a million to nine million (World Bank 2016)1. As education
expanded, so did the need for accurate data collection in order to facilitate proper allocation of
resources and effective decision-making. Recognizing this need, the United States Agency for
International Development (USAID) and other development partners supported the Ministry of
Education (MoE) to strengthen the reliability of MoE data, collected through the Education
Management Information System (EMIS).

Findings from the 2015 USAID-commissioned assessment of EMIS cite the absence of independent
third-party verification of EMIS data, the dependence of MoE-EMIS on source data collected at the
district and school levels where training and procedures remain inadequate, and the lack of a data
verification mechanism. In addition, the President’s office and the international donor community
raised concerns about the dearth of information about school enrollment, actual attendance of
students and teachers, and even the existence of schools. To assess these matters, USAID
commissioned this second Data Verification and Quality Assessment (DVQA) of the MoE-EMIS 2016
data.

1.1 Assessment Purpose and Questions

This assessment focuses on verifying data collected by the MoE’s EMIS in Afghanistan for 1395. The
aim is to assess how reliable EMIS data is, and to identify inconsistencies in EMIS data at the national,
provincial, and school levels. The study assessed information about:

• School status (operational), type, location, and structure/building;
• Actual students enrolled and attending on the day of the visit (disaggregated by gender);
• Number of teachers at the school by contract and taskheel (disaggregated by gender).

1.2 Methods and Limitations

This assessment, based largely on a probability sample, relies on quantitative survey data collected
from schools. The study draws on 1,067 schools, of approximately 16,000 total schools, stratified by
province and school level, i.e., primary, secondary, and high school. The questionnaire was based on
the MoE existing data collection form to ensure that survey and EMIS data would be comparable.
The survey collected information about; (i) school operational status, type, and structure; (ii) number
of teachers by Tashkeel and contract, and number of teachers actually present at school at the time
of the visit; and, (iii) student enrollment from grades 1 to 12, number of present students according
to school register, and number of students actually found at school.

The study also employed qualitative interviews (23) with district and provincial MoE officials focusing
on how the EMIS system functions and the challenges faced by MoE officials at district and provincial
levels in data collection. The DVQA data collection began on August 20 and ended on October 30,
2016.

High risk areas2 in over a dozen provinces were excluded when the sample was being drawn. In
addition, during data collection, a large number of sampled schools, approximately 30% of the total,

1 World Bank (2016). Afghanistan Overview. Retrieved from http://www.worldbank.org/en/country/afghanistan/overview
2 The team excluded high risk areas that included neighborhoods, villages, and districts based on security reports and information from
local enumerators.
1
had to be replaced for security reasons. Very likely, the schools that could not be surveyed differ
from the schools that were surveyed in reporting reliable data, and on other measures, such as
teacher and student counts, but this is not possible to verify. In addition, school closure may be
more likely in remote or conflict-affected areas.

1.3 Key Findings

A. Qualitative Findings: Data Collection Process and Constraints

 The EMIS data collection process at the school-level varies across provinces. Qualitative data
indicate that school officials fill out the EMIS data collection forms. In some cases, academic
officials at the District Education Departments (DEDs) complete the forms during their visits
to schools. Generally, DEDs are unable to monitor data collection or provide support to
schools in filling out the forms due to a lack of resources.

 School officials lack a clear understanding of the EMIS form and certain technical terms used
in it. They find the form too detailed and face difficulties filling it out, particularly the student
and teacher data. Some DED officials report similar concerns and an urgent need for capacity
building.

 Data reported by schools contain discrepancies, including inaccurate information about
teachers by category and student breakdown. If inaccuracies are detected at the DED or
Provincial Education Department (PED) levels, clarifications are requested from schools.

 Officials interviewed unanimously identify lack of training and capacity gaps at the provincial,
and particularly at the DED and school levels, as significant challenges for improving data
quality and process.

B. Survey Findings: School Status, Type, Level, and Building

 There are small but statistically significant differences in the percentage of boys’ and mixed
schools reported by the two data sets. The MoE data report 3% more boys’ schools and 4%
fewer mixed schools than the total survey sample. By school level - i.e. primary, secondary
or high school - the MoE reports a slightly higher number of primary girls’ and mixed schools
and a lower number of high schools relative to the survey data.

 The percentage of schools with boundary walls is significantly lower (9%) in the EMIS data as
compared to the survey data. This difference is reflected between the two data sets at the
primary and secondary school levels.

 Relative to the survey data, the percentage of schools with drinking water availability is
reported significantly higher (24%) in the EMIS data at the national and school levels. In
contrast, the average number of toilets reported by the MoE data is significantly lower (3.3%)
than the average number in the survey data.

 The percentage of schools located in rural areas is smaller in the MoE data; on average the
MoE reports 7% fewer schools located in rural areas compared to the survey data. This
difference is reflected at secondary and high school levels.

 A total of 24 schools were closed at the time of the survey, of which 14 have been
permanently closed for durations that reportedly range from six months to over a year. Nine
of these permanently closed schools are reported open in the MoE data. A majority of these
nine are closed due to poor security and the ongoing conflict; two are closed due to
2
nonpayment of rent; and three are closed for unknown reasons. Of the 10 temporarily
closed schools, nine were closed for less than a month for extended periods, either for
holidays or teacher absence.

C. Survey Findings: Teachers

1. Number of Teachers by Type and Gender

 The MoE data reports a significantly higher number of male Tashkeel teachers overall and at
high school level as compared to the survey data; the total estimated numbers of female
teachers reported in the two data sets are the same.

 In contrast, the numbers of both male and female contract teachers are lower in the MoE
data than the survey data. These differences are similarly reflected at primary, secondary and
high school levels for both male and female teachers.

 Differences also exist between the two datasets in the reported numbers of Ajeer (by
Tashkeel) teachers. The number of male Ajeer teachers is higher and the number of female
Ajeer teachers is significantly lower in the MoE data than in the survey data.

 Once aggregated, the composite and uncategorized teacher numbers reach near parity at the
total sample level. In other words, the MoE and survey data sets record no statistical
difference in the total number of teachers. This trend continues at the school level - primary,
secondary, high - where the differences are nominal and insignificant.

II. Teachers by Attendance Register vs Actually Present at School

 More teachers are marked present in attendance registers than the number actually found at
school. These differences correspond to 8% additional male teachers, 6% additional female
teachers, and overall 7% additional teachers reported present in the attendance register
relative to the headcount numbers. On average, less than 1 additional male and female
teachers (or 0.43 and 0.21, respectively) per school were reported present in school
registers as compared to the number of teachers actually found at school.

 Of the total sampled schools (N=1,067), only 34 (3% of the total sample), comprising 6 high
schools, 11 secondary schools, and seven primary schools, reported one or more teachers
who never showed up at school. This number of absent teachers varies from 1 to 4, with the
majority of the 34 schools reporting one permanently absent teacher. Of these schools, four
were girls only, 10 were boys only, and 20 were mixed schools.

III. Head Teachers by Type and Gender - Tashkeel and Ajeer (by Tashkeel)

 The number of male Tashkeel head teachers between the two data sets is not significantly
different. In contrast, the number of female Tashkeel and Ajeer (by Tashkeel) head teachers
reported by the MoE is significantly lower than the survey data.

 There are differences in the composite and uncategorized head teacher number at the total
sample level; the MoE number is significantly lower than those recorded in the survey data.

D. Survey Findings: Students

I. Student Enrollment by School Level and Gender

3
 Small to moderate, but significant, differences exist between the MoE and survey tallies of
both girls and boys for every primary class. Cumulatively, the MoE data reports 18% more
boys, 12% more girls, and overall 15% more primary school students than the survey data.
Similarly, the MoE data includes 15% more boys and 7% more girls at the secondary school
level relative to the survey numbers. Overall, the MoE reports approximately 11% more
students than the survey data. At the high school level, differences are smaller than at the
secondary school level. While the MoE reports 9% more boys than the survey data, there is
no difference in female student numbers between the two. Overall the MoE reports 4%
more high school students than the survey data.

 The total numbers of boys and girls reported in the MoE data are significantly higher -15%
more boys and 9% more girls - as compared to the survey data. The overall MoE and survey
totals of students, boys and girls combined, are significantly different, i.e. the MoE reports
12% more students than the survey data.

II. Student Enrollment by Attendance Register Versus Actually Present at School

 More students are found present at school than marked present in student attendance
registers. The small to moderate differences correspond to approximately 3% additional
students in grades 1 and 3, 11% in grade 7, and 12% in grade 10 who were actually present as
compared to the attendance record. There were 9.5% more boys and 7% girls actually found
at school than reported by school attendance register. This means, on average, that the data
from the headcounts report 2.5 additional boys and 1.9 additional girls actually present per
school as compared to the school attendance record.

1.4 Conclusions

A. Qualitative: Data Collection Process and Constraints

Inaccuracies in teacher and student information are not uncommon. The variation in the data
collection process at the school level, underdeveloped capacity of school officials, absence of capacity
building programs for district and provincial officials, inadequate monitoring at the district and school
levels due to budgetary constraints, and limited human resources likely affect both the process and
quality of data collection.

B. Survey: School Status, Type, Level, and Building

 Beyond marginal differences, the MoE data about school type and structure does not
significantly vary from the survey data on key indicators. In other words, the MoE reports
mostly reliable data on school type and structure.

 Contrary to broad speculation, e.g., Buzzfeed 20153, the number of MoE permanently closed
schools is small. Yet, although the number is small, the fact that the MoE data
underrepresents the actual number - 09/14 reported by the survey are marked operational
in the MoE data - indicates a gap in reporting.

C. Survey: Teachers

 There are significant discrepancies in the two data sets at the teachers’ subcategory levels,
i.e. teacher type, gender, and school level, with the exception of female Tashkeel teachers.

3 BuzzFeed (2015). Ghost Students, Ghost Teachers, Ghost School. Retrieved from BuzzFeed website:
https://www.buzzfeed.com/azmatkhan/the-big-lie-that-helped-justify-americas-war-in-afghanistan?utm_term=.qqlyyMzk1o#.ukepp821V0
4
Qualitative data indicate that variation at the subcategory levels may stem from inaccurate
categorization of teacher data at the school or district levels.

 Despite the discrepancies in the MoE data across the majority of the subcategories, these
discrepancies almost disappear at the total sample and total teachers by school level. In
other words, while the reliability of the MoE data at the subcategory level is questionable,
the data is, by and large, consistent at the aggregate level.

 Except for the male head teachers (Tashkeel) figures, the MoE head teacher data is less
reliable than the teacher data due to discrepancies in this data present at both the
subcategory and composite levels.

 Teacher attendance records do not accurately account for all the teachers who are actually
absent from school. The practice of marking teachers present who are actually absent is
more common for absent male teachers than for absent female teachers; for primary schools
relative to secondary and high schools; and for larger schools compared to smaller ones.

 The proportion of schools reporting teachers who never show up at school, i.e. permanently
absent teachers, is small. The practice tends to be more common at the high school level
than at the secondary and primary levels.

D. Survey: Students

 The MoE student enrollment numbers are larger than the survey data for every grade for
boys and for all but grade 12 for girls - overall, the MoE reports 12% more students than the
survey data. The discrepancies are more common at the primary school level relative to
secondary and high school levels and more common for boys than for girls. Relatively smaller
discrepancies in overall girl enrollment numbers as compared to boys indicate that the girls’
data is slightly more reliable than boys.

 Student attendance records are inconsistent when compared to the number of students
actually present. Schools generally underreport the number of students present. The practice
of inaccurately recording student attendance is more pronounced at the secondary and high
school levels than at primary school, and slightly more common for boys than for girls
overall.

1.5 Recommendations

1.5.1 For USAID

 USAID should consider continued support to future independent assessments of the EMIS
data, which may also compare EMIS data between provinces, e.g., secure and insecure,
inaccessible, remote. Such assessments, requiring large samples from provinces, could
provide valuable insights into solutions for variations in EMIS data quality between regions or
provinces.

 Future USAID support to the MoE should include capacity-building measures for the MoE-
EMIS at the DED, PED, and EMIS Kabul levels. Such a program might include:
a. Training programs specifically focused on planning and EMIS data collection for quality
assurance, sorting, cleaning, analysis, and management.
b. Funding or providing technical support to the MoE to develop a strong capacity building
program directed toward EMIS at Kabul, provincial, and district levels. The capacity-
building might include conducting needs assessments of school officials, DEDs, PEDs, and
5