EVALUATION OF LEARNING ACHIEVEMENT IN SELECTED WOREDAS IN AMHARA AND ADDIS ...

139 Pages·2015·3.24 MB·English

EVALUATION OF LEARNING ACHIEVEMENT IN SELECTED WOREDAS IN
AMHARA AND ADDIS ABABA SUB-CITIES

MID-TERM REPORT

Submitted By:

IN ASSOCIATION WITH
FROMSEAS EDUCATION & TRAINING /FEAT/ S.C

Submitted To:
JUNE 2015
ADDIS ABABA, ETHIOPIA
Evaluation of Learning Achievement in Selected Woredas in Amhara
and Sub-Cities in Addis Ababa, Ethiopia
Mid-Term Report

This document was prepared by
CfBT Education Trust and FEAT S.C.
for the Ministry of Education
and UNICEF Ethiopia
as part of Contract No. 43156492
June 2015
Addis Ababa, Ethiopia
2
ACKNOWLEDGEMENTS
The evaluation team appreciates all the support it has received during the Mid-Term Evaluation
study from the Government of Ethiopia, its Ministry of Education and the Regional Education
Bureaus of Amhara and Addis Ababa for mobilisation of the assessors, data collectors and the
mapping of schools. The team is also very grateful for all the support provided by the UNICEF
Ethiopia country office Chief of Education Section, Education Specialist and all the members of
the team. In addition we thank the management and staff of all the schools that participated in
this exercise for their cooperation.

3
LIST OF ACRONYMS AND ABBREVIATIONS

ABECs - Alternative Basic Education Centres
AIR - American Institute for Research
CPD - Continuing Professional Development
ECCE - Early Childhood Care and Education
EGRA - Early Grade Reading Assessment
EMIS - Education Management Information System
ESDP - Education Sector Development Programme
ESNLA - Ethiopia Second National Learning Assessment
ETNLA - Ethiopia Third National Learning Assessment
FGD - Focus Group Discussions
GDP - Gross Domestic Product
GEQIP - General Education Quality Improvement Programme
GER - Gross Enrolment Ratio
GoE - Government of Ethiopia
MLCs - Minimum Learning Competencies
MoE - Ministry of Education
NAEAE National Agency for Educational Assessment and Examination (NAEAE)
NER - Net Enrolment Ratio
NLA - National Learning Assessment
PRC - Pedagogical Resource Centre
PTA - Parent-Teacher Association
REB - Regional Education Bureaus (REB)
SIP - School Improvement Programme

4
TABLE OF CONTENTS
ACKNOWLEDGEMENTS ............................................................................................................ 3
LIST OF ACRONYMS AND ABBREVIATIONS ...................................................................... 4
LIST OF TABLES .......................................................................................................................... 9
LIST OF FIGURES ...................................................................................................................... 12
EXECUTIVE SUMMARY ............................................................................................................ 13
INTRODUCTION ......................................................................................................................... 19
1.0 THE MID-TERM EVALUATION .................................................................................... 20
1.1 Evaluation Purpose and Questions ...................................................................... 20
2.0 BACKGROUND ............................................................................................................... 22
2.1 Project Goal ................................................................................................................ 22
2.2 Education in Ethiopia ............................................................................................... 22
2.2.1 General Education in Ethiopia ........................................................................... 23
2.2.2 Primary Education in Ethiopia ............................................................................ 24
2.2.3 Developments in Primary Education 1999-2010 ............................................ 25
2.2.4 The Education Sector Development Programme IV 2010/11 to 2014/15 ... 28
2.2.5 Developments in Primary Education under EDSP IV 2010 - 2013 .............. 29
2.2.6 General Education Quality Improvement Package ........................................ 30
2.3 Accelerated Development of Literacy, Numeracy and Life Skills among
first cycle learners ................................................................................................................. 31
2.3.1 Objectives ............................................................................................................. 32
2.3.2 Inputs ..................................................................................................................... 32
2.3.3 Development hypothesis .................................................................................... 33
2.3.4 Outputs .................................................................................................................. 35
3.0 EVALUATION DESIGN .................................................................................................. 37
3.1 Evaluation Aims ......................................................................................................... 37
3.2 Evaluation Strategy ................................................................................................... 37
3.3 Data Collection Tools ............................................................................................... 39
3.3.1 Review of Project Documents and Materials ................................................... 39
3.3.2 Student Achievement Tests ............................................................................... 39
3.3.3 Structured Focus Group Discussions with Learners ...................................... 41
3.3.4 Teacher Observation and Interview .................................................................. 41
3.3.5 Interviews with Trainers ...................................................................................... 42
3.3.6 Semi Structured Interviews with Education Stakeholders ............................. 42
5
3.4 Pilot Study ................................................................................................................... 43
3.5 Data Collection .......................................................................................................... 44
3.6 Sample Selection ...................................................................................................... 45
3.6.1 Schools ................................................................................................................. 45
3.6.2 Students ................................................................................................................ 46
3.6.3 Teachers ............................................................................................................... 47
3.6.4 Trainers ................................................................................................................. 47
3.6.5 Education Stakeholders ..................................................................................... 47
3.7 Data Entry and Analysis .......................................................................................... 48
3.8 Data Quality Assurance ........................................................................................... 49
3.8.8 Data Quality .......................................................................................................... 50
3.9 Report Validation ..................................................................................................... 50
3.10 Limitations .............................................................................................................. 50
3.11 Evaluation Timeline .................................................................................................. 52
4.0 FINDINGS ........................................................................................................................ 53
4.1 The Scope of Project Interventions ...................................................................... 53
4.1.1 Teachers Perspectives ....................................................................................... 55
4.1.3 Trainers Perspectives ......................................................................................... 56
4.1.4 Community members’ perspectives ................................................................. 57
4.1.5 Education Officials perspectives ................................................................... 57
4.2 The Quality and Relevance of project materials ............................................... 58
4.2.1 Validated Test Items ........................................................................................... 58
4.2.1 (a) High Quality .................................................................................................. 59
4.2.1 (b) Alignment with the Minimum Learning Competencies: ............................. 65
4.2.1 (c) Adequate number of items in different formats ............................................... 65
4.2.1 (d) Varied difficulty and cognitive complexity ....................................................... 68
4.2.2 Training and Support Materials ......................................................................... 68
4.3 Student Performance ............................................................................................... 70
4.3.1 Grade One ............................................................................................................ 70
1.3.2 Grade Two ............................................................................................................ 77
4.3.3 Grade Three ......................................................................................................... 84
4.3.4 Grade 4 ................................................................................................................. 91
4.4 Teacher Performance ............................................................................................. 101
4.4.1 Lesson Structure ............................................................................................... 101
6
4.4.2 Student Participation ......................................................................................... 102
4.4.3 Relevance of the Lessons ................................................................................ 103
4.4.4. Inclusivity of the lesson .......................................................................................... 104
4.4.5 Student Interest and motivation in the lesson ................................................... 105
4.4.6 The use of teaching and learning resources ................................................. 106
4.4.7 Students Perspectives ...................................................................................... 106
4.5 The Use of Continuous Assessment .................................................................. 107
4.5.1 Classroom Observation .................................................................................... 108
4.5.2 Teachers Perspectives ..................................................................................... 109
4.5.3 Students Perspectives ...................................................................................... 112
4.5.4 Education Officials perspectives...................................................................... 114
4.5.5 Community Members perspectives ................................................................. 114
4.6 Strengthening Project Interventions .................................................................. 114
4.6.1 Teachers perspectives ...................................................................................... 115
4.6.2 Trainers perspectives ........................................................................................ 115
4.6.3 Education officials’ perspectives...................................................................... 116
5.0 CONCLUSIONS AND RECOMMENDATIONS ........................................................ 116
5.1 The Challenge of measuring difference in the evaluation ............................ 117
5.1.1 Comparing baseline (2013) to midline (2015) ............................................... 117
5.1.2 Comparing programme and ‘control’ schools. ............................................... 118
5.2 To what extent have project inputs made a difference to the learning
achievements of students?............................................................................................... 118
5.4 To what extent have project inputs made a difference to teachers’ use of
learner centered pedagogical approaches? ................................................................. 123
5.5 To what extent have project inputs made a difference to teachers’ use of
continuous assessment techniques? ............................................................................ 124
5.6 What is the quality and relevance of the project materials? ....................... 125
5.7 What can the project do to improve interventions for increased impact
and to promote the sustainability and scale up of promising practices? ............ 127
5.7.1 Ensure teacher training and support takes place ......................................... 127
5.7.2 Engage directly with school leaders................................................................ 129
5.7.3 Fine-tune the project materials ........................................................................ 130
5.7.4 Strengthen project documentation .................................................................. 131
ANNEXES ................................................................................................................................... 138

7
8
LIST OF TABLES

Table 1: Three Stage Evaluation of the ELA project ....................................................................... 20
Table 2: The Structure of General Education in Ethiopia ............................................................... 23
Table 3: The Ethiopian National Learning Assessments ................................................................. 26
Table 4:Average Composite Scores (%) in NLA by Year and Grade ................................................ 26
Table 5: Targets established under ESDP IV ................................................................................... 28
Table 6: The Components of GEQIP II ............................................................................................ 30
Table 7: Student Tests for the Mid-Term Evaluation ..................................................................... 40
Table 8: Outline of Student Tests for the Mid-Term Evaluation .................................................... 41
Table 9: General Schedule of Data Collection at the local level ..................................................... 45
Table 10: Final Sample of Woredas and Schools ............................................................................ 46
Table 11: Sample of Students and Number of Tests Administered ............................................... 46
Table 12: : Summary of Evaluation Sample by Sub Population ...................................................... 48
Table 13:Schedule of key activities in the mid-term evaluation .................................................... 52
Table 14: Planned and Actual Activities ......................................................................................... 53
Table 15: Minimum Learning Competencies Covered by Grade 1 English Test Items ................... 66
Table 16: Breakdown of the student sample by sub population ................................................... 70
Table 17: Grade 1 Amharic average scores (%) by sex, region and group ..................................... 71
Table 18: Easiest and most difficult questions in Grade 1 Amharic ............................................... 71
Table 19: Grade 1 English average scores (%) by sex, region and group ...................................... 72
Table 20: Easiest and most difficult questions in Grade 1 English ................................................. 72
Table 21: Grade 1 Mathematics average scores (%) by sex, region and group ............................. 73
Table 22: Easiest and most difficult questions in Grade 1 Mathematics ....................................... 74
Table 23: Grade 1 Environmental Science average scores (%) by sex, group and region .............. 74
Table 24: Easiest and most difficult questions in Grade 1 Environmental Science ........................ 75
Table 25:Average Grade 1 Composite scores (%) in grade 1 by sex, group and region ................. 76
Table 26: Average grade 1 scores and Standard Deviation by subject, group and region ............. 76
Table 27: Distribution of Grade 1 Average Scores by Percentiles, region and group .................... 77
Table 28: Grade 2 Amharic average scores (%) by sex, group and region ..................................... 77
Table 29: Easiest and most difficult questions in Grade 2 Amharic ............................................... 78
Table 30: Grade 2 English average scores (%) by sex, group and region ....................................... 79
Table 31: Easiest and most difficult questions in Grade 2 English ................................................. 79
Table 32: Grade 2 Mathematics average scores (%) by sex, group and region ............................. 80
Table 33: Easiest and most difficult questions in Grade 2 Mathematics ....................................... 80
Table 34: Grade 2 Environmental Science average scores (%) by sex, group and region .............. 81
Table 35: Easiest and most difficult questions in Grade 2 Environmental Science ........................ 81
Table 36: Average Composite scores (%) in grade 2 by sex, group and region .............................. 82
Table 37: Average grade 2 scores and standard deviation by subject, group and region ............. 83
Table 38: Distribution of Grade 2 Average Scores by Percentiles, region and group .................... 83
Table 39: Grade 3 Amharic average scores (%) by sex, group and region ..................................... 84
Table 40: Easiest and most difficult questions in Grade 3 Amharic ............................................... 84
Table 41: Grade 3 English average scores (%) by sex, group and region ....................................... 85
Table 42: Easiest and most difficult questions in Grade 3 English ................................................. 86
Table 43: Grade 3 Mathematics average scores (%) by sex, group and region ............................. 87
Table 44: Easiest and most difficult questions in Grade 3 Mathematics ....................................... 87
Table 45: Grade 3 Environmental Science average scores (%) by sex, group and region .............. 88
9
Table 46: Easiest and most difficult questions in Grade 3 Environmental Science ....................... 89
Table 47: Average Composite scores (%) in grade 3 by sex, group and region ............................. 89
Table 48: Average grade 3 scores and standard deviation by subject, group and region ............. 90
Table 49: Distribution of Grade 3 Average Scores by percentile, region and group ..................... 90
Table 50: Grade 4 Amharic average scores (%) by sex, group and region ..................................... 91
Table 51: Easiest and most difficult questions in Grade 4 Amharic.............................................. 91
Table 52: Grade 4 English average scores (%) by sex, group and region ....................................... 92
Table 53: Easiest and most difficult questions in Grade 4 English................................................. 93
Table 54: Grade 4 Mathematics average scores (%) by sex, group and region ............................. 93
Table 55: Easiest and most difficult questions in Grade 4 Mathematics ....................................... 94
Table 56: Grade 4 Environmental Science average scores (%) by sex, group and region ............. 95
Table 57: Grade 4 Easiest and most difficult questions in Environmental Science ....................... 95
Table 58: Average Composite scores (%) in grade 4 by sex, group and region ............................. 96
Table 59: Average grade 4 scores and standard deviation by subject, group and region ............. 97
Table 60: Distribution of Grade 4 Average Scores by percentile, region and group ..................... 97
Table 61: Comparison of Grade 4 Results with the NLA (2012)..................................................... 97
Table 62: Overall test results by school type, region and sex ........................................................ 98
Table 63: Frequency of Overall Highest and Lowest Test Scores by Grade and School Type ....... 99
Table 64: Frequency of Overall Highest and Lowest Test Scores by Grade and Region ................ 99
Table 65: Frequency of Overall Highest and Lowest Test Scores by Grade and Sex ..................... 99
Table 66: Frequency of Overall Highest and Lowest Test Scores by Grade and Subject ............. 100
Table 67: The structure of lessons observed by region and school group .................................. 101
Table 68: How students participated in lessons by region and school group ............................. 102
Table 69: The relevance of lessons observed by region and school group ................................. 103
Table 70: The Inclusivity of lessons observed by school group ................................................... 104
Table 71: Student interest and enjoyment of lessons observed by region and school group .... 105
Table 72: The use of teaching and learning aids in observed lessons by region and school group
..................................................................................................................................................... 106
Table 73: Children’s opinions on school by region and school group ......................................... 107
Table 74: Teachers use of classroom based continuous assessment techniques ....................... 108
Table 75: Students experience of assessment ............................................................................. 113
Table 76: Baseline Survey Data collection tools and purpose ..................................................... 117
Table 77: Comparison of results of grade 1 student achievement tests in programme schools in
in the two point study .................................................................................................................. 119
Table 78: Comparison of results of grade 4 student achievement tests in programme schools in
the two point study ...................................................................................................................... 119
Table 79: Comparison of the percentage of students meeting the national benchmark ........... 120
Table 80: Comparison of standard deviations in grade 1 test at baseline (2013) and mid-term
(2015) by subject and school group ............................................................................................. 122
Table 81: Comparison of standard deviations in grade 4 tests at baseline (2013) and mid line
(2015) by subject and school group ............................................................................................. 123

10