End-Project Evaluation of the Youth Leadership, Empowerment, Advocacy and Development in ...

145 Pages·2016·5.73 MB·English

UNICEF SUDAN

End-Project Evaluation
of the Youth Leadership, Empowerment,
Advocacy and Development in Sudan
Final Report
The Youth LEAD Project in Sudan
funded by Canada-CIDA from 2012 to 2015 about USD 11 million,
has been evaluated by the independent consultant team of :

 Mrs Oumoul Khayri Ba Tall, International Consultant Team
Leader
 Dr Rania Elfatih Elsheikh, National Consultant
 Dr Reema Abass, Nataional Consultant for qualitative data
analysis

In consultations with UNICEF Country Office, the Regional
Office and the Ministry of Youth and the Embassy of Canada
in Sudan

2
Table of Contents

ACRONYMS ....................................................................................................................................................... 5
ACKNOWLEDGEMENT ............................................................................................................................... 6
EXECUTIVE SUMMARY .............................................................................................................................. 8
I. OBJECT OF THE EVALUATION .................................................................................................. 15
1.1. INTRODUCING THE CONTEXT OF THE INTERVENTION .............................................................................. 15
1.2. THE PROJECT AREA AND TARGET GROUPS .................................................................................................. 16
1.3. THE PROJECT OUTCOMES, COMPONENTS AND RESULTS FRAMEWORKS .............................................. 17
1.4. THE IMPLEMENTATION APPROACH AND MANAGEMENT STRATEGY ..................................................... 18
II. PURPOSE, OBJECTIVE AND SCOPE OF THE EVALUATION ............................................. 21
2.1. PURPOSE & OBJECTIVE OF THE EVALUATION ............................................................................................ 21
2.2. EVALUATION SCOPE AND KEY QUESTIONS .................................................................................................. 21
2.3. THE EVALUATION METHODOLOGY .............................................................................................................. 24
2.4. COMMENTS ON THE TORS, ADJUSTMENTS AND LIMITATIONS ...................................................................... 30
III. EVALUATION FINDINGS ........................................................................................................... 31
3.1. RELEVANCE ...................................................................................................................................................... 31
3.2. EFFECTIVENESS & LIKELY IMPACT .............................................................................................................. 32
3.3. EFFICIENCY ...................................................................................................................................................... 60
3.4. SUSTAINABILITY .............................................................................................................................................. 65
IV. MAIN CONCLUSIONS & LESSONS LEARNED .................................................................. 68
6.1. CONCLUSIONS .......................................................................................................................................................... 68
6.2. LESSONS LEARNED ................................................................................................................................................. 72
V. RECOMMENDATIONS ..................................................................................................................... 72
6.1. RECOMMENDATIONS TO UNICEF AND THE GOVERNMENT ........................................................................... 72
6.2. RECOMMENDATIONS TO UNICEF ....................................................................................................................... 73
6.3. RECOMMENDATIONS TO THE GOVERNMENT ..................................................................................................... 74
APPENDIX ....................................................................................................................................................... 76
APPENDIX 1 : EVALUATION TORS .............................................................................................................................. 77
APPENDIX 2 : LIST OF DOCUMENTS AND OTHER REFERENCES CONSULTED ......................................................... 84
APPENDIX 3 : LIST OF PERSONS INTERVIEWED, AND SITES VISITED ..................................................................... 86
APPENDIX 4 : THE EVALUATION MATRIX .................................................................................................................. 88
APPENDIX 5 : THE PROJECT RESULTS FRAMEWORK ............................................................................................... 92
APPENDIX 6 : THE PROJECT BUDGET AND EXPENSE REPORT ............................................................................. 113
APPENDIX 7 : THE EVALUATION CORE TEAM ........................................................................................................ 114
APPENDIX 8 : DETAILED DATA COLLECTION METHODS & TOOLS, SAMPLING, DATA ANALYSIS ................... 114
APPENDIX 8.1 : SAMPLING (ESTIMATING THE SAMPLE SIZE, ALLOCATION TO LOCALITIES, SAMPLE METHODS)
........................................................................................................................................................................................ 118
APPENDIX 9 : SUMMARY OF YOUTH FRIENDLY SERVICES AND TRAININGS ...................................................... 119
APPENDIX 10: INSTITUTIONAL CAPACITY BUILDING: LIST OF TRAININGS TO MOYS AND YOUTH CSO STAFF
MEMBERS ...................................................................................................................................................................... 120
APPENDIX 11: TRAININGS PROVIDED BY TO CSO TO YOUTH .............................................................................. 122
APPENDIX 12: YOUTH EMPLOYMENT SURVEY DATA ............................................................................................ 123
APPENDIX 13 : SURVEY POPULATION DATA (SELECTED DATA) .......................................................................... 125
APPENDIX 14 : POPULATION SIZE IN TARGET LOCALITIES .................................................................................. 126
APPENDIX 15 : SURVEY SAMPLE (ENGLISH VERSION) ......................................................................................... 127
APPENDIX 16 : KEY INFORMANT INTERVIEW QUESTIONNAIRE (KII) -YOUTH ............................................... 131
APPENDIX 17 : FOCUS GROUP DISCUSSION GUIDE (FGD) .................................................................................. 143
APPENDIX 18: CASE STUDY WORLD CHILD CANADA (WCC) ............................................................................. 145
3
Table of Illustrations - Photographs
FIGURE 1: FATUMA FROM GOKER, TRAINED IN MAÇONRY .................................................................................................................. 47
FIGURE 2: FATUMA REBUILT HER DESTROYED HOUSE, AFTER TRAINING .......................................................................................... 47

Table of Illustrations – Tables & Graphs
TABLE 1 : SAMPLE SIZE CALCULATION FOR THE SURVEY ........................................................................................................... 29
TABLE 2 : DATA COLLECTION TOOLS ................................................................................................................................................. 29
TABLE 3 : YOUTH FRIENDLY TRAININGS & SERVICES TO YOUTH BENEFICIARIES, PER STATE (SUMMARY) .............. 33
TABLE 4 : TARGET POPULATION AND REACH, PER STATE (SUMMARY) .................................................................................. 33
TABLE 12 : YOUTH INVOLVEMENT IN COMMUNITY (Q9) ............................................................................................................ 37
TABLE 4 : ALP ENROLMENTS PER YEAR, BY SEX.......................................................................................................................... 41
TABLE 13 : TRAININGS ACHIEVED VS PLANNED VS EMPLOYABILITY .................................................................................... 43
GRAPH 1 : PROJECT TRAININGS (VCT), KASSALA STATE .......................................................................................................... 44
GRAPH 2 : PROJECT TRAININGS (VCT), WEST DARFUR STATE ................................................................................................ 44
TABLE 5 : RELATE CURRENT OCCUPATION TO VCT ..................................................................................................................... 45
TABLE 6 : IMPROVED EMPLOYABILITY BY VCT ............................................................................................................................ 45
TABLE 11: CONTRIBUTIONS OF YOUTH TO INCREASED FAMILY EARNINGS.......................................................................... 48
TABLE 8 : PARTICIPATION TO STOP AND ACT EVENTS (PROJECT FINAL REPORT) ............................................................. 49
TABLE 15 : PARTICIPANTS TO STOP AND ACT TRAININGS & OCCUPATION (SURVEY) ...................................................... 50
TABLE 10 : NUMBER OF TRAINEES AND PARTICIPANTS COMPARED TO PLANS ..................................................................... 64
TABLE 7 : SUMMARY OF VOCATIONAL TRAININGS (NBR) PAR LOCALITY, (ACTUAL VS PLAN) ....................................... 65

Report Submitted by the Evaluation Core Team:
-Ms Oumoul Khayri Ba Tall
-Dr Rania Elfatih Elsheikh

The Final version of the report is endorsed by UNICEF on 26 February 2016 following
technical quality assusrance and the national workshop organized in December 2015 jointly
by the Ministry of Youth and UNICEF for discussions of findings, lessons learned and
recommndations.
4
Acronyms

ALP Accelerated Learning Programme
AUW Ahfad University for Women
CIDA Canadian International Development Agency
CPA Comprehensive Peace Agreement
CSO Civil Society Organization
DPA Darfur Peace Agreement
FGD Focus Group Discussion
FGM/C Female Genital Mutilation / Cutting
GBV Gender Based Violence
GE Gender Equality
HIV/AIDS Human Immunodeficiency Virus / Acquired Immunodeficiency Syndrome
IDPs Internally Displaced Person(s)
IGA Income Generating Activity
KII Key Informant Interview
MICS Multiple Indicator Cluster Survey
MOE Ministry of Education
MOH Ministry of Health
MOSD Ministry of Social Development
MOYS Ministry of Youth and Sports
MTR Mid-Term Review
NGO Non-Governmental Organization
PCA Programme Cooperative Agreement
PMF Performance Measurement Framework
RH Reproductive Health
STIs Sexually Transmitted Infections
TOC Theory of Change
UNDAF United Nations Development Assistance Framework
UNFPA United Nations Population Fund
UNICEF United Nations Children‘s Fund
UXO Unexploded Ordinance
VCT Vocational Training
WASH Water, Sanitation and Hygiene
WD West Darfour
WHO World Health Organization
YC Youth Center
YFS Youth Friendly Space
YFSS Youth Friendly Space and Service
Youth LEAD Youth Leadership, Empowerment and Development
5
Acknowledgement
The Evaluation Team is grateful to UNICEF Representative in Sudan (Mr Geert Cappelaere),
the former UNICEF Deputy Representative (Mrs Shaya Asindua) and the Ministry of Youth
who approved and provided strong support to this evaluation as part of UNICEF’s
accountability to the Donor.
The Evaluation team is grateful to all the people who have taken part or contributed to this
work in one way or another, and would like to acknowledge their valuable contributions in
helping to improve the work. From the early days of the planning of the field work to the final
evaluation report, many people were involved in ways that made it possible to start and
complete this assignment, they belong to:
 The UNICEF Evaluation Office in Khartoum led by Robert Ndamobissi Chief of
Planning, Monitoring & Evaluation, Aala Mahmoud Monitoring & Evaluation
Specialist who assured the Technical direction, guidance, management and quality
assurance of the Evaluation. The Youth LEAD management team composed of Mrs
Jumana Haj-Ahmad former Manager of Youth LEAD Project, Mellese Bedanie and
Buthayna Osman Abdel Hakam Youth Project Officers, and UNICEF field offices in
Kassala (Ali Abu ElGasim, Abdel Raouf Ahmed) and in Geneina Waala Hussein
Awad who provided adequate informations related to the project and assistance to the
field data collection in selected communities.
 The experts of Central Bureau of Statistics (CBS), data collectors and enumerators in
Kassala, WD states, and Khartoum (Dr Reema Abass) provided technical support for
the survey. They reviewed the survey questionnaire to make it compatible with the
field language, helped in the selection of the sites within each locality, organised
expert and qualified teams to collect, compile and summarize data, providing even a
first level analysis, including a debriefing meeting (in Kassala) to share their field
observations.
 The Ministry of Youth-MOYs focal points in Khartoum and Kassala, implementing
agencies, CSOs for their contributions in the data collection, and their feedback from
the validation workshop
 The UNICEF Regional Office in Jordan (Mrs Roumiana Gantcheva): The quality
review assesment performed by UNICEF regional office helped focus the evaluation
report and streamline responses to the initial evaluation’s questions of the ToR. The
report was reshaped after this review to adopt an adapted definition of « impact » vis-
à-vis the five OECD-DAC evaluation criterias by incorporating the longer-term
horizon for an unfinished business (the project implementation was ongoing).
Acknowledging that impact should be considered in the longer term horizon, we
agreed with the Evaluation Office at UNICEF Sudan to include the “likely impact”
under the assesment of effectiveness of the project, to echo the qualitative evidence
collected from triangulated FGD, KII and the opinion survey pointing to potential
sustainable impact in the lives of young people and an improved enabling environment
for youth programming in Sudan.
 A Theory of Change has been developped aposteriori by the independent Consultant
Team as requested by the UNICEF Evaluation Office in Sudan in order to show the
potential expected benefits of this large investment on Youth well-being in the view of
Value For Money (chain of results) in a challenging operating context.
The evaluation findings were extensively reviewed by project stakeholders all of whom have
taken part to the national validation workshop organized by UNICEF and the Ministry of
Youth following in December 2015 based on an earlier evaluation draft report. They provided
additional information that was taken into account in this final report and also endorsed the
final conclusions and recommendations in consensus. All those contributions should be
considered as part of the evaluation process as a good practice of great ownserhip of an
evaluation for learning, accountability and improving strategies for greater impact on Youth.
The evaluation team is also grateful to the UNICEF operations and logistics staff who worked
so hard to make us travel to the field in very difficult circumstances…up to the remote Sirba
under stormy water and security escort.
Most of all, we would like to recognize all the young boys and girls, and those we met in the
Youth Centers (YCs), for their strong enthusiasm and appreciation of what the project aims to
provide them, we hope we have been able to convey at least some level of their faith in the
YouthLEAD aspirations.

We acknowledge some limitations of performing the assessment of impact using the
comparative community group in the absence of initial base line survey of the project and
taking into consideration the challenges of conflicts affected areas. We apply adapted
methods and tools that enabled to generate valuable combined quantitative and qualitative
evidence that support conclusions and recommendations that have been approved by all
parties. This end-project evaluation is a concrete example of “real world evaluation of
development end-project in a difficult context of country facing protracted emergency crisis”.

We appreciate UNICEF’s and the MOY willingness and openness to undertake this exercice
for better evidence-based policy & strategy and accountability of results for children in
application of the programmatic shifts adopted by Government and UNICEF at the 2014 Mid-
Term Review for doing business differently in Sudan.

The Evaluation Core Team:
-Ms Oumoul Khayri Ba Tall
-Dr Rania Elfatih Elsheikh
7
Executive Summary

1. Presentation of the Evaluation Object
UNICEF has completed the independent ex-post evaluation of the Youth Leadership,
Empowerment and Development project (YouthLEAD) implemented during 3 years (March
2012-June 2015) in Sudan. The project ultimate goal is to achieve « Enhanced contribution to
socio economic development, and to the development of peace, for 56 000 young women and
men aged 10-24 years, in Kassala and Aroma in Kassala Sate, and Geneina and Sirba in West
Darfur State ».
YouthLEAD is a holistic and innovative pilot project, designed and implemented in a
participatory way, building on the vision of young people as potential agents of social change
and peace. The 2013 gender analysis confirms that youth is a central player « to ensuring
sustainable recovery and peace building in Sudan », while mainstreaming gender, when it’s
potential for development and peace is valued.
Most of the population in Sudan is considered young, as illustrated in the 2014 MICS survey
with 50.6% under 18 years. Young people in Sudan have been « living in a country that has
been into war for the most part of their lives », and they face multiple challenges such as high
level of poverty (50% from UNDAF 2009-2011), high unemployment level (66% Sudanese
aged 15-29 are unemployed, while youth unemployment reaches 88% in our Survey); high
level of out of school (only 20% (1 in 5) children completed primary school, …), high level of
illiteracy particularly among women (36,2% of males and 63,8% of women have never been
to school) ; vulnerability to HIV and other sexually-transmitted infections, and particularly
women (UNDP, 2009), in addition to associated risks such as early pregnancy, unsafe
abortions, and paid sex.
In order to help young people face the challenges and become productive and responsible
citizens, two intermediate results were targeted: (A) Increased equitable engagement in
sustainable livelihoods, peace building, and use of economic and social services by young
people, and (B) Enhanced ability of targeted government Ministries and CSOs to design and
implement participatory, gender responsive policies, programmes and services. The project
was implemented from 2012 to mid-2015, in four localities in four localities of two states:
Aroma and Kassala in Kassala State, and Sirba and Geneina in West Darfur State.
To achieve these results, a variety of activities aimed at the youth and youth service providers
were identified and organised in five components: (i) Youth Friendly Spaces and Services, (ii)
Basic Education, Vocational Skills Training, (iii) Youth Voices in Governance and Peace, (iv)
Capacity building in Policy and Advocacy, and (V) Institutional Capacity Building of Civil
Society Organisations (CSOs) and Ministries supporting youth.
The project activities include youth friendly spaces, life skills development, youth friendly
trainings/services, vocational trainings, ALP, gender awareness/advocacy and gender-
sensitive youth programming trainings, management trainings and institutional capacity
building for YSP. The identification of services was based on needs assesment, and used a
participatory approach with the active involvement of youth, and other project stakeholders
such as community leaders, parents and YSP.
UNICEF developed partnership agreements with INGOs, CSOs and other national
institutions, in addition to government line Ministries at federal and state levels, to carry out
specific activities, and for the coordination at state and federal levels. Project Cooperation
Agreements (PCA) were signed with Plan International (who sub-contracted Concern
8
Worldwide, and CDA in WD), War Child Canada (WCC), and Child Development
Foundation (CDF) a well established national NGO. Besides CDF, National Partners include
the Afhad University for Women, and NNGOs subcontracted as implementation partners
(CDA, SORD, FPDO), to conduct a number of specialized skilled activities. Small Scale
Funding Agreements (SSFA) were signed with 13 youth CSOs to strengthen their capacity as
Youth Service Provider (YSP) while they were tasked to implement activities related to their
respective fields.

2. The Evaluation Design
To measure outcomes and assess the potential for impact, UNICEF seeks to provide solid
evidence on what difference the YouthLEAD investment made for the personal development
of the target youth and adolescents, for their role in the communities, and for the institutional
arrangements regarding youth affairs.
The evaluation was designed using the DAC criteria of Relevance, Effectiveness, Efficiency,
Impact, and Sustainability; in addition to the criteria of Coherence considered important
given the diverse and complex nature of the intervention, and we have adressed working thru
the logic of the intervention. The evaluation assessed the project relevance, its contribution
to improved social and economic development of youth, to the development of peace, to
greater equity and gender balance, and questionned the validity of the intervention logic. The
Evaluation reviewed how girls and young vulnerable groups were reached, and how gender
equality and equity were promoted. With regard to the Impact criteria, it was agreed to look
into emerging and likely impact only, under the assesment of project outcomes, as the project
was just closing, and the timing was not suitable to an assesment any impact as defined by the
OECD-DAC. The evaluation also assessed the project contributions to enhanced capacities of
youth service providers to design and implement gender-responsive policies, programs and
services to the youth. Again, the evaluation also assessed important signs of improvement in
the youth environnement and service delivery, looking for the likelihood of development
impact.
We designed a mixed method evaluation approach, with a quasi-experimental impact
assesment survey, focus group discussions, Key informant and other semi-structured
interviews with diverse project stakeholders (youth, youth commitee members, community
leaders, implementation partners, and UNICEF staff involved in the project), in addition to a
number of documentation reviews, including financial reports. The sample size for the survey
is estimated at 600 individuals (by application of the statistical formula of probabilitst sample
design performed with technical assistance of the CBS) using the highest number of youth
involved in any given activity, which was then divided between the two states, for 300 for
each, with 100 dedicated to a non-intervention locality to serve as a comparison group. The
sample was allocated to different sites where interviews were to take place, prorated by the
population size. Individuals surveyed within each site were then randomly selected, with a
purposeful criteria to target the age range and young girls.
The table below provides a list of sites where the interviews have taken place within each
selected locality in the project and the comparison areas:
Kassala State Frequency WD State Frequency
Hay Alshaty 50 Alzohor 25
Dar Alsalam 25 Alemtidad 25
Ard Alhajar 25 Alrriad 10
Hay Alarab 20 Om Alqora 10
Alhalanga 20 Ardamga 10

9
Alssoq Alshaaby 15 Kerandaq 10
Almorabaat 15 Aljabal 10
Alkormota 15 Sirba 50
Alamiriah 15 Katarby 30
Aroma 50 Bir Daqiaq 20
Albatary 25 Morni 100
CMacaly 25
Total 300 Total 300

The control locality for Kassala is Khashmelgerba and Keriank in WD, where the Morni IDP
camp is located. The first choice was not possible due to logistics and security obstacles,
therefore, the choice of Keriank was by default, and it happened that the locality is
challenging for the quality of the comparative data as it is an IDP camp, and one that has been
receiving important donor assistance, which included many of the same Youth project
activities. Therefore we made extensive use of other data sources, from KII and FGD as
triangulation and for further exploration on the causes of the observations. Despite the limits
due to imperfect comparison, we believe that the data collected will still provide valuable
information if properly triangulated.

3. The Evaluation Findings and Conclusions

3.1. Relevance of the YouthLEAD response to youth priorities
The availability of relevant comprehensive set of interventions as key priorities for youth and
enhanced institutional services have been well appreciated both by young people and their
communities. All services offered fall into their priority needs, and most appreciated was the
YFSS model. The project model is unique so far in Sudan, in that it adresses both the demand
(thru youth empowerment and leadership) and the supply side (from YSP improved plans and
delivery) of youth programming, in a harmonious, inclusive and equitable manner.

3.2. General Effectiveness
Evidence-based findings revealed that the YouthLEAD project created a unique space for
young people, to meet first, and exercise their newly acquired knowledge, skills and
competencies to involve themselves into the community events and processes, from and
around the new centers. Their active involvement in the project allowed young people to gain
greater attention, from their peers and from the community leaders and officials, which made
them able to foster the new thinking and skills embedded into the teachings they have
received, to push youth leadership roles and GE forward.
Major improvements were noted, although not always at a sizeable scale, but important
enough to deserve acknowledgement and attention given the potential for impact.

3.3. Effectiveness of Youth Friendly Centers (YFC) and Youth Friendly Spaces and Services
(YFSS)
The project was succesful in making the YFCs the focal point for Youth friendly activities
and Youth involvment into the community, including girls. Girls were able to find a suitable
space to accomodate their needs, which all stakeholders acknowledged. The YFC conveys a
new vision with a broader set of activities and new roles (YFSS), such as education,
awareness and capacity building in addition to the traditional entertainment and sports. The
stigma of « boys-only Clubs » strongly attached to the old fashion Youth Clubs is seriously
10