Download 2010 in itinere evaluation report - Programme Med

81 Pages·2012·2.21 MB·English

1

0

1
0
2
T
R
O
P

E E
R R N
O
E IS
N R
N E
I O V H
T I S
T IL
I
G
N A N
E
U
I
L
A

V
E

Parcourir L’Europe
Sandra Ungemach-Bensaid,
Jeanne Geoffroy, Daniel Poulenard
28 Avenue Léon Blum
31500 Toulouse/France
www.reseau-parcourir.eu

In itinere evaluation Programme MED Parcourir- 09/2011
2

TABLE OF CONTENTS
PART 0: INTRODUCTION/METHOD .......................................................................................................................................................... 3
MED PROGRAMME ........................................................................................................................................................................................ 4
INTRODUCTION PRESENTATION ................................................................................................................................................................... 5
GENERAL INTRODUCTION TO THE METHOD .............................................................................................................................................................. 6
AUDIT OF USE OF EVALUATION TOOLS .................................................................................................................................................................... 7
PART 1: THE INTERREG IV MED PROGRAMME IN PERSPECTIVE ................................................................................................................ 9
PERSPECTIVE, ISSUES AND COOPERATION AREA ......................................................................................................................................... 10
definition of the cooperation area ......................................................................................................................................................... 10
PERSPECTIVE, GOVERNANCE AND IMPLEMENTATION ................................................................................................................................ 12
The 2000-2006 legacy .......................................................................................................................................................................... 12
Challenges facing the MED programme since 2006-2007. ..................................................................................................................... 13
Response to the challenges ................................................................................................................................................................. 13
PART 2: PROGRAMME EVALUATION ..................................................................................................................................................... 17
ANALYSIS OF PROGRAMME LOGICAL FRAMEWORK ................................................................................................................................................. 18
Introduction ........................................................................................................................................................................................... 18
State of implementation of Programme................................................................................................................................................. 18
Conclusions ............................................................................................................................................................................................ 36
BENCHMARKING COMPONENTS (RATING) ............................................................................................................................................................. 37
Methodological rating principle ............................................................................................................................................................. 37
Analysis of questions submitted for rating ............................................................................................................................................. 40
conclusions ............................................................................................................................................................................................. 44
PART 3: PROJECT EVALUATION .............................................................................................................................................................. 45
REMINDER OF METHODOLOGY ................................................................................................................................................................... 46
OVERVIEW OF PROJECT SELECTION ............................................................................................................................................................. 46
CLASSIFICATION OF PROJECTS BY TYPOLOGY .............................................................................................................................................. 47
INTERVIEW METHOD .................................................................................................................................................................................. 49
Establishing a sample group of projects to be evaluated ....................................................................................................................... 49
Interview Methods ................................................................................................................................................................................. 53
ANALYSIS OF THE "PROJECT LIFE".................................................................................................................................................................... 55
ANALYSIS OF COMMUNICATION .......................................................................................................................................................................... 59
ANALYSIS OF COMMUNICATION .......................................................................................................................................................................... 60
CONCLUSIONS ................................................................................................................................................................................................. 61
PART 4: RECOMMANDATIONS’ SYNTHESIS ............................................................................................................................................ 62
METHOD REVIEW ............................................................................................................................................................................................ 63
THE OVERALL PROGRAMME IMPLEMENTATION PROCESS .......................................................................................................................................... 63
ANALYSIS OF THE PROGRAMME-PROJECTS COHERENCE ............................................................................................................................................ 65
CAPITALISATION, MUTUALISATION, FEEDBACK ....................................................................................................................................................... 66
PROPOSALS, PRECONISATIONS ............................................................................................................................................................................ 67
Proposed general organisation: programme management, project monitoring .................................................................................... 67
General recommendations ..................................................................................................................................................................... 69
Short-term recommendation: Evaluating scenarios for using available funds in the last phase of programming ............................. 70
PART 5: ANNEXES.................................................................................................................................................................................. 72
GUIDES D’ENTRETIENS ................................................................................................................................................................................... 73
pROJETS ................................................................................................................................................................................................. 73
aUTORITES NATIONALES ........................................................................................................................................................................ 75
stc /aug- JTS /MA ................................................................................................................................................................................... 77
INTERVIEWS LIST ......................................................................................................................................................................................... 81

In itinere evaluation Programme MED Parcourir- 09/2011
3

PART 0: INTRODUCTION/METHOD

In itinere evaluation Programme MED Parcourir- 09/2011
4

MED PROGRAMME

Programme agreed the 20th December 2007 by the European Commission:
Eligible Regions
Managing structures
 Cyprus: the entire country  Managing Authority : Provence Alpes Côte d’Azur
 France: 4 regions - Corse, Languedoc-Roussillon, Region
Provence Alpes Côte d'Azur, Rhône-Alpes  Joint technical secretary : Provence Alpes Côte d’Azur
 Greece : the entire country Region
 Italy : 18 regions : Abruzzo, Apulia, Basilicata,  Liaison office between Med et ENPI CBC
Calabria, Campania, Emilia-Romagna, Friuli-Venezia Mediterranean programmes : Valence
Giulia, Lazio, Liguria, Lombardy, Marche, Molise,  Liaison office between the programme and IPA
Umbria, Piedmonte, Sardinia, Sicily, Tuscany, Veneto. partners : Thessalonique
 Malta: the entire country  Certifying Authority : Caisse des dépôts et
 Portugal : 2 regions - Algarve, Alentejo consignations
 Slovenia: the entire country  Audit Authority : Commission Interministérielle de
 Spain: 6 autonomous regions - Andalusia, coordination des contrôles
Aragon, Catalonia, Balearic islands, Murcia, Valencia -  Monitoring Committee: representatives of each
and the two autonomous cities - Ceuta and Melill Member States, MA, Audit Authority, Certifying
 United-Kingdom : 1 region of economic Authority, representatives from liaison offices, and
programming - Gibraltar economic abd social partners.

Priority axes and financial allocation
At the programme level : 256 million €, of which 193 millions are ERDF
 Axe 1 : Strengthening innovation capacities : 77 millions €, of which 58 millions are ERDF
 Axe 2: Protection of the environment and promotion of a sustainable territorial development: 87 millions €, of
which 66 millions are ERDF.
 Axe 3 : Improvement of mobility and territorial accessibility : 51 millions €, of which 38 millions are ERDF
 Axe 4 : Promotion of a polycentric and integrated development of the Med space: 26 millions €, of which 19
millions are ERDF
 Axe 5 : Technical assistance : 15 millions €, of which 12 millions are ERDF.

In itinere evaluation Programme MED Parcourir- 09/2011
5

INTRODUCTION PRESENTATION
The 2010 in itinere evaluation report represents the second stage of the 2010 evaluation work. The first stage
was carried out internally with the programme bodies to define the method and terms of reference. This report
constitutes the first deliverable for dissemination. In mid-2011 a further document shall set out specifications
for the implementation of the recommendations presented in the fourth part of this report.
The work of evaluation started in February 2010 and the following tasks have been completed:
Production of evaluation guidelines March April 2010
Documentary analysis of the programme documents and the two calls for February June 2010
projects (2008 and 2009) and benchmarking.
Production of a "Programme authorities" maintenance guide
May 2010
Discussions with the authorities in charge of the programme.
May - September 2010
Production of the list of projects to assess and an interview guide
June 2010
Interviews (Traditional projects only)
September 2010 - January 2011
JTS/MA Interviews
December 2010 - January 2011
Drafting of evaluation report
December 2010 - February 2011
Working notes were produced during this first evaluation period
May 2010 - July 2011

In parallel with this, the evaluation team took part in transnational meetings during the whole period, including
the ones dedicated to the Strategic projects (Santorini, Chania, Thessalonica, Marseille, and Madrid).
Implementation of evaluation work, progress and limitations:
The evaluation work is fitted to the implementation schedule of the program. Given the launch period for the
first call for Strategic project submissions, the projects from the first two calls for Traditional projects were
assessed. During 2011, the Strategic projects shall be assessed.
The visits to the projects’ partners and lead partners were, globally very much welcomed. The project
interviews required more time than initially anticipated (from September 2010 to January 2011) . Some
projects were late in launching their transnational activities. For this reason, the evaluation team prioritised
interviews with the leaders and some partners. Participation in transnational activities have benn partially
postponed untill 2011. The analysis devoted to the rating of projects shall be carried out in 2011 for Traditional
and Strategic projects.
In 2011 specific evaluations regarding categories of projects, partners or results may be carried out at the
request of the programme's managing authorities.
We are extremely grateful for the availability of the persons met during this first, 2010, stage of the in itinere
evaluation and their welcome

In itinere evaluation Programme MED Parcourir- 09/2011
6

GENERAL INTRODUCTION TO THE METHOD
GOALS OF THE EVALUATION: IN ITINERE STRATEGIC EVALUATION (CE DOC 5)
The terms of reference, as well as our proposal, make refer to the recommendations of CE Doc 5 concerning
the in itinere strategic evaluation and, in particular, to the following points
• Macroeconomic impact of Structural Funds
• Relevance and consistency of strategies at national/regional level
• Proposals for adjustments of context
• Thematic and strategic targeting
• Inventory of good practice
• Impact indicators
• Other means of achieving the impact sought (rating)

As the evaluation process is anticipated to take 36 months, we have methodologically and chronologically set
specific goals for our initiative:
– Main stages of the evaluation process

• 2010: overview
• 2011: strategic adjustments
• 2012: monitoring and audit
OUR WORKING ASSUMPTIONS
Our method is focussed on the following points:
• Importance of taking project evaluation into account during the process of Programme evaluation:
project typology, state of progress etc. This implies doing a cross-evaluation:
– Between programme and projects,
– Resulting from different perspectives of the beneficiaries of the programme, national bodies,
the MA (Managing Authority) and the JTS (Joint Technical Secretariat)
• Rating projects: comparison of the sample of projects with:
– MED Programme "atypical" projects
– Similar projects outside the MED Programme
• Putting the evaluation in context:
– Analysing the Programme environment, mainly at the level of regional, national and
European reference policies
– Comparing the Programme governance regime with similar programmes in terms of
territorial cooperation or thematic targeting.
The domain of in itinere evaluation covers the analysis of projects and of the Programme, across three distinct
themes included in our approach (Programme/projects different perspectives):
 Evaluation of projects (Traditional/Strategic): As the programme can mainly be assessed through its
principal outputs, i.e. the MED projects, an essential part of the evaluation shall be concentrated on
the processes of project selection and implementation
 Programme Governance: Our cross-disciplinary approach enables a comparison between the selection
and implementation process of projects in one section, the visions and decision-making process of
Programme bodies concerning its implementation.
 Communications and capitalisation of projects and of the Programme constitute the third pillar of our
themes of evaluation

In itinere evaluation Programme MED Parcourir- 09/2011
7

Overall evaluation methodology
Evaluation team
Analysis
Analysis
Process Opinions Projects
& organisation Goals Relevance
Consistency Consistency

Relevance

Interviews Cross-evaluation Interviews
Programme Programme & Project partners
instances projects
Performance
Intervention

approach
Benchmarking
Rating Reference
Reference
programmes and
documents(Program
policies (local,
me, territorial,
Reference projects and national,
thematic)
players (stakeholders, key transnational)
players, other types of
project)

AUDIT OF USE OF EVALUATION TOOLS
The work of evaluation was based on two elements:
 The proposal drafted in the context of the call for tenders. A series of evaluation tools were proposed
 The available Programme components and development of the Programme implementation schedule
made it necessary to adapt the tools initially proposed.
We have deliberately described each tool and method used at the beginning of each Section of this
Evaluation Report, rather than commenting on them here.

Proposal of assignment provision Problems encountered Adapting the use of tools in the first
evaluation tools (Service 5: phase of the evaluation
Performance of evaluations of the
2010 and 2012 Programme)
Analysis of call for projects procedures None Performed for 2008 and 2009 calls for
projects
Analysis of decision-making process None Performed
Analysis of national and transnational None Performed
structures and bodies
Analysis of allocated resources Absence of suitable documents Evaluation chart proposed by the current
evaluation team
Cost analysis Collection of sources to be Collection of documents to be carried
completed out
Producing a set of indicators Collection of sources to be In progress
completed
Selection and analysis of reference None Performed for this evaluation
documents
Contacting people Difficulty organising meetings with Carried out for the projects from June
project workers/eployees due to 2010
their lateness in starting activities
(2nd call)
In itinere evaluation Programme MED Parcourir- 09/2011
8

Implementation 2010 (according to project evaluation dates) cf. evaluation proposal
MA interviews Change to initial schedule Interview carried out at end of first stage of
evaluation
National Authorities interviews Change to initial schedule Carried out first in order to have an analysis
of the general perception of the Programme
and to prepare project evaluations. Increase
in the number of interviews from that in the
initial proposal
JTS interviews Change to initial schedule Carried out after the project interviews to
facilitate correlation of information.
Correlation of project evaluations None Performed
Documentary analysis None Performed

Initial evaluation report including Change to initial schedule For a complete 2010 evaluation in
compiling project data compliance with the goal, the production of
the report was put back in order to have the
returns of all project interviews and of the
Programme.

Service 4 cf. assignment Included cf. Assignment tender Problems encountered and
tender "Production of change to performance
methodological tools"
Proposals document Description of issues, underlying assumptions, Presentation to the Monitoring
evaluation principles Committee, Chania June 2010
Setting evaluation goals
Courses of action to achieve these goals
Project evaluation materials Memorandum of reference documentation Carried out (Traditional
Sampling projects only) interviews were,
Evaluation chart as a priority, targeted on the
Lead partner interview guide lead partners
Partners questionnaire
Activity/site visit audit sheet
Programme evaluation Memorandum of reference documentation Completed except for chart of
materials Indicators chart indicators in progress
Interview guide
Interview audit sheet
Initial projects evaluation Summary of projects evaluation campaign: description Information included in the
report of projects, possible lines of development, analysis of 2010 report because of delay
project/Programme connections due to the time taken for
project interviews
Initial Programme evaluation Interview summary sheets Ditto
report Summary of the evaluation campaign: projects
evaluation return, possible lines of development,
analysis of project/Programme impressions etc.
Advice note Summary note analysing the projects and Programme Note summarising
evaluation campaigns: context analysed, recommendations presented
recommendations for amendments and/or corrections to the Monitoring Committee,
Valencia, February 2011
2010 Report Summary of approach followed Submission early March 2011
Summary tables and diagrams
Documentary appendices (methodology guide,
standard tools, projects/Programme evaluation charts)
In itinere evaluation Programme MED Parcourir- 09/2011
9

PART 1: THE INTERREG IV MED
PROGRAMME IN PERSPECTIVE
In itinere evaluation Programme MED Parcourir- 09/2011
10

PERSPECTIVE, ISSUES AND COOPERATION AREA
DEFINITION OF THE COOPERATION AREA

2000-2006 2 distinct programmes,
MEDOCC and ARCHIMEDES, in the
same cooperation area, aiming to
bring together adjacent non-
member states or adjacent
candidate countries, essentially in
the east and south.
2007-2013 the 2 areas brought
together in one and the same
programme, the dual purpose of
which is to bring together pre-

accession assistance countries (IPA
€199 900 000 - Balkans) in coordination with € 79.536.208
ENPI MED (southern
Mediterranean shore)
The problem to be solved for this INTERREG cooperation area is one of the most complex for the EU 27,
exceeding the complexity which all previous programmes have attempted to address:
The first difficulty is that the cooperation area is located in a zone of major geopolitical and economic issues
exceeding the scope of the objectives of an INTERREG programme. Unlike in certain contact areas with non-
member states such as the Baltic, for example, major transformations and differences are involved here. For
example, the ex-ante evaluation of the Archimedes programme in 2001 concluded that the only two common
issues between the countries of the northern shore and the others were the environment and culture!
The second difficulty which is specific to the MED programme lies in the geographical extent of the
cooperation area. The diversity of territories and socio-economic situations between Member States emerges
as soon as we go beyond the phase of analysing the major indicators. The previous programmes have dealt
with this issue for each programming period. The intermediate MEDOCC evaluation included annexes to
measure developments since the start of the programme. The analysis instruments available in the Operational
Programmes (OPs) are either too simplistic (such as the SWOT analysis) or out of date (ESDP1). The MEDOCC
programme was the last to make significant reference to this work.
The third difficulty is in what lies ahead. What types of work can support the OP in building realistic
operational objectives. Work is currently in hand (ESPON or example) or is continually in operation, such as
EUROSTAT statistics. These types of work have been little used so far. For its part, the INTERACT programme
addresses the need to analyse good practices and to establish links between programmes and between
projects. Conversely, it makes little contribution to the issue of the territorial impact of the programme.

1
European Spatial Development Perspective
In itinere evaluation Programme MED Parcourir- 09/2011