Independent Evaluation of the Japan Administered Account for Selected IMF Activities

263 Pages·2010·1.38 MB·English

G L B A L
P A R T N E R S H I P S
Shared Objectives (cid:76)(cid:0)Joint Action (cid:76)(cid:0)Real Impact
Independent External Evaluation
February 2010
Japan
Administered Account for
Selected IMF Activities (JSA)
I n t e r n a t i o n a l M o n e t a r y F u n d
International Monetary Fund (IMF)
Office of Technical Assistance Management (OTM)
Independent Evaluation of the Japan Administered
Account for Selected IMF Activities (JSA)
February, 2010
(MURC)

Table of Contents
List of Abbreviations ..............................................................................................................iii
Executive Summary Evaluation Results and Recommendations .......................................... v
I. Evaluation Results…………………………………………………………………….……...v
II. Recommendations…………………………..………………………………...……..…….xiii
Chapter 1 Outline of the Independent Evaluation ......................................................... 1
1.1 Background and Objectives .................................................................................. 1
1.2 Implementation and Study Activities ..................................................................... 1
1.3 Documents, Data and References ........................................................................ 3
1.4 Evaluation Criteria and Verification Method .......................................................... 5
1.5 Review of Evaluation Reports .............................................................................. 7
Chapter 2 Outline of JSA and Japanese ODA ............................................................ 12
2.1 Outline of JSA .................................................................................................... 12
2.2 ODA Policy of Japan ........................................................................................... 21
2.3 Consistency of IMF/JSA TA and Japanese ODA ................................................ 23
Chapter 3 Evaluation Results through Data Analysis and Questionnaires .................. 27
3.1 Review of Evaluation Reporting System of JSA ................................................. 27
3.2 Efficiency and Cost Considerations .................................................................... 34
3.3 Evaluation Results of the Questionnaire Surveys ............................................... 40
Chapter 4 Evaluation Results through Field Survey .................................................... 59
4.1 Evaluation Results of Field Interviews ................................................................ 59
4.2 Cases of Good Practices .................................................................................... 79
Chapter 5 Overall Evaluation Results and Proposed Rating System .......................... 95
5.1 Overall Evaluation Results ................................................................................. 95
5.2 Proposed Rating System .................................................................................. 102
Chapter 6 Recommendations to IMF and MOF ........................................................ 109
6.1 Recommendation to IMF .................................................................................. 109
6.2 Recommendations to MOF ............................................................................... 111
i
Appendix
A. Questionnaire Survey Sheet: Beneficiaries
B. Questionnaire Survey Sheet: Experts
C. Results of Questionnaire Survey: Beneficiaries
D. Results of Questionnaire Survey: Experts
E. JSA Evaluation Questionnaire (Attachment IV of JSA Guidelines)
F. Results of Collected JSA Evaluation Questionnaire
G. Schedule of IMF Headquarter meetings
H. Schedule of the Field Survey conducted
I. List of JSA-funded TAs in the evaluation period
ii
List of Abbreviations
ADB -- Asian Development Bank
AfDB -- African Development Bank
AFRITAC -- Africa Regional Technical Assistance Center
AML -- Anti-Money Laundering
APD -- Asia Pacific Department
BI -- Bank Indonesia
BOT -- Bank of Tanzania
CFT -- Combating the Financing of Terrorism
DFID -- Department for International Development (UK)
EBRD -- European Bank for Reconstruction and Development
FAD -- Fiscal Affairs Department
FDI -- Foreign Direct Investment
FY -- Financial Year
GBS -- General Budget Support
HQ -- Headquarters
IDB -- Inter-American Development Bank
IEO -- Independent Evaluation Office
IFC -- International Finance Corporation
INS -- IMF Institute
IT -- Information Technology
JICA -- Japan International Cooperation Agency
JSA -- Japan Administered Account for Selected IMF Activities
LEG -- Legal Department
MCM -- Monetary and Capital Market Department
MOF -- Ministry of Finance
MOU -- Memorandum of Understanding
MURC -- Mitsubishi UFJ Research & Consulting
NBC -- National Bank of Cambodia
NIS -- National Institute of Statistics
OAP -- Regional Office for Asia and Pacific
OED -- Office of Executive Directors
OTM -- Office of Technical Assistance Management
PFM -- Public Financial Management
PHRD -- Policy and Human Resources Development
PRGF -- Poverty Reduction and Growth Facility
RA -- Resident Advisor
RAP -- Resource Allocation Plan
ROSC -- Report on the Observance of Standards and Codes
RSN -- Regional Strategy Note
SBV -- State Bank of Vietnam
iii
SIDA -- Swedish International Development Cooperation Agency
STA -- Statistics Department
SWAP -- Sector Wide Approach
TA -- Technical Assistance
TAIMS -- Technical Assistance Information Management System
TCAP -- Technical Cooperation Assistance Program
TOR -- Terms of Reference
UN -- United Nations
UNDP -- United Nations Development Programme
UNFPA -- United Nations Population Fund
WB -- World Bank
WTO -- World Trade Organization
iv
Executive Summary
Evaluation Results and Recommendations
The evaluation results and recommendations in this study are based on the following
sources of information and the analyses conducted by the study team.
(cid:122) The project-level evaluation results from the current IMF/JSA reporting system, i.e.
“Project Assessment Report” and “JSA Evaluation Questionnaire”1.
(cid:122) The Questionnaire Survey conducted under this evaluation, addressed to the
beneficiaries (recipient governments) of JSA and experts employed by IMF for JSA TA
projects.
(cid:122) The results of a field survey conducted by the study team for four countries, Cambodia,
Indonesia, Vietnam and Tanzania (East AFRITAC), in August 2009.
(cid:122) The interviews conducted at IMF headquarters mainly with OTM, TA departments and
regional departments and OED Japan, in July 2009.
(cid:122) The interviews in Japan with Japanese MOF, OAP and ex-advisors to central banks in
Asia under IMF and JICA TA.
(cid:122) The desk analyses of the relevant information collected during this study.
I. Evaluation Results
1. For this evaluation study, the evaluation of IMF/JSA TA for “effectiveness and
benefit to the recipient countries” and “accountability to the Japanese general public” are
categorized by the terms of reference (TOR) with six criteria: 1) Quality of Activities; 2)
Effectiveness and Sustainability; 3) Efficiency; 4) Consistency with Japanese ODA Policy; 5)
Visibility of Japan as Donor; and 6) Advantage of IMF Scheme. On the other hand there
are six criteria by “OECD DAC Criteria for Evaluating Development Assistance” (relevance,
effectiveness, efficiency, impact and sustainability), widely applied to development aid
evaluation, have been modified for this evaluation, in seven criteria, namely 1) Relevance,
2) Effectiveness (Impact), 3) Efficiency, 4) Sustainability, 5) Transparency, 6) Consistency
with Japanese ODA and 7) Visibility of Japan as a donor. These sets of criteria were applied
to conduct this evaluation work.
1. Evaluation Results based on OECD Evaluation Criteria
1) Relevance
2. The Questionnaire Survey conducted by the study team, for both the recipient
countries and experts, clearly indicated that they recognize IMF/JSA TA projects as being
relevant to recipient government needs and priorities. The answers for consistency with

1 One-page questionnaires collected by OTM within five weeks of the completion of each JSA
project, as defined by Guidelines for the Use of Resources for Technical Assistance Activities
from the Administered Account for Selected Fund Activities-Japan (JSA Guidelines).
v
the recipient government policy were either “Excellent” (64%) or “Good” (36%), and there
was no answer for “Modest” or “Poor”, as shown below.
Answers by beneficiaries for consistency with government policy
Answer Response Share
Excellent 25 64.1%
Good 14 35.9%
Modest 0 0
Poor 0 0
No knowledge 0 0
3. The field survey results indicated that IMF/JSA TA projects are relevant to the
recipient countries’ developing priorities, based on their country contexts, and also relevant
to IMF priorities, whose regional strategy is indicated in RSNs.
4. According to the interviews with TA departments at IMF headquarters, IMF TA
projects are highly specialized and quite focused on the relevant development issues in
monetary and finance policies, legislation process and procedures, which make the TA
highly relevant compared to TA by other donors.
2) Effectiveness (Impacts)
5. Overall results of the evaluation surveys indicate that IMF/JSA TA projects are
basically effective and achieved the desired goals. The results of the “Project Assessment”
by TA departments indicate that TA projects have been regarded as effective in light of the
targeted objectives. The results of the “JSA Evaluation Questionnaire” (by the
beneficiaries) indicated an 80% “yes” response regarding the usefulness of the TA, as
shown below.
Answers by beneficiaries for usefulness
Answer Response Share
Yes 25 80.6%
Partially Yes 6 19.4%
6. Based on our questionnaire surveys for the beneficiary countries and experts, the
IMF/JSA TA projects have mostly achieved their expected results, with most answers as
“excellent” (39%) or “good” (62%), and with no answers for Modest or Poor. TA outputs
have been used by authorities and contributed to improvement of knowledge, awareness
and motivation for reforms. In addition, TA outputs, to some extent, have directly led to
improvement of policies and their instruments in the recipient countries. Major reasons for
their effectiveness are: 1) expertise of advisors, 2) experience of experts in other countries,
3) communication skills of experts in understanding country context, and 4) backstopping by
headquarters.
7. The field survey results show that IMF/JSA TAs are quite effective in generating (or
contributing towards generation) of tangible results. Main reasons for the effectiveness
vi
are: 1) expertise of experts, 2) focus on competitive sectors, 3) selection method (roster
system) of experts, 4) communication skills of experts, and 5) backstopping by IMF
headquarters.
3) Efficiency
8. Based on our field survey results, IMF/JSA TAs seem to be efficient due to their
well-focused approach and selection of competent experts. Overall evaluation results of
the study are basically positive in regard to the efficiency of IMF/JSA TA projects.
9. Regarding the efficiency in terms of cost, JSA is well managed through
cost-effective means such as the roster-based selection of experts, smooth implementation
supported by strong backstopping from headquarters, and repeated assignments by the
same expert whose ability is proven. The budget per project and standard cost of long-term
experts indicate that these are well within the reasonable range, compared with TA provided
by Japanese ODA agencies and other international organizations.
10. The questionnaire survey results for JSA experts indicate that in comparison with
other international organizations, IMF/JSA TA is higher in quality by 92%, while the cost is
higher by 46%, indicating comparative higher efficiency in terms of quality and cost
consideration, as shown below.
Answers by Experts for Comparison with Other International Organizations
Points of Difference Answered as “yes”
Quality is higher for IMF/JSA 92%
Monthly remuneration is higher for IMF/JSA 46%

4) Sustainability
11. According to our questionnaire surveys for both recipient countries and JSA
experts, major factors for sustainability of TA projects such as institutional memory, staff
capabilities, or budget allocation are achieved in most of the counterpart authorities, as
shown below. The majority of the experts have expressed their shared views that
sustainability of IMF TA depends on a willingness to reform by authorities.
Answers by Experts for Levels of Sustainability
Answer Response Share
Excellent 21 25.6%
Good 42 51.2%
Modest 11 13.4%
Poor 1 1.2%
No knowledge 7 8.5%
12. The field survey results indicate that the sense of ownership by recipients varies by
country, e.g. Vietnam with strong ownership. The concept of increasing the sense of
vii