Background and rationale of the study
Objectives of the study
Methodology
This report
The FAO/Italy Inter-regional Project for Participatory Upland Conservation and Development (PUCD Project) aims at identifying and field-testing strategies, methods and techniques for promoting and consolidating people's participation in the integrated management of upland watersheds The project was begun in 1992 and, until 1997, was implemented in two phases in selected watersheds of Bolivia, Burundi, Nepal, Pakistan and Tunisia A two-year third phase is currently being conducted in the three countries that the Tripartite Review Mission indicated as the most promising for institutionalising the project experience at the national level (Bolivia, Nepal and Tunisia)
Since its inception, the PUCD Project has been a pilot, process-oriented, learning initiative It was designed to be flexible in terms of the nature and type of activities and outputs An iterative process of assessment, planning, implementation, monitoring, evaluation, and re-planning has been developed to ensure that field activities and specific outputs are consistent with the needs of participating communities and the capabilities of local counterparts and partner institutions1.
1 See COO/94/1. For the sake of brevity, all references to project materials included in this paper are denoted using the following code: the PUCD field component (COO = Coordination Unit; PUB = publications; BOL = Bolivia; NEP = Nepal; PAK = Pakistan; and TUN = Tunisia); the year of preparation (94, 95, 96, etc.); and the month of delivery (1 = January; 2 = February; 3 = March; etc.). The list of documents consulted is presented in Annex II.
During the second phase of the project (1994-97), a specialist consultant (the author of the present report) provided technical support to National Field Teams (NFTs) and the Coordination Unit in developing systems for internal planning, monitoring and evaluation (PME). Based on a needs-assessment, and following a preliminary reconnaissance to Bolivia, the specialist consultant prepared a framework for these systems in 1994 2. This framework was meant to be structured enough for establishing a common PME practice among the different field components of the project, yet flexible enough to be adaptable to the variety of institutional and social environments
2 See COO/94/8.
From 1994 to 1996, a number of field missions to Bolivia, Nepal, Pakistan and Tunisia allowed this framework to be adapted to each specific national component 3 During these missions, interaction with the NFTs facilitated the development of several amendments to the consultant's initial proposal. Further suggestions for improvement were provided by discussions held during yearly Internal Technical Review Meetings (ITRM) and informal exchanges among NFTs. The PME systems implemented in each country can thus be considered as the outcome of a joint elaboration process.
3 Unfortunately, it was not possible to directly involve the NFT in Burundi in this process. In fact, two missions of the consultant to this country were canceled due to prevailing security conditions. Consequently, the NFT in Burundi autonomously developed its own project-level PME practice. However, the unique conditions in which the Burundi NFT's PME experience developed and the lack of substantial documentation of this experience led to the decision to not include Burundi in this review.
Discussions held at the 1996 PUCD project's ITRM allowed common strengths and weaknesses of this experience to be identified 4. The following strengths were identified; high flexibility and adaptability to local conditions; progressive improvement through experience; contribution to better planning; involvement of field-staff; improved availability of information on the functioning of the project and its achievements; contribution to local capacity-building; and support to the dissemination of the project's approach. The main weaknesses included: technical problems in the operation of the system; controversial value of certain key indicators; limited reliability and validity of data from monitoring and evaluation (M&E); uncertain sustainability at the institutional level; insufficient investment in the operation of the system, unsatisfactory performance in processing and applying M&E data, and inconsistent community involvement.
4 Se COO/97/7.
Discussions also revealed significant differences in the NFTs' experience with the system and in opinions about its actual performance, usefulness and adaptability to the local environment It was thus deemed necessary to conduct a more in-depth review of the PME experience carried out to date in each country in order to identify areas for improvement. This review was found to be especially urgent in view of the termination of Phase 2 of the project and the start-up of Phase 3, aimed at replicating the project's approach on a wider scale and at it institutionalising it. Establishing a PME system adapted to the institutional culture and operational capability of national counterparts was, in fact, identified as a basic task to be carried out by the project after August 1997 (the termination date of Phase 2).
The present study is meant to contribute to this endeavour by reviewing the NFTs' experience to date with PME in Bolivia, Nepal, Pakistan and Tunisia, based on the opinions of actors in the field-projects (managers, staff and members of participating grassroots organizations). Its main purpose is to develop technical recommendations (covering both amendments in the system's design and operational requirements) for improving the NFTs' PME practice during the forthcoming third phase
This study focuses on the PME experience gained by four NFTs (i.e. Bolivia, Nepal, Pakistan and Tunisia) from 1994 to 1997.
The specific objectives of the study are as follows:
1. To describe the initial design of the project- and community-level PME systems and the variants actually implemented by each NFT involved.2. To assess the implementation of the above PME systems in terms of their overall performance, usefulness, and adaptation to the local institutional and social environment, according to the opinions of project management, staff, and participants (i.e. self-assessment).
3. To discuss common problems in PME practice elicited from the above mentioned self-assessment exercise.
4 To propose technical recommendations for improving the design and operation of the PME systems
The methodology of the study is based on an adapted version of the Focused Rapid Assessment of Monitoring and Evaluation Systems (FRAMES), developed in the early 1990s by the Evaluation and Research Department of the International Fund for Agricultural Development 5 A twofold research method has been adopted, including:
5 See O. N. Feinstein. 1993. Focused Rapid Assessment of Monitoring and Evaluation Systems (FRAMES). IFAD M&E Division Special Studies. Report N° 0449. IFAD: Rome.a) a review of all relevant PME documents (mainly for achieving objectives 1 and 3); and
b) interviews with NET management, field-staff, and local participants (for achieving objective 2).
a) Review of PME documents
The documents reviewed included the following PME materials produced from 1993 to 1997 by the four NFTs and the Coordination Unit:
* PME design documents;
* mission reports by consultants;
* relevant materials submitted or produced during the 1995 and 1996 ITRMs;
* PME manuals and guidelines prepared for/by NFTs; and
* internal monitoring and evaluation reports prepared by NFTs.
Based on these materials, the overall framework of the PUCD Project's PME was outlined, and profiles of the systems actually implemented in each field-project were prepared. This review also contributed to identifying elements to be considered in the discussion of self-assessment findings.
b) Interviews with PME actors
These interviews were conducted to obtain the opinions of project management [Chief Technical Advisers (CTAs), National Project Directors (NPD), and the staff in-charge of M&E activities (M&E Officers)], and of a sample of field-staff and participants, on the following aspects of the PME system:
· functioning (i.e performance related to the collection, storage, analysis, feedback, and use of PME information).· usefulness (i.e. relevance of PME information to project- or community-level decision-making); and
· adaptation to the local environment (including participation of different PME actors and replicability of certain elements of the system without assistance from the project).
Different means of data collection were used with each of the above categories of respondents. A questionnaire, including both scoring-scale questions and open-ended questions, was submitted individually to CTAs, NPDs, and M&E Officers. The perceptions of field-staff were collected through group exercises in strengths and weaknesses analysis. The participants' opinions were obtained through focus-group interviews (involving at least two different groups of project participants for each national component).
Processing of interview data included;
· a quantitative analysis of scoring-scale questions included in the questionnaire for project management. Average scores were calculated for each item of the questionnaire and for the overall performance and usefulness of project-level and community-level PME 6; and6 Quantitative data generated by scoring-scale questions are presented in table format in Annex I.· a qualitative analysis of answers to open-ended question and statements from strengths and weaknesses analysis group exercises and focus-group interviews. To this end, several techniques were used, including cross-referencing of statements, identification and classification of key themes, and validation by triangulation.
Based on this information, self-assessment profiles of the PME practice developed by each NFT were created. Subsequently, these profiles were compared with the aim of identifying and resolving technical problems affecting PME practice
In this report, research findings have been organized into three chapters.
Chapter I provides an overview of the PUCD Project's PME system design, covering:
· the managerial needs the system was meant to address;· a "normative" description of the different components and elements of the system;
· a review of the variations in the design, subsequently developed for adjusting the system to the specific needs, assets and constraints of each national component.
Chapter 2 summarises the results of the evaluation exercise in the form of four self-assessment profiles, one for each national component. These profiles are meant to provide detailed diagnostic indications for future improvement of critical aspects of each local PME system, This detailed presentation addresses the specific needs of the concerned NFTs. Certain readers of this document may not be interested in such an in-depth analysis and are advised to move on to Chapter 3.
Chapter 3 describes the main comparative findings of the study and provides general recommendations.
