Since its inception, the FAO/Italy Inter-regional Project for Participatory Upland Conservation and Development (PUCD Project) has been a pilot, process-oriented, learning initiative, based on an iterative process of assessment, planning, implementation, monitoring, evaluation, and re-planning Based on the preliminary experiences of the project's National Field Teams (NET), a specialist consultant developed, in 1994, a framework for designing and operating a comprehensive planning, monitoring, and evaluation (PME) system. NFTs in four participating countries (Bolivia, Nepal, Pakistan, and Tunisia) have used this framework to develop a PME system fitting local needs, assets, and constraints.
Discussions held at the 1996 PUCD Project's Internal Technical Review Meeting revealed some strengths and weaknesses of this ongoing experience, as well as significant differences in the NFTs' experience with the system and in opinions about its actual performance, usefulness, and adaptability to the local environment It was thus deemed necessary to conduct a more in-depth review of the PME practice in each country, so that areas for improvement could be identified.
The present study is meant to contribute to fulfilling this need by pursuing the following objectives:
1 To describe the initial design of the project- and community-level PME systems and the variants actually implemented by each NET.2 To assess the implementation of the PME systems in terms of their overall performance, usefulness, and adaptation to the local institutional and social environment, according to the opinions of project management, staff and participants (i.e. self-assessment);
3. To discuss common problems in PME practice elicited from the above mentioned self-assessment exercise; and
4. To develop technical recommendations for improving the design and operation of the PME systems.
The methodology of the study is based on an adapted version of the Focused Rapid Assessment of Monitoring and Evaluation Systems (FRAMES), developed in the early 1990s by the Evaluation and Research Department of the International Fund for Agricultural Development (IFAD). A twofold method strategy has been adopted, including:
a) a review of all relevant documents produced by the project's NFTs and the Coordination Unit (mainly for achieving objectives 1 and 3); andb) interviews with project management, field-staff, and local participants (for achieving objective 2).
In this report, research findings have been organized into three chapters.
Chapter 1 provides an overview of the PUCD Project's PME systems, covering:
· the identification of the managerial needs the PME systems are meant to address;· a "normative" description of the framework of the PME systems, including the different components (i.e. project- and community-level PME) and modules (i.e. planning, monitoring, process evaluation, and result evaluation);
· a review of the variations in the design and operation of the PME systems developed by each national component, according to local needs, assets, and constraints.
Chapter 2 summarises the results of the self-assessment exercises in the form of four detailed self-assessment profiles, one for each national component. These profiles are meant to provide detailed diagnostic indications for future improvement of critical aspects of each local PME system.
Chapter 3 provides a comparative discussion of major self-assessment findings, whose main conclusions can be summarised as follows:
1) During the second phase of the project, the four NFTs made major efforts towards developing sound project- and community-level PME systems.2) All NFTs positively evaluated the usefulness of outputs of both project- and community-level PME practice, as well as the adaptation of the overall system to the local environment.
3) There are some general problems regarding the functioning of the different components and modules of the PME systems, specifically:
· Priority-setting mechanisms built into project-level planning have not always succeeded in preventing over-planning (i.e. planning commitments could not be fulfilled with the available resources and time).· Most NFTs have encountered difficulties in using time allocation indicators, as suggested by the consultant, for project-level process evaluation, and some NFTs have questioned the relevance and validity of these indicators.
· To date, project-level PME systems have paid insufficient attention to assessing the project's environmental, social and economic impact.
· According to project management, the field workers' insufficient skills in data collection have been a major constraint in implementing project-level monitoring and evaluation (M&E) practice. This has often been complicated by the field-staffs misunderstandings about the function and meaning of M&E.
· Project managers have also reported difficulties in terms of the field staff's ability to act as facilitators of community-level PME exercises
· All NFTs have found that the analysis and application of project-level M&E information were inadequate.
· The field-staff in Bolivia and Nepal have reported that time requirements are a major constraint in community-level PME practice.
· According to all NFTs, local participants and communities are not skilled or motivated enough to begin organizing and implementing autonomously their community-level PME practice This opinion, however, contradicts the participants' perception of their own performance in this area.
· Based on these conclusions, the following operational recommendations have been developed with the aim of improving the PME practice of the PUCD Project:
Stricter priority-setting mechanisms should he built into the project-level planning module. To this end, information on cost-effectiveness and time required for different activities should he elicited from evaluation exercises and considered, together with technical, social and institutional factors, in assessing the actual feasibility of each activity to he included in local workplans.
Project-level process-evaluation techniques should he simplified. When needed, time-allocation analysis should be replaced by simpler qualitative exercises, such as the Strengths, Weaknesses, Opportunities, and Limitations analysis.
The Coordination Unit and the NFTs expected to participate in the third phase of the project should reconsider the possibility of including an impact evaluation module in the project-level PME system.
The supervision of data collection by field workers should be strengthened. This supervision should have a "formative" orientation (i.e. offering field workers the opportunity to consolidate and upgrade their knowledge, skills, and attitudes).
Higher priority should be given to the processing of M&E data. Whenever possible, this responsibility should he assigned on a full-time basis to one member of the NFT, who should possess the necessary skills or receive appropriate training. If this is not feasible, well defined M&E data processing tasks should be included in the terms of reference of the CTA or other senior staff, and the time needed to perform these tasks should he properly considered in project workplans and individual agendas.
All M&E reports, whether formal or informal, should include a number of operational recommendations for re-planning relevant activities. After thorough discussion with the concerned staff, these recommendations should he fully taken into consideration in designing, programming, and implementing subsequent activities.
Staff members contributing to data collection should receive timely feedback of the findings of evaluation in special sessions or workshops. Appropriate facilitation techniques should he used to stimulate field workers to interpret this information and to improve its implementation
The Coordination Unit, in collaboration with centres of excellence in this area, should facilitate the preparation of a comprehensive training package on community-level PME. This training package should address the learning needs of both field-staff and leaders of selected grassroots organizations.
· During the third phase of the project, the role of project staff in the operation of community-level PME exercises should he progressively re-oriented. Less work and time should he invested in organizing and implementing community-level PME events, while more emphasis should be placed on providing training and formative supervision on participatory PME to local interest groups and grassroots organizations.
The responsibility for organizing and implementing community-level PME events should he progressively handed over to trained community members. However, methodological backstopping should continue to he provided to interest groups and grassroots organizations. Arrangements should also he made with national centres with relevant expertise, to ensure that they will continue to provide their assistance after the project's termination.
