This chapter reports the findings of the NFTs' and communities' self-assessment of PME practice, with the aim of outlining a diagnostic profile of both the positive and negative aspects of the established PME systems. This information is meant to be used by each NFT to identify practical measures for improving its PME practice, according to local managerial needs and existing operational assets and constraints 23.
23 To this end, in this chapter, the information provided by each self-assessment exercise is reported and reviewed in detail, addressing the specific needs of each concerned NFT. Certain readers of this document may not be interested in such an in-depth analysis and are advised to move on to Chapter 3, which presents the main comparative findings of the study and general recommendations.
The chapter is organized into four sections, one per field component, each of which is divided into four subsections (designated by the letters a, b, c, and d). Each section's first two subsections address the NFT's analysis of the performance and usefulness of the project- and community-level PME practice; subsection © highlights the participants' perception of the community-level PME practice; and subsection (d) focuses on the adaptation of the overall system to the local environment.
The NFTs' self-assessment of project- and community-level practice takes into consideration the following three main sets of information:
i) average scores given to the closed-ended questions included in the questionnaires for project management 24;24 These data are also presented in table format in Annex I. Scores referring to the PME system's usefulness should be interpreted as follows: 0 = not applicable; 1 = of no use; 2 = of little use; 3 = of adequate use; 4 = of great use. Scores referring to the system's functioning should be interpreted as follows: 0 = not applicable; 1 = unsolved problems; 2 = major problems, though being dealt with; 3 = minor problems; 4 = problem-free.ii) lists of assets/constraints in the operation of the system and major/insufficient contributions of the system to managing the project (in the case of project-level PME), or to the self-reliance of communities/interest groups (in the case of community-level PME), as identified by project management; and
iii) lists of strengths and weaknesses of various project- and community-level components of the system, as identified by field-staff.
The description of the participants' perception of community-level PME practice is based on focus-group interviews.
The analysis of the adaptation of the overall system to the local institutional environment is based on answers provided in the relevant section of the questionnaire completed by the project management.
2.1.a. NFT's assessment of the performance and usefulness of project-level PME
2.1.b. NFT's assessment of the performance and usefulness of community-level PME
2.1.c. Participants' views on community-level PME
2.1.d Project management's views on the adaptation of the overall PME system to the local social and institutional environment
The project management's quantitative assessment of the performance of project-level PME is very positive, with an overall average score of 3.41 (out of a maximum of 4). The highest scores were given to the use of project- and community-level evaluation information for further planning (score: 3.5); M&E data analysis (all relevant items scoring 4), and quality (i.e. validity and reliability) of M&E data (scores ranging from 3 to 3.5) In general, data collection in the field was also positively assessed, with the exception of the collection of process-evaluation information (i.e. raw data for calculating time-allocation indicators), which scored the lowest (2.5).
Assets facilitating the start-up and development of project-level PME include both design features (such as user-friendliness, progressive adjustment to local conditions, and formative orientation) and operational features (appropriate allocation of responsibilities in managing the system, staff participation and commitment, and reliability of the established operational mechanisms) Constraints tend to focus on operational difficulties such as lack of relevant skills and attitudes of field-staff, and on problems in the use of forms for individual planning, monitoring and process evaluation.
Project management also perceives the overall usefulness of project-level PME as quite high (total average score: 3.6). The maximum score was given to planning items, with the exception of allocation of resources, which scored the lowest (2.5). The scores for monitoring and evaluation items range between 3 and 4. Usefulness of project-level PME system for reporting was rated 3.5.
The results of this scoring exercise are consistent with the lists, developed by the same respondents, of major/insufficient contributions to managing the project. Major contributions include detailed planning, timely identification of problems in implementation, feedback on project performance, documentation of achievements, and improved communication between management and field-staff. Insufficient contributions include insufficient attention paid to determining the costs of activities and to other financial information, and a need for fine-tuning the system through further experience.
Among members of the field-staff, there are greater differences of opinion as compared with project management, as demonstrated by the group strengths and weaknesses analysis exercise. According to field-staff, the main strengths of the system lie in the monitoring design, which allows for the continuous feedback of the progress in implementation and which results in the work being well organized. However, the mechanism for facilitating the allocation of individual responsibilities foreseen by the system's planning procedures was mentioned both as a strength (in terms of efficiency) and as a weakness (considered by some staff members as a factor negatively affecting team work).
Regarding project-level evaluation practice, project field-workers identified the following strengths: relevant contribution in assessing the overall progress of project implementation; support in analysing team performance; and provision of formative feedback. Weaknesses include the design of the evaluation module, which is considered to be too quantitatively-oriented and, at the same time, not sufficiently focused on the project's social and environmental impact
Other weaknesses concern the operation of the system, including inconsistencies in planning (excessively ambitious planning; unclear links between some short-term plan results and project objectives, and unexpected changes in planning decisions); inadequate use of monitoring and process evaluation tools (forms not filled-out in due time, different criteria used in estimating implementation time); insufficient duration of evaluation and re-planning workshops; and poor systematisation of lessons learned through implementation. Participation of community representatives in project-level evaluation and re-planning events (which began to be promoted in late 1995) was identified as a strength by some team members, while other colleagues found that it was still insufficient to be meaningful.
The project managers' quantitative assessment of the performance of community-level PME is less positive than that of project-level PME. The overall average score is 2.06. Higher scores were given to the functioning of community-level exercises for planning, monitoring and evaluation (2.5, 3.5 and 3, respectively). Use of community-level monitoring and evaluation information also scored quite high (score: 3). However, major problems were identified with regard to the consistency of the results of community-level planning with the project's mandate (score 1.5), and community self-reliance in operating the system (0.5, for all relevant items).
A rationale for this scoring is suggested by the project management's list of assets and constraints in community-level PME practice. According to this source, community-level planning (or re-planning) and evaluation have been facilitated by a simple and easily understandable methodology, by a good organizational and logistic support, and by the trust existing between the project and the community (which these exercises have contributed to build). The institutional environment created by the "Ley de Participación Popular" was also identified as a major asset 25.
25 Promulgated in 1995, the "Ley de Participación Popular" (People Participation Law) institutionalises grassroots participation in sustainable development. To this end, communities are provided with incentives to organize themselves into "Organizaciones Territoriales de Base" (OTB), entitling them to receive a small share of the State's fiscal revenue. These funds are meant to be spent for implementing activities identified and planned by the community according to local needs, which usually include the development of basic services. The PUCD Project has made major efforts towards promoting the integration of initiatives for natural resource management into the planning conducted by project-area OTBs.
On the other hand, lack of communities and user groups' self-reliance in running these exercises was identified as a major constraint. This can be attributed to insufficient methodological know-how, creativity and commitment by the staff in-charge, as well as to farmers' limited understanding of the usefulness of PME, lack of time, lack of relevant skills, and deceptive experience with other development institutions.
The project management's quantitative assessment of the usefulness of community-level PME also reveals dissatisfaction. The overall average score (2.44) is similar to the score for performance, with most items in the list scoring 2.5 or less. Identification of needs by local actors is considered to be totally sufficient. The project management's lists of major and insufficient contributions of community-level PME practice to communities' self-reliance portray a more positive picture. Major contributions include development of communities' problem-solving skills, involvement of community members in decision-making, strengthening of grassroots organizations, facilitation of the development of community plans (foreseen by the "Ley de Participación Popular"), increased ability to negotiate with local authorities, and increasing interest for natural resource management activities. However, the contribution to the understanding of PME and to the development of a self-reliant monitoring, evaluation and re-planning practice is still assessed as insufficient.
The field staff's strengths and weaknesses analysis shows that almost all strengths of participatory planning and re-planning focus on design features, including: adjustment of participants' expectations to opportunities the project can actually provide, sharing of responsibilities, identification and prioritisation of results to be achieved, and exchange of experience among participants. Weaknesses tend to relate to poor performance, including over-commitment in project planning (over-planning), insufficient time allocated to the implementation of participatory planning exercises, insufficient attention to organizational aspects, and inadequate use of evaluation information for re-planning.
Field staff acknowledge that the project has not yet developed a true participatory monitoring practice.
Several design features were identified as main strengths in community-level evaluation. These include joint (participants/staff) analysis of field data, use of methods based on direct observation, and publicity of results (made possible by the workshop format). However, the methodology used for participatory evaluation is considered to be insufficiently developed Further weaknesses concern operations (including poor attitude and commitment of the staff in-charge, lack of an interdisciplinary approach, and excessive time needed for organizing the workshops).
Though the field staff criticised community-level PME practice, community members participating in focus-group interviews 26 expressed their appreciation for the way in which the project has applied participatory methods to facilitate their own decision-making and action-taking
26 Two focus-group interviews were earned out in Bolivia in the framework of this study. Both interviews involved randomly selected project participants belonging to different communities and different user groups. The first interview was conducted among farmers who were financially better-off, whereas the second interview focused on a more disadvantaged social group.
As suggested by the following statements on decision-making, project participants perceive initial participatory planning as a process in which community-felt needs are matched with technical solutions identified by the project
"We, the peasants of Paredones, identify our needs. Based on that, we took the decision to invite project staff to assist us. We did not know from where to start. So we tell them: This is what we need. How can we do it?' They told us 'you may do so and so, and we said 'OK, we are going to do it' We decided to do it and we are ready to continue.""For instance, in Floripondio, the decision of improving our water supply did not come from some project staff telling us: 'You should do this'. Rather, it happens that project staff come to visit us during the dry season and we told them: 'Somebody has to come to assist us, to tell us how to improve the water supply situation.' In this way we decided together to implement a water scheme to supply water to households and to provide irrigation to the fields. This was not project staff decision; it was our decision".
Some project participants have highlighted the important role played by subsequent evaluation and re-planning workshops in creating awareness of the importance of improving both natural resource management practices and the consistency between community felt needs and project mandate. This sometimes took the form of discussion on a pre-set "menu" of working hypotheses proposed by the project, For instance, in Vallecito, during one of these workshops:
"The project sociologist came and asked us if we were interested in re-afforestation activities, if we need to build wells, if we were interested in pisciculture or in rangeland improvement. And, after discussion, those who were interested applied to these activities and started to work with the project as a group "
The focus group interviews also revealed that some interest groups were able to fill-in gaps in the project's support to participatory monitoring by developing their own procedures and tools:
"In Paredones we keep a record book in which we write down the decisions made. Project staff commitments are written, as well as farmers' commitments. And when somebody is not fulfilling what is written, we tell to the extensionist: 'You see, here there's an unfulfilled commitment'. This is our way to monitor because what is not written, cannot be claimed. When something is written we can immediately claim it to the extension worker. And the extensionist is obliged to discuss the issues with project management."
Finally, project participants emphasised the pivotal role played by participatory evaluation exercises in facilitating a "learning by doing" process, in which the points of view of the community and technical staff are identified and harmonised:
"Evaluation is done with the active participation of the community, of those who actually work and are in need of project support. There is also a technical part, which is carried out by the project. In this way. we, the participants, and project staff, see what has been done, if it has been done properly, if it has been followed up properly, if agreements have been accomplished or not. In case of poor achievement, we assess if this depended on bad luck or other factors. In this way, the project and us can find the reasons of success and failure and make better decisions for the future".
The project management provided its opinion on the adaptation of the overall PME practice to the local social and institutional context based on its assessment of the involvement in PME exercises of staff, counterparts, partner institutions, and communities.
The project management identified staff involvement as a major strength in the implementation of both project- and community-level PME practices, as indicated by the following statement of the staff in-charge of M&E:
"Staff participation is an intrinsic element of our PME system. This has been well understood by all staff members and has made everybody committed to providing reliable information and to fulfilling their data collection tasks (even though sometimes with some delay). In evaluation and re-planning workshops (both at the project- and community-level) participation is quite high. Everybody contributes with questions and suggestions"
Counterpart representatives other than team members have shown much less interest in participating in PME activities; on several occasions they have ignored the invitation to take part in project- and community-level evaluation and re-planning workshops. However, the NPD has pointed out that:
"Time has been insufficient to show to our SEARPI 27 colleagues (not directly involved in the project) the relevance of community-level PME for overall community development. It seems that the conditions to promote interest in project- and community-level PME events are still not there. Sensitisalion of SEARPI to this issue will be carried out in the next phase of the project, during which enough time should be allocated to this task".27 SEARPI is the Piraí River watershed authority acting as project counterpart.
The involvement of other governmental and non-governmental partner institutions in project-and community-level PME has also been assessed as poor. However, the NPD points out that:
"In the last years, due to the socio-political changes on-going in the country through the 'Ley de Participación Popular', participatory planning has become a current need for most governmental and non-governmental local institutions, and the same will soon happen for participatory monitoring and evaluation"
This statement may also explain why, as stressed by the project's CTA, despite the technical difficulties and limitations identified by project staff, community members have "always played a very active role in community-level PME events" Furthermore:
"Participation of community representatives in project-level evaluation and re-planning workshops has been increasing since 1995. As shown by the relevant reports, their contribution focused on re-planning. However, there is a need to better define community representatives role in project-level evaluation. In particular, a way should be found to present evaluation information in a more easily understandable manner to our community partners, than the current one (based on display of quantitative and qualitative tables and graphs)".
Based on these considerations, CTA, NPD and staff in-charge of M&E are quite optimistic about the actual possibility of transferring the overall PME system to local institutions and grassroots organizations:
Staff in-charge for M&E: "So far the system has involved more than 20 staff and community members from different places, working on different subjects. Planning and evaluation exercises have proven to be effective. The system is flexible, easily understandable and easy to operate. For these reasons, it can be almost totally replicated. However, some adjustment should be made in order to make the system more relevant to specific institutional needs" 2828 In particular, according to the opinion of the field-staff, "Too much time is needed to collect field data, process them and implement an evaluation workshop. Such an effort will not be sustainable in a 'without project situation'".NPD: "In my opinion, the whole system could be transferred, with some adaptation and adjustments, according to user needs. This is possible because of the high flexibility built into the system".
CTA: "The overall design of the project-level PME system could be transferred with some adaptations. In particular, the methods for evaluating community participation and natural resource management conservation activities could be transferred. The community-level PME methodology is likely to be transferred, even though it has no(been fully adopted by all communities so far. This may need some lime. Up to now, community-level PME events still need the support of a project facilitator and are not perceived as a felt need by most community members. There is a need for training community facilitators who, in the future, may replace project staff. Perhaps the increasingly important role assigned to 'OTBs' and 'Comités de Vigilancia' in the implementation of the 'Plan Municipal' will create among community members the need for strengthening their PME practice".
2.2.a. NFT's assessment of the performance and usefulness of project-level PME
2.2.b. NFT's assessment of the performance and usefulness of community-level PME
2.2.c. Participants' views on community-level PME
2.2.d. Project management's views on the adaptation of the overall PME system to the local social and institutional environment
The project management team's 29 quantitative assessment indicates great dissatisfaction with the performance of project-level PME. The overall average score is 1.69 (out of a maximum of 4). Planning items scored higher than monitoring and evaluation items, and data analysis items have the lowest scores.
29 In Nepal, the project management team includes the NPD, the CTA, the APO, the Assistant Soil Conservation Officer, and the National Expert in Sociology and Capacity-Building, who are jointly responsible for M&E and who thus all completed the project management questionnaire.
These results are validated by the lists of assets and constraints. Major constraints refer to operational aspects, including lack of consensus among project staff on the focus of M&E, poor management of the information flow, and inadequate data analysis. Two design features were identified as assets: planning and monitoring procedures and data collection tools.
The overall average score is slightly better for the usefulness of the system (score 2.53). The monitoring activities (including adjustment of the workplan to contingencies, allocation of resources, and time management) and preparation of progress reports (all scoring 3,2) are considered to be especially useful. Planning items scored between 2.2 and 2.4, whereas scores for evaluation items range between 1.8 and 2.
The lists of major/insufficient contributions to project management, developed by the same respondents, are highly consistent with the above results. Major contributions include resource allocation, adjustment of the yearly action plan to contingencies (by means of monthly plan revision), and use of community-level information for project-level planning (facilitated by the Community Action Plans Database). Insufficient contributions are analysis, reporting and use of field data.
According to the field-workers' analysis of strengths and weaknesses of project-level PME 30, the strengths of project-level planning and re-planning seem to include: use of a participatory approach, decentralisation of the project's planning/decision-making at the area level 31, and feeling of ownership by staff and participants. Major weaknesses include over-planning (i.e. planning beyond project's operational capability), farmers' difficulties in fitting the project's initiatives into their busy schedules, and lack of communication between project staff and other development institutions working in the area.
30 In Nepal, two separate exercises in strengths and weaknesses analysis, focusing on both project- and community level PME, were carried out in the framework of this study. The former was carried out with mid-level technicians; the latter involved group promoters.31 See note 15.
Regarding project-level monitoring and evaluation, major strengths include: availability of data collection tools, good and frank communication among field team and project management, contribution to definition of workable implementation strategies, and orientation in re-planning. Major weaknesses include: difficulties and delays in data analysis and lack of involvement of participants in project-level evaluation exercises.
The project management team has expressed moderate satisfaction with the overall performance of the community-level PME, with an overall average score of 2.69. Particular appreciation (score 2.5 or above) has been shown for community-level planning (functioning, self-reliance and consistency of results with project mandate) and for community-level evaluation (functioning, self-reliance and use of relevant information for re-planning). Lower scores were given to community-level monitoring, particularly the use of relevant information (score 1.4). As pointed out by the CTA, this is due to the fact that "few formal links exist in monitoring between user groups and the project. User Groups organize their own meetings and do not feel that it is important to report to the project"
The main assets in community-level PME practice include its contribution to empowerment, establishment of democratic and transparent decision-making, technical and managerial capacity-building, and development of feelings of ownership, Establishment of an autonomous monitoring practice by some user groups is a further important asset.
Constraints in community-level PME practice include insufficient organization and managerial skills, excessively ambitious planning (risking discouragement), time needed to carry out PME exercises (possibly preventing more disadvantaged social groups from participating), and lack of comprehensiveness in user groups' M&E practice. All of these elements suggest the need for continuous input by highly experienced facilitators.
The project management team's scores for usefulness of community-level PME for community capacity-building substantially triangulate with the above quantitative and qualitative information. The overall average score is 2.75. All planning items scored 3, with the exception of negotiation with local institutions (score: 2). All monitoring and evaluation items scored around 2.5.
The project management's lists of major and insufficient contributions of community-level PME practice to communities' self-reliance further support the above perception. Major contributions include preparation of better and more realistic plans, improvement of decision-making capability, increased capacity to run PME exercises autonomously, and increased cohesion of user groups. Insufficient contributions refer to limited capability in focusing planning on integrated natural resource management, carrying out sound assessments of costs and benefits, managing funds, and facilitate mobilisation of resources from other local development agencies.
Regarding community-level planning and re-planning, the field staffs analysis of strengths and weaknesses of community-level PME shows that most strengths are related to intrinsic features of the participatory approach (empowerment, ownership, consensus building, "learning by doing", etc.). Weaknesses tend to focus on equity related aspects, such as the risk of exclusion and of a local elite playing a dominant role. Other weaknesses include difficulties in estimating costs and identifying institutional partners (other than the project). Once again, the pivotal role of skilled facilitators for the successful completion of participatory planning exercises was highlighted.
For community-level monitoring, the strengths and weaknesses refer almost exclusively to operational aspects. The main strengths include upgrading of user group managerial skills, allowing for proper use of resources and timely accomplishment of activities. The major weaknesses are insufficient motivation and skills of the communities, lack of commitment by staff (due to time constraints), and poor record keeping.
With respect to community-level evaluation, the strengths identified are the feedback of positive and negative aspects of implementation to both the user groups and project staff. Weaknesses include design features (little attention paid to quantification, and, more specifically, economic analysis of gains and losses), as well as operational aspects (low level of participation).
Regarding community-level PME, all groups of participants interviewed 32 showed general satisfaction with participatory planning and re-planning established with the support of the project. Methodological support provided by the project's facilitator is found to be instrumental in developing decision-making skills. In the words of one member of the Pragati Women User Group:
32 In Nepal, three focus-group interviews were carried out in the framework of the study with three different User Groups participating in the project. These included: the Janjagriti User Group, which was established more than four years earlier and was one of the first male, high-caste dominated groups formed during phase 1 of the project; the Pragati Women's User Group, established three years earlier and one of the first women's groups (high-caste dominated) formed at the beginning of phase 2; and the Bhawana Women's User Group, one of the recently formed women's groups of disadvantaged caste."The whole group decides about the plan that is to be implemented. Every member has equal say in the decision making process. (...) Initially, we could not make decisions. But after receiving training from the PUCD, we are now capable to make decisions, since almost all the members attend the monthly meetings"
An important finding of these interviews is that even though the project was not able to provide continued methodological support to participatory monitoring practice, two of the three groups interviewed organized themselves for ensuring the continued monitoring of their activities:
"At the monthly meetings, we review previous months' decisions and the attendance register. We also assess the progress. If everything is satisfactory, we take decisions for the next month. We are satisfied with this system" (Janjagriti User Group)."At the end of each month, we hold a general meeting. The meeting reviews the mistakes made and the reasons behind the failure of the activities, and attempts to correct and not to repeat the same mistakes in the near future" (Pragati Women User Group).
One member of the Bhawana Women User Group identified the lack of participatory monitoring as a major shortcoming in group functioning:
"Due to the absence of monitoring, the programme has come to a halt now. The monitoring job is the responsibility of the group, but so far we have not been able to monitor. Yet, in the near future we expect to accomplish this task as well."
Participants provided only a few comments on community-level evaluation during focus group interviews. One member of the Janagriti User Group said that "evaluation after the completion of activities is not realistic" and recommended establishing an on-going evaluation practice, built into community-level monitoring. A member of the Bhawana Women's User Group pointed out that: "Evaluating any task through group discussion is beneficial". The following statement of one member of the Pragati Women's User Group demonstrates an appreciation of the project-promoted community-level evaluation and re-planning workshop:
"We evaluate each activity at the general body meetings; we carefully assess whether the activity (such as trail construction, water tap, vegetable farming, etc.) has benefited us or not; we also consider whether we are just wasting our time or invested time is giving fruitful results to us. If we feel that we have benefited from the project, then we continue the activities".
The following statement, made by the project's APO, thoroughly describes the staffs involvement in PME operations:
"All staff (field and project management) are involved in a monthly staff meeting where status of field activities, problems encountered, ways to overcome them, and the implementation plan for the coining month(s) are discussed. Group promoters are the ones collecting the information for water source protection forms, user group assessment, etc. Both group promoters and area facilitators arc involved in community level PME."
According to the Assistant Soil Conservation Officer, Area Facilitators are playing a special and increasing role in the system's operation:
"Project staff involvement in project- and community-level PME is increasing. Area Facilitators are fully responsible for the overall monitoring of field-activities of their respective sites. They also facilitate community-level evaluation and re-planning workshops."
Furthermore, all local staff of the District Soil Conservation Office take part in major project-level planning, monitoring and evaluation events, though not all of them are official members of the NFT. By contrast, officers from the central-level Soil Conservation Department have so far not directly contributed to project planning, monitoring and evaluation.
As pointed out by the NPD and the APO, the local partner institutions have considered project-level PME as "PUCD Project's business" and have thus been only minimally involved. However, the APO has reported that, since 1996, representatives of line agencies and local NGOs have begun to take part in coordination meetings promoted by the project, These meetings are found to contribute significantly to the project-level planning process. In the APO's opinion, this practice is likely to continue after the completion of the project. Furthermore, the National Consultant in Capacity Building recalls that Village Development Committees have often been important actors in community-level evaluation and re-planning workshops.
The involvement of local communities has been limited to community-level PME meetings and workshops. Most members of the project management team agree that user groups and grassroots organizations are increasingly motivated to participate in these events.
Based on these considerations, respondents agree that most of the community-level PME elements (including PRA diagnostic tools and procedures) will likely be replicated by local institutions and user groups in a "without project" situation. According to the National Consultant in Capacity Building:
"The methodology for running evaluation and re-planning workshops, developed by the team in collaboration with consultants, is gaining much popularity in the project area (...). I think that this method can be widely used all over the country"
2.3.a. NFT's assessment of the performance and usefulness of project-level PME
2.3.b. NFT's assessment of the performance and usefulness of community-level PME
2.3.c. Participants' views on community-level PME
2.3.d. Project management's views on the adaptation of the overall PME system to the local social and institutional environment
The project managers 33 have found the performance of project-level PME in Pakistan to be nearly acceptable (overall average score: 2.55, out of a maximum of 4). High-scoring aspects include the use of community-level information for project-level planning (3.67), collection of monitoring information (3), and collection and quality (i.e. validity and reliability) of information for evaluating results (3.33 and 3, respectively). Analysis and quality of information for process evaluation scored low (1.57), for which, according to the CTA, "no satisfactory solution has been found so far"
In their analysis of assets and constraints in project-level PME, the project managers found the planning and monitoring method based on identification of results and step-wise scheduling to be an asset. This method is considered easy to understand and follow, acceptable by field-staff, and helpful in better organizing the work. A further asset is the implementation of a number of evaluation exercises documenting project outcomes. Constraints focus on the overall complexity of the system and, in particular, on the limited relevance, poor reliability and low acceptability of process evaluation information (namely, time allocation indicators). The need for qualified personnel and the insufficient analysis of project-level M&E data are main constraints
33 In Pakistan, the questionnaire for project management was answered by the CTA (who is also responsible for Men's Team's M&E activities), the NPD, and the staff-in-charge of the Women' Team's M&E activities.
The overall usefulness of project-level PME system is perceived as quite high, with an average score of 3.03. The highest score (3.67) was given to allocation of resources and preparation of progress reports. Planning and monitoring items scored 3 or above, with the exception of identification and planning of actions (i.e. steps in implementation, and time management (scoring 2.33 and 2.67, respectively). Like performance, the technical quality of results achieves sufficiency, while dissatisfaction (2 33) was once again expressed about the usefulness of process evaluation.
According to the same respondents, main contributions of the system to project management include provision of information on progress in implementation and control over field-staff performance. Insufficient contributions include inadequate quantification of outcomes of field activities and unsatisfactory evaluation of the quality of the participatory process.
The field staff's strengths and weaknesses analysis of the project-level PME system 34 produced results consistent with the project managers' views. The main strengths in project-level planning include involvement of the entire staff in decision-making, appropriateness of step-wise planning, and progressive adjustment of the plan according to implementation contingencies. Lack of coordination with other line agencies is the only weakness reported.
34 Strengths and weakness analysis exercises were carried out by a mixed group of extensionists, field technicians, and group promoters belonging to both the Men's and Women's Teams.
As far as project-level monitoring is concerned, one strength is the fact that planning and monitoring activities become progressively easier with experience. Other strengths include user friendliness and flexibility of the relevant forms. Weaknesses include the two-year delay in the start-up of project-level monitoring and irregularities and delays in the preparation of the monthly monitoring data base.
The field-staff found that project-level evaluation promotes the participatory approach by putting the project in the position of showing the results achieved through its application. It is also considered instrumental in extracting lessons learned from implementation and thus in improving re-planning. Weaknesses of project-level evaluation practice include technical difficulties in gathering data for determining indicators of time allocation; collection of these data is also time consuming.
The project managers showed moderate satisfaction with the performance of community-level PME. The overall average score is 2.96. All items score 3 or above, with the exception of communities' self-reliance in planning (2) and evaluation (2.67). The creation of Village Associations and of savings/credit schemes are identified as important assets for the development of an iterative and comprehensive PME process at the village level. This process is perceived as enhancing villagers' analytical skills and creativity and as promoting feelings of ownership towards the work done in collaboration with the project. The importance of community-level PME in orienting project-level planning is also acknowledged. Constraints include insufficient exposure of project staff to relevant methodological expertise, inequalities in local social structure, and poor attitude and skills of villagers. The validity of the results of community-level evaluation exercises is also questioned.
The project managers expressed satisfaction with the usefulness of community-level PME practice for the development of managerial skills among project participants. All items scored 3 or above, with an overall average of 3.33.
Regarding the major/insufficient contributions to project management of community-level PME, the CTA found that regular meetings and joint discussions promote the community spirit needed to work together towards a common goal. However, the CTA also found that Village Associations still have difficulties in networking and in autonomously approaching projects and institutions which could support them in reaching their objectives.
The field-staff's strengths and weaknesses analysis exercise showed that major strengths of participatory planning include promotion of villagers' self-reliance, building of the consensus necessary to implement activities, clarification of implementation modalities and responsibilities, project's adaptation to local culture, improved efficiency, and mobilisation of hidden resources. Weaknesses focus on relationships between project and communities, specifically: difficulties in clarifying to the people the limits of project mandate, risk of raising unrealistic expectations, difficulties in maintaining people's trust when faced with unexpected technical or financial difficulties, and risk of excluding village members not involved in interest groups.
The strengths of community-level monitoring practice, as perceived by field staff, are effectively conveyed by the following two statements:
"Progress can be checked step by step, and people can understand the scope and importance of the activity. The faith which they might have lost (due to non-completion of other activities) is regained.""At the beginning of the project, a lot of people had volunteered to work. Gradually a few who were really interested remained. However, thanks to participatory monitoring, most people saw that something was really happening, and those who had left came back again."
Major weaknesses are related to social factors, including illiteracy, which acts as a barrier to understanding the importance of monitoring practice, and the pivotal role played by community's kinship system in shaping and orienting participation in Village Associations.
Regarding community-level evaluation, strengths include the contribution to communities' learning-by-doing process and improved re-planning, whereas a major weakness is expressed in the following statement:
"Evaluation exercises may lead to hold somebody responsible for failure. This is a sensitive issue. When somebody is held responsible for an en-or, the community can be divided and the activity is hampered."
With regard to community-level PME, focus group interviews with representatives of Men's and Women's Village Associations 35 convey feelings of interest, commitment and satisfaction, as expressed by one participant who quoted the Urdu saying, "Working together attracts blessing" ('Ittefaq main barkat hai').
35 Two focus-group interviews were carried out in the framework of this study with representatives of Men's and Women's Associations belonging to different villages.
Participatory planning is described by representatives of both Men's and Women's Village Associations as a process of joint decision making, in which the overall community's concerns and individual interests are harmonised:
"There are 32 members in our Association. All of them are contributing; all of them have an equal say in decision making and planning. Some activities are planned, regarding village's common land. Yet, if somebody is facing a special problem in agriculture or orchards, or is having financial constraints, or wants to open up a grocery shop, we discuss it in the Association meeting and, if feasible, give him loans. The project staff also help us in coming to a conclusion."
Though conflicts may arise, participants feel capable of managing them and of finding a "win-win" solution:
"There is a conflict on certain issues sometimes. I think it helps us also. If the majority has decided to do something and some are not willing we can make them understand."
Among Men's Village Associations, participatory monitoring is carried out during monthly ad hoc meetings, focusing on progress made and problems met in implementation. According to one member of a Men Village Association, this "step by step checking has really benefited us" According to another participant, this is because:
"When the association members are constantly asking each other about the progress, then the problems being faced come up. Discussions resolve the problem"."Nothing can be finished without monitoring. In this way if there is a conflict within the members or there is a problem in the implementation, than it can be removed. If somebody needs guidance, than other members can help him"
Among Women's Village Associations, community-level monitoring is coordinated by an Association "monitor". According to one of the interviewed individuals, this arrangement has "made the work run smoothly and successfully."
Most statements on participatory evaluation from representatives of Men Village Associations are descriptive rather than judgmental. However, a good understanding of the rationale and importance of this practice can be elicited from the following quote:
"Some benefits (elicited by participatory evaluation) can be seen soon and some take time, maybe years. (...) The results of new ways of pruning or the use of proper medicines are now so obvious that everybody has started to adopt them. However, when the project staff did pruning in my orchard I was angry. I could not do anything and had to leave them to do it. They cut almost all the branches. My heart wept. But, when the season of the flowers came (and the evaluation was done) my heart was overjoyed".
The relevance of community-level evaluation is expressed by one woman who participated in the final evaluation of the work done by Village Associations during the second phase of the project:
"For the first time, we have evaluated all the project activities together. We have found results very encouraging, and we would like to continue this process in the future because it helps to know what are the weaknesses and what are the strong points of our activities."
Major problems were met in the past in adapting the PUCD project's approach to the "top-down" approach prevailing among Pakistani institutions. However, the project managers' suggest that significant progress has been made in the past year, namely in terms of staff involvement in both project- and community-level PME. According to the project's CTA:
"There has been an increasing involvement of project staff in PME events (...). In 1996, for the first time all project staff participated in the planning workshop to prepare the yearly workplan for 1997. Planning/monitoring meetings are held weekly by the men's and women's teams and a joint PME meeting is held monthly by all field and senior staff. (...) Moreover, project staff participates regularly in all PME community-level events (such as planning workshops, monitoring and evaluation meetings, conventions)".
As far as the project's counterpart is concerned, only staff assigned to the project participate in PME events. However, the NPD has found that regional and local institutions are paying increasing attention to participatory planning, monitoring and evaluation. Officers from the Forestry Department started to regularly attend workshops and meetings at the district or 'tehsil' (sub-district) level.
Involvement of partner institutions in project-promoted PME events has been limited to the participation of local government leaders in Area Conventions. Recently, representatives from two FAO sister-projects and one local NGO (AURAT) have taken part in community-level evaluation exercises.
As indicated above, and according to all respondents, communities' participation in PME events seems to be increasing. In particular, the Village Association leaders are playing a more active role in organizing and implementing these exercises. However, community representatives are not yet involved in project-level PME.
Opinions differ with regard to the replicability of the overall PME practice developed by the project in a "without project" situation. According to the NPD:
"The PME system as a whole can be transferred to the counterpart institution. However, the PRA methodology (for need assessment) and the elements of conflict resolution and planning of natural resource management at the village level are key elements to be transferred".
By contrast, the CTA states:
"Most of the elements of PUCD Project's PME systems are also operated by the Forest Department staff assigned to the project. However, this system is far away from being institutionalised by the Department and at present no element can be realistically expected to be transferred. Some elements of the community-level PME system (PRA methodology, preparation of Village Upland Use Plans, and monitoring and evaluation meetings) could be transferred, provided that the Department moves towards a social forestry programme. In this case a simplified version of the stepwise monitoring system (project-level) could also be transferred".
Finally, in the opinion of the staff in-charge for M&E of the Women Team:
"None of the elements of PUCD project can be transferred to counterpart institutions".
2.4.a. NFT's assessment of the performance and usefulness of project-level PME
2.4.b. NFT's assessment of the performance and usefulness of community-level PME
2.4.c. Participants' views on community-level PME
2.4.d. Project management's' views on the adaptation of the overall PME system to the local social and institutional environment
The project managers gave an overall average score of 2.24 (out of a maximum of 4) to the performance of project-level PME system. They show moderate satisfaction (score 3) with the collection and quality (i.e. validity and reliability) of information for both project-level monitoring and result evaluation. Analysis of information for project-level process evaluation (score 1) and use of community-level evaluation information for project-level planning (score 0.67) suffer from especially poor performance. All other items have scores that are close to the overall average.
The project managers failed to identify any evident asset in the project-level PME, yet they did list many constraints). Some of these concern staff performance, such as limited understanding of the system's potential usefulness, perception of allocation of individual responsibilities as a means of control, and insufficient motivation for (and commitment to) project-level M&E practice. Other constraints include the affiliation of NFT members with different administrative bodies, time needed to operate the system and, as far as planning is concerned, insufficient capability to estimate the time needed to implement different activities.
The project managers express a more positive opinion regarding the usefulness of project-level PME system, which has a sufficient overall average score (3). Planning and monitoring score slightly above the overall average, whereas evaluation items are significantly below average (2.33).
Major contributions concern planning and monitoring issues, including: organization of work through quarterly planning, efficient use of human and financial resources, progressive adjustment of workplans to implementation contingencies, and availability of updated information on project implementation (also useful in preparing progress reports). No areas of insufficient contribution are identified.
The field-staff strengths and weaknesses analysis conveys differing). No strengths are identified in project-level planning. As far as weaknesses are concerned, some field staff feel that the system is too demanding and too detail oriented. Others think that it does not cover all the information needs of different staff and that identification of implementation responsibilities is not precise enough.
With regard to project-level monitoring and evaluation, the only strength identified is the provision of information on the schedule for expected/actual implementation. Weaknesses include poor interpretation and use of monitoring information and limited relevance to project objectives of process evaluation information (based on time allocation indicators).
The average score of the project managers' assessment of community-level PME performance is 2.04, though the range (0 - 3.33) is too wide to make this figure truly meaningful. Contradictory opinions are expressed with regard to the functioning of planning, monitoring and evaluation exercises. Items describing the communities' self-reliance range around 2.5. Community-level has not been experimented to date.
The main asset is the planning approach focusing on felt needs of specific interest groups. However, working with interest groups and not with the community as a whole is also mentioned by one respondent as a constraint. Further constraints include: lack of field staff, insufficient importance given to participatory M&E, and little systematisation of participatory methodology.
The results of the assessment of usefulness of community-level PME for communities' capacity building are more consistent. The average score is 2.88. Most planning and monitoring items score above average, with the exception of usefulness for conflict resolution and joint decision making, which scores slightly below average. Evaluation items have the lowest scores (2.67).
The main contributions of community-level PME to communities' self-reliance include the facilitation of an exchange of views and opinions (namely among women), identification of activities and related needs for training and technical support, and progressive adjustment of activities to constraints met in implementation. An insufficient contribution is the development of an autonomous M&E practice.
The latter opinion is shared by field workers, who, for this reason, felt that they were not in the position to comment on community-level monitoring and evaluation practice. Therefore, their strengths and weaknesses analysis exercise focused on participatory planning. Main strengths include promotion of involvement of local people, facilitation of productive group dynamics among participants and start-up of implementation activities. An important weakness is the difficulties in matching national administrative procedures with needs expressed by project participants.
In general, the interviews with the men's and women revealed a positive view of the project's approach to involving local communities in analysing their situation and facilitating relevant decision-making 36 In particular, women stressed that methods used for participatory planning have been highly instrumental in allowing them "to become aware of problems, identify our needs and make suggestions which may lead to improve our living conditions".
36 In Tunisia, four focus-group interviews were carried out among both participating and non-participating Men's and Women's Groups, belonging to the different douars in which the participatory process was launched in late 1995. In accordance with the study design, and as done by the other NFTs, only the opinions of community members actually involved in project activities were taken into consideration. However, it is important to observe that the interviews conducted among non-participants convey a sense of discomfort about not having participated in project activities. The NFT in Tunisia should strive to understand the factors preventing these persons from participating.
As a major strength of the participatory planning process, both groups identify the consensus building process leading to the formation of interest groups for different activities. For instance, one participant in the women group interview points out that the selection of beneficiaries for the micro-credit programme:
"...has been done with our participation, following criteria which have allowed us to understand (the pros and cons of the initiative) and decide about the opportunity of get the credit. Final decision has been made by ourselves, according to our capacity of taking the responsibility to implement the project (for which the credit is given) and reimburse the credit."
A further strength of participatory planning, highlighted by a men's group, is the contribution given to improved communication among communities and local institutions. As stated by one of the participants:
"When the project started, there was a problem among the people and the Forestry Service concerning the conservation of the Sidi Salem forest. Discussions facilitated by project staff have allowed us to find out a different way to envisage this problem and to start up a negotiation. This has led to release strains and find out a practical solution, in which our point of view has been considered."
Unfortunately, due to the limited development of participatory M&E activities, the focus group interviews on the subsequent stages of the participatory process have provided few (and not very informative) elements.
According to the project managers, adaptation of the system to the local institutional environment is still in progress. In their opinion, a learning process has taken place in recent years, leading to increased staff commitment to PME activities. Project-level quarterly planning sessions are regularly implemented with the active participation of the entire team, including staff from the Water and Soil Conservation Department (the national counterpart) working with the project on a part-time basis. This has led to the planning practice being progressively adapted to the operational capability of the project (namely, to the human and material resources available). However, it was not possible to establish a regular follow-up routine, and project-level monitoring meetings are still carried out on an as-needed basis. Furthermore, in spite of efforts made by individual staff members, project- and community-level evaluation practice still needs to be systematised.
Despite these limitations, involvement of the local counterpart and partners in PME activities has been strengthened. During the past year, technicians and Service Managers from different branches of the CRDA (Regional Agriculture Development Authority, the project's local counterpart) have actively participated in some project-level and community-level PME events.
The participation of local communities in PME activities is promising, though not yet fully developed. This is basically due to the project's delay in establishing a proper monitoring, evaluation and re-planning practice. However, the response of some women interest groups exposed to participatory evaluation exercises is considered to be encouraging.
Obviously, the limited experience gained so far makes it difficult to assess which elements of the system are likely to be replicated in a "without project" situation. According to the staff in-charge of M&E, it is unlikely that the entire system in its current state can be transferred to local institutions. However:
"Some key elements of current PME practice should deserve special attention in view of the institutionalisation of the participator)' approach. These include the link to be established between community-level planning (as practised by the project) and CRDA's regional level planning, and the use of time allocation indicators (in terms of working days) for monitoring and evaluating the implementation process "
