The self-assessment presented above portrays quite a heterogeneous and complex situation which allows little room for sound generalisations about the overall performance, usefulness and adaptation of the PUCD Project's PME practice, However, several common problems in the system's design and operation can be elicited.
In this chapter, these problems are analysed in light of both statements made during the self-assessment and the consultant's field observations. Based on the triangulation among perceptions of "users" and a "resource person", a number of operational recommendations are made to the Coordination Unit and the NFTs. The aim of these recommendations is to outline a general strategy for upgrading the system's design and operation, in view of the forthcoming third phase.
3.1.a. Over-planning
3.1.b. Relevance and appropriateness of time-allocation indicators
3.1.c. Focus of evaluation practice
3.1.d. Skills and attitudes of field staff
3.1.e. Analysis and use of M&E information
Five main common problems in project-level PME practice can be elicited from cross-referencing constraints, insufficient contributions, and weaknesses, identified by project management and field staff:
a) over-planning (i.e. planning beyond capabilities);
b) relevance and appropriateness of time-allocation indicators;
c) focus of evaluation practice;
d) M&E skills and attitudes of field-staff; and
e) analysis and use of M&E information.
Each of these problems is briefly reviewed below, together with recommendations for facilitating their solution.
Only the field-staff in Bolivia and Nepal have identified over-planning as a weakness of project-level planning practice. However, the consultant's field observations suggest that matching workplans with actual operational capabilities (in terms of human and financial resources and available time) has been an important problem for all national components.
The risk of over-planning is indeed intrinsic to several features of the project's design, such as the open-ended "menu" of general outputs and activities foreseen by the ProDoc, the limited availability of funds for direct implementation of field activities, and the PRA-based start-up mechanism (leading to the formulation of long and very heterogeneous lists of felt needs). This risk is compounded by participatory implementation, which entails the continuous adjustment of field initiatives to the farmers' busy schedules.
To try to control over-planning, all field components have developed priority-setting mechanisms, based on a preliminary assessment of the consistency of field activities with the project's mandate and technical feasibility, and of the relevance to the felt needs of a significant part of the participating community. Furthermore, the project has, in certain cases, limited its coverage to communities settled within a "concentration area" (smaller than the official project area).
In spite of these measures, monitoring and evaluation exercises have, on several occasions, revealed a significant gap between planned deadlines and actual dates of accomplishment of field activities 37. This suggests that the number of field activities planned for the period under consideration went beyond the capacity of staff and participants and/or project resources.
37 Specific problems related to over-planning, such as delays and low rates of achievement, are described in the following documents: BOL/94/2; BOL/94/10b; BOL/95/8; BOL/96/6b; BOL/96/11a; NEP/96/10c; NEP/97/7; PAK/95/10b; TUN/96/8a; and TUN/96/12b.
Based on these observations, the following recommendations can be proposed with the aim of facilitating the formulation of sounder workplans:
Stricter priority-setting mechanisms should be built into the project-level planning module. To this end, information on cost-effectiveness and required time for different activities should be elicited from evaluation exercises and considered (together with technical, social and institutional factors) in assessing the actual feasibility of each activity to be included in the collaborative workplans.
The above recommendation raises a number of issues regarding the collection and use of information on staff and participants' time allocation. The consultant has recommended that all four NFTs systematically collect this type of information, with the aim of facilitating both process evaluation (by providing a concrete measure of staff and participants' involvement in the implementation process) and re-planning (by providing indications on the amount of time actually needed by staff and participants to implement a given activity).
The three NFTs who decided to follow this recommendation encountered several problems, including poor acceptability by field staff (who, especially at the beginning, perceived the request to report the amount of time spent on different activities as a means of control by project management) and technical difficulties in producing reliable estimates of the amount of hours needed to carry out a given step or achieve a given result. Furthermore, a controversy over the relevance and validity of time-allocation indicators for assessing the implementation process developed among and within NFTs. In Nepal, due to the large population in the project area and the number of User Groups involved in the project, this practice was perceived as unfeasible, after a brief (and quite incomplete) field test.
This situation is reflected by most of the opinions expressed on this subject during the PME self-assessment exercise. The NET in Bolivia was generally satisfied with the approach, though both the staff in-charge of M&E and the field-staff reported poor use of this information for re-planning. One member of the management team in Nepal stressed his disagreement (though the technique is no longer part of project-level PME practice in this country). Project management in Pakistan has questioned the relevance and usefulness of this technique, and the field-staff have identified practical problems in gathering relevant data. The Tunisian project management has stressed the need for relying on sound time-allocation estimates for planning, though the field-staff have questioned the relevance of these data. Finally, the project management of both Pakistan and Tunisia are uncertain as to the time needed to collect and analyse relevant data.
What conclusions can be drawn? The consultant still believes that time-allocation indicators are among the most valid and useful measures of implementation efforts for a project that spends about 80% of its budget on staff remuneration and that expects to rely on the participation of local communities to achieve its core objective (i.e. improved management of upland watersheds). As shown by the Bolivian experience, related operational and acceptability problems can be progressively solved by proper training and involvement of field-staff in data analysis and interpretation. However, it should also be acknowledged that these indicators are perhaps too abstract to be easily understood and that the suggested data collection and analysis techniques may appear to be too sophisticated for considering them for replication on a wider scale.
A compromise must thus be reached, which may help to satisfactorily meet the twofold need of (i) relying on timing estimates to be taken into account in the preparation of workplans and (ii) developing a more user-friendly approach to process evaluation. To this end, the following recommendations can be proposed to the NFTs willing to facilitate, and render more acceptable, the project-level process evaluation during the next project phase:
Whenever possible, standard estimates of working days needed to implement different activities, or steps, should be calculated on the basis of previous experience.
Process-evaluation exercises involving field-staff should be carried out on a regular basis by means of simple qualitative techniques, such as SWOL analysis.
According to the ProDoc for Phase 2, "due to the process-oriented nature of the project, the review and evaluation exercises will focus primarily on the qualitative aspects of this process, while not disregarding quantitative physical outputs in terms of work accomplished, people trained, etc.". Together with a number of practical and contingent considerations, this orientation has led the consultant to recommend to NFTs the adoption of an evaluation approach, known by specialists as 'implementation assessment 38.
38 For the design of implementation assessment, see King, J.A., L. Lyons Morris, C.T. Fitz-Gibbon. 1987. How to Assess Program Implementation. Program Evaluation Kit of the Center for the Study of Evaluation, University of California, Los Angeles. Newbury Park: Sage Publications.
Implementation assessment focuses on the implementation process and on its preliminary results (including achievement, quality, and short-term effects). I cannot provide any measure of the degree of achievement of project's immediate or development objectives (i.e. of project's long-term effects and impact). Nonetheless, this approach presents several practical advantages in comparison with more sophisticated, impact-focused, evaluation designs. These include user-friendliness of procedures for data collection and analysis, rapidity of execution, and capability to provide actors with periodical feedback on their performance during the life-time of the project.
Discussions held among project management, field-staff, and the consultant during PME inception missions led to a general consensus on the suitability of keeping project-level evaluation practice within the limits of the implementation assessment approach. Nevertheless, during the present self-assessment exercise, some participants criticised the lack of concern for impact evaluation or for the assessment of the degree of achievement of project objectives at the project-level.
Though lack of sound baseline information (which is a basic condition for carrying out impact evaluation exercises), together with the technical and financial constraints affecting project PME practice, actually justify the initial design of the system, insufficient attention paid to the long-term effects of the project should be acknowledged as a major limitation 39. Based on these considerations, the following recommendations can be proposed:
39 The NFT in Bolivia has begun to address this problem with the establishment of a Geographic Information System (see BOL/97/8).
The Coordination Unit and the NFTs participating in the third phase of the project should reconsider the possibility of including an impact evaluation module in the project-level PME system, in light of the current assets and constraints. If the decision is made to include this module, then relevant action should be taken as soon as possible to facilitate impact evaluation exercises.
The project management of all four NFTs indicated insufficient data collection skills of field workers as a major constraint in the implementation of project-level M&E practice. Moreover, project management in Nepal, Pakistan and Tunisia identified specific problems regarding the field workers' attitudes towards M&E.
Given that most field-staff had not been previously exposed to any kind of M&E practice, the development of data collection skills is considered to be directly related to the amount and quality of relevant training provided in the framework of the project. It should be stressed that during the start-up missions, the consultant provided inductive training on this subject to all NFT members. This included discussion of the relevance and practical use of the information to be collected and role-playing exercises for facilitating the acquisition of skills needed to gather and report the data to be recorded on the forms. Subsequent follow-up missions revealed that these inputs were effective only when field-staff were asked immediately to start using these tools in the field, and when, as recommended by the consultant, the project management ensured steady supervision of data collection. Unfortunately, these conditions could not always be met. The problem of the staffs attitudes towards M&E is more complex. Specifically, the project management has identified the following two major difficulties: feelings of being controlled (Pakistan and Tunisia) and perception of M&E tasks as an additional burden (all four NFTs).
The first difficulty basically relates to the collection of data on staff time allocation data. As already pointed out, this is a sensitive aspect of the suggested process-evaluation practice and needs to be carefully addressed by project management. If strong evidence of poor acceptability continues to be a problem (as seems to be the case in Pakistan and Tunisia), discontinuing the collection of these data and shifting to a different approach to process evaluation, as suggested in paragraph b, should be considered.
The perception of M&E tasks as an additional burden is probably related to a poor understanding of the usefulness of M&E practice. In the consultant's opinion, the best and perhaps only way of changing this attitude is to provide systematic and timely feedback of M&E findings to field workers and to involve them in making sense and use of this information.
These considerations led to the following two main recommendations:
Field staff data collection activities should be regularly supervised by the staff in-charge of M&E and/or other senior staff. This supervision should have a formative orientation (i.e. aimed at providing field staff with the opportunity to consolidate and upgrade their relevant knowledge, skills and attitudes).
A timely feedback of the findings of the evaluation exercises should be provided to staff who contributed to data collection in ad hoc sessions or workshops. Their contribution in interpreting this information and in applying it to improved implementation should be promoted through appropriate facilitation techniques.
All teams identified specific problems in the analysis and practical use of M&E information, including delays in processing field data (Nepal), insufficient analysis and reporting (Nepal and Pakistan), and inadequate use of findings for better re-planning (Bolivia and Tunisia).
The first two problems seem to be related to the difficulties met in Nepal and Pakistan in identifying a team member who could be entrusted with these tasks on a regular. As a consequence, M&E analysis and reporting had to be shared among managers and senior staff, leading to the overburdening of their already complex terms of reference. Underestimation of the time needed to carry out this task and, in some cases, the low priority given to its fulfillment, are likely to have contributed to slowing down the overall process.
A timely and steady feedback of M&E findings to the entire NFT has been ensured in Bolivia and Tunisia, where a better staffing situation allowed data analysis and reporting to be assigned to a single staff member, working on a full-time (or almost full-time) basis on this subject 40. However, in both countries, practical use of this information is still found to be a problem by field-staff (Bolivia) or by project management (Tunisia). Comments specifically refer to the use of estimates of time needed to implement different activities which, according to respondents' opinions, are not taken into consideration in subsequent planning exercises. Indeed, over-planning is still a major problem for both of these national components.
40 In Bolivia and Tunisia, reporting was facilitated by the fact that forms were filled-out in the official project languages (Spanish and French, respectively), whereas in Pakistan and Nepal, translating from Nepali and Urdu (respectively) represented an additional burden.
As stressed in the previous paragraph, provision of timely feedback on project performance allowing for improved re-planning is supposed to be one of the major advantages of the implementation assessment approach to project-level evaluation, However, this is possible only if arrangements are made to facilitate appropriate data analysis and use and if project management and staff commit to making consistent use of relevant findings.
Therefore, the following recommendations can be proposed, with the aim of improving M&E data analysis procedures and of ensuring timely application of relevant findings:
Higher priority should be given in the overall organization of the staff's work to the processing of M&E data. Whenever possible, this responsibility should be assigned on a full-time basis to one NFT member, who should possess the necessary skills or receive appropriate training. If this is not feasible, well defined M&E data processing tasks should be included in the official terms of reference of relevant staff, and time needed to perform these tasks should be adequately considered in project workplans and individual agendas.
All M&E reports (both formal and informal) should include a number of operational recommendations for re-planning relevant activities. After thorough discussion with the concerned staff, these recommendations should be fully taken into consideration in designing, programming and implementing future activities of the same kind.
3.2.a. Availability of participatory PME expertise within the NFTs
3.2.b. Time requirements for community-level PME practice
3.2.c. Transfer of community-level PME skills to interest groups and grassroots organizations
Three main technical problems can be elicited from a comparison of the weakest aspects of community-level PME practice identified by NFTs.
a) availability of participatory PME expertise within the NFT;
b) time requirements for community-level PME practice; and
c) transfer of participatory PME skills to interest groups and communities.
As in the previous section, the following paragraphs present a brief discussion of each one of these problems, leading to the formulation of operational recommendations.
The experience of all teams during the project's second phase shows that very specific expertise is needed to design, promote and facilitate community-level PME exercises. The key elements of this expertise include a clear understanding of the rationale and purpose of participatory action - research methods, the capacity to make sense of quantitative and qualitative information, strong organizational and communication skills (including fluency in the local language), and a creative and committed attitude towards the accomplishment of tasks.
Until recently, national academic and development institutions paid little attention to participatory methodology, thus finding local staff with the necessary qualifications has often been more difficult than expected. For this reason, project management has promoted in-service training initiatives. Specifically, participation of national senior staff in brief courses on PRA and participatory methods was facilitated. Brief seminars and workshops have been organized to address the training needs of extensionists and group promoters. Moreover, in Nepal and Bolivia, participatory PME field manuals have been prepared or translated
In spite of these efforts, project management has highlighted several difficulties in this area. In Bolivia, the field staffs lack of technical, organizational and communication skills for community-level PME is one of the main constraints to the development of this component of the overall PME system. In Nepal, the need for high quality PME facilitators is still felt. In Tunisia, poor performance and motivation of some staff have been identified as the most important factors affecting the development of community-level PME practice.
These findings confirm the consultant's impression that in-service training activities often have not been sufficiently relevant, effective or long to successfully address the staffs learning needs in this area. Considering the essential role that community-level PME is meant to play in the implementation of the project's approach, it can be concluded that major efforts must be made to allow NFTs to successfully face the challenges of replication and institutionalisation which will be posed by the forthcoming third phase.
To this end, the following operational recommendations can be forwarded:
Preparation of a comprehensive training package on community-level PME should be facilitated by the project's Coordination Unit in collaboration with centres of excellence in this area. Such a package should include: a community-level PME curriculum (to he developed according to learning needs identified by project management and field-staff); a series of guidelines for implementing training sessions; and a set of learning materials (including readings and self-assessment exercises).
Implementation of a solid in-service training course (with a duration of two or three months) based on the above package, should he considered as a priority activity, to he included in the third phase workplans. This course should be attended by NFTs and selected staff of local or national partner institutions. Coordination and facilitation of teaching/learning activities should be entrusted to a consultant hired ad hoc, with expertise in both training and participatory methods.
The NFTs' self-assessment exercises explicitly identified time requirements as a main constraint in community-level PME practice.
In particular, in Bolivia, insufficient availability of time has prevented the staff in-charge from establishing and operating a consistent participatory monitoring practice. In Nepal, the time needed to organize and implement evaluation and re-planning workshops has led this practice being limited to eight pilot villages (out of the 26 villages which prepared a Community Action Plan). In Tunisia, the length of the initial participatory planning process has contributed to the decision of limiting the number of douars involved in the project's first planning cycle.
Though the field-staff, through continued experience, may eventually increase their efficiency in dealing with this task, the situation is obviously not sustainable in the long-run and is thus incompatible with the objectives of institutionalisation and replication to be pursued during the project's third phase. In fact, it is unlikely that such a time-consuming practice could be successfully transferred to already under-staffed and over-committed national and local counterparts, or any other institutional partners.
In the consultant's opinion, to overcome this difficulty, a substantial change must take place in the strategy followed to promote community-level PME. In particular, greater emphasis should be placed on making interest-groups and other grassroots organizations responsible for the organization and implementation of community-level PME events, and the project should make a more solid commitment to implementing relevant capacity-building activities.
Based on the latter consideration, the following recommendation can be proposed:
During the third phase of the project, the role of project staff in the operation of community-level PME exercise should he progressively re-oriented. Less work and lime should he invested in organizing and implementing community-level PME events, whereas more emphasis should he given to providing training and formative supervision on participatory PME to interest-groups and grassroots organizations.
After a number of years of exposure to the project, are local participants skilled and motivated enough to autonomously begin organizing and implementing their own community-level PME practice? Based on the findings of the self-assessment exercise, the answer to this question seems to be "no" Project management gave rather low scores to questionnaire items related to community self-reliance. Moreover, though the NET members' statements on this subject are often controversial, the quantitative information collected indicates a lack of trust in the communities' ability to run PME exercises without the direct support of field-staff. According to members of the NFT in Bolivia, some progress has been made in participatory planning, yet the people still lack a complete understanding of the potential of community-level M&E exercises. According to the NFT in Nepal, for the time-being, communities and User Groups lack the proper capacity to assess their own activities in terms of costs and benefits, and they are only erratically involved in M&E practice Some members of the NET in Pakistan feel that communities are not skilled enough to autonomously carry out PME exercises and that the validity of community-level evaluation is highly disputable
However, the negative opinions of project management and staff should be weighed against the rather different views conveyed by focus-group interviews with participants. Relevant statements reported in Chapter 2 suggest that in all countries, project participants have developed a rather good understanding of the rationale and purpose of the different elements of community-level PME practice; they have been able to deal autonomously with problems not solved by project staff, such as the establishment and operation of participatory monitoring in Bolivia and Nepal, and are actually interested in continuing to practice PME, as well as in learning more about the subject.
It is difficult to determined the reasons for the gap between the project staffs views and the outcomes of participant focus-group interviews. Nonetheless, it must be pointed out that, in most cases, the exposure of community members to PME practice has been limited to their involvement as participants in exercises designed and implemented by project staff. To date, little effort has been made by all field component (with the exception of Nepal) to train representatives of interest groups and grassroots organizations in running PME events or managing relevant techniques and tools. Obviously, this lack of training is the most important constraint in handing over community-level PME practice to project participants.
Based on these considerations, two main recommendations can be proposed:
A comprehensive training programme on community-level PME should he implemented for the entire duration of (he project's third phase, addressing selected members of interest groups and grassroots organizations of both new and old project areas. This training should include both residential events, in which basic PME facilitation skills will be transferred, and community-based activities, which will provide trainees with the opportunity to practice the new skills with the advice and support of qualified (or re-qualified; see above) pro/eel staff.
As soon as possible, the responsibility of organizing and implementing community-level PME events should he handed over to trained community members. However, a backstopping action aimed at controlling the quality of community-level PME practice and at providing relevant advice should be established and operated during the entire life of the project. Arrangements with local institutions should also he made, to ensure the continuation of this backstopping action after the project's termination
