PC 83/4 (b)
|
Eighty-third Session |
Rome, 8-12 May 2000 |
Synthesis of Recent Field Project Evaluations |
Summary of Main Findings on Areas for Improvement
Synthesis of Field Project Evaluations 1994-99 (Excluding TCP)
COMPARISON OF PRPJECTS: 1985-91 VS. 1994-99
EFFECTS AND IMPACT OF PROJECTS
POTENTIAL FOR IMPROVEMENT IN PROJECT PERFORMANCE
QUALITATIVE ANALYSIS OF MISSION FINDINGS AND RECOMMENDATIONS
FUTURE REPORTING TO THE GOVERNING BODIES ON SYNTHESIS OF PROJECT
EVALUATIONS
Thematic Evaluations of TCP Projects:
Apiculture and Sericulture (1998) and Legislation (1999)
The analysis indicated that results were good in terms of achieving the projects' outputs and effects for the benefit of the target populations. Nevertheless, particular areas were identified for attention in project design and implementation. These include:
1. The note brings together an analysis of the findings of field project evaluations of UNDP and trust fund projects by independent missions and thematic evaluations of FAO Technical Cooperation Programme (TCP) projects.
2. This is the fourth report to the Governing Bodies covering the results of individual field project evaluations since the 1992-93 Programme Evaluation Report (PER). In that document (C 93/4, Chapter 4), a synthesis was made of all project evaluations in the period 1985-91. Subsequent reports on the results of project evaluations were contained in the Programme Implementation Report (PIR) 1994-95 (covering evaluations from 1992-94) and the PIR 1996-97 (covering evaluations through 1997). This report for 1994-99 includes the time periods reported in the previous PIRs and is updated to include the biennium 1998-99. The year 1994 is appropriate as the starting point for the study because in that year a number of improvements were introduced in the FAO evaluation process, including specific requirements to address the issues of cost-effectiveness and greater focus on effects and impact and contributing factors to project sustainability.
3. The note includes quantitative information, as presented in the PIRs, but also includes more in-depth analysis of factors associated with effects and impact and qualitative assessments based on reports covering particular subject areas. The focus is on finding areas for improvement, particularly in order to achieve project effects and impact. The analysis is based on the findings of independent tripartite evaluation missions for 279 projects evaluated between 1994 and 19991. In terms of areas of work, the projects covered were in reasonable harmony with the distribution of FAO's overall field programme (see Table 1). Theme-oriented TCP2 evaluations began in 1997, with a study of food quality control projects, the results of which were reported in the Programme Evaluation Report 1998-99 (Chapter Four). This report covers apiculture and sericulture (1998) and assistance in legislation (1999). Groups of TCP projects are evaluated by the Evaluation Service around particular themes, in order to draw lessons for future design of similar projects, determine achievements at project level, improve cost-effectiveness and provide a basis for accountability reporting.
4. The Technical Cooperation Department, in consultation with the Evaluation Service, makes the choice of topics for evaluation. Apiculture and sericulture were selected for evaluation in order to address a "problem area" and given the large volume of requests. While important for small farmers in many parts of the world, these topics represent a very small part of FAO's overall programme. Nonetheless, a large number of requests for TCP projects in these areas had been received and approved over the three biennia preceding the study. A few of these projects had notable implementation difficulties that were known to the Technical Cooperation Department before the study was undertaken. Assistance in legislation is one of the major areas for TCP requests and this was the main reason to select this topic. The evaluation examined not only projects that were exclusively in the legal sphere, but also reviewed projects in other subject areas that had a legal component.
Percentage of total projects evaluated | Percentage Share in FAO's Total Extra-budgetary Development Field Programme of this Technical Programme Area* | ||
Agricultural Production and Support Systems | 40 | 46 | |
of which: | Natural Resources | 10 | 12 |
Crops | 17 | 17 | |
Livestock | 4 | 7 | |
Food and Agriculture Policy, Planning, Nutrition & Legal | 7 | 11 | |
Fisheries | 7 | 8 | |
Forestry | 31 | 22 | |
Sustainable Development | 15 | 13 | |
of which: | Technology Development and Transfer | 7 | n.a. |
Women in Development and Population | 5 | n.a. | |
Rural Development and Agrarian Reform | 2 | n.a. | |
Total | 100 | 100 | |
* By value 1996-97, based on PIR1996-97 |
5. Table 2 below presents statistical information on evaluations undertaken by region and source of funding. The Table also presents global information on the types of evaluation undertaken (at mid-term, terminal, or ex-post).
Region | UNDP | TF | TOTAL | ||||||
94-95 | 96-97 | 98-99 | 94-95 | 96-97 | 98-99 | 94-95 | 96-97 | 98-99 | |
Africa | 43 | 40 | 37 | 40 | 47 | 41 | 41 | 44 | |
Asia and the Pacific | 49 | 50 | 57 | 16 | 25 | 22 | 36 | 36 | 34 |
Latin America and Caribbean | 2 | 2 | 3 | 28 | 21 | 22 | 12 | 12 | 16 |
North Africa, Near East and Europe | 6 | 6 | - | 12 | 5 | 2 | 8 | 6 | 1 |
Inter-regional | - | - | - | 7 | 9 | 7 | 3 | 5 | 5 |
Total | 100 | 100 | 100 | 100 | 100 | 100 | 100 | 100 | 100 |
By type of Evaluation | |||||||||
Mid-term | 51 | 42 | 40 | ||||||
Terminal | 48 | 54 | 45 | ||||||
Ex-post | 1 | 4 | 15 | ||||||
TOTAL |
100 | 100 | 100 |
6. As noted above, project evaluation report formats were amended in 1994 to be more comprehensive in their coverage and to better reflect concerns expressed by the Governing Bodies. Table 3 examines the results of projects over the last three biennia against those that were reported to the Governing Bodies in the PER 1992-93.
Aspect | Percentage rated 1985-91 |
Percentage rated 1994-99 |
||||
Poor |
Satisfactory | Good |
Poor |
Satisfactory | Good |
|
Project Design | 26 |
56 |
18 |
21 |
40 |
39 |
Implementation | 29 |
49 |
22 |
9 |
39 |
52 |
Project Outputs: | ||||||
quality | 9 |
62 |
29 |
7 |
43 |
50 |
quantity | 13 |
57 |
30 |
8 |
44 |
48 |
Project Effects | 16 |
60 |
24 |
11 |
39 |
50 |
7. The results point to an improvement over time. Considerable effort has been put into improving project design, for example, by upgrading staff skills through training programmes and improving programming and evaluation guidelines. Equally important have been the stringent standards established by donors for accepting proposals for funding, including requirements for project design. Similarly, project effects are increasingly rated "good".
8. Table 4 attests to very reasonable results from projects in terms of their benefits to countries, as determined by evaluation missions. It is against these basically positive findings that subsequent analyses of areas for improvement need to be viewed.
Projects were found to have: | ||||
Addressed a genuine development problem | A problem of major importance | 85% | A significant problem | 98% |
Effects in terms of use expected to be made of outputs | At least 80% of outputs expected to be used as foreseen | 50% | At least 60% expected to be used as foreseen | 89% |
Expectations of sustainable impact | Considerable impact | 45% | Some or more sustainable impact | 86% |
Cost-effectiveness for sustainable effects | The most cost-effective approach | 73% |
9. Only 18 evaluations (6%) were ex-post. Thus, most evaluations were looking forward towards probable eventual impact but effects (use of outputs by beneficiaries) were frequently already evident, especially for projects evaluated at or near the end of project implementation (terminal evaluations). There was some divergence between regions (see Annex Table 1). In particular, missions had lower expectation for use of outputs and sustainable impact in projects evaluated in the Near East and Europe. Although the reasons for this were not uniform, missions found that project designs consistently showed extreme over-optimism as to what could be achieved within project budgets and time-frames and also over-estimated the capacities of national staff4.
10. Projects were examined for impact against different types of objectives, i.e. a) Policy, planning and legislative improvements b) Strengthening national institutional capacity c) Uptake of technical improvements d) Expansion of pilot activities and e) Follow-up investment. Follow-up investment was the weakest and uptake of technical improvements and institution building the strongest. As might be expected, the risk of no impact was greatest in projects involving uptake of policy planning and legislative improvements, replication of pilot activities and investment follow-up. In such projects, achievement of impact usually depends on positive decisions at higher echelons, beyond where the project may have been working (if there is no follow-up, there can be little impact).
11. The analysis below focuses on the identification of potential areas for improvement, based on evaluation missions' assessments on the quality of project selection, design and management. It also examines the specific factors that contributed most to limiting project effects and impact, and cost-effectiveness.
Areas where there was potential for improvement, in approximate order of importance | Percentages of Projects with Scores of: (scores on a 5 point scale with 1 indicating the greatest potential for improvement and 5 the least need for improvement) |
||
1+2 | 4+5 | 3 | |
Project Design - Realism | |||
� identification of pre-requisites and risks for project success | 37 | 24 | 39 |
� project duration too short | 515 | n.a. | n.a. |
� workplanning | 34 | 28 | 38 |
� expectation of national resource commitments | 21 | 38 | 41 |
� expectation that outputs would continue to be used after project completion | 15 | 53 | 32 |
Project Design Quality - overall assessment | 21 | 39 | 40 |
� clarity of immediate objectives | 25 | 46 | 29 |
� specification of outputs and targets | 21 | 45 | 34 |
� cost-effectiveness of the project approach | 13 | 51 | 36 |
Implementation Management- overall assessment | 9 | 52 | 39 |
� Government Input to management | 23 | 37 | 40 |
� FAO input to management, of which | 9 | 54 | 37 |
- FAO - written comments | 20 | 39 | 41 |
- FAO - country visits | 16 | 50 | 34 |
A. Aspects of the Project Process (Cycle) Where Missions Found the Greatest Need for Improvement: | % of projects | ||
Project Selection | 13 | ||
Project Design | 33 | ||
Project Supervision | 24 | ||
B. Deficiencies in Projects Found Most Likely to Limit Sustainable Effects and Impact - in approximate order of importance | % of cases identified | ||
Project Design | Weaknesses in national institutions | 49 | |
Lack of national financial resources for follow-up | 42 | ||
Insufficient national manpower | 17 | ||
Design and Implementation | Insufficient involvement of beneficiaries | 18 | |
Lack of national commitment | 13 | ||
C. Factors Found to Have Particularly Limited Cost-effectiveness | % of cases identified | ||
Scheduling - work planning | 21 | ||
Government procedures | 27 | ||
FAO procedures | 19 | ||
D. Mission Findings on Aspects on Features of Projects Which Could have increased cost-effectiveness | More | Less | |
Project Design |
Focus/concentration in the project on central objectives | 39 | 3 |
Design and Implementation | Use of national training | 37 | 2 |
Use of the private sector | 41 | 5 | |
Use of NGOs | 36 | 7 | |
Use of government capacity | 24 | 11 | |
Use of national experts | 25 | 4 | |
Use of short-term staff | 27 | 10 | |
Procedures | Delegation of authority | 30 | 3 |
12. Project Design: Quantitative analysis showed that project design was the weakest aspect of the projects evaluated: missions reported in 33% of cases that design was the aspect of the project where there was the greatest need for improvement (Table 6). As can be seen from Table 5, missions found that project designs suffered from over-optimism as to what could be accomplished in the project duration. Fifty-one percent of projects were considered too short and analysis showed lack of realism on duration was co-related with reduced effects and impact. In projects performing below the optimum in terms of effects and impact, analysis showed formulators were particularly over-optimistic as to the use that could be made of project outputs. Projects also tended to have unrealistic expectations of the capabilities and resources of national institutions (Table 6). Further, there was inadequate attention to risks and prerequisites for project success (Table 5).
13. In separate questions on factors having the greatest negative impact on project cost-effectiveness, scheduling was found to have been the most important factor in 21% of cases (Table 6). There tended to be over-expectation of what inputs governments could reasonably be expected to provide to the project (Table 5). Other faults were less pervasive, but significant proportions of projects could have been considerably improved with respect to clarity of immediate objectives and targets. There was also found to be a need for more focus in projects in 39% of cases (Table 6). In further analysis, failure to adequately specify beneficiaries was found to be linked to sub-optimal performance in terms of effects and impact.
14. Missions found that 20-40% of projects would have been more cost-effective had there been more reliance on national training, more use of the private sector, NGOs, national experts and short-term staff and a greater reliance on government capacities. However, it was noticeable that there could also be too much of these features and it appears that where for instance, there was a heavy use of NGOs, short-term staff or government capacity, this was considered counter-productive (Table 6).
15. Project Implementation and Management: Projects which performed sub-optimally in terms of effects and impact suffered particularly from inadequate management, with internal management being the weakest point but FAO supervision also being an area for improvement. Both Government and FAO procedures were found by missions to have constrained performance and insufficient delegation of authority was identified as a problem in 30% of cases (Table 6). In what can be both a design and implementation fault, the involvement of beneficiaries could have been improved in 18% of cases (Table 6).
16. Some of the issues identified above were amplified by a qualitative review of evaluation findings and recommendations, in particular project areas where there was a substantial volume of similar findings.
17. Capacity-Building: Institution strengthening projects evaluated in recent years indicate that the focus has been shifting increasingly from establishing or enlarging government departments to strengthening capacity of existing institutions for new functions, such as community forestry or for environmental planning. Missions were not generally optimistic for the sustainability of results of earlier institutional expansion projects due to shortages of government resources. More recent projects have been designed to develop people's self-reliance and ownership through participatory and group approaches. For both types of institution building, missions generally concluded that the project duration was too short and a further phase was essential to consolidate results. This points to:
18. Policy support: The relatively weak performance of projects in the area of policy support in realising their effects and impact has been noted above. Projects attempting to strengthen policy and planning directly were most influential not through direct provision of advice, but rather when they identified the major issues and supported national dialogue between the community and the political level. It was also found that most policy outputs were not from projects specialised in policy but from projects carrying out institutional development and development support. Thus a project for area development might have an influence on land tenure policy, one on community forestry might influence the approach to extension.
19. People's Participation: Projects where people's participation was important were on occasion criticised for giving excessive attention to establishing new participatory groups, while failing to successfully initiate other improvements through the groups. There were also criticisms that groups were not necessarily the best development tool in some situations and small-scale private initiative could have been encouraged as an alternative. As with production projects, unless people saw a definite benefit to themselves from the groups (usually economic), progress was limited. People's participation projects were sometimes found to be both too short and to be spread over an excessively wide area. This tends to indicate that participatory projects should initially work intensively in a small area with a few groups, thus leaving a lasting capacity in the groups themselves and in the support personnel who have worked with the groups.
20. In institution-building, people's participation and policy projects the qualitative analysis found linkages needed to be strengthened. There was a particular danger of projects in people's participation working as separate entities rather than forging sustainable local partnerships with NGOs and government agencies concerned. Similarly, missions found training to be among the projects' most valuable outputs for capacity-building. There appeared to be an underlying assumption that individuals with increased skills would contribute to development, even if the specific endeavour for which they were trained was not sustainable.
21. Development of Production and Land Management: For projects designed for the development of production and improved land management, there were frequent findings that insufficient attention had been given to economic and marketing aspects. There were a few examples of pilot demonstrations failing to achieve impact because of lack of a legal basis, particularly lack of clear tenure to the resource. There were also a large number of calls by missions for improved monitoring and evaluation to determine which approaches and packages were in reality proving acceptable to people and were producing the expected improvements. Similarly a few missions highlighted attention to these aspects as having been important factors in achieving sustainable effects and impact.
22. Gender: While there were not a large body of findings on reasons for the effectiveness of gender related activities, several missions noted a failure of projects to specifically target women, even when they were the main actors, and others pointed to success when packages and extension training were specifically designed for women. In some situations the use of female staff to reach women had been advantageous, even in societies where there were no particular taboos on men communicating with women.
23. Regional Projects: Some qualitative conclusions emerged with respect to regional projects, in particular:
24. Overall Conclusions and Recommendations: The present report shows considerable improvement overall in project design, implementation and production of outputs from the last examination over a period of time (1985-91). Nonetheless, it will be noticed that most of the findings and issues arising are very similar to those made in the PER 1992-93. It is recommended that given the persistent weaknesses in project design, further efforts should be pursued in the context of decentralized arrangements for the operational activities, including in particular: (a) preparation of updated guidelines in project formulation and design with particular attention to the areas needing improvements; (b) further training of the FAO staff, especially those in the decentralized offices, on the subject; and (c) strengthening the existing mechanisms for reviewing and appraising project proposals, both at the decentralized offices at at Headquarters.
25. This report also presented a limited examination of the qualitative aspects of particular types of projects and drew some conclusions in these various areas. It is suggested that for the immediate future, reporting on results of evaluations goes further in this direction and that syntheses of evaluation reports be based on particular programme areas and cross-cutting themes in line with the priorities of the Strategic Framework or possibly on the work in particular regions. Programmatic evaluations will also systematically assess the Regular Programme and related field activities. This approach would be consistent with that followed for evaluation of TCP. It is believed that such analysis, if based upon a sufficiently large sample of projects, will provide more focussed analysis and recommendations for consideration by Management and the Governing Bodies. A broader analysis may be repeated every 2 or 3 biennia when a further substantial body of evaluations have been completed.
26. Both evaluations were implemented following similar methodologies. In keeping with FAO policy, evaluations were carried out with the assistance of external consultants. Two consultants (one each for apiculture and sericulture) were used in the 1998 exercise and four consultants, each specialised in the region being visited, took part in the 1999 legislation review. For the apiculture/sericulture review, all projects (22) approved in the previous three biennia were evaluated while for legislation, a sample of 31 (34%) projects in 17 countries was reviewed.
Apiculture and Sericulture
27. Apiculture and sericulture are potentially significant sources of income for small farmers in developing countries. Apiculture in particular can be important for the landless and is highly significant to agricultural production and ecosystem conservation in general due to its role in pollination. Sericulture is an important economic activity particularly for women, as they are most involved in the labour-intensive work of placing mature silkworms in frames and spinning yarn.
28. The majority of the projects dealt with significant development problems, usually relating to disease control (varroa in bees, pebrine disease in silkworms). However, some were not well justified on technical grounds or would not have met TCP criteria for approval had conditions on the ground been better known. Most of the projects were found to be stand-alone efforts and not part of any larger Government or donor-funded programme.
29. Four of the projects were considered to have had serious implementation difficulties. Performance of international consultants, with a few exceptions, was good. Nonetheless, it was noted that economies could have been achieved in some projects (particularly in sericulture) which used several experts with similar qualifications. Also, national consultants used in some countries tended to mirror the expertise of international consultants (although some were lacking in technical qualifications). Technical backstopping from FAO for the sericulture projects was found to be generally better than for apiculture. A serious problem was the late or non-production of terminal statements or letters of completion.
30. Evaluation missions rated the individual projects overall, taking into account all aspects and against the following criteria, using a 3-point scale (Good, Satisfactory, Unsatisfactory).
31. The results were as follows6:
Quality | Overall | Relevance | Design | Implementa-tion | Effects/Use of Outputs | Impact/ Sustainability |
Good | 8 | 7 | 2 | 6 | 8 | 8 |
Satisfactory | 6 | 6 | 12 | 10 | 6 | 2 |
Unsatisfactory | 5 | 7 | 6 | 3 | 5 | 9 |
Not Rated7 | 1 | - | - | 1 | 1 | 1 |
32. The number of unsatisfactory projects was felt to be too high. On the whole, the best apiculture projects were those that dealt with disease outbreak (varroa) in countries where apiculture is economically significant. However, several apiculture projects were deemed unsuccessful because there was little scope for spreading their results among beekeepers in the country or their technical justification was poor. Because of much lower costs of entry, apiculture is likely to be of more interest than sericulture to most small farmers. It was found, however, that projects should be made more relevant to the majority of beekeepers in countries, rather than aiming at the most modern production methods. A consequence of this approach is that training did not have the desired impact at producer level.
33. Sericulture could be a relevant topic for TCP assistance, provided there is a plan for development of the sector and the role of the TCP project is clearly established. The evaluation found that chances for sustainable impact of sericulture projects are much less in countries with a limited tradition of sericulture and where there were no marketing opportunities.
34. It was found that there was insufficient linkage of lessons from projects to normative work at FAO and inadequate transfer of lessons from one project to another.
Legislation
35. The projects reviewed were found to be highly relevant to development problems in the recipient countries and the assistance received from FAO was greatly appreciated. Project design tended to be rather general in terms of description of problems to be tackled and the approach taken to implementation, but there are rather limited prospects for improving the amount of information made available. Any deficiencies in project design were generally remedied by the initial mission to the country, when issues were identified in greater depth and project implementation strategies decided.
36. Quality of the international consultants and Legal Office (LEG) staff (who served as legal consultants in 13 of the projects reviewed) was generally very high. The fact that FAO has a great deal of international expertise in various aspects of natural resources legislation was found to be its primary comparative advantage and the main reason for TCP project requests. Technical backstopping of legal components of projects has been generally excellent, with active interchange between LEG officers, consultants and national counterpart staff. In some cases, there has been continued post-project follow-up by individual LEG officers as legislation has worked its way through national processes. This was greatly appreciated by member governments.
37. Using the criteria mentioned for the apiculture/sericulture evaluation, legal projects (or legal components of projects) were scored as follows:
Quality | Overall | Relevance | Design | Implementation | Effects/Use of Outputs | Impact/ Sustainability |
Good | 11 | 22 | 15 | 17 | 11 | 9 |
Satisfactory | 17 | 9 | 15 | 10 | 12 | 12 |
Unsatisfactory | 3 | 0 | 1 | 4 | 8 | 10 |
38. While most projects were relevant and well-implemented, eventual project impact was found less satisfactory. In most countries where follow-up and impact were not satisfactory, the reason could be found in particular national circumstances such as economic upheaval or the failure to advance drafted legislation, which had been found acceptable by the individual Department concerned at the time of project implementation, through the legislative system. In order to ensure a complete package of assistance, the evaluation recommended that implementing regulations should normally be prepared in projects along with laws.
39. The two most important factors for likely project success were a high degree of stakeholder involvement in project implementation and a favourable policy framework for achieving the desired legislative reforms. Most projects made considerable effort to ensure broad consultation with potential stakeholders. Some notably successful projects also included a policy component and in these projects, there was close association between the legal and policy activities. Factors found that could negatively impact on project success included replacement of the project's main promoter or "champion", projects which are asked to mediate internal disputes and projects where legislative components are add-on measures outside the main project thrust. It also must be appreciated that proposed legislative reforms that seek to alter long-standing administrative arrangements often encounter resistance.
40. The Evaluation Service has now conducted three thematic evaluations of TCP projects and some general recommendations and lessons have emerged which are highlighted here:
Annex Table 1: Project Performance by Region of the World |
||||||||||||
Projects were found to have: | ||||||||||||
Africa | Asia & Pacific | Latin America & Caribbean | Near East & Europe | Inter-Regional | Africa | Asia & Pacific | Latin America & Caribbean | Near East & Europe | Inter-Regional | |||
Number of project evaluation questionnaires analysed | 115 | 100 | 38 | 16 | 10 | |||||||
Extent to Which Project addressed a genuine development problem |
||||||||||||
A problem of major importance | 88% | 88% | 78% | 81% | 70% | A significant problem | 98% | 99% | 100% | 88% | 100% | |
Effects in terms of use expected to be made of outputs | ||||||||||||
At least 80% of target | 46% | 55% | 59% | 8% | 62% | At least 60% of target | 91% | 89% | 91% | 50% | 100% | |
Expectations of sustainable impact | ||||||||||||
Considerable impact | 48% | 47% | 44% | 19% | 40% | Some or more sustainable impact | 91% | 83% | 88% | 69% | 90% | |
Cost-effectiveness for sustainable effects | ||||||||||||
The most cost-effective approach | 79% | 69% | 79% | 36% | 70% |
Annex Table 2: Project Performance by Sector/Sub-sector |
|||||||||||||
Projects were found to have: | |||||||||||||
Natural Resources | Crops | Livestock | Forestry | Fisheries | Natural Resources | Crops | Livestock | Forestry | Fisheries | ||||
Number of project evaluation questionnaires analysed | 29 | 49 | 12 | 85 | 19 | ||||||||
Extent to Which Project addressed a genuine development problem | |||||||||||||
A problem of major importance | 86% | 84% | 92% | 90% | 94% | A significant problem | 100% | 94% | 100% | 99% | 100% | ||
Effects in terms of use expected to be made of outputs | |||||||||||||
At least 80% of target | 38% | 65% | 45% | 51% | 53% | At least 60% of target | 77% | 100% | 91% | 95% | 87% | ||
Expectations of sustainable impact | |||||||||||||
Considerable impact | 41% | 60% | 45% | 46% | 59% | Some or more sustainable impact | 85% | 92% | 82% | 91% | 82% | ||
Cost-effectiveness for sustainable effects | |||||||||||||
The most cost-effective approach | 74% | 77% | 90% | 75% | 75% |
Annex Table 3 was compiled based on a review of mission reports for qualitative lessons in predicting likely achievement of effects and impact. It includes those factors that had a particularly high percentage of scores in the range of 1 to 3 for effects and impact (column b), i.e. the range of nil to moderate effects and impact. Column (c) compares these scores with the percentage of scores (also in the range of 1 to 3, i.e. poor to satisfactory but not good) that the factors had in the total population of projects (column a). The bigger the difference in column (c), the more closely the factor in question is a predictor of sub-optimal effects and impact.
Annex Table 3: Factors Found to be Particularly Correlated with Sub-Optimal Performance in Achieving Effects and Impact |
|||
Percentage of Projects scored in the lower range (1, 2 or 3 for this factor on 5 point scale) |
|||
Factor | In the Overall Population (a) |
In projects with only moderate or inadequate Effects and Impact (b) |
(b) - (a) (a) |
Realistic expectations for output use at the time of project design | 47 | 65 | 38 |
Project design, of which | |||
� project duration too short | 51 | 59 | 15 |
� specification of beneficiaries | 44 | 55 | 25 |
Implementation issues, of which: | |||
� internal management | 49 | 69 | 41 |
� FAO supervision visits | 50 | 59 | 18 |
� Government input | 63 | 78 | 24 |
____________________________
1 A total of 301 projects were evaluated and reports met standards for comprehensiveness and quality of assessment in 279 cases (93% of the total).
2 The TCP is part of the Organization's Regular Budget, funded by assessed contributions from member states. TCP projects are of short duration (not over 24 months) with a maximum budget of $400,000 and are aimed at solving urgent problems faced by developing member nations of FAO. The projects are intended to be catalytic and to result in significant follow-up action by the Government.
3 Analysis hereafter refers to the period 1994-1999.
4 While the results for the Near East and Europe are considerably below those for other regions, they are based on a much lower sample of projects.
5 For this particular question, missions are offered only 3 choices: too short (51%), too long (3%), about right (46%).
6 Two of the projects evaluated were continuations of previous projects; thus the total number of results in each column will be 20, rather than 22.