ANNEX 1: SPFS EVALUATION TERMS OF REFERENCE

A. Purpose of the Evaluation

1. The evaluation is to be undertaken both in response to the request of the Governing Bodies and to meet internal management needs. The independent external evaluation will serve two aims:

• to provide a credible accountability report on the Programme, containing in-depth analysis and assessment of its continuing relevance, effectiveness in achieving the results and overall cost-effectiveness; and
  
  
• to consolidate and enhance the knowledge basis of the Programme for the future by learning from the experience to date, especially by identifying emerging key issues, strengths and weaknesses.

B. Issues to be Covered

2. The evaluation will cover the following main aspects in line with the standard evaluation approach:

(i) Review of the efficiency and adequacy of the programme implementation process over the period, covering:

1. programme planning and coordination within FAO;
   
   
2. technical backstopping and administrative support to the implementation in FAO (including guidelines and advisory notes, etc);
   
   
3. role and participation of the target LIFD countries;
   
   
4. collaboration with international agencies and donors;
   
   
5. financial resources mobilized (national, FAO and other external sources);
   
   
6. monitoring, review and oversight (by FAO and participating countries).

(ii) Analysis and assessment of overall achievements and results, covering:

1. Number of countries participating and their status, including features of their pilot operations (target crops, farming systems, types of farmers, integration of the four programme components, etc);
   
   
2. Results achieved in participating countries in terms of:

- number of sites covered, including the components implemented in each site;

- number and nature of participating farmers and communities in each site, including their resource commitment;

- adoption of improved production methods and practices as well as their effects on on-farm productivity;

- contribution of the pilot operations to supply and availability of food to the farmers and local communities (not only increases in supply but also in reduced variability of annual supplies);

- contribution of pilot operations to the food security of vulnerable rural population groups;

- enhanced participation of farmers and other local stakeholders, in particular women, including institutional arrangements;

- improvements in extension and other support services to the rural producers in the context of pilot operations;

- strengthening of national capacity and commitment for food security, including policy and institutional measures;

- effects and impact on mobilizing external aid and resources for food security in the participating countries, including assistance in support of the pilot operations;

- governments' plans for future operations;

3. Quality of design and implementation of the pilot operations, including:

- the use of the four Programme components (water, intensification, diversification and constraints analysis) and South-South cooperation;

- effectiveness of national programme management (including coordination, monitoring, reporting and problem solving);

- participation and mobilization of farmers, local actors and government agencies;

- support by, and collaboration between, the governments and FAO;

4. Key constraints experienced in the pilot operations to date.

(iii) Assessment of viability/sustainability and replicability of the results achieved in the participating countries, covering:

1. prospects for various countries, especially in those with longer implementation periods;
   
   
2. key positive and negative factors influencing viability (i.e. in terms of on-farm cost/benefit), broad sustainability of the results and replicability of the approach as a whole;
   
   
3. main lessons and issues for the future.

(iv) Overall assessment on the Programme, including South-South cooperation, where applicable, covering:

1. its cost-effectiveness and validity in the light of implementation performance and achievements;
   
   
2. strengths and weaknesses in the design, implementation and management;
   
   
3. main issues requiring action, especially in terms of:

- further improvement of the efficiency, effectiveness and sustainability of the pilot operations;

- considering the transition from pilot phase to its extension phase and into the expansion phase;

4. prospects and options for the future (by groups of countries at different stages of progress).

(v) Based on the above, make recommendations regarding how the Programme may be improved (these could be addressed to FAO, participating countries, and the international donor community).

C. Conduct of the Evaluation

3. At present, a total of 86 countries are participating in the SPFS at various stages, including 60 in the operational stage (some 30 countries with a 3-year or longer period of implementation). The evaluation will be based largely on field visits to selected countries where the SPFS is in operation. In order keep the exercise manageable physically and financially, field visits will be limited to 12 countries, most of which have more mature programmes.

4. To facilitate in-depth analysis and to ensure a common approach in the field visits, more detailed and operational criteria and methods for evaluating individual pilot operations will be prepared during the preliminary phase of the evaluation. The criteria and methods will be based on the methodology used for FAO field project evaluations (logical framework and result-oriented approach, including if required definition of indicators for gauging key achievements and results).

D. Implementation Schedule

5. The evaluation exercise will be implemented during 2001 as follows:

(i) Preparatory phase (present-March 2001), covering:

- preparation of final Terms of Reference and detailed evaluation methodology for field visits;

- identification potential team members, selection and their engagement;

- collection of documents and information for the evaluation;

- preparation of proposed workplan.

(ii) Desk review (March-May 2001) for the preparation of studies for use by the evaluation team, covering:

- a concise summary of the Programme;

- desk studies of the performance and achievement in countries under operational status. The studies will take advantage of available information, including Field Inspector Reports;

- a summary analysis of perceived achievements and emerging issues under the Programme;

- proposals on the selected countries for field visits to be decided finally by the evaluation core team.

(iii) Four field visits (June-October 2001) to the 12 selected SPFS countries. To the extent possible, the team should visit the Regional or Sub-regional Offices concerned. Each visit, covering about 3 countries, may last for about three weeks, including report writing.

(iv) Preparation of a synthesis evaluation report (November-December 2001 for about 2 weeks by the core team members in Rome). The team will be debriefed by the Director-General and other FAO staff.

(v) Preparation of the Director-General's comments on the Evaluation Report's findings and recommendations for submission to the Governing Bodies through the Programme Committee (May 2002).

E. Composition of the Evaluation Team

6. The team will comprise some 9-10 external consultants, including a team leader recruited by FAO. The team leader will take overall responsibility for the evaluation report, ensuring that it reflects the views of team members. Each mission will prepare a report on findings of its country visits.

7. The team members should have international recognition and no involvement in the planning and implementation of the Programme. They should also combine as a team the range of expertise necessary to address the substantive aspects and issues to be covered.

8. The evaluation team will be backstopped by the Chief, Evaluation Service throughout the entire exercise and one staff member of the Evaluation Service will accompany each mission visiting the regions, participating in report writing as directed by the mission leader.

ANNEX 2: SUMMARY CURRICULUM VITAE OF SPFS EVALUATON TEAM MEMBERS

Dunstan S.C. SPENCER (Sierra Leone) - Team Leader for the SPFS evaluation. He has a Ph.D. in Agricultural Economics from the University of Illinois, Champaign-Urbana, with graduate level training in animal science. He did his B.Sc. in Agriculture (with Hons) at Wye College, London University. He taught agricultural and development economics to undergraduate students in Sierra Leone, and graduate students in the USA. He has led major farming systems research projects in international research organizations of the CGIAR system including the International Institute of Tropical Agriculture (IITA), where he was the Director of the Resource and Crop Management Division, the International Crops Research Institute for the Semi-arid Tropics (ICRISAT), and the West Africa Rice Development Association (WARDA). He is currently an independent Consultant, based in Freetown, Sierra Leone, in which capacity he has been a team member and leader in a number of agricultural project identification, design, appraisal supervision and evaluation missions on behalf of international organizations such as AfDB, FAO, IFAD, the World Bank and UNDP, as well as a number of Foundations such as Rockefeller and Ford. He is an expert in Women in Development, environmental assessment, impact evaluations, and poverty alleviation issues.

Pierre SPITZ (France), Mission Leader for the SPFS evaluation in Asia and West Africa. Graduated from the National Institute of Agriculture (INA, Paris) in Agricultural Economics. As a Research Fellow of the Indian Statistical Institute, he conducted research on livelihood systems in a group of villages in Bihar. He was later an Adviser to the Ministry of Planning, Economy and Finance in Tunisia, Harkness Fellow of the Commonwealth Fund, New York, Research Fellow at M.I.T. (Centre for International Studies) and Harvard University (Centre for International Affairs). He was co-founder of the Research Group on International Economic Relations at the National Institute of Agricultural Research (INRA), and later he co-directed the Food Systems and Society/Food Security in the Modern World Programme at the UN Research Institute for Social Development (1977-85). He was, thereafter, Research Coordinator, the Independent Commission on International Humanitarian Issues, and Director of Evaluation, the International Fund for Agricultural Development (IFAD). He was Chairman of the Monitoring and Evaluation Panel of the UN/ACC Sub-Committee on Rural Development. He has worked with a dozen multilateral and bilateral agencies, in addition to FAO, IFAD and WFP. He is Director of Research (UMR Public Economics) at INRA, based at INA, Paris. His writings concern food policies, rural development, the Right to Food, system analysis and evaluation methodology.

Frank Malcolm ANDERSON (Australia) was awarded PhD in Agricultural Economics by Oregon State University (1972). He has been: Team Leader of Highlands Research Program of the International Livestock Centre for Africa; Foundation Professor, Agricultural and Horticultural Systems Management, Massey University, New Zealand; Consultant Agricultural Economist to the Turkish Agricultural Research Project; and Senior Consultant to the Eastern Anatolia Watershed Rehabilitation Project, Turkey. He has also undertaken various consultancies in Asia and Africa for FAO, ADB, IBRD and the Australian Centre for International Agricultural Research. Main professional themes are agricultural research and natural resource management, and the integration of agricultural research and extension.

Manuel CONTIJOCH ESCONTRIA (Mexico) has a degree in civil engineering from the National University of Mexico and studied development planning at University College, London. He has 28 years of experience with the Mexican Government, participating in the integration of the National Water Plan; Coordinator of the Program for the Rural Development of the Tropical Areas of Mexico; responsible for the Irrigation and Drainage Research Centre of the Mexican Institute for Water Technology; Regional Manager of the National Water Commission NWC in the South East Region of Mexico; Sub-director General of the NWC; General Director of FIRCO, the agency responsible for the Agricultural Alliance and the Rural Employment Programme for Mexico. He has undertaken international assignments in water and rural development with the World Bank, USAID, SUEZ, PA Consulting Group and FAO.

Antanas MAZILIAUSKAS (Lithuania) has a doctor's degree in hydraulic engineering. He is currently Dean of Water and Land Management Faculty at the Lithuanian University of Agriculture and President of the Lithuanian National Committee on Irrigation and Drainage. He has served as Director of Agriculture and Food Department, Ministry of Agriculture of the Republic of Lithuania and has international experience in Algeria, Ivory Cost, Kazakhstan, Nigeria and Uganda, including work in project development and evaluation with FAO, World Bank, EU, British Council and SIDA. His main professional themes are land and water resources management, irrigation and drainage and policy development in the agriculture and food sectors.

David W. NORMAN (USA) is Professor of Agricultural Economics at Kansas State University where he teaches courses in poverty and international agricultural development. He did his B.Sc. in Agriculture at Wye College, University of London, UK and M.S. and Ph.D. in Agricultural Economics at the Ontario Agricultural College, University of Toronto, Canada and Oregon State University, USA. He has spent twenty years working in national agricultural research systems in Africa: including Head of the Ford Foundation sponsored Rural Economy Research Unit, Nigeria and Head of the Department of Agricultural Economics and Rural Sociology at Ahmadu Bello University, Nigeria; and Chief of Party of the Mid-America International Agricultural Consortium USAID sponsored Agricultural Technology Improvement Project under the Ministry of Agriculture - Botswana. He currently undertakes short-term assignments (mostly as team leader) and has worked in over 60 countries of Africa, Asia and the South Pacific for many different agencies including UNDP, FAO, CGIAR system, DFID, DGIS, and the Rockefeller and Ford Foundations. He played a prominent role in the development and popularisation of the farming systems research approach, including the initiation of the International Farming Systems Association of which he is a past president. He is the author/part author of about 185 publications.

Maija SALA (Finland) has a background in business administration and economics. She worked for over twenty years as an adult education specialist in cooperative movements in Africa with ten years attachment to several large national level programmes. During the past eleven years she has undertaken a wide range of short-term assignments in various aspects of rural development, as a mission member and team leader, in nearly twenty developing countries funded by the World Bank, Danida, EU, Finnida, ILO, SIDA, UNDP, IFAD and FAO.

Vijay Shankar VYAS (India) Professor Emeritus IDS, Jaipur, has a Ph.D. in Economics. He has taught in the University of Bombay, Sardar Patel University, and Indian Institute of Management, Ahmedabad, and has occupied senior positions in academic and policy making bodies in his country and abroad. He was a Member of Agricultural Prices Commission of India, Director of Indian Institute of Management, Ahmedabad, and Senior Advisor in Agriculture and Rural Development in the World Bank, Washington D.C. He has consulted with CGIAR, FAO, ILO, UNDP and other multilateral and bilateral agencies. He has been on the Board of Trustees of IFPRI and CIAT and a member of the Board of Governors of IDS, Sussex. Currently, he is a member of the Central Board of the Reserve Bank of India. He has written extensively on agricultural policies and rural development and has been honoured in his country and abroad for his contributions. He is the current President of the Asian Association of Agricultural Economists.

Mahgoub G. ZAROUG (Sudan) is a collaborating lecturer and supervisor of post-graduate students at the universities of Khartoum, and Omdurman Ahlia. He has his BSc in Agriculture, University of Khartoum and M.Sc. - Range Management and Ph.D. - Ecology from the University of California, Davis. He rose in his Government service to be Director-General of Range and Pasture. He has worked as Range Ecologist and Senior Technical Advisor for FAO in Range and Forestry in Oman. In addition to his university work he now undertakes consultancy assignments, including with FAO, UNDP, UNHCR, CGIAR, OXFAM, AOAD, BADEA, UNICEF, IDA and IFAD.

ANNEX 3: SUMMARY OF BASIC DATA FOR SPFS COUNTRY PROGRAMMES

Note: Countries in bold were visited by the Evaluation Team.

Source: FAO-TCOS

Note by Evaluation Team: The data in this table were obtained by TCOS from Information Sheets submitted by country projects. They should be treated with caution since there was no consistency in the definition of "sites" and "number of farmers" as far as the different countries are concerned.

SENIOR MANAGEMENT RESPONSE

1. We commend the Independent External Evaluation Team for preparing a succinct, helpfully critical and constructive review of the Special Programme. From the outset, when the SPFS was launched in 1994, we have seen the need to adapt the Programme to respond both to the lessons emerging from experience gained in its implementation as well as to changes in the broader development environment. We welcome this report as a most important contribution to this learning process.

2. The report provides a wealth of useful observations on the Special Programme which will enable the Programme Committee to have a well-informed debate on its achievements and its future. We expect that the representatives of those countries which are hosting SPFS activities, financing them or contributing to their implementation through providing South-South Cooperation will have much to say in this debate, and we will be particularly interested in their observations.

3. The Evaluation Team's work was deliberately focussed on countries which had entered the SPFS early in its existence because it would be here that there would be the longest track record for review. Many of the concerns expressed in the report about the design and implementation of these early SPFS projects have, we believe, been largely addressed both in later projects which were not covered in this Evaluation as well as in SPFS Phase I Extensions. We accept, however, that there is room in the SPFS for further improvement and evolution. Some of this is a matter of closing the gap - which exists in any large-scale development programme and from which the SPFS is not immune - between intent, as set out in guidelines, and what actually happens on the ground. But there is also a need for more fundamental adjustments.

4. In responding, we do not wish to burden the Programme Committee with detailed comments on the Team's analysis but to concentrate mainly on the report's recommendations as to how the SPFS might be further improved in future. While we need to learn from the past and take note of the Team's observations, especially when these are critical of certain aspects of the Programme, it is on the future that we need to focus our efforts. Should members of the Committee, however, wish to seek our views on specific comments and statements in the analytical sections of the report, we will be pleased to share them.

5. However, there are three broad themes on which our perception of the SPFS does not converge entirely with that of the Independent Review Team.

• The first concerns the processes of innovation. A major purpose of the Programme is to foster creativity amongst farmers and to promote the testing and uptake of new and affordable ways of farming better, applying participatory methods (para 164 and 165 of the report). We feel that it is necessary, however, to make the point that, even in demand-driven projects and programmes founded on principles of people's participation such as the SPFS, the Organization, based on its technical competence, must retain the right to decide upon the purposes for which its resources - or those entrusted to it - are utilized. It must also be prepared to exclude the use of funds for activities which it perceives are unlikely to be viable. FAO management also believes that it needs to encourage the testing and uptake by farmers of innovations which could respond appropriately to their needs but which are not "demanded", if only because they are not known to participants. Indeed, one of the Organization's comparative advantages is its international character and ability to promote technology transfer between countries.

It is in this context that we feel that the results to date of the South-South Cooperation (SSC) initiative and its impact on the processes of innovation are encouraging, especially if compared with other more conventional modes of technical assistance. We believe that this justifies the continued application of the main principles and elements of the current guidelines, adapting them, as required, to country-specific situations. What distinguishes the South-South Cooperation model being applied by FAO in the SPFS from more traditional forms of technical assistance is that it places strongly committed technicians with good practical skills as change agents out in rural communities where they can interact directly with front-line extension workers, local leaders and farmers. In this way, they can break the conservatism which often inhibits innovation, bring on a daily basis new and very practical ideas directly into the farming environment and encourage groups of farmers, fishermen and animal producers to adapt and test approaches to livelihood improvement that have been found to work well.

To increase the multiplier effect of SSC and to avoid creating long-term dependence, all SSC technicians are expected to share their experiences after practical work in rural communities with national staff in training of trainers. This is a part of an exit strategy which normally limits engagement periods to 3 years. The strategy, adapted to local conditions in each participating country, includes the training by SSC experts and technicians of trainers at the national level who in turn train other trainers at regional and community level, thus creating a snowball effect.

The costs per SSC expert and technician (about US$12 000 and US$7 200 per year respectively, shared between the source, host countries and FAO initially but later paid by bilateral and multilateral donors) are very low relative to conventional technical assistance (typically in the range of US$120 000 to US$200 000 per expert per year depending on funding source). This makes it possible to field in stages a critical mass of technicians under SSC arrangements, thereby stimulating a process of locally adapted change from below in many rural communities.

This formula is still new and we shall clearly need to make adjustments on the basis of feedback from the countries involved, but we see it as an exciting and affordable way through which developing countries can transfer successful experiences amongst each other, contributing in a very practical manner to food security and agricultural development. We, therefore, intend to continue playing a catalytic role in helping interested countries engage in SSC agreements and in assisting them in mobilising the necessary financial resources.

• Secondly, the Team has advanced a number of proposals on criteria that might be applied in prioritizing countries for SPFS participation. Their perception that there is, indeed, a need for selecting countries implies a recognition that there are many countries which wish to join the SPFS: this is especially evident from the requests from countries outside the LIFDC list which are committed to using their own national resources to finance SPFS implementation ^*. The Programme Committee does not need to be reminded that the Organization's field programmes, including the SPFS, are driven by the requests of member countries. To the extent that legitimate demands for assistance exceed the Organization's financial and institutional response capacity, it obliges FAO to explore means of increasing its delivery. It is for this reason that the Organization is actively seeking resources in consultation with LIFDCs from bilateral and multilateral sources.
  
  
• Thirdly, we feel that the Evaluation Team may have underestimated the impact of the SPFS on national policies for food and agriculture. This is, of course, difficult to judge because policies are determined through an amalgam of different influences including, of course, the broader political orientation of those in power as well as the advice given to governments by external sources of funds. The direction of policies also tends to shift only slowly except when there are fundamental changes in power structures. In this context, however, we believe that there are signs that many developing countries, especially those engaged in the formulation of Poverty Reduction Strategy Papers (PRSP), are increasingly prepared to adopt policies which will lead to improved food security and to rapid growth in agriculture. This will be clearer when we meet for WFS:fyl. It is, however, already visible at the regional and sub-regional levels, whether one considers NEPAD or the understandings reached between FAO and UEMOA and other sub-regional bodies on regional food security programmes.

It would be presumptuous to imply that the SPFS alone has prompted a shift in policy orientation. We believe, however, that through its high visibility, its demonstration that it lies within the capacity of countries to bring about rapid improvements in the output of small farmers, and its links into the World Food Summit process, the Programme can claim some responsibility. At national level, the decisions of a number of LIFDCs to extend the programme nationwide (e.g. China, Pakistan, etc.) as well as of nine other countries to commit their own funds on a significant scale to implement the SPFS country-wide, would seem to indicate that it is beginning to impact on policies.

6. We agree fully with the Team's assessment of the strengths of the SPFS and with the recommendation that it is on these that future SPFS activities should be built. We concur with their views on the importance of flexibility in the Programme and on the need to balance microlevel measures to improve production and livelihoods with addressing meso and macro issues which could impinge on both the production and the distribution of benefits. We accept the case for a longer time horizon for pilot activities under Phase I and for larger and more sites, representative of all major agro-ecological regions of a country. Indeed, wherever resources allow, this is the direction in which recent SPFS initiatives, including those financed by developing countries from their own resources, are moving.

7. We also agree on the need to update and improve guidelines and to ensure that they are widely accessible and used. In line with the Team's proposal, an SPFS Guideline Technical Committee will be established. Work is already in progress to develop cost-effective methodologies for impact monitoring, covering both production gains and their impact on household income and food security. FAO management also accepts that subsidies, whether on inputs or on interest rates, should be avoided except where these are consistent with national policies or might be required to indemnify participants against pioneering risks (which would otherwise be met by the state through their funding of additional on-station research). The widespread use of subsidies noted in the report is not in line with the underlying concept of the Programme that it should promote replicable innovations. One of the practical problems faced by many farmers interested in taking up new practices, however, has been the collapse of credit, input supply and marketing systems in a large number of developing countries. In such situations, it has been necessary to intervene in a pragmatic way to bridge the gap between what should be done and what can be done to enable innovators to have access to the necessary inputs and equipment.

8. In considering Options for the Future of the SPFS, our observations are as follows:

• Greater Priority to Household Food Security. The SPFS is a household and community focussed programme. It is built on the understanding that improvements in local food security, especially where these stem from production gains by large numbers of small farmers, will also contribute to a higher level of food security at the national level. We take the view that in all countries in which the SPFS is operational, whether or not they are close to self-sufficiency in food, the primary focus should be on communities in agricultural areas where the greatest number of food insecure households live. But we attach one proviso: this is that the Programme should only go where it can offer practical, sustainable and viable solutions to the problems facing families, regardless of whether the areas have high or low agricultural potential. FAO management, however, notes that in many countries, contrary to conventional perceptions, the largest numbers of households suffering from malnutrition and chronic food insecurity live in areas of high agricultural potential, and hence a focus on high potential areas is not usually incompatible with the improvement of both household and national food security.

For the future, the aim is to extend SPFS Phase I (pilot) activities to respond to demands from food-insecure communities in all agro-ecological zones of a country: the range of activities supported by the Programme would be progressively widened as institutional capacities grow. We also envisage that increasing attention will be given within SPFS operations to empowering communities to address the underlying factors which determine the distribution of food between households within the community. We see the need for a special focus on issues related to women and households in which, as a result of HIV-AIDS and other diseases, there have been adverse shifts in dependency ratios with disastrous effects on agriculture. Some experience is being gained (with funding from telefood sources) in targeting increases in food production where these are most needed and in linking these to nutrition education, particularly through promoting school and health centre garden programmes; these examples can be more widely replicated in future under the SPFS.

• SPFS Design Considerations. We welcome the Team's recommendations set out in Section 7.3.3 and intend to make sure that they are all taken up in the design of future SPFS initiatives, including planning for the extension of existing Phase I activities. We share with the Team the concept of a people-centred approach which is designed to empower food-insecure communities to analyse the constraints and opportunities which they face and to arrive at their own choices on strategies and actions for reducing hunger and for bringing about sustainable improvements in their livelihood systems.

FAO management also strongly subscribes to the need for strengthened partnerships, building these up from the outset of project design. Furthermore we fully accept the recommendation that more explicit attention be given in project design to means of addressing seasonality issues, ecological sustainability, gender equality and linkages. Mainstreaming these approaches will require increased attention to capacity building at all levels, but with special emphasis on farm leaders and front-line extension staff, building particularly on the Organization's experience with Farmers' Field Schools which has been commended by the Team.

In addition to revising SPFS guidelines to respond to these recommendations as well as those on design processes contained in Section 7.3.4 (paras. 230 and 231), the Organization will provide training for staff and government officials engaged in SPFS design.

• Complementary Strategies. FAO management concurs with the need to strengthen the capacity of FIVIMS to identify foci of food insecurity and to link this to the targeting of SPFS initiatives. We also share the Team's view of the need both to improve monitoring systems, keeping these simple and cost-effective, and to support training programmes in project formulation: this will be greatly facilitated by the Project Formulation Tool Kit, currently in its test phase.

9. The development context and environment in which the SPFS is now operating has changed considerably since it was launched almost eight years ago. The proclamation of the Millennium Development Goals, the expansion of debt relief programmes, the launching of the Comprehensive Development Framework and of Poverty Reduction Strategy Papers and the thinking on new modes of development financing emerging from the International Conference on Financing for Development, all pose challenges and offer new opportunities for the SPFS which the Organization will address.