2. Methodological aspects of impact evaluation

2.1 Efforts to develop a conceptual framework

The methodology for impact evaluation of the IPM Farmer Field School is still under development. At present, there is no agreed framework for IPM impact assessment. The complex nature of farmer decision-making processes together with the diversity of results has posed a major challenge to capturing impact. As explained further on, there is a variety of possible parameters for study, a range of agro-ecological and socio-economic settings, and a dilemma between being statistically rigorous and being comprehensive.

A contribution towards developing methodology on impact assessment of IPM is being made by the System-wide Program on IPM of the Consultative Group on International Agricultural Research. The System-wide Program recognized that conventional methods to assess the impact of IPM tend to underestimate the true costs of pesticide use on the one hand, and the various benefits that can accrue from adoption of effective IPM strategies on the other. This is because conventional methods focused narrowly on inputs, yields and productivity. In 2001, the System-wide Program made a start in developing a new conceptual framework and methodological approach to evaluating the impact of IPM programs. A workshop identified different phases in the development of an IPM program and defined indicators of impact at levels of farm household, community, and institutional policy-making, in the economic, social and ecological domains. The program is planning to undertake case studies to further develop this new framework.

In addition, the Global IPM Facility is supporting an ongoing series of workshops organized by the University of Hanover and committed to the development of guidelines on good practices for IPM impact assessment and evaluation, including the IPM Farmer Field School. A first workshop was held in March 1998 and a second in May 1999. These focused on concepts and methodologies for the evaluation of IPM programs. Combining economic and social science approaches was seen as a major challenge. Within the economist approach there is a need to make cost-benefit analysis more comprehensive e.g. by adding economic evaluations of environment and health impacts. The sociologist approach calls for an increased involvement of project stakeholders (including farmers) in the design and implementation of impact evaluation. A third workshop is planned for early 2004 with a focus on reviewing longer term impact.

Concentrating on the IPM Farmer Field School, some of the key issues appearing in discussions on impact assessment are briefly discussed below.

2.2 Defining impact

What is seen as impact of IPM depends on a project’s objective. What do IPM initiatives attempt to achieve? Is the purpose to reduce insecticide use, to enhance sustainable pest management, or to enhance adaptive crop management? Is it to increase yields, to increase profits, or to improve livelihoods?

Although initially, pest resurgence was the problem that triggered the emergence of the IPM Farmer Field School, the objective has been to enable farmers to become better managers of their fields. Crop health – not pest control – was the central theme in most training. Later still, an objective was added to help farmers become better trainers, organizers and experimenters within their own locally developed programs. The training often went further than increasing farmers’ technical capabilities and also helped enhance their educational, social and political capabilities. This raises the question of what should be considered impact: the immediate impacts such as farmer knowledge, decision capabilities, pesticide use or yield, or the indirect developmental impacts such as reduced poisoning, improved biodiversity, community agenda setting or policy change (Table 1).

The IPM Farmer Field School combines an approach to pest management and an approach to farmer education. This combination compounds the difficulties in assessing and measuring impacts. Although impacts in terms of efficiency and effectiveness of pest control are most quoted, assessing the returns to the Farmer Field School as an educational investment is equally important.

So far, most impact studies concentrated on measuring immediate impacts, most notably, the effects on pesticide use and yield. This may have partly been due to difficulties in quantifying and measuring other parameters, due to the lack of methodologies that are accepted by the broader scientific community, or due to the short time-line for many evaluation studies. A number of studies, however, have attempted to capture a broad range of developmental impacts, including changes in the social and political domain.

2.3 Who defines impact

Every project stakeholder has a role in impact evaluation, each from a different perspective. Farmers, as primary stakeholders, are perhaps in the most appropriate position to describe the impact they experienced or valued in their livelihood situation. Program staff, as secondary stakeholders, need to determine whether their training effort has the desired effect, to suggest ways of improvement. External stakeholders, such as governments and donor agencies, generally want to know whether targets are met, whether the approach is cost-effective, whether it compares favorably to alternative approaches, whether the project contributes to rural development, or whether it contributes to solving environmental and health problems or impediments to export of crops.

In this paper, three levels of evaluation are differentiated: Self-evaluation by farmers (reported in 2 case studies), self-evaluation by a project (16 case studies) and external evaluation (7 case studies). Each has its own merits and disadvantages. Anyone’s direct or indirect stake in a program can cause results (e.g. benefits, disadvantages, costs) to be overstated or understated. Self-evaluation is potentially relevant and comprehensive, especially when primary stakeholders are involved in designing the evaluation. External evaluations, aimed to provide an independent perspective, are costly, while their lack of association with a program can hamper sample selection, the choice of parameters and the interpretation of results. There is a tendency among development agencies towards a greater appreciation of self-evaluation.

Future impact evaluations of IPM Farmer Field Schools need to emphasize participatory evaluation involving farmers. After all, a project’s objective to increase farmer expertise implies that control over implementation shifts from project staff to farmers. Hence, farmers determine what is best practice (in the broad sense), based on their values and on their local conditions. Participatory evaluation furthermore stimulates learning and enhances the engagement of stakeholders in a program’s course. Local processes and structural changes are best described in open-ended qualitative studies (e.g. case studies, photo reportage, non-structured interviews, and participant observation). Hence, reasons for, and meaning of changes become apparent. The shift from external to local control over a project, however, will inevitably increase the variation and scope of project impacts, especially with regard to developmental impacts.

2.4 Measuring impact

The measurement of impact is complicated for several reasons. First, IPM involves more than one field variable and context-specific decision-making. Thus, practicing IPM is not merely a matter of adoption or non-adoption of a technology, but field-level decisions are made at various levels of advancement based on someone’s understanding. The Farmer Field School is often seen as an extension method, which it is not. Extension sets out to deliver, and its effects are measurable by the level of adoption of specific practices or technologies. Conversely, the Farmer Field School sets out to educate local people to enhance their capability for informed decision-making in response to what are always context-dependent pest problems, and thus also for adaptive management. The former is easy to evaluate, the latter not.

Second, it has proven difficult to establish good comparisons. The selection of participants or locations for Farmer Field Schools is potentially biased towards farmers who are motivated, or towards locations with favorable conditions. This bias can influence the results of a latitudinal comparison in which the contrast between trained and untrained farmers is determined, unless caution is taken to ensure that the treatment groups are comparable (e.g. by collecting background data on the comparison groups). In case comparison groups are dissimilar (e.g. the literacy rate or access to irrigation was higher for trained than for untrained farmers), statistical modeling can be attempted to correct for a particular factor (e.g. by adding literacy rate into the regression), although in practice, it is difficult to contain the sources of variation involved in a flawed comparison. Further, diffusion effects may blur the contrast between comparison groups, causing an underestimation of impact. Longitudinal comparisons (e.g. a comparison before and after training) avoid these problems of comparison groups but introduce temporal variation, such as variable yearly rainfall or fluctuating market prices. A combination of a latitudinal and a longitudinal comparison is generally considered most robust (although it could pose restrictions to sample size), and is currently being encouraged in FAO-supported IPM programs.

Third, as explained above, there are numerous possible parameters for impact assessment, and some of these are difficult to measure. Simple measurements of success are, for example, pesticide use (volume, spray frequency, chemical compounds), yield, input costs and profit, but also, the variation in yield or profit. Other, less tangible, but not less important, parameters are the quality of produce, marketability, ground water contamination, pesticide-related health symptoms, agricultural biodiversity, agricultural sustainability, policy change, gender roles, farmer-to-farmer diffusion, education and empowerment indicators.

An inevitable dilemma in impact assessment is the need to be rigorous and the need to be comprehensive. Here, ‘rigorous’ is meant in the narrow sense – in the way of obtaining representative and convincing statistical data – implying a restricted scope. Comprehensiveness is meant in terms of a broad range of technical, educational, social and political impacts. Studies have been designed to be either rigorous or comprehensive but never both. This suggests there is a need to combine the methods or results of different data sources to evaluate benefits from more than one perspective.

Long-term impact is particularly difficult to capture, because of the progression from immediate impacts to developmental impacts. Moreover, in the course of time, the contrast between comparison groups may fade due to two-way influences: (i) a diffusion of training impacts, causing bias in control farmers, and (ii) an influence of the prevailing (socio-economic, political, natural) environment, causing bias in isolated groups of trained farmers. These influences may be direct (e.g. through farmer-to-farmer contact), or indirect (e.g. through cooperatives phasing out the general supply of certain chemicals as a result of farmer lobbying, or through established, non-supportive policies inhibiting IPM). Studies aiming to look at long-term impacts should take account of these influences. Particularly, in vegetable farming, there is the additional risk that farmers change to new crops after some years.

As pointed out earlier, the interface between economic and social assessment requires special attention. The two sciences, by having different objects of study, often have different requirements with respect to experimental design. This is described in the following example. The parameter yield can show a high degree of variation (is highly context-dependent). Conversely, the way people learn will be less dependent on context (i.e. learning capability is rather universal). Hence, a study to compare knowledge or concepts between IPM farmers and control farmers can suffice with a small sample size, even in a so-called pseudo-replicated design, where individual measurements are part of the same experimental unit (for example 30 FFS farmers from one village compared with 30 control farmers from another village). However, a problem arises when knowledge is subsequently related to yield, e.g. if the two villages had different access to irrigation water. A study on yield thus requires a different experimental set-up, with a sufficient number of true replications. One possible solution is to design the study for the most demanding parameter (e.g. yield), and then accommodate other parameters (e.g. knowledge) accordingly. Another approach is through cross-verification with qualitative studies, as will be discussed later.