FAO.org

C9 - 2.1 Overview of the planning, monitoring, evaluation and learning cycle within a climate-smart agriculture programme 

Monitoring and evaluation start with the design process and the identification of objectives. Before describing monitoring and evaluation indicators and tools, some important design processes need to be outlined. A shared process for setting objectives and determining indicators among intervention stakeholders is key to obtaining feedback, learning and re-strategizing – all of which are important for climate-smart agriculture.

This chapter largely refers to outcomes and impacts because they tend to be specific to climate change and agriculture, and because behavioural, institutional and policy changes, outcomes and impacts are pertinent to evaluation. On the other hand, the project and programme outcomes, outputs and activities to be monitored, which often focus on capacity development, organizational systems change, infrastructure and policy support, will be highly intervention-specific and will fall within more regular planning and monitoring guidance. Also, under changing climatic conditions it is important to also allow for adaptive management, and enhance capacities in this area, both for implementers and the affected communities. This demands that a balance be struck between formal structured frameworks and more flexible approaches. 

Intervention planning frameworks are used to map out links between action and results. Much has been written about different kinds of project and programme frameworks (e.g. logical framework and results frameworks) as key tools for planning and establishing monitoring and evaluation indicators. These frameworks do not need to be treated in detail here (see, for example, IFAD, 2002). What needs to be noted is that logical frameworks are very intervention-specific and cannot be prescriptive. There is no single model for a logical framework that can work for the very large range of climate-smart agriculture interventions, many of which will be nested within broader programmes and projects (see FAO, 2012a for additional guidance). 

A different organizing framework is the Driving forces - Pressure - State - Impact - Response (DPSIR) framework, which has been extensively applied in the context of environmental management. This framework follows a causal chain from the causes of an environmental issue (the driving forces), to its effect (impact) and required responses. Monitoring indicators are then identified in relation to the different elements of the causal chain. For an example, consult the FAO Land Degradation Assessment (LADA). The Theory of Change (TOC) approach similarly helps to lay out the broader cause and effect links in projects and programmes with several components. 

Project and programme frameworks are useful in delineating the expected outputs and outcomes that result from stakeholder participation. Indicators are more easily developed and organized around such a framework. The CCAFS Climate-smart Agriculture Programing and Indicator Tool (Box. C9.1) provides a guidance tool to help planners identify interventions and related indicators that can maximize the potential benefits related to the different climate-smart agriculture objectives (i.e. productivity and income, climate change adaptation and mitigation). 

The importance of tracking intervention processes 

It is important to distinguish between objectives that are oriented towards direct impacts and results ‘on the ground’ and those that are oriented toward processes. Even though an understanding of underlying processes is critical for climate-smart agriculture interventions, these processes are often neglected, as they are less easy to measure. Implementing climate-smart agriculture cannot be done in a strictly linear way from interventions to results. With rapid changes in the environment and the need to continuously address capacities for adaptation at both the institutional and household level, it is crucial to measure changes in processes and participation (Villanueva, 2010). For example, understanding why behavioural changes are taking place or not (Villanueva, 2010) is a process that is worthy of monitoring and evaluation. In this area, it is possible to draw on work from the broader agricultural development field (FAO, 2012b) and other disciplines.

Outcome mapping, developed by Canada’s International Development Research Center in a research context, has been adopted by a range of programmes for fostering institutional change. It is particularly helpful in delineating the expected outcomes among the different project partners and stakeholders. Outcome mapping is suited for monitoring institutional changes, capturing changes in capacity and the resulting delivery of services.

Ex-Ante economic and climate change analysis 

The basic design for large-scale programmes requires a sound financial and economic analysis to provide an economic justification for proposed interventions and gain a better understanding of their long-term sustainability. A financial and economic analysis, which examines the returns to costs at the farm and project level, provides input for decision-making regarding ex-ante project investment. Financial and economic and analyses can be used to complement specific climate change analytical tools, such as the Ex-Ante Carbon-balance Tool (EX-ACT). For more information on the use of EX-ACT in financing the adoption of climate-smart agriculture, see in chapter C3.5.3). It assesses the feasibility of reaching objectives and indicator targets for a range of possible interventions and their required resources and costs. This analysis helps ensure expectations remain realistic. See Module C8 on climate impact assessments and appraisals of climate-smart agriculture options.

Baselines provide important data on the starting conditions against which the impacts of policy and programme intervention can be compared. This also includes capturing qualitative and quantitative baselines for system-wide capacity enhancement for climate-smart agriculture (See module C1 on system-wide capacity development). If climate impact assessments and climate-smart agriculture options assessments are carried out for a given point in time or for the projected future conditions, they constitute a baseline or baseline projection that is relevant for the evaluation of impacts of a particular policy, project or programme. These assessments can also be used for monitoring outputs and progress towards outcomes and impacts along the way. Examples of variables used to measure baselines are shown in Figure C9.3.

Indicators are identified in the design stage, particularly when preparing a results-based framework. They are then further refined when there is greater clarity on the specific interventions and their expected scope of action within a programme or a project. Specific indicators for system-wide capacity development are discussed in module C1.

This section provides an overview of the range of monitoring and evaluation methods applied to climate-smart agriculture programmes and projects, which are generally not different from the standard range of methods and tools. For more information in this area, consult the FAO Investment Learning Platform on Monitoring and Evaluation. These methods refer to the implementation phase of the monitoring and evaluation process, but also need to be considered during the preparation phase.

Methods for measuring results of outcomes and impacts (mainly for evaluation) and progress against expected targets (mainly for monitoring) are substantial topics that are best examined in relation to literature on specific indicators and the broader literature on monitoring and evaluation. Many indicators related to implementing climate-smart agriculture deal with well-established measures of change (e.g. technology adoption, land-use change, household livelihoods and institutional change) that have been dealt with extensively in development literature and through a range of methods and good practices. 

Methods for programme and project preparatory assessment for Monitoring and Evaluation 

Technical aspects of baselines are addressed in module C8 on climate impact assessments and appraisals of climate-smart agriculture options and module C1 on system-wide capacity development. Here it is important to note that methods and process for baseline assessments ideally need to consider the type of information that will be required for doing robust end-of-project evaluations. 

Project and programme baseline assessments are done through surveys in intervention and control areas. The surveys, which measure food security, livelihoods and agriculture systems, incomes, basic household assets and services, and environmental parameters, have to be designed with appropriate sampling to ensure that useful end of project analysis can be done.

Participatory poverty and vulnerability assessments help determine who are the most vulnerable groups in the community as defined by community members’ own criteria. This serves to identify key intervention target groups, and assesses how these groups have may have changed as a result of climate-smart agriculture interventions.

Participatory stakeholder, capacity, institutional and legal assessments consider the starting situation and needs for changes in capacities, human resources, organizational systems, institutional coordination, and laws and policies (See module C1 on system-wide capacity development). 

Methods for project monitoring 

Monitoring needs to have in-built and integrated systems for tracking financial transactions, expected outputs, activity targets and achievements. Monitoring must also incorporate feedback and learning into programme and project management. To track and manage data, many of these monitoring systems make use of computerized management information systems. A critical element of successful monitoring and evaluation is the internalization of its importance in planning and decision-making by management and other stakeholders. Monitoring and evaluation tasks are too often merely seen as reporting for governments or donors. Basic methods and tools for monitoring are:

• Regular project monitoring involves the gathering of activity and output progress data, financial management information, and signalling emerging issues or good practices.
• Management information systems are web-based support systems that are increasingly being managed through remote devices, linked to financial management (FMS) and geographic information (GIS) systems. FMS and GIS take on a greater importance for climate-smart agriculture as they may be linked to accounting for payments for environmental services, greenhouse gas accounting and accurate measurements of spatial changes in natural resources.
• Agriculture and natural resource management monitoring is carried out at frequencies and scales significant enough to provide meaningful information. The measurements can be done by a range of methods from structured crop to participatory transect walks.
• Process monitoring is often done to support regular monitoring, assess project process and institutional changes and relationships, and rapidly identify management responses. If sensitively and rigorously done, often by an external third party with independent insights, it can provide a powerful tool for management to quickly learn about capacity issues, the changes that may be required, and whether participation mechanisms are working, and respond appropriately. This is particularly relevant for climate-smart agriculture interventions, which follow new development approaches, need to be flexible, and are looking for ways to scale up their activities.
• Participatory monitoring and evaluation methods include a wide range of methods for enhancing the engagement of the communities involved in programmes and projects, often as beneficiaries and partners, in gathering information, and for increasing their ownership and adoption of the project goals.

Methods for evaluation

Evaluation, at key milestones in the programme, but especially at its conclusion or after its completion, involves gathering specific data relating to project indicators, and focuses on outcomes and impacts. It is important to revisit these indicators using the same methods for collecting data and information that were employed in the baselines assessments. However, evaluation is not only about the impact evaluation surveys. It also assesses the ongoing relevance of the design, objectives and implementation mechanisms; reviews the efficiency, performance of different actors; and highlights lessons learned. Much of this evaluation work consists of reviewing existing reports and data, interviews, small studies and consultations. 

Impact evaluation methodology

Impact evaluation assesses the impact of an intervention using counterfactual analysis. The estimated impact of the intervention is calculated as the difference in mean outcomes between a ‘treatment group’ (i.e. those receiving the intervention) and a ‘control group’ (i.e. those who don’t). This is done through randomization (experimental design), pipeline and matching. To date, impact evaluation has not been used extensively in the climate change context, and it faces some challenges in this area, due to the scales involved, the externalities and process orientation. Nevertheless, impact evaluation is being increasingly advocated to understand the attribution of impacts. It has been used in mitigation programmes, and is often applied in agricultural and rural development projects, and natural resource management projects (Prowse and Snilstveit, 2009). Given the complexities involved when working under changing climatic conditions and the need to constantly adapt the project and practices to deliver benefits to smallholder agricultural producers, finding a true counterfactual becomes nearly impossible and may not add much real value.

As mentioned earlier, a particular methodological issue regarding impact evaluation is the importance of attribution. The challenge of attribution arises when attempting to ascribe observed changes and results specifically to a project, in cases where these changes could also be due to other external factors and interventions. This is a major concern for climate change programmes since they are potentially affected by long-term and large-scale climate and economic processes. In the context of projects, the issue is addressed through the design of rigorous project baselines and impact evaluation surveys, which take into account external effects. These activities include ‘control’ areas and households in the survey samples, against which changes in land use and the livelihoods of the project's beneficiaries’ can be compared. As mentioned above, impact evaluations can be challenging for climate change interventions. There are also other factors to consider, which add to the complexity. For example a given project site is situated within a larger landscape, and this landscape, which affects ecosystem dynamics in the project site, may not be taken into consideration in the impact study. Also, with large-scale projects involving several thousand households it is difficult to find similar households that have not been affected by the project given local political, market or social dynamics. Impact evaluations are also more costly. However, opportunities may be found for impact evaluations, and these opportunities should be taken advantage of. 

Contribution analysis, which is the predominant approach of international development evaluations, can be valuable (Mayne, 2008). Under this approach, the causality behind an outcome from an internal or external factor, or earlier outcome, is inferred using evidence. Rather than saying “A is attributable to B” (based on experimental methods), one would say, using contribution analysis, that “it can be plausibly concluded, with some level of confidence, that B contributed to A.” However, this approach depends upon an articulated and fixed theory of change, and climate change may require constant reformulations of the theory of change. Nevertheless, making inferences based on evidence on an ongoing basis and in real time might help ensure that the project is meeting climate-smart agriculture goals.

Developmental and utilization-oriented evaluation

The standard approach for evaluating projects and programmes, involves conducting a formative, or mid-term, evaluation to assess whether the intervention model is working and gauge the results achieved to date. A summative evaluation is carried out at the end of the project to determine whether the objectives were achieved and to draw lessons. These evaluations have proven to be useful. The OECD/DAC evaluative criteria, applied in or adapted for many international development evaluations, have succeeded in drawing out key findings of projects and programmes. However, these interventions, as complex as some may be, have been regarded as linear, straightforward and fixed in nature. Developmental evaluation, on the other hand, involves using evaluative thinking and evidence on an ongoing basis during the intervention to assess the results of innovations and learn by doing, and guide decision-making by stakeholders in an environment of complexity. The 'model' never stops developing. This makes monitoring and evaluation more useful, since it not only gauges project effectiveness, but also identifies in real time what is and is not working, or changing, for communities, and what adaptations are needed. 

Developmental evaluation also constitutes a form of utilization-oriented evaluation. A standard evaluation’s consultation with stakeholders at the design stage and its lessons for them at the end can be a useful exercise. However, a utilization-oriented evaluation that goes well beyond accountability, “requires moving from the general and abstract, that is, possible audiences and potential uses, to the real and specific: actual primary intended users and their explicit commitments to concrete, specific uses” (Patton, 2011). Financial and economic analyses, which use mainly agricultural, environmental and socio-economic data, as well as detailed market, labour and trade information, consider the economic and financial returns at the household, farm and system levels. This is most often used at the design stage for feasibility assessment, and for an evaluation of the final economic returns at end of the project. However, financial and economic analyses can be used also for ongoing assessment of the viability of economic activities related to climate-smart agriculture. 

Monitoring and evaluation system-wide capacity development for climate-smart agriculture

Module C.1 on system-wide capacity development provides further insights into the importance of tracking system-wide capacities for individuals, organizations and institutions and throughout the enabling policy environment to facilitate the transition towards climate-smart agriculture. It also describes the key mechanisms and tools for enhancing monitoring capacities to foster country-ownership and commitment. It is important to complement the socio-economic and institutional monitoring and evaluation elements described above, with activities that collect information and measurements on the programme's expected aims and results with regard to applied knowledge, effective organizations, institutions, networks and an enabling policy environment that is committed to promoting the uptake of climate-smart agriculture practices. Module C.1 outlines the process to establish a qualitative and quantitative baseline for a proposed capacity assessment for climate-smart agriculture. This process incorporates FAO normative work that has been published in the Organization's Capacity Development Learning Modules, particularly module C1 on Capacity Development Programming: Processes and Tools, which provides a guide to assess existing capacities, set priorities and objectives for interventions, track capacity development results and ensure that these are sustained and scaled up by national and/or regional institutions. Done jointly with stakeholders, this process maximizes country-ownership and commitment. The methods and tools cover:

• joint analysis of the context and assessing capacity needs for individuals, organizations and institutions and throughout the enabling policy environment using problem trees, stakeholder mapping and analysis, and review of drivers of change;
• measuring capacity development ('what' and 'how') by providing guidance on formulating capacity-development outputs, outcomes and indicators, outcome mapping , developing monitoring and evaluations plans for capacity change, and knowledge, attitudes and practices (KAP) surveys; and
• building sustainability using checklists for sustainability, concerted capacity-building components and exit strategies within projects and programmes.

Above all, it is important to involve the stakeholders throughout the process to foster learning and maximize country-ownership and commitment to achieve the identified results. Particular attention needs to be given to the 'jump' from output to outcomes to ensure that skills and knowledge are being applied, organizations are performing better, policies are being implemented. 

Efforts to address the impacts of climate change unite various sectors and stakeholders. Climate-smart agriculture requires reinforcing and extending the dialogue and engagement of key players. In the context of climate change, a typical agricultural development project can be relevant for a wide range of issues, including public health and migration. Climate-smart agriculture will not be successful unless all stakeholders are engaged at all levels and across all relevant sectors (UNFCCC, 2010; Hedger et al., 2008; Villanueva, 2010; GIZ, 2011a). The stakeholders to be involved vary from local communities to international organizations (Hedger et al., 2008). Ideally a landscape or territorial approach,  which encompass key ecosystems and communities engaged and affected by planned interventions, should be followed. In general, stakeholders must be representative of the target population. Men and women from different socio-economic groups should be involved, and different actors along the value chain should be consulted. Integrated frameworks for climate-smart agriculture across all levels will allow for clear and effective feedback mechanisms.

Especially with regards to mitigation, national-level monitoring and evaluation may need to be linked to UNFCCC requirements for indicators, such as greenhouse gas inventories and measuring, reporting and verification requirements for reduced emissions from deforestation and forest degradation (REDD+), At the project level, monitoring and evaluation should ideally be based on a national annual planning, monitoring, evaluation and budgeting system that provides orientation and harmonizes different projects under national programmes and policies for poverty alleviation, natural resources management and agricultural development.

Participatory, gender-responsive approaches in assessment, monitoring and evaluation, which recognize that local men and women are best suited to understand the local agricultural conditions, should be promoted. In these approaches, men and women contribute to assessing the impacts of climate change on their livelihoods and food security, and identify and measure their own indicators of change for monitoring and evaluation. This creates opportunities to develop a learning partnership that involves all the implementing partners and the participating communities. Participation in the assessment by the communities that are affected is critical, not only for gathering as much information as possible on the local situation, but for building ownership and commitment to the process by the community and increasing the likelihood of successful implementation of climate-smart agriculture practices.

Participatory planning, monitoring and evaluation have been recognized as important in the broader development context. They are used widely in rural community development interventions, and are particularly helpful for understanding community perspectives on the opportunities and constraints associated with climate-smart agriculture; dealing with fine-scaled variability in changes in climate and other factors; and empowering local communities to engage in community-based natural resources management (see Guijt, 1999 for methodologies, and FAO, 2008 for an example of adaptation to drought through community groundwater monitoring). The ADAPT process proposed by Villanueva (2010) also assists in organizational learning, monitor perceptions and promote organizational reflection and change. 

Community participation becomes essential for adaptive management and developmental evaluation under complexity. It contributes to building the capacities of beneficiaries to gather data and engage in evaluative thinking. Participation and ongoing feedback from communities is the sole means by which the intervention, and government institutions involved in programme implementation and research in the longer run, can receive information from the field on how climate-smart agriculture practices are performing in terms of their three main objectives and what the trade-offs appear to be. In developing countries, establishing or strengthening farmers’ organizations has been an important activity for a number of reasons. Under changing climatic conditions, and from a broader systems perspective, farmers' organizations will need to assume the additional role of drawing conclusions on the practices they have applied and communicating their findings not only to project managers to help assess the project's results and identify needs for adaptation, but to research institutions to develop the knowledge base in the emerging climate change field. Similarly, agricultural extension services, in addition to promoting the dissemination and exchange of technology to support adaptation to climate change (Simpson and Burpee 2014), can also assist in gathering the data required for monitoring and evaluation, and transferring this information to research institutions. This will require greater coordination between agricultural extension services and farmers’ organizations, as well as between the different sectoral line departments working at the landscape level. In most countries, fostering this type of coordination has been problematic. 

Tools that are particularly important for monitoring and evaluation of climate-smart agriculture are those that address vulnerability, climate change mitigation, and the monitoring of resources. Some examples of practical tools have been used by FAO and other support agencies to assist countries in this area are listed below. 

FAO has developed and assisted countries and partners in applying the FAO Resilience Index Measurement and Analysis (RIMA). Measuring resilience is challenging, since it is multidimensional and cannot be observed or quantified directly. RIMA is a quantitative approach that allows for a rigorous analysis of how households cope with shocks and stressors. The results from resilience analyses using RIMA provide the necessary evidence to improve the design, delivery, monitoring and evaluation of the assistance offered to vulnerable populations, based on what they need most. RIMA methodologies estimate resilience through a set of pillars, which are then aggregated through latent variable models. Comparisons can be made between different types of households (e.g. male-headed households versus female-headed households or urban versus rural) in a given country or area. RIMA measurements help to answer questions such as: who is most in need? where should investment focus in terms of geographical location? which dimensions of resilience need to be supported? to what extent have interventions increased or decreased the target populations’ resilience?

EX-ACT provides ex-ante estimations of the impact of agriculture and forestry development projects on greenhouse gas emissions and carbon sequestration. EX-ACT is a land-based accounting system, measuring carbon stocks and stock changes per unit of land, expressed in tonnes of carbon dioxide equivalent per hectare and year. The ex-ante carbon balance appraisal guides the project design process and decision-making on funding aspects and complements the usual ex-ante economic analysis of investment projects. EX-ACT has the potential to support project designers in selecting project activities with higher benefits both in economic and climate change mitigation terms. EX-ACT outputs can also be used in financial and economic analysis. 

Technical expertise in forestry has been provided for direct support to individual developing countries in the design and implementation of National Programmes for the UN-REDD Programme and targeted support upon request on thematic areas, such as National Forest Monitoring Systems and measuring, reporting and verification (MRV) systems (See module B3 on climate-smart forestry). This technical expertise assists countries in developing their capacities to build robust national monitoring forest systems for REDD+ that also provide broader monitoring needs for sustainable development. It supports countries on how to gather information on safeguards, design safeguard information systems, and carry out governance assessments.  Countries are also given advice on land tenure regimes and legal preparedness and identification of best practices for sustainable forest management. Participatory approaches where forest resource users are directly involved in monitoring programmes can increase efficiency of data collection and buy-in to programmes (see boxes 9.7 and 9.8). A strong MRV system can be also combined with EX-ACT tools for example for national development banks to make assessments on the feasibility of investing in agriculture and forestry development, which both reduce emissions and increase resiliency (Bockel et al., 2016). 

Support for rapid remote sensing has been developed with Collect Earth, a FAO-Google collaboration for monitoring and evaluation that uses existing open access remote sensing data. It has been used for national forestry assessments. Collect Earth has also been applied to measuring programme and project baseline situations and evaluating impacts, for example in the ex-post assessment of land-use changes comparing project and non-project control areas.