MODULE 10: MONITORING AND EVALUATION

OBJECTIVES

Provide some basic participatory and gender-oriented guidelines to improve existing on-going, mid-term and/or final monitoring and evaluation systems.

KEY CONCEPTS

Aims, Beneficiary contact monitoring, Engendered logical framework, Participatory monitoring and evaluation, Results based management, Steps and criteria.

Monitoring and evaluation processes enable staff to analyze the performance of emergency operations, and to adjust the programme, if needed, in order to obtain the desired results.

Monitoring is a surveillance system, used by those responsible for an operation, to see that everything goes as nearly as possible according to plan, and that resources are not wasted. It is a continuous feedback system, on going throughout the life of the intervention, and involves the overseeing or periodic review of each activity, at every level.

Monitoring Aims

Relief packages are ready on time
Workplans are followed as closely as possible
Adjustments can be made and corrective action taken where necessary

Those who need to know are kept informed
Resources are used efficiently and effectively
Constraints and bottlenecks can be foreseen, and timely solutions found

Data collected during monitoring provides the basis for evaluation analysis, which concerns the assessment of the effects of the intervention on or for the beneficiaries. These include the benefits at a certain term (periodic evaluation) and the full impact of the activities and the inputs when carrying out the evaluation ex-post.

Evaluation is the systematic analysis of operations by management. Beneficiaries should be involved to enable them to adjust or redefine objectives, reorganise institutional arrangements or redistribute resources up to the extent possible.

Aspects of Monitoring and Evaluation Systems

Selection of indicators
Identification of priorities
Design of data collection systems

Collection of data
Analysis of results
Information use/dissemination

Four Step System Design Process

Check the intervention objectives to see if they are specific, needs based and useful to assess the actual outcomes
Identify a set of indicators to measure the actual outcomes

Plan how information collection can be done and by whom, according to the selected indicators
Explain why the information is being reported, who will use it and what action can be undertaken or anticipated

The selection of indicators for monitoring and evaluations is important, but also difficult. Though time consuming, the more stakeholders that are involved in the selection and design of indicators, the more respected will be their sense of ownership and responsibility.

One of the most important tasks in project design is how realistic the targeting efficiency is. A too optimistic assessment can often lead to conflicts and supply shortages during the execution stage.

Results-based Management

There is an increasing emphasis on the visibility and the accountability of the intervention’s consequences, instead of looking only at process indicators related to emergency operations, like food distribution (e.g., tons of food distributed and number of beneficiaries reached). Output Indicators (e.g., km of road constructed), the effects and impact of the intervention, are the focus of Results-based Management.

The aim is to improve management effectiveness and accountability by defining realistic expected results, monitoring progress toward the achievement of expected results using Key Performance Indicators^[22], integrating lessons learned into management decisions and reporting on performance^[23].

Outcome and impact are long-term project results, which can only partly be influenced. The objectives should be realistic and if possible quantitative and qualitative indicators should be combined. In general qualitative indicators reflect the perceptions and the level of participation, and are therefore very relevant for analysing the gender impacts, while quantitative indicators are easy to be measured.

Overall Monitoring and Evaluation Criteria

Appropriateness	Design of the relief operation as formulated during the impact assessment phase
Relevance	Whether the intervention addresses the needs and priorities of the most vulnerable population
Efficiency	Use of available resources, and outputs obtained in relation to the inputs
Effectiveness	Extent to which expected results were achieved
Impact	Contribution of intervention to farming systems and beneficiaries’ livelihoods
Perspective	Short-term emergency activities should take into account longer-term and interconnected problems. (e.g. the sustainability of improved access to land and resources by female-headed household beneficiaries). SEAGA places emergency operations into a sustainable development perspective
Timeliness	Implicit in the efficiency and effectiveness criteria, but important considering that if the delivery of relief packages is significantly delayed they might not be useful. If food assistance does not reach the targeted people in due time their nutritional status will decline
Responsiveness	A cross-cutting criterion referring to the capacity of the relief mechanism to address in time the different needs of all the affected vulnerable people. This is intrinsically related to the speed by which recovery from the disaster situation takes place
Adherence	Whether the project it is in line with the policy and targets of the agency and donor

Participatory Monitoring and Evaluation

Participatory monitoring involves beneficiaries in measuring, recording, collecting, processing and communicating information to assist both operation management staff and the beneficiary group members in decision making. A participatory approach facilitates a gender perspective in the emergency intervention design among the beneficiary community.

Gender analysis addresses "whose needs" and "whose participation", ensuring the representation of women and men in emergency operations as well as in monitoring and evaluation analytical tools and processes (e.g. the logframe).

Generating and sharing information with all involved parties on the progress and impacts of relief operations is essential for coordination between donors, NGOs, governments and local beneficiaries. The appraisal of evolving needs is also necessary for the achievement of sustainable longer-term rehabilitation and recovery. For a comprehensive view of the status of the overall intervention, there is a need for an efficient monitoring and evaluation system in rapidly evolving situations. Progress and terminal reports are prepared by each agency and/or institution based on monitoring information received from the field.

Approaches used to collect data vary according to the resources available. Typically, beneficiaries are questioned at specific locations by the operating staff (e.g. UN Organisation or Government) using topic-focused interviews. Responses are investigated for the programme as a whole, using a questionnaire on a sample of beneficiaries. Follow-up action for future implementation and/or readjustments then takes place according to the response of the beneficiaries.

Two Levels of Beneficiary Contact Monitoring

Country staff (e.g., WFP, FAO) and emergency operators carry out field visits to contact beneficiaries and explore their response to the intervention. These interviews are part of a regular field supervision
Rapid Rural Appraisal participatory techniques can be used when a more detailed investigation is required, and where sufficient resources and management capacity are available

The organisation of workshops with all stakeholders, with an adequate representation of the beneficiaries, during the intervention implementation phase, is considered a useful method for generating participatory information and redefining policies and objectives of the emergency operations.

For "slow onset disasters", such as droughts, project designers should investigate who (men and women) manages and controls household resources. Where women are responsible for daily monitoring of food stock levels, Early Warning systems should consider their knowledge of food stock levels in relation to requirements. Women’s participation is encouraged to monitor and feed back information to VAM and GIEWS systems.

The Engendered Logical Framework Approach

The logical framework (or logframe) is an analytical tool used to plan, monitor and evaluate projects or programmes. Its name derives from the logical linkages between the project’s means and its ends. The logframe aims to ensure that all factors, linkages and causal relationships associated with the intervention and its context (social, economic, cultural, geographical, ecological, and political) are properly taken into consideration in the operation planning, implementation and evaluation.

Example of Indicative Logframe^[24]

Operation structure	Indicators of achievement	How indicators can be quantified or assessed	Important assumptions
Long-term objectives	Quantitative/qualitative measures	Information Sources (existing or to be obtained)	External conditions to the operation are necessary if the immediate objectives will contribute to longer-term ones
Problems?
Solutions?
Immediate objectives	Quantitative/qualitative evidence	Information Sources (existing or to be obtained)	External factors, which can restrict the progress from outputs creation to achievement of immediate objectives
Immediate effects?
Benefits? Who?		Input/output provision for information collection
Improvements or changes?

Outputs: Which outputs (kind, quantity, purpose, by whom, by when) should be produced in relation to the objectives?		Sources of information	External factors must be considered to achieve the planned outputs on time
Inputs: Materials, equipment, services, commodities, resources should be provided, by whom and at what cost?		Sources of information.	Decisions or actions outside the control of the Organisation that are necessary for the operation's inception

The use of the logframe does not pre-empt other monitoring and evaluation tools. It should be encouraged as part of documentation required.

An engendered logframe requires that the project’s planning and each component of the logframe matrix is reviewed with the tools of socio-economic analysis, incorporating the gender approach in the project management process. The preparation of an engendered logical framework matrix involves the participation of project planners, stakeholders and beneficiaries in analyzing the gender relations and addressing questions at each level of the framework.

This analysis takes place not only at the launching of the project, but throughout the course of monitoring and evaluation, keeping in mind that the logframe is both adjustable and applicable to long-term management.

Review Aspects to Present in Reports

Input	Human, physical and financial resources (both quality and quantity) used in the operation (e.g. number of workers, amount of food distributed, and contribution of counterpart personnel, operating expenses).
Output	Immediate tangible result obtained after the introduction of the inputs in the operation (e.g. road, pond, number of beneficiaries receiving food).
Effect	A positive or negative response from the beneficiaries to the intervention, against the outputs obtained. This will have different effects on the affected population and area (e.g. improved access to markets and nutritional status, increased school attendance, new situations of conflict linked to the use of one output). Effects are often difficult to anticipate and measure.
Impact	The sum of individual/community effects will define the overall impact of an intervention on the operation area and population (e.g. employment, children’s health, women’s literacy rate).

Question Tanks - Monitoring and Evaluation

Reporting System Checklist

Does the existing reporting system generate information concerning men and women separately on:
Project staff at various levels Implementing agency staff Numbers of entitled persons Category (e.g., internally displaced, refugee, returnee) Total persons receiving aid	Committees at each level Registration committees Distribution committees Participants in reconstruction work Heads of households
Was the operation designed properly to focus on the differential effect of the disaster on men and women? Is their situation improved? Did we use the available resources efficiently, measuring the outputs in relation to the inputs? Did we achieve the expected results effectively? How can we adjust assistance to the specific needs of women and men? Was the type of aid provided really tailored to the real and different needs of the affected men and women?	Could the needs of men and women have been met more efficiently following a different approach? Does it incorporate a participatory approach among project staff at different levels to assess the progress? Does the monitoring and evaluation system incorporate participatory feedback from village women? What kind of specific changes in livelihood systems of benefited male and female-headed households occurred? Have the achieved results been perceived as effective for men and women? What are their perceptions in terms of their livelihood and farming systems?

Roles of Women Checklist

Are women already represented in the Village Committee and in what proportion? Are they elected or appointed? If women had to be added to achieve gender-balanced representation, did this really happen? If not, why? Is the distribution of individual household entitlements transparent and fair? How close did the project actually come to achieving its targets (give reasons)?			What was the role of women members in registration committees in distribution (e.g. checking identity cards and household size)? In the view of villagers, did this make the distribution fairer? Does being on the committee have a positive impact on women’s self-esteem and respect from the other villagers? If so, does it last over time? What were the roles of women members in distribution committees (e.g. weighing, re-bagging, and monitoring that people actually got their entitlements)? Did this make the distribution fairer?
Analyse the gender situation and what is missing against the following criteria:
	Current status	Constraints to decision-making		Change possibilities
Distribution
Receipt
Control
Consumption

Beneficiary Contact Monitoring Checklist

How many women and men are being saved by the relief project? Who participates?
What is the impact of changed migration on the recovery pace within the village?
What is the impact on men and women’s workload?
What is the overall impact on the access to and control of resources, by gender?
What is the impact of women’s participation in committees in terms of leaders’ transparency and accountability?
What is the impact on the human capital value of men and women (as recipients of training)?
How much food actually reached the target group, compared with the total amount of food distributed?
How adequate were resources received by the needy (amount, type, quality, frequency)?
Is the programme reaching the targeted beneficiaries?
Are the activities useful to them and how (with a gender breakdown)?

What is the impact on the migration of women and men?
What is the impact on the recovery of men and women’s assets (e.g. replacement of women's livestock).
What is the impact of UN agency policies on the workload and food control of men and women?
What is the impact of participation in the project by women and men (e.g. self-esteem and status in the community)?
What is the impact of the emergency relief project (food aid in particular) on school attendance by girls and boys?
What is the impact on women’s income and livelihood options (e.g., income-generating activities and new employment opportunities)?
What share of the total number of actual food aid recipients were truly needy (entitled and non-entitled ones)?
How adequate was the timing of food and non-food inputs distribution?
In what way do beneficiaries see their lives improving or changing as a result of the relief intervention?
Are the beneficiaries encountering specific problems related to gender?

^[22] Key Performance Indicators, A Working Menu For Key Areas Of WFP Assistance, June 2000.
^[23] Results Based Management in Canadian International Development Agency, January 1999.
^[24] Source: Hambly, et al. ISNAR. Seminar on Engendering Monitoring and Evaluation. January 2001. FAO Rome.