SECTION I: ASSESSING PROGRAMME DESIGN

We begin with an examination of the programme itself. This section focuses on those important aspects of programme design that have been found to have a significant impact on programme performance. You are asked to examine the programme design to assess the following:

• programme relevance;
• programme targeting;
• programme interventions;
• community activities;
• community mobilizers;
• programme management;
• programme monitoring and evaluation;
• programme linkages.

1) Assessing programme relevance

To assess programme relevance, you need to discover whether:

1. causes of malnutrition were investigated, and
2. whether the programme's objectives address the identified causes.

Ideally you need to talk to the persons who formulated the programme, or at least to senior programme staff, and examine the programme document and relevant files.

Broad causes of malnutrition such as ‘poor diet’ are not especially helpful: you need to know in what way the diet is poor. You also need to know why the diet is poor. Answers to these questions should have guided the nature of the interventions undertaken by the programme.

Planning the programme (problem analysis, selection and design of interventions) should have taken place in a participatory fashion, working closely with the programme's target communities. If it did not, not only are errors in analysis and design more probable, but also the target communities are unlikely to have any sense of ownership or involvement.

Ultimately all nutrition programmes must aim to improve nutritional status (reduce wasting, stunting, obesity or micronutrient deficiencies, as appropriate) and this improvement must be measurable using accepted indicators, such as anthropometric status and/or biochemical indicators of micronutrient status. Improving nutritional status must be the primary goal or objective of any comprehensive, national nutrition programme. All objectives need to be relevant, specific, measurable, realistic and time-bound⁴. It is against its objectives that a programme's success or failure is evaluated.

Many smaller or focused programmes set objectives that relate only to one or a few of the impediments to improving nutritional status. For example, a programme may aim to improve nutritional knowledge. This is an intermediate objective, addressing one impediment to improving nutritional status. Improving nutrition knowledge will only lead to better nutritional status⁵ if it is accompanied by, for example, improved child feeding practices, better access to food, and activities to reduce morbidity, as appropriate. The primary objective of improving nutritional status must therefore be accompanied by a set of intermediate objectives that address the specific causes of malnutrition in the programme's catchment area.

A nutrition programme will benefit by setting also objectives relating to community participation, improving the macro and microenvironment, and building capacity. Setting such objectives will ensure that these important issues are not forgotten during the implementation of the programme.

Here are the questions you need to answer to assess the programme's relevance:

• Was any form of problem analysis undertaken before the programme was designed? If so, is there evidence that a participatory approach was used for the problem analysis? How specific are the causes of malnutrition identified by the problem analysis?

• Does the programme have clear, realistic, relevant, measurable and time-bound objectives? Are there objectives relating to:

  -   Nutrition? If so, do they address the causes of malnutrition identified in the problem analysis?

  -   Community participation?

  -   Improving the macro and microenvironment?

  -   Capacity building?

2) Assessing programme targeting

Targeting is a mechanism to ensure that the programme reaches only those beneficiaries for whom it is intended. Good targeting can reduce costs because resources are not wasted on beneficiaries who do not need the programme interventions.

Targeting can take the form of geographical targeting (to depressed areas, or areas with particular agricultural or climatic problems), of socio-economic targeting (to low income areas of cities, or to households that fall below a specified poverty line, poor elderly people or landless households), or of vulnerable group targeting (to weaning-aged children, single mothers or elderly people for example).

If the programme is targeted to specific beneficiaries, you need to find out if the programme is failing to reach the intended beneficiaries (undercoverage), or if it is including beneficiaries who do not meet the selection criteria (leakage):

• Are programme activities targeted to specific households, communities or areas? If so, what are the criteria for inclusion in the programme?

• If the programme is targeted, is the system working? Is the programme reaching all the intended beneficiaries? Are beneficiaries included who do not meet the selection criteria?

3) Assessing programme interventions

Programme interventions must address the causes of malnutrition identified by the problem analysis. They need to meet the objectives of the programme, within the specified time-frame, and be as cost-effective as possible, bearing in mind issues of equity. Costs borne by the communities, in terms of both material resources and time, must be considered. You should consider too that the most cost-effective programme is not necessarily the most sustainable, nor is a programme that is cost-effective in the short-term, necessarily cost-effective in the longer term. Consider a clinic-based vitamin A supplementation programme, for example, in comparison with a food-based approach that attempts to improve consumption of vitamin A rich foods. The supplementation programme will be more cost-effective in the short term, but the food-based approach will not only be more sustainable, but in the long run may prove less costly⁶.

There is now a considerable body of literature on experiences of nutrition interventions worldwide. This presents an excellent resource and should be consulted at the programme formulation stage, to help select the most appropriate and cost-effective programme design. Interventions must not only address the causes of malnutrition in the programme area, they must also be based on the most up-to-date scientific knowledge (see footnote 30), they must engage all appropriate sectors and they must take into account local resources, conditions, food availability and cultural practices.

You should check that the interventions undertaken by the programme do not contradict the services that are offered to the communities. An example of this would be a programme that distributed infant formula, while the local health center attempts to promote breastfeeding. The programme should reinforce rather than undermine the activities of the local health and agricultural authorities, provided these are indeed appropriate.

No programme will meet its objectives if the quantity and intensity of resources allocated are inadequate. Here are some examples:

-   A one-off short course to improve nutrition knowledge is unlikely to be sufficient to change behaviour: it needs re-enforcement.

-   Community mobilizers or health workers with too many families to cover cannot provide the intensive support needed to achieve change.

-   Micronutrient supplementation or food supplementation needs to be provided at a sufficiently high level to achieve the intended improvement in nutritional status.

-   Agricultural extension workers need to visit farmers frequently to improve food production.

To assess the appropriateness and adequacy of programme interventions, obtain answers to the following questions:

• Were appropriate interventions selected and implemented?

• Are the interventions in line with the basic nutrition services offered in the programme area?

• Are the programme's resources sufficient to achieve its objectives?

4) Assessing community activities

This subcomponent asks you to assess the extent of community participation, whether staff have been adequately trained in the participatory approach and how it has been used and supported in the implementation of community activities.

Community participation in nutrition programmes is now accepted as a key prerequisite for success and sustainability. The aim of the participatory approach is to assist communities to become more self-reliant, with the capacity to analyse their own food and nutrition situation, identify their needs, plan activities to address these needs, secure funding and technical expertise, and implement and manage the activities. Achieving a fully participatory approach, whereby communities have a true sense of ownership of the programme, demands considerable investment of time and resources. However, once achieved, it can be maintained at little cost to the programme, provided that communities continue to have access to adequate basic services (health, nutrition, agricultural extension services, for example), technical expertise to help in their selection and design of activities, and funding support for their activities.

Participation ranges from passive participation to full community mobilization. To assess the degree to which the programme has achieved a participatory approach, you are asked firstly to assess where in this range the target communities of your programme fall. Secondly, you are asked to assess whether programme staff are adequately trained in the participatory approach.

Visit a representative range of communities, observe community meetings in progress and have discussions with community leaders and community members. Make sure you meet the most vulnerable members of the community, to assess the extent of input they have had in decision-making.

Ask the following questions:

• Does the programme work with active, representative community action groups? Community groups should meet regularly, have well-defined plans, priorities and activities. They should include representation from all sections of the community and women must be active members.
• Is the programme-building capacity in the community? What training has been provided and what skills have been acquired (e.g. management, conducting meetings, planning)? What additional training needs are there?
• Do communities have access to funding and technical advice for their identified projects/activities?
• Has the programme made use of appropriate cultural practices?

Then situate the communities within the following range:

Another method of assessing the extent of community participation is to measure participation in five key areas: needs assessment, leadership, organization, resource mobilization, and management. To do this we suggest you use the ‘spidergram’ approach described in S. Rifkin and P. Pridmore: “Partners in Planning” (see Annex 2 for full reference). Details of how to use this method are reproduced in Annex 4.

Now organize a few focus group discussions with programme staff from different levels and different regions covered by the programme, to ask the following questions:

• Do you think communities can identify their own needs, and plan and implement activities to meet their needs? If not, why not?
• What training have you received in the participatory approach?
• What additional training do you feel you need?

Based on your observations and discussions, you should begin to have a good idea of the programme staff's perceptions of community participation. This information will help you decide whether additional training is needed.

You should now be able to answer the following questions:

• What is the level of participation achieved by the programme?

• How adequate was the training in community participation received by programme staff and their understanding of the approach?

• Has the programme encouraged community organization, self-reliance and empowerment?

5) Assessing the effectiveness of community mobilizers

Community mobilizers⁷ are an essential part of any project that employs a participatory approach. Some are paid by the programme, by local authorities or by the communities themselves. Others are volunteers, generally serving their own community only. Paid mobilizers are often responsible for a number of communities. Some programmes also have supervisors who monitor the work of a number of mobilizers or volunteers. Whatever the system, community workers, whether paid or unpaid, play a crucial role in community development and programme delivery. The success or failure of the programme relies heavily on their performance. Unfortunately, their selection, training and supervision are often given scant attention. Broadly, mobilizers must have strong technical support and supervision, but must also be accountable to the community they serve.

Another aspect that needs attention is the career aspirations and expectations of community workers. We tend to assume that having identified and recruited the workers, they will be willing to undertake the same work for the same pay for an indefinite period of time. Inevitably, dissatisfaction sets in and work performance falls off.

There are ways around this: a basic career and salary structure, training opportunities, regular feedback to highlight achievements, public recognition and awards⁸, and additional responsibilities. Field visits for observation are essential to assess their role and effectiveness, as well as focus group discussions with a sample of community groups and discussions with mobilizers and with supervisors, if these exist. Here are the questions that need to be answered:

• Have mobilizers received any training in community participatory methods? In leadership and group dynamics? Are they effective in the communities?

• Is there a system of accountability? If so, to whom are mobilizers accountable and does the system work?

• Can they help communities secure funding for their selected activities? Do they know when and where to seek technical expertise⁹?

• Is their workload manageable? Are they well supervised? Do they receive any feedback on their performance?

• How are they rewarded? Are they paid? If so, by whom¹⁰? Is there any system of social recognition or motivation?

• Are there opportunities for advancement? Is there a programme of workshops or further training opportunities, when they can also interact with other mobilizers?

6) Assessing programme management

Management problems beset a surprising number of programmes. FAO's in-depth study of nine programmes (2003) highlights the following:

• poor supervision and quality control;
• poor staff motivation;
• excessive control over community leadership;
• absence of feasibility studies for income- generating activities, leading to failure and disappointment;
• failure to fund community activities;
• operational difficulties with credit schemes;
• delays in the release of funding and resources;
• no feedback to communities or programme staff;
• political interference in selection of programme staff and beneficiaries;
• overly elaborate management structures leading to excessive bureaucracy.

The challenge with good management is to establish a structure that promotes transparency, that defines roles and responsibilities clearly, that permits quick response and limits bureaucratic formalities but that at the same time is able to check misuse of programme resources. Perhaps one of the most important features of good management is the ability to maintain a committed and motivated staff. For this, frequent feedback is needed and a recognition of achievement and good performance.

Through an examination of programme reports and discussions with programme staff at all levels, answer the following questions:

• Is there an adequate management system? Is support and supervision adequate?

• Is the staff committed and motivated? Are there well-defined roles, job descriptions and lines of responsibility?

• Does the programme demonstrate financial transparency? Are programme resources well-utilized and monitored?

• Has management training been provided at all levels?

7) Assessing programme monitoring and evaluation

All programme planners agree that monitoring and evaluation is an essential component of good programme design. Yet few programmes make provision for adequate monitoring and evaluation. Monitoring and evaluation should be viewed as an integral part of the programme design. Monitoring and evaluation should therefore be:

• included in the programme design;
• designed at the start of the programme with advice from a statistician and epidemiologist;
• funded within the programme budget.

Monitoring and evaluation can be divided into three parts:

1. monitoring (a process management tool);
2. evaluation (to measure programme performance and impact);
3. participatory monitoring (community-based, for the community's use).

The literature on monitoring and evaluation is extensive. Two such publications are included in the reference list in Annex 2 (IADB 1997, ACC/SCN 2001).

a) Monitoring

Monitoring is the periodic and routine collection of information throughout the life of the programme to determine whether programme delivery is proceeding smoothly. It is first and foremost a management tool for programme staff, but also provides essential information to understand and explain the results of programme evaluation. As a management tool, it answers questions such as:

• Are programme inputs delivered on time, inputs such as equipment, supplements, funds and training exercises? If not, why not?
• Is coverage of intended programme participants good? This includes, for example, attendance at antenatal clinics or growth monitoring and promotion (GMP) sessions. If not, why not?
• Are community mobilization activities proceeding on schedule? Have community groups been established, do they meet regularly, have they developed action plans, are they implementing activities? If not, why not?

Information should also be recorded on external events that can affect programme impact. This includes events such as drought or floods, civil disorder, the state of roads and bridges. As a management tool, there are two other important aspects of programme monitoring:

• There must be a system of rapid response to identified problems;
• There must be a system of information flow.

b) Evaluation

Evaluation attempts to determine and document, as systematically and as objectively as possible, the relevance, effectiveness and impact of a programme in the light of its objectives. Appropriate indicators¹¹ must be identified, an epidemiologist must contribute to evaluation design and a statistician to data analysis. Both qualitative and quantitative information are important components of a good evaluation system.

c) Participatory monitoring

The information provided by a programme's monitoring and evaluation system is largely of little interest to communities. What they need is a system to monitor their own progress towards achieving their own specific developmental goals. For this, it is recommended that community groups be encouraged to establish a simple system of participatory monitoring that relates closely to their own identified priorities and activities. Examples of simple community monitoring tools include:

• a chart showing the growth of community children;
• a community map showing, for example, which households have built latrines, established kitchen gardens or participated in credit schemes;
• the community action plan indicating the status of activities.

To decide if the programme has an adequate monitoring and evaluation system, answer the following questions:

• Does the programme have an adequately-funded monitoring and evaluation system?

• Are monitoring reports scrutinized? Is feedback provided to programme staff?

• Is there evidence of timely response to monitoring information on bottlenecks and other operational problems?

• Has statistical and epidemiological advice been sought in the design of the evaluation system?

• Are the indicators in line with the programme's objectives? Has evaluation been implemented as scheduled?

• Does the evaluation data allow assessment of programme impact?

• Have the communities designed and implemented a participatory monitoring system that relates to their specific development priorities¹²?

• Do community members understand and use the information it provides?

8) Assessing programme linkages

No programme can function well in isolation. Linkages with subnational authorities are essential to its eventual institutionalization. Linkages with other programmes and the establishment of useful partnerships can enrich a programme substantially and make it more cost-effective. You are asked here to seek information in order to answer the following questions:

• Has the programme established good working relations with subnational authorities, bodies or committees?

• Does the programme have collaborative linkages with other relevant programmes?

• Has the programme established useful partnerships? Partnerships¹³ with non-governmental organizations, the private sector and with research and training institutions are important to access technical expertise and supplementary funding, especially funding for community activities.

9) Summarizing the assessment of programme design and action

You now have the information you need to make an assessment of the programme design. Turn now to the Summary Report provided in Annex 1, and answer the questions in Section I. Then carry out a SWOC analysis.

Programme effectiveness or the extent to which your programme is able to improve nutritional status in the area covered by the programme will be seriously undermined if your programme design is flawed. If you judge the programme design to be poor, then you need to consider whether the programme is worth continuing, and whether you should not rather formulate a new programme. In any event, you will need to discuss your options with the programme funders, especially if the programme is externally funded.

If you have identified some weaknesses, you can take any or all of the following actions, as appropriate:

1. Re-formulate programme objectives to make them relevant, achievable and measurable;

2. Develop the necessary conceptual frameworks to ensure that your programme addresses the causes of malnutrition in the different areas covered by the programme;

3. Consider targeting, or improve the system for targeting, as appropriate;

4. Examine alternative programme designs to see if any are more cost-effective and appropriate for your situation;

5. Strengthen community participation by providing additional training to all staff and by addressing ways and means of enhancing the effectiveness of community mobilizers;

6. Find ways to motivate community mobilizers. Discuss with them how you can improve their job satisfaction: the problems they face in their jobs, their career aspirations, their response to the ideas of social recognition schemes and additional training opportunities;

7. Establish clear guidelines for community-level activities, covering the following aspects: identifying suitable community action groups, building community capacity, helping communities access technical expertise and funding sources, encouraging inter-community collaboration and the use of appropriate cultural practices;

8. Introduce a system of monitoring and evaluation, and secure funding for it, or re-design the existing one. Establish a system of information flows and use the information for programme management;

9. Address identified weaknesses in programme management: simplify if necessary; discuss job descriptions, responsibilities, career structures and recognition of achievements with staff and modify as needed; provide (additional) training in management to supervisors;

10. Improve relations with local authorities; establish or strengthen links and partnerships with other programmes, non-governmental organizations, the private sector and research and training institutions, as appropriate.

FAO's in-depth study found a number of strengths, weaknesses, opportunities and constraints in the nine programmes it examined:

Strengths:

• high level of community involvement;
• well-trained and committed community workers;
• effective and appropriate community activities;
• appropriate use of local tradition and culture;
• well targeted (socio-economic or geographic targeting);
• good programme management;
• good monitoring and evaluation system;
• feedback to communities, recognition of achievements, motivation of communities and community workers;

Weaknesses:

• political interference in targeting of programme activities;
• operational problems and delays;
• limited community participation;
• poor community capacity development, or some leaders not used to full potential;
• some inadequate or inappropriate community activities;
• lack of conceptual framework, leading to root causes of malnutrition not being addressed, short-term interventions and curative rather than preventive approach;
• no clear objectives;
• weak monitoring and evaluation system;
• poor management.

Opportunities:

• planned strengthening of community involvement and capacity building;
• planned change of programme offers opportunities for improvement.

Constraints (and threats):

• political interference in programme operations (targeting; staff selection);
• delays due to government bureaucracy, leading to operational problems;
• programme expansion leading to inadequate supervision and poor quality control;
• failure to secure funding for community activities leading to disillusionment;
• excessive programme control over community leadership; limited flexibility.

⁴ An acronym that may be helpful is SMART: Specific, Measurable, Achievable, Relevant, Time-bound

⁵ In some unique situations, lack of knowledge is the only impediment to nutritional improvement.

⁶ This statement assumes that supplementation will continue to be needed ad infinitum, whereas if the food-based approach achieves a real behavioural change in dietary practices these will continue with no further inputs, or with no additional inputs that are not already provided by a good GMP programme.

⁷ Various titles are used for community-level workers whose task is to work with communities, to help them organize, identify needs, plan and implement activities: popular titles are mobilizers, promoters, facilitators, community or village workers, development workers. If unpaid, they may be called community volunteers. Existing staff, such as community health workers, extension staff, can also play a mobilizing role.

⁸ Thailand made good use of this approach and found it essential to the smooth running of the programme. Social recognition can take the form of badges or t-shirts, public recognition through various ceremonies, or training opportunities even outside the community. Arguably, funding spent on a social recognition programme is more effective than funding spent on salaries for the volunteers.

⁹ More important than providing technical training, which of necessity will be basic, is training mobilizers to know when, where and how to seek expert assistance, and to recognize the limits of their knowledge.

¹⁰ If paid by the programme, sustainability after the programme ends may be in doubt.

¹¹ If child anthropometric status is selected as an indicator (and it must be if reducing malnutrition is an objective of the programme), we strongly urge you not to use clinic-based GMP data. Such data are likely to be biased because of incomplete coverage and age of attendance at GMP sessions. Moreover, GMP data generally provides weights only, hence levels of wasting and stunting cannot be assessed.

¹² You should ask to see evidence of the system when you visit communities.

¹³ FAO's in-depth study of nine programmes (2003) found that the partnership with the Catholic Church in Brazil's Child Pastorate programme led to a high degree of commitment and motivation on the part of community mobilizers.