Over the last decade, the countries of the Latin America and Caribbean region have pioneered a series of innovative programmes in rural development. These innovations have covered the whole span of rural activities and assets including human capital formation, the development of off-farm activities as well as the more traditional agricultural and livestock production. Specific interventions have ranged from conditional cash transfer programmes linking rural anti- poverty alleviation to human capital building, conditional cash transfer programmes linking trade liberalization compensation to continued agricultural production, demand-driven agricultural technology transfer schemes, the reform of agricultural research, school decentralization programmes and social investment funds.
In general, these programmes reflect current thinking on social and agricultural policies, which has moved away from universal supports for either consumers or producers to targeted cash transfers, financing or other non-monetary interventions. They represent an important shift in how rural policy is carried out in Latin America and are thus controversial. They imply the construction of new institutions, some of which are highly centralized, while others are decentralized and participatory. The experience to date and in coming years of these programmes is very relevant for the context of other countries.
The objective of this chapter is to provide an introduction to one set of this new generation of programmes. Thus this chapter focuses on those programmes with all or most of the following characteristics: (i) targeted to specific households; (ii) use cash as an incentive to modify individual behaviour; (iii) make receipt of transfers conditional on certain actions by beneficiaries; (iv) channel transfers to women; and (v) are rigorously evaluated. Thus while we leave aside other important innovations in social policy and rural development from the last decade, we take stock of some of their experiences which are relevant for the programmes under discussion here.[24]
Targeted: Targeting involves directing the intervention to specific regions, communities, households and/or individuals. The level of targeting will vary with the nature of the programme and the characteristics of potential beneficiaries, for example the relative level of homogeneity or heterogeneity within the potential beneficiary population. Targeting may be carried out by different levels of government, include participation from the local community or even involve the potential beneficiary her or him self.[25] The rationale for targeting stems from both efficiency and equity concerns. Properly targeted programmes should have a greater and more focused impact by providing assistance to those who need it most. The reduction in inequality brought about by targeting also may have a positive effect on overall economic growth (Ravallion, 2002). Targeting, however, is not necessarily easy to accomplish in political terms, as the target population is usually the poorest and most politically disfranchised segment of the national population.
Cash transfer: Cash is often preferred over in-kind transfers for a number of reasons.[26] First, cash transfers are generally a more efficient means of achieving a desired impact. In-kind aid, whether it is food or agricultural inputs, involves significant transaction costs and may distort local markets for those goods that are distributed. For example, local agricultural producers may be negatively impacted by the influx of free food into a region. Second, cash transfers are considered to lead to the greatest increase in household welfare, because presumably the household knows best how to spend its money. While in most of the examples discussed here beneficiaries are free to spend their transfers as they please, some programmes restrict spending to certain items, in particular food. This is the nexus of the debate over the introduction of the new Fome Zero anti-hunger programme in Brazil, for example.
Conditionality: Conditionality forms part of the new generation of programmes for a variety of reasons. First, conditionality assigns responsibilities to the beneficiary family. Households are not passive recipients of state aid, but must take action in a consistent and responsible fashion in order to receive payment. Second, conditionality constitutes an effective way of enticing poor households to carry out actions that have public externalities, such as receiving preventative health checkups and sending their children to school. Third, conditionality is seen as an effective way of encouraging investment in human capital accumulation, which parents may be unable or unwilling to provide to their children due to poverty or lack of information. Finally, conditionality may also serve as a means of self-targeting, as not-so-needy families, or at least those that do not value the transfers as much, may be unwilling to fulfil these responsibilities.
Payments directed to women: In almost all of the cases described here, payments are made to mothers, in order to assure compliance with conditionality and assure maximum impact on positive spending. This concept, which has become conventional wisdom in the development arena, is based on empirical evidence found in the development literature that females spend income differently than men. In particular, women are more likely to spend own-earned income on nutrition and children’s health and education while men are more likely to allocate income under their control to tobacco and alcohol. These gender differences in the allocation of income seem to be especially relevant among poor households.[27]
Evaluation: A crucial component of most of these innovative programmes is rigorous impact evaluations that are built into the design of the intervention. In most cases this has entailed the collection of experimental panel household data, as well as case study qualitative data. These evaluations tend to be conducted by external organizations as a way of ensuring legitimacy and accountability in the context of historical political use of transfer programmes. They provide invaluable insight into the incentive structure and processes of an intervention, and as such form an essential part of policy design and of the agricultural and rural development process itself.
Of the five aspects characterizing these programmes, this chapter will focus primarily on the issue of evaluation: the importance of evaluation forming part of the original design process, what can be obtained from a well-conceived evaluation and, finally, some results from recent evaluations. The conditional cash transfer programmes’ experience with rigorous evaluations is of paramount importance, as it presents a good example of what other programmes in the area of rural development could adopt. Through fits and starts, projects in other sectors, particularly in agriculture and livestock, are attempting to apply these concepts where appropriate.[28]
The programmes discussed in this chapter provide a wealth of experience as well as data to analyse and reflect upon. The ultimate goal here is to identify unresolved and/or debated design and implementation issues, as well as impact analysis, which merit further reflection and research. The chapter is organized as follows. Following this introduction we briefly describe the major existing conditional cash transfer programmes in the Latin American and Caribbean regions. Next, we focus on the role of evaluation in these innovative programmes and provide a summary of the results of recent evaluations. The chapter closes with a discussion of future research directions.
The bulk of the conditional cash transfer programmes link cash transfers to human capital formation in the form of education, health and nutrition. One programme, however, PROCAMPO from Mexico, was conceived as a mechanism for compensating basic grain producers under trade liberalization. Chronologically the first of this new generation of conditional cash transfer programmes in the Latin American and the Caribbean region, payments in this programme are linked to continued agricultural production. Below we provide a detailed description of conditional cash transfer programmes currently in existence. They are listed in chronological order from year of inception.
PROCAMPO (Programme of Direct Payments to the Countryside) in Mexico
Cash transfer schemes were part of policy changes that included the ending of government subsidies in the agricultural sector, on the theory that terminating subsidies on agricultural inputs and outputs removes market distortions, allowing resources to be allocated more efficiently. PROCAMPO was initiated at the same time as the NAFTA agreement (1994) and was designed as a 15-year transition to free trade. It is expected to terminate in 2008. The level of eligibility is dependent on the total hectares of nine key grains and oilseeds (corn, beans, rice, wheat, sorghum, barley, soybeans, cotton and cardamom) that were planted in the three agricultural years prior to and including August 1993. The prices of these crops were expected to drop as a result of the trade agreement, and PROCAMPO was designed to compensate for losses incurred by producers. Eligibility was actually given to land parcels and those with usufruct over those land parcels, not to particular farmers, and payment is meant to go to whomever is planting the property, whether owner, renter or sharecropper. Theoretically, the farmer receiving payment for a particular property may change, depending on who is using the land, though in practice most benefits accrue to the owner, either directly or through the rental price. The eligibility roster was fixed prior to commencement of the programme; no new properties have been added since 1994. Since PROCAMPO is geared toward farmers, the primary recipients of the cash transfers are male landowners. Although not explicitly a poverty alleviation programme, the transfers are widely distributed across rural Mexico and many recipients are categorized as poor. Payments are made directly to producers, and can be used as collateral against which to borrow from lenders or input retailers.
Since there are potentially two agricultural seasons per year, PROCAMPO payments may be made up to twice a year, though in general only farmers with access to irrigation can take advantage of the second agricultural season. Payments correspond to the amount of land currently under production, which cannot exceed the amount of land registered in the eligibility roster. Farmers must prove that the parcel is currently under production, but until recently monitoring of actual planting was haphazard, and many devices are employed to skirt this requirement. In any case, given that the programme is based on past agricultural production and the requirement that farmers continue producing or participate in an official environmental management programme,[29] the intervention is closely and intentionally linked to agricultural production.
The real value of PROCAMPO payments fell 35 percent between 1994 and 1996. Only in 2001 did payments per hectare, as well as total PROCAMPO expenditures, in real terms reach the original 1994 levels. The per-hectare payment in 2002 was set at 875 Pesos, or US$90. PROCAMPO annually reaches almost three million producers each year, covering almost 14 million hectares of land. The programme has a budget of US$1.24 billion for fiscal year 2002 (Fox, 2002).
Since PROCAMPO is distributed on a per-hectare basis, larger farms tend to get higher total transfers. Overall payments are regressively distributed; the 45 percent of producers with farms smaller than 5 hectares receive only 10 percent of total PROCAMPO transfers (SAGAR, 1998). Payments are progressively distributed on per hectare basis, however, as they are uniform per hectare, and thus unrelated to yields achieved and whether households were selling basic crops before NAFTA. That is, poor subsistence farmers received the same amount, per hectare, as wealthy modernized farmers. Transfers thus reach (and overcompensate) producers who had never benefited from pre-NAFTA price support programmes due to lack of marketed surplus, but undercompensate farmers who had benefited fully from subsidies.
Bolsa Escola in Brazil
Bolsa Escola programmes were first introduced in 1995 in Campinas in the State of Sao Paolo and in the Federal District, Brasilia.[30] Bolsa Escola has four objectives: i) increase educational attainment, thus reducing poverty in the long run; ii) alleviate short-term poverty; iii) reduce child labour; and iv) serve as a safety net for the amelioration of the impact of negative shocks. Since the administration of public policy in Brazil is highly decentralized, in its early stages various permutations of the original Bolsa Escola were implemented in different municipalities and states. By 1998, more than 60 programmes were in operation covering a total of 200 000 mostly urban families (World Bank, 2001a). Decentralization has meant not only local control, but also local financing, as municipalities were responsible for part of the total cost of the programme. This arrangement created perverse inequalities as the poorer municipalities were the least likely to be able to afford to cover all needy families.
In all cases household cash transfers were tied to children’s school attendance. For the most part the programme targeted children aged 7-14, though this varied by municipality. Payments were channelled directly to the mother, and the actual level of payment varied by municipality. Other programme details differed as well; for example, in Brasilia a sum equivalent to one minimum wage was deposited into a savings account in the name of the child beneficiary. The money could be withdrawn only when the student completed the eighth grade.
Selection of beneficiaries entailed two stages of targeting. First, the pro-gramme identified the poorest neighbourhoods in a given municipality. Next, beneficiary households were selected based on a combination of an income test and a score system, which is similar to a proxy means text. A series of household characteristics were assigned points; weighting of each characteristic varied by municipality. The income test - half a minimum wage - did not vary by municipality, though it probably should have, given regional variance in prices. In most cases, only families with school age children and who had lived in a municipality for a specified period of time were allowed to participate (World Bank, 2001a).
The programme was greatly expanded in 2001 with the creation of the federal Bolsa Escola. This new programme involved the integration of previous Bolsa Escola experiences with another major federal programme, the FGRM (Guaranteed Minimum Income Fund), which had begun operations in 1997. Payments are now made directly to the mothers’ account at a public bank, Caixa Economica Federal, and withdrawn using electronic cards. The federal government is now responsible for 100 percent of the funding for the programme. The ages have been expanded to 6-15 years of age, and the goal of the federal Bolsa Escola is to reach 10.7 million children drawn from 5.8 million households with monthly per capita incomes of less then R$90 (or US$36). The transfer is R$15 per month per child, for up to three children per household. As of December 2001, the programme reached 4.8 million households, covering 8.2 million children, in 5 470 municipalities (or 98 percent of all municipalities), with a monthly budget of R$123.7 million (Bolsa Escola, 2002). Municipalities still introduce innovations; in the Renda Cidada programme in the State of Goias, for example, households may spend the transfer only on food, and the city of Sao Paolo provides additional funding and stricter verification of targeting.
PETI (Child Labour Eradication Programme) in Brazil
The PETI programme was introduced in 1996 by the federal government with the goal of reducing the incidence of child labour in rural Brazil through education-related grants. While it is illegal in Brazil for children under the age of 14 to be employed in labour activities, including family economic activities, many children still work. In 1999, approximately 37 percent of children age 10-14 living in rural areas, and 8 percent in urban areas, were engaged in labour activities. Most of these labour activities were unpaid, family agricultural labour, with large regional differences in incidence, and boys were almost twice as likely as girls to be working (World Bank, 2001b).
The PETI was thus originally designed to remove and protect children aged 7-14 from the worst forms of child labour, and thereby targeted children already working or those at risk of working. The programme is implemented in rural areas and specifically in areas with the highest incidence of risky child labour. This included, for example, agave production in Bahia and sugar cane production in Pernambuco. As with Bolsa Escola and most of the other conditional cash transfer programmes, payments are directed towards the mother and are conditional on two actions by the family. As in Bolsa Escola children must attend school, but in addition they must attend a jornada ampliada, or after-school programme. This after-school programme is intended to provide additional educational and recreational stimulus for the children and most importantly hinder them from combining work and school. The actual content of this after-school programme varies by municipality. PETI also provides the children with a school lunch (World Bank, 2001b). Parents are also encouraged to participate in complementary programmes, such as PRONAGER (Generation of Employment and Income in Poor Areas Programme), in order to improve household incomes and thus reduce in the long term household dependence on income from child labour.
Targeting was carried out in two steps. First, municipalities with a large incidence of the worst forms of child labour were selected by the State Commission for the Prevention and Eradication of Child Labour. Second, Municipal Councils for the Eradication of Child Labour carried out the selection and registration of families. Households did not have to enrol all eligible children in a family, thus creating the possibility that participating children gain at the expense of non-participating siblings who may receive a greater work load. By 1999 PETI covered over 130 000 children, and the programme is expected to increase coverage in the next phase to over 800 000 children (World Bank, 2001b).
PROGRESA (National Programme for Education, Health and Nutrition) in Mexico
PROGRESA is a national anti-poverty programme begun in 1996 with the goal of developing the human capital of poor rural households by improving education, health and nutrition outcomes. Cash transfers are conditional on school attendance by household children in the third through eighth grade, basic health care for all family members and public health classes for adults. Households receive separate monetary transfers for education scholarships, educational materials and general consumption. In addition, pregnant and lactating mothers and infant children receive nutritional supplements. The required health checkups are subsidized. Payment for the scholarship component varies by gender and grade of the child beneficiary, and transfers are provided directly to mothers. Payments are made directly to beneficiaries via periodic cash disbursements through a private financial network. Households receiving PROGRESA are not permitted to receive other forms of anti-poverty or education subsidies.
The selection of eligible households was done in three stages. First, potential recipient communities, the smallest administrative unit in Mexico, were identified as poor based on a marginality index developed from the national population census. This marginality index was constructed using community data including the share of illiterate adults, access to water, drainage and electricity, number of occupants per room, dwellings with a dirt floor and population working in the primary sector. The more marginal communities were considered potential target locations and were further evaluated based on location and existence of health and school facilities. After communities were identified, the second step was to select households for participation in PROGRESA based on data collected from a household census carried out within the community. Scores were produced for each household using a statistical procedure, discriminate analysis, and households above a certain line were included as beneficiaries. After households were initially identified as potential participants, the third and last step was to present a list of these households to the community assemblies for review and discussion, though in practice this review was perfunctory and the lists were rarely modified (see Skoufias, Davis and de la Vega, 2001 and PROGRESA, 1998).
With the Fox administration in 2001, PROGRESA changed its name to OPORTUNIDADES and expanded operations to urban and semi-urban areas (into communities with a population of over 2 500 inhabitants). High school age children were incorporated into the programme, and targeting techniques have been adapted to urban realities. From the beginning of 2002, public media campaigns were conducted inviting households living in poor neighbourhoods to apply for the programme. Potential beneficiary households were then visited at home (Gutiérrez, Bertozzi and Gertler, 2002).
The PROGRESA budget for 2002 reached US$1.9 billion, covering almost three million rural families and over 1.2 million urban and semi-urban families. For girls in the first year of secondary school, the 2002 monthly payment was US$31. Since 1997, this payment has increased, in real terms, by 47 percent (Fox, 2002).
PRAF-II (Family Allowance Programme) in Honduras
PRAF-II began in 2000 implementing a programme of household cash transfers covering 40 municipalities with a total population of approximately 400 000. The primary objective of the programme is to increase human capital accumulation among the poor, through increased demand for and quality of primary education and health services, as well as improved feeding and hygienic practices of mothers with small children and improved nutritional status of small children. Given availability of funds, the second phase of PRAF was limited to the 80 poorest municipalities in the country. Municipalities were selected based on the nutritional status (age- and sex-standardized height) of children attending first grade, which is almost universal in Honduras. Height-for-age is considered a good indication of chronic malnutrition, and thus serves as a proxy for both poverty and food insecurity. Data were taken from the 1997 First Grade School-children Height Census, and Z scores were standardized with reference to a healthy United States population, as is customary. After municipalities were ranked by Z scores, they were randomly allocated into three treatment groups and one control group for evaluation purposes. This evaluation design is quite unique in that three distinct treatment groups are constructed. One receives just cash transfers, a second cash transfer plus improvements in service (education and health) quality and the third only improvements in service quality.
The PRAF-II interventions comprise two distinct sets of household transfers: the first for primary school age children (aged 6 to 12) and the second for pregnant women and children under three. Within the forty municipalities randomized to this intervention, all households with young or primary school age children or pregnant women are eligible to receive the transfer, but they face removal if they do not comply with required prenatal check-ups, growth monitoring, vaccinations and school attendance, depending on the case. A maximum of two beneficiaries per household is permitted for the health transfer and three for the education transfer. The programme also provided funds to schools and health centres to improve the supply of services commensurately with increased demand. Benefits are channelled to mothers, and the programme is administratively centralized (MNPTSG, 2001; IDB, 2001; and IFPRI, 2001a).
RPS (Social Protection Network) in Nicaragua
The RPS began in late 2000 to operate a pilot phase and currently reaches approximately 6 000 geographically targeted rural households located in northern Nicaragua. A marginality index was required in order to rank census segments for targeting purposes. The index was composed of four variables (household size, percentage of households without potable water, percentage of households without latrines and percentage of illiterate adults), which were weighted by the coefficients derived from ordinary least squares regression analysis of the determinants of extreme poverty, using household data and a larger group of variables (Arcia, 1999). The expansion phase of the RPS currently underway, incorporating another 4 000 households, will utilize poverty maps based on a small area estimation strategy.
The interventions include two sets of household transfers: one for primary school age children (aged 7 to 13) and one universal transfer. Virtually all (97.5 percent) households in 21 pilot census districts are eligible to receive the transfer, the exceptions being those that reported owning a vehicle of some kind or more than fourteen hectares of farming or grazing land. Beneficiary households are required to take part in a programme of health education, attend child growth and development monitoring sessions and keep vaccinations up to date if they have children of the appropriate age group. School aged children are required to be in school. The RPS also specifically targets the mother, or responsible female adult, of the family to receive the benefits. The programme was originally centralized in an autonomous agency of the federal government, but has since been relocated in 2002 to the new Ministry of the Family (IFPRI, 2002; Government of Nicaragua, 2001 and MNPTSG, 2001). Despite the success of the pilot, described below, political support for the expansion of the RPS has been lukewarm.
Bolsa Alimentação in Brazil
Bolsa Alimentação is a recently created federal anti-poverty programme in Brazil, begun in 2001. The design is very similar to health components of PROGRESA and the Central American interventions, though cash transfers are limited to poor families with pregnant and lactating women and/or infants and young children under six years of age. Programme objectives include i) reducing the incidence of infant malnutrition and mortality; ii) linking families at risk, and particularly mothers and infants, with the health system; and iii) focusing beneficiary families on other actions that can help improve their socio-economic status.
Conditionality is linked only to health care usage (prenatal care and growth monitoring, vaccinations and health education) and not education, which is instead dealt with by the Bolsa Escola programme. Given the targeting of different aged children and the split between health and education, Bolsa Escola and Bolsa Alimentação are complimentary. As with the other Brazilian programmes, each Bolsa Alimentação beneficiary will receive R$15 a month, for a maximum of R$45 per month per family. Similarly, payments are made through direct deposit to the mothers’ account at the Caixa Economica Federal and withdrawn using electronic cards. As of December 2002, the programme is operating in over 4 000 municipalities, reaching more then 1.4 million beneficiaries (IFPRI, 2001b; Bolsa Alimentação Web page, 2003).
PATH (Programme for Advancement through Health and Education) in Jamaica
PATH is a new conditional transfer programme, initiated as a pilot in 2002, designed to transform the existing social safety net system in Jamaica into a more efficient system of social assistance for the poor and vulnerable. Overall, PATH is very similar in terms of content and objectives to the other conditional cash transfer programmes described above. What is unique about the programme is that in addition to focusing on families with children of school age (6-18 years), small children and pregnant and lactating mothers, PATH specifically targets the elderly and disabled. The programme also uses a somewhat different targeting system. Households are included based on a proxy means test. Households are not visited, however, to collect the requisite information, but instead, as with the urban version of PROGRESA, families are invited to visit PATH offices and apply for the programme. This process is then verified afterwards using random sample techniques. Payments are made through the Post Office, and the transfer is the same for all types of beneficiaries. The programme is scheduled to reach 236 000 individuals over a period of four years, at a total cost of US$77.5 million.
Cartão Alimentação in Brazil
The Cartão Alimentação, launched in February of 2003, is a key component of the Fome Zero programme, which itself forms the centrepiece of recently-elected President Luiz Inácio Lula da Silva’s efforts to end hunger in Brazil. Like its predecessor conditional cash transfer programmes Bolsa Alimentação and Bolsa Escola, Cartão Alimentação provides monthly a fixed amount of cash (R50) to the mother or responsible female in families whose per capita income is less then one half the minimum wage. This payment is guaranteed for at least six months, with the same payment mechanism as the other programmes. However, unlike Bolsa Alimentação and Bolsa Escola, households are restricted to spending the transfers only on food items, which is verified by the household providing receipts for the amount of the transfer. No other conditionality is imposed. Under the auspices of the larger Fome Zero programme, Cartão Alimentação is accompanied by other local development initiatives at the municipal level, including for example adult literacy, cistern provision, school feeding, as well as programmes more regional or national in scope, including land reform and support for small-scale agriculture. While currently using the same basic targeting instrument as Bolsa Alimentação and Bolsa Escola, the Cartão Alimentação gives a much larger role to local administrative bodies, made up of representatives from local government and civil society, in correcting targeting errors. These local administrative bodies also play a key role in verifying restriction on spending of the transfer (Presidencia da Republica, 2003).
The objectives of Cartão Alimentação are ambitious; initially begun as a pilot in two municipalities of the semi-arid northeast, as of May 2003 the transfer was reaching 39 051 households in 111 municipalities (Fome Zero, 2003), with the goal of reaching 1 million households by the end of 2003. At the time of this writing, however, debate was continuing in Brazil regarding the possible integration of the different cash transfer programmes described in this chapter. Some type of integration or coordination seems likely and desirable, given that the programmes share targeting mechanisms and information systems and given the overlap of target populations.
The important role that evaluation techniques should play in the selection, design, implementation and impact evaluation of rural programmes has gained increasing recognition in recent years. This is evident in the growing number of government and international organizations that require serious evaluations of specific programmes. Evaluation techniques can serve to improve implementation and efficiency of programmes after interventions have begun, provide evidence as to the cost efficiency and impact of a specific intervention and provide information on comparison of interventions within and between policy sectors. Consensus is growing on the value of spending relatively small amounts of resources (typically around 1 to 5 percent of project costs) for programme evaluations. While the new generation of targeted cash transfer programmes has led the way in terms of incorporating rigorous and comprehensive evaluations, the concept is slowly expanding to other sectors, most notably agricultural technology transfer programmes. In this section we discuss a variety of evaluation techniques, including the appropriateness of quantitative and qualitative methods for different kinds of interventions; the identification of the appropriate persons or institutions to carry out evaluations of public sector programmes and the implications of the recent trend towards legally mandated evaluations in a number of LAC countries. We close with a presentation of results from a number of these evaluations.
Components and methods
A project evaluation can incorporate part or all of a series of components, depending on budget, objectives and the nature of the intervention.[31] These components include (Baker, 2000):
Process evaluation: how a programme operates, focusing on problems in service delivery;
Theory-based evaluation: assessment during intervention of the explicit or implicit assumptions of programme design, in order to pinpoint where this process may break down (where intended or unintended incentives may lead people to behave in an unexpected fashion);
Impact evaluation: attributing causality to an intervention;
Cost benefit/effectiveness: programme costs, in relation to alternative uses of the same funds.
A variety of methods, both qualitative and quantitative, are available for a project evaluation, and any one evaluation may include a different combination of methods, again depending on budget, objectives and the nature of the intervention. The choice is also dependent on the intended audience, as different kinds of decision-makers need to be confident to lesser or greater degrees that any observed effects are in fact due to the project or programme. Quantitative methods use primary or secondary data found in surveys to reach conclusions, hopefully based on some type of statistical rigour and certainty; we discuss difficulties associated with quantitative methods below. Qualitative methods (focal groups, open ended interviews, expert opinion) can provide a different perspective (Baker, 2000; Rawlings, Sherburne-Benz and Van Domelen, 2001):
Understanding of processes, behaviours and conditions as they are perceived by the individuals or groups being studied, or by specialists or other outside observers;
Different perspectives on what are the important outcomes (for example, poverty verses vulnerability);
Analysis of social processes (conflict and capital) and institutional development;
Interpretation, providing detail and richness to quantitative data/conclusions.
A new generation of evaluations
The new generation of evaluations is noteworthy for two reasons. First, new rigour and techniques have been brought to bear in quantitative evaluations, particularly in terms of attributing causality to specific interventions where historically impact evaluations have not been conducted. Governments and international organizations are increasingly willing to commit the necessary resources for this type of evaluation. Second, barriers between evaluation practitioners from different schools (particularly between economists, nutritionists and sociologists/anthropologists) are slowly being reduced and as a result quantitative and qualitative methods have been brought together in innovative ways. Enough experience has been gained with these new evaluations to suggest some best practices.
Impact evaluation
The purpose of an impact evaluation is to establish the known probability that the findings of the impact of a specific intervention on intended beneficiaries may be a chance configuration of the data. The identification of the counterfactual is the organizing principle of an impact evaluation; that is, what would have happened to the beneficiaries if they had not received a particular intervention. The “with” data are observed in a household survey that records outcomes for recipients of the intervention. The “without” data, however, are fundamentally unobserved since a household cannot be both a participant and a non-participant of the same programme.[32]
The counterfactual is identified by selecting a control group. A group of control households should be chosen from non-beneficiaries to be representative of the group of participants with one key difference: the control households did not receive the intervention. The outcomes of non-beneficiaries may differ systematically from what the outcomes of participants would have been without the programme, producing selection bias in the estimated impacts. This bias may derive from differences in observable characteristics between beneficiaries and non-beneficiaries (location, demographic composition, access to infrastructure, wealth, etc.) or unobservable characteristics (natural ability, willingness to work, etc.).
The history of project impact evaluation, however, is littered with evaluations which claim to attribute causality but without correctly creating a counterfactual. This is particularly true when evaluations are hastily arranged ex post. In these cases, causality is incorrectly attributed to a specific intervention. The distinguishing characteristic of the new generation of evaluations involves attempting to set correctly these counterfactuals. A variety of designs are available, each with different advantages and disadvantages in terms of ease of use, cost and robustness of results.
Establishing the known probability is only possible with a randomized (experimental) design. An experimental design, if done correctly, is the only method that can eliminate selection bias and allows the most ease of use following collection of the data. In this case, households are allocated randomly to treatment and control groups. Assuming that this process of randomization is done correctly, both types of selection bias described above are minimized, and in a technical sense there is no bias. There is a known (and low) probability that observed between-group differences will be due to something other than the intervention. This permits analysis of the results without recourse to sophisticated econometric techniques and their accompanying assumptions. The evaluations of PROGRESA, PRAF-II and RPS use experimental design.
An experimental design, however, is often difficult to implement for institutional and political reasons. Withholding benefits from eligible individuals or households can be considered unethical and is very difficult to justify politically. In some cases, budget limitations (Honduras and Nicaragua) or the phasing-in of eligible beneficiaries (Mexico) may permit use of an experimental design for the policy maker with worries about the morality of conducting social experiments, but making this justification understood publicly is very problematic. Further, the transparency inherent in randomization may be considered appropriate for programmes with limited budgets that must decide how to allocate scarce resources. Both Honduras and Nicaragua conducted the randomized selection of beneficiary municipalities in public events.[33] Even when the randomization is correctly applied, the implementing agency must be careful that control communities are not “contaminated” by other public programmes or even NGOs, and that benefits are given as planned to treatment households. Providing benefits to control communities too early has apparently caused problems in the PRAF-II evaluation, and a number of treatment households in the PROGRESA evaluation started receiving benefits much later then originally planned (Skoufias, 2001).
Quasi experimental design constitutes a second best option when experimental design is not feasible. Any design with a non-randomized control group may be termed quasi-experimental. One strategy involves using as controls communities (or households or some other grouping) excluded from the programme for budgetary or administrative reasons. One example, employed by the PETI programme as discussed below, consists of using as controls, ex post, communities originally excluded from the intervention due to budget constraints. In a second strategy, control groups may be “created” through econometric techniques, in particular propensity score matching, which has become very popular in recent years. In matching methods, the counterfactual group is constructed by matching programme participants to non-participants on the basis of similarities in observed characteristics that are predictive of participation in the programme (Heckman, Ichimura and Todd, 1997, 1998). Depending on the data sources used and on the relative comparability of beneficiaries and non-beneficiaries, it may require a large reservoir of data on non-beneficiaries from which to select. Further, Heckman, Ichimura and Todd (1997 and 1998) find that the biggest source of bias in propensity score matching stems not from selection bias but from bias associated with differences in observable characteristics. Thus for propensity score matching to be made workable, it must effectively control for observed characteristics and use similar survey instruments for beneficiaries and non-beneficiaries. These results have been tentatively confirmed in preliminary head to head testing of propensity score matching and randomized methods using the data from the PROGRESA evaluation and another household survey in Mexico (Diaz, Handa and Orozco, 2003).
Depending on the data and the nature of the selection bias present, other econometric techniques such as instrumental variables may also be appropriate. A number of the social investment fund evaluations (Rawlings, Sherburne-Benz and Van Domelen, 2001), an evaluation of the Workfare programme in Argentina (Jalan and Ravallion, 1999), a comparison of PROGRESA and PROCAMPO recipients in Mexico (Davis et al., 2002) and evaluations of technology transfer programmes under way in Nicaragua (World Bank, 2000) and Mexico (Gonzalez and Davis, 2002) use or propose using versions of these techniques. The PATH programme in Jamaica is considering using either propensity score matching or administratively excluded households as two alternatives in the construction of a comparison group (Rawlings and Rubio, 2003). While this type of quasi experimental design is often cheaper and quicker, may or may not require collection of original data and constitutes the second best alternative, the econometric methods are complex and results are dependent on sometimes excessively stringent statistical assumptions.[34]
Integrating quantitative and qualitative methods
Successful evaluations are generally able to integrate quantitative and qualitative techniques in a complementary fashion. Some of the social investment fund evaluations, particularly Nicaragua and Honduras (see Rawlings, Sherburne-Benz and Van Domelen, 2001 for a review), and to a lesser extent the PROGRESA evaluation (see Adato et al., 2000), provide good examples. In these cases rigorous quantitative impact evaluations were combined with many of the qualitative methods described earlier. While the quantitative components established achievement of numerical objectives, qualitative methods, when done correctly, provide the details and story behind the empirical results. While quantitative methods show convincingly if a programme had a measurable impact, qualitative methods help understand why, or the process of how this change came about. Qualitative methods are also more appropriate for aspects of the programme that quantitative methods may not capture, for example the influence of politics and corruption, the dynamic between men and women and the incentives driving beneficiary behaviour in response to the intervention. Despite these obvious synergies, professionals involved in each methodology have often been at odds, each viewing the other with scepticism and disdain. Fortunately, efforts made at international organizations such the World Bank, the IDB and IFPRI, historically dominated by economists, to open up the vision of an evaluation - as well as other types of analytical work such as poverty assessments - has helped to break down these barriers.
Cost
The cost of these types of evaluations varies greatly depending on the components and methodologies employed. Typically the biggest expense is data collection; when original data are required costs increase substantially. Experimental designs also imply collection of original data, pre and ex post, making it a relatively more expensive option, though this is true only if advantage cannot be taken of regular data collection efforts.
Total cost of course depends on the size of the data collection required; for small sample sizes, the cost of the analytical work may be greater, as this cost is relatively fixed compared to survey costs. For this reason experimental designs tend to be the most expensive, requiring at least before and after household surveys which may cost anywhere from US$25 to US$125 per observation. The evaluations of social investment funds cost on average 1 percent of total programme cost, though this percentage varied greatly by method (Rawlings, Sherburne-Benz and Van Domelen, 2001). The cost of quasi experimental designs varies widely as well, again depending on the need for original data. Qualitative methods, which require primary data collection, though of a more concentrated, less extensive sort, are by comparison relatively inexpensive. As such, they are almost invariably worth the added value they provide to the quantitative data.
Rawlings, Sherburne-Benz and Van Domelen (2001) and Baker (2000) argue that serious impact evaluations should be considered public goods. Not only do the interventions in question benefit from the experience of a given programme, but also other interventions in the same sector and indeed in other countries. Thus when an intervention is part of a loan with an international organization or donor country, it is reasonable to expect some sort of cost sharing on the part of the donor to help pay for the evaluation.
The political economy of evaluation
Successful evaluations require political and institutional support. Historically a number of elements have limited the incorporation of evaluations into development projects:
No value: government officials and staff from international organizations may not see the value of evaluations and are unwilling to invest sufficient resources.
Distrust: evaluators may have their own agenda to push.
Fear: government officials and staff from international organizations do not want to see bad results.
Political/institutional change: turnover, policy change, political infighting and battles over turf often inhibit planning and development of long-term evaluations.
Appropriateness: Some goals cannot be measured by impact evaluations (which should not preclude other types of evaluations).
The recent push for more serious impact evaluation has come from a variety of sources. On the one hand, donor countries and international organizations are slowly recognizing the importance of including evaluations in the initial design of an intervention, instead of waiting until the intervention is over, when it is very difficult to perform an evaluation. In the Latin America and the Caribbean region, this trend is most evident in the series of evaluations of social funds and the targeted cash transfer programmes in Nicaragua and Honduras.
On the other hand, in some countries such as Mexico and Brazil, federally mandated evaluations of all government programmes have forced state agencies to develop evaluations for major programmes. The quality of these state mandated evaluations varies widely, however, even within countries, ranging from the rigorous experimental design of PROGRESA, to massive perception of impact evaluations with Alianza para el Campo, to very poor quality impact evaluations such as that of PROCAMPO.
Mexico case study
The case of Mexico provides a very good example of both the potential and pitfalls of state-mandated evaluations. Beginning in 2001, major government programmes were required to conduct annual evaluations. The law itself does not provide much incentive for producing quality evaluations; most initiative comes from individuals at the different state agencies. The PROGRESA programme was developed under the guidance of a group of public officials with experience and interest in evaluation and statistical techniques. They opted for a randomized experimental design, carried out in conjunction with IFPRI, the most rigorous possible in terms of impact evaluation and in fact quite an innovation in impact evaluation design in Latin America. This type of design, as was described earlier, required treatment and control groups, with withholding of benefits from the control group for a certain amount of time (1.5 years in this case). While this was justified given the incremental expansion of PROGRESA over the first few years of existence, PROGRESA officials feared (correctly) that this would be misunderstood in Congress. This element of the design was left purposely vague until the evaluation was practically completed, at which time it did come under political attack from Congress and the media, but without affecting the evaluation. The evaluation that was presented to Congress included a review of the IFPRI studies, some further original research using the same surveys and a review of state monitoring information (Parker and Scott, 2001). The cost of the evaluation required by Congress was thus minimal, having depended on data (as well as studies) collected in previous years.
The evaluations of Alianza para el Campo and PROCAMPO followed very different paths. Prior to the implementation of the law, in 1999 officials in the Ministry of Agriculture (currently with the acronym SAGARPA) had contracted with the FAO for assistance in designing its evaluation. Together they chose essentially a quantitative assessment of perceived impact, where beneficiaries are asked their opinions as to how a particular programme had affected productivity and income, as well as monitoring questions and beneficiary satisfaction. This was done on a grand scale, with a complete evaluation for almost all of Alianza’s programmes (starting with 25 in 1999) in each of Mexico’s 32 states, requiring tens of thousands of informants. This massive collection of data was quite costly, at over US$1 million a year. The results are relatively limited given the amount of money spent, as it was not possible to determine with statistical certainty whether Alianza actually had an impact on a variety of socio-economic indicators. SAGARPA operated under the belief that evaluations of each programme in each state is necessary to comply with the congressional directive.
In this case, the SAGARPA interpretation of the law involves collecting essentially monitoring data on as many programmes as possible, representative at the state level when feasible. This interpretation has led SAGARPA to collect much more information than is needed to evaluate effectively a programme, and the evaluation process was geared more to collecting and processing these massive amounts of data than to analysis. The cost of the report to the Congress is thus very expensive, given the large amount of original data collected. Fortunately, in successive years the concept of evaluation has broadened, with gradually better analysis, the inclusion of a process evaluation and qualitative methods and the incorporation in 2002 of a pilot quasi experimental design evaluation in two states (FAO, 2001 and Gonzalez and Davis, 2002).
PROCAMPO has been less fortunate in terms of official evaluation. While PROCAMPO has been lucky, as discussed in detail below, in that researchers at academic institutions and international organizations have creatively used a variety of data sources to measure its impact, the two official evaluations have been limited to perception of impact evaluations (SAGAR, 1998 and Colegio de Ingenieros Agrónomos de México, 2001). The latter was commissioned hastily in order to comply with the congressional requirements, and it is the least useful of the three examples provided here. This survey was unable to establish, with any known probability, that PROCAMPO had an impact over any socio-economic indicator of interest. Beneficiaries were instead asked how they spent their payments, if they hired more workers, etc. The evaluation served more as a monitoring tool, providing information on such issues as whether payments had arrived as promised.
Some conclusions on evaluation design
Experience to date from the evaluation of targeted cash transfers and social investment funds provide some guidelines for designing an evaluation system. First and foremost, a good evaluation is most efficiently achieved and taken advantage of when it is part of the design of the intervention. Ex post evaluations constitute a much more difficult technical challenge, and in most cases the resulting analysis is not up to the task. Second, if externally mandated, national policymakers must feel ownership over the evaluation process, or as Rawlings, Sherburne-Benz and Van Domelen (2001) put it, the evaluation should have an in-country “champion”. This will more likely ensure government commitment to the entire process and that results will be discussed and influence the policy-making process. Third, the quantitative impact evaluations as described here (i.e. obtaining the correct counterfactual) are much more technically challenging. Professional experts are required in sampling, surveying and analytical techniques. This often requires depending on international experts, though in some of the bigger middle-income countries, such as Mexico and Brazil, the numbers of trained nationals are expanding. Fourth, and connected to the previous two comments, evaluations conducted with international assistance should include cost sharing and must also have the objective of building capacity within country and within national institutions.
Examples of impact evaluations
Below we present a summary of the results of a number of the impact evaluations of conditional cash transfer programmes in Latin America and the Caribbean. These include the PROGRESA programme, PROCAMPO, RPS and PETI. Evaluations have also been completed for Bolsa Alimentação and PRAF-II, in both cases conducted by IFPRI. Those results, however, were not yet publicly available at the time of writing. Impact evaluations of both PATH and Cartão Alimentação are still in the design phase.
PROGRESA
Given its sheer size and its status as the first of its kind in Latin America, the PROGRESA evaluation is the pioneer of randomized impact evaluation design in social policy. Data from its evaluation surveys have been analysed by dozens of researchers both inside and outside of Mexico. “Before” and “after” repeated observations were collected from 24 000 households in 506 localities at six-month intervals from November 1997 to November 2000. Of these communities, 320 were randomly assigned as treatment communities and 186 as control communities.[35]
Skoufias and McClafferty (2001) summarize the results of IFPRI, the research institute contracted to carry out the first PROGRESA evaluation.[36] They report that, comparing treatment and control households, following a year and a half of benefits, PROGRESA has been shown to:
keep children in secondary school, including a 20 percent increase in enrolment for girls and 10 percent for boys;
increase educational attainment (years of schooling) of poor children by 10 percent;
lower incidence of illness among children ages one to five by 12 percent;
lead adult beneficiaries to have:
- 19 percent fewer days of difficulty with daily activities
- 17 percent fewer days incapacitated by illness
- 22 percent fewer days in bed sick;
increase the number of first visits in first trimester of pregnancy by 8 percent;
increase median food expenditures by 13 percent, with most of this additional expenditures on fruit, vegetables and meat;
increase child growth and reduce the probability of child stunting;
provide no evidence of crowding out of private transfers, allaying fears that the programme would displace informal assistance networks;
have no apparent negative effect on labour force participation rates for men or women, countering concerns that transfer payments would provide a disincentive to work;
have low administrative costs relative to the costs incurred in previous programmes:
- for every 100 pesos of budget allocated to the programme, an estimated 8.9 pesos are absorbed by administration costs;
- dropping the household targeting procedures would reduce programme costs from 8.9 to 6.2 pesos (primarily through reduced survey and analysis costs);
- dropping conditionality requirements would reduce the programme costs from 8.9 to 6.6 pesos (primarily through lower data verification costs);
- dropping both of these elements of PROGRESA would reduce administrative costs to 3.9 pesos per 100 transferred.
A multitude of other papers outside the formal IFPRI evaluation have been written using the PROGRESA data, focusing on different aspects of the programme, including targeting, schooling and health. Some examples include Behrman, Sengupta and Todd, 2001; Handa et al., 2001; Dubois, de Janvry and Sadoulet, 2002; Raymond and Sadoulet, 2002; Skoufias and Coady, 2002; and de Janvry and Sadoulet, 2002, to name just a few.
The qualitative evaluation of PROGRESA (Adato et al., 2000 and Adato, 2000) found different levels of divisions between beneficiaries and non-beneficiaries in participating communities, leading to social tension, social conflict and political conflict in some cases. Elsewhere PROGRESA was shown in some instances to build up social capital and in other cases to break it down. In general, PROGRESA was not shown to allow local communities to play a significant role in the conceptualization, administration or targeting of the programme. On the other hand, PROGRESA was shown to empower women, particularly through the increase in women’s inter-household negotiating power.
PROCAMPO
PROCAMPO, on the other hand, does not have much in the way of a serious formal impact evaluation. As mentioned above, the legally mandated evaluations that have been carried out are of limited use. PROCAMPO has been extraordinarily lucky, however, in that a large number of PROCAMPO households can be found in (at least) two national household surveys in Mexico: first, the 1994 and 1997 national ejido household surveys, which coincidentally provide a natural experiment on participation in PROCAMPO; and second, the surveys from the PROGRESA evaluation described above.
With the ejido survey, the first year in this panel was carried out just prior to the disbursal of initial PROCAMPO payments, while the second year captured households after receiving two years of assistance. Programme placement bias was shown not to be an issue, allowing this data to be used as before and after experimental data. Sadoulet, de Janvry and Davis (2001) found that PROCAMPO transfers created large indirect effects among ejidatarios through multiplier effects in the range of 1.5 to 2.6 (depending on the econometric model employed) pesos per each peso transferred. That is, for every peso of PROCAMPO transfers, approximately two pesos in total income were generated through household economic activities. This multiplier reflects marginal income opportunities that go unrealized due to household liquidity constraints, which are relaxed by the transfers.
The multiplier works through a number of productive channels. PROCAMPO transfers created a positive multiplier effect in livestock, with every peso of transfer generating 0.28 pesos of livestock income. Similarly, PROCAMPO transfers created a positive multiplier effect in agriculture, with every peso generating 0.33 pesos in agricultural income. The availability of technical assistance and ownership of irrigated land increases the agricultural income multiplier, while the availability of credit decreases it. Thus, households that control more irrigated land and have access to technical assistance and no access to credit take greatest advantage of this multiplier effect.
A PROGRESA dataset was used to evaluate the impact of PROCAMPO on migration among rural households. PROCAMPO potentially could have a negative or positive impact on migration. While PROCAMPO may ease credit constraints and thus increase agricultural profitability, or simply tie farmers to the land in order to receive the benefits, PROCAMPO could also serve as a source of financing migration for one or more family members. Gonzalez, Konig and Wodon (2002) find that PROCAMPO had a negative impact on the probability of both permanent and temporary migration.
Direct comparative analysis of PROCAMPO and PROGRESA
Direct comparison of two cash transfer programmes in the same country is rare, if only because few countries have more then one kind of conditional cash transfer programme. In Mexico this is the case with PROGRESA and PROCAMPO, which affords a unique opportunity to compare the impacts of two conditional cash transfer programmes with different conditionality requirements.
First, in order to compare the poverty impact of these two programmes with similar numbers of beneficiaries and budget resources, a simulation exercise was carried out using data from the 1996 Mexican national household survey of income and expenditure (Davis, Handa and Soto, 2001). Specifically, the exercise posited what would have been the impact of PROCAMPO not existing during the devaluation crisis of 1994-1995, which led to a large increase in poverty in Mexico, and conversely, what would have been the impact if PROGRESA had existed during the crisis. If PROCAMPO had not existed in 1996, the national headcount index would have been 5 percent higher and the squared poverty gap 5.7 percent higher. If PROGRESA had distributed benefits to its 1999 beneficiaries in 1996, the headcount index would have been 8 percent lower and the squared poverty gap 22.5 percent lower. This shows that PROGRESA is much better targeted to the poorest households compared to PROCAMPO, which is to be expected since PROCAMPO makes no pretence of targeting based on poverty criteria.
Second, a direct econometric comparison of PROCAMPO and PROGRESA has been carried out by Davis et al. (2002). This study was interesting in that it took advantage of the existence of households in the PROGRESA evaluation data set who received payments from PROCAMPO. The study found that an agricultural support programme like PROCAMPO has an impact on household welfare similar to that of an anti-poverty programme like PROGRESA. Both programmes are also associated with increased spending on productive activities: PROCAMPO on agricultural production and PROGRESA on non-agricultural business activities. Male beneficiaries thus invest a substantial part of cash transfers, contrary to anecdotal evidence and conventional wisdom.
A companion study by Ruiz et al. (2002), focusing on food security, finds that both programmes boost total food consumption and caloric intake in similar proportions. Increased food security is achieved through different channels: PROGRESA through purchases, while PROCAMPO through investment in home production (then translated either into increased cash income through sales or directly through consumption of home-produced crops). The results also show that both programmes increase nutritional diversity.
RPS
With the assistance of IFPRI, an experimental design with randomized treatment and control groups was used to evaluate the pilot phase of the RPS. In this case, 42 census areas located in six highly marginal municipalities were randomly allocated into treatment and control groups of similar size. A baseline household survey composed of 1 700 observations was carried out in August 2000 prior to the intervention and a follow-up survey of the same households in October 2001.
The intervention was shown to have had a significant positive impact on total household consumption per capita, as well as on household food consumption and the share of food consumption in total spending. The level of food consumption for control group households actually fell during the reference period of the evaluation, while that of treatment households increased, showing that the RPS effectively protected the household food security of beneficiaries during this period. Beneficiaries also showed a significantly greater variety, as well as quality, of foods consumed.
Beneficiary households showed significantly higher incidence of participation in infant and child health monetary programmes, as well as a variety of health practices, including check-ups and vaccinations. Children in beneficiary households also showed a significantly higher level of school attendance and continuation in classes. The programme showed no negative impact over work effort of either adult men or women as a result of receiving the transfers (IFPRI, 2002).
PETI
PETI was evaluated in 1999, in three states, using a quasi experimental design. In each state 200 eligible households were randomly selected from three treatment and three control municipalities, for a total of 3 600 observations. Treatment and control communities were not randomized (as the evaluation was conducted ex post). Control municipalities were eligible for participation in PETI but had been held back due either to budget limitations or an initial refusal to participate in the programme. Only a single cross section of data was collected. The analysis found that in each state covered in the evaluation, PETI increased the number of hours beneficiary children spent in school, with no negative impact for non-participating children. However, the results show that while PETI significantly lowers the probability of a child working, it is less successful at lowering the incidence of working long hours (>10 hours). Worse, the probability that non-participating children work long hours actually increases, suggesting further specialization in child labour for these children. Finally, the results show that participating children had a lower probability of working in a risky job (Yap, Sedlacek and Orazem, 2002; World Bank, 2001a).
This section presents a number of issues related to programme implementation and impact in which further research and analysis is warranted. In terms of conditional cash transfer programmes, research has focused on quantitative impact. Given the copious amounts of data collected by these programmes, much can still be done in terms of quantitative impact and cost benefit analysis. This is particularly true in the area of the relationship between conditional cash transfers and productive investment and moving the discussion from the analysis of average impact to a discussion of which kinds of households, and under what circumstances, these programmes have a bigger impact. Other areas of potential research lie primarily in the area of qualitative or institutional analysis and have important implications for design and implementation. These topics range from the tradeoffs between centralization, decentralization and participatory administration; the compatibility of safety net and capital accumulation objectives; when should households “graduate” from cash or technology transfer programmes; the debate between supply and demand driven interventions; the impact on social capital and community organization; and the political economy of these programmes.
What are the tradeoffs between centralization, decentralization and participatory administration?
PROGRESA is highly centralized, with all targeting decisions and selections made in Mexico City headquarters. Transfers are made directly to household beneficiaries, bypassing all local intermediaries. Given the history of social programmes in Mexico, which has been characterized by corruption, lack of transparency and political manipulation, particularly at the local level, this was a logical choice. By law, Bolsa Escola and the other Brazilian programmes are administratively decentralized and are run by municipalities under a common set of general rules and federal financing. Neither Bolsa Escola nor PROGRESA provides much space for civil community actors to participate in the design or administration. On the other hand, the Cartão Alimentação programme in Brazil gives a significant role to local authorities and civil society in targeting and administering the programme, while the supply-side intervention in PRAF-II also provides a role. Here, community members form local Quality Improvement Teams in the health centres and Parents Associations in the primary schools and are charged with drawing up mission and vision statements, developing these into strategic plans and annual plans and submitting budgets for the projects they want to carry out each year.
While popular participation and decision-making are often considered the ideal, not all local participation is optimal. Lack of democracy may allow local elites to capture gains; local cultural beliefs may conflict with progressive social policy; and even in the presence of democratic structures, exclusionary policies such as machismo or racism against a minority may exist. This is a particular concern with the institutional arrangement being used to gather local participation in the Fome Zero programme. Further, decentralization of administration leaves more space for the possibility of unequal implementation across implementing units, as was found recently for PETI (Presidencia da Republica, 2002). Again, the challenge is determining the proper balance between government and local initiative and control and between different levels of government.
Similar debate is found with the design and implementation of social investment funds. For example, open menus of eligible interventions provide more choice and perhaps a better match between community preferences and investments. The drawback is that community preferences may not be in line with national priorities and, depending on the community, may be prone to capture by the better-off for their own benefit. Second, community preferences may not be technically or financially feasible. Third, national objectives in terms of success indicators and community preferences may not coincide. For example, a community may place high priority on improving the physical characteristics of an existing school, without necessarily affecting the enrolment rate, which may be the national measure of success (Rawlings, Sherburne-Benz and Van Domelen, 2001).
What are the advantages and disadvantages of centralization, decentralization and participatory processes in terms of transparency, efficiency, equality and financing? A real challenge facing countries with a political and institutional history like Mexico and Brazil is how to assure transparency and freedom from political manipulation while at the same time taking advantage of a potential local contribution. The appropriateness of centralization versus decentralization seems to depend on such factors as the administrative structure; the legacy of corruption and public transparency; the legacy of political use of social programmes; and the nature of the specific programme. The decision to choose one system or another is thus empirical, depending on the particularities of each country. Research needs to be done comparing experiences across countries to discover what works and what does not.
Are safety net and human capital accumulation functions compatible?
The compatibility of safety net and capital accumulation (broadly defined) functions is open to debate. This compatibility has two dimensions. First, a basic contradiction exists between short-term safety net and human capital functions; one programme cannot be the “best” possible solution for both.[37] Safety nets are intended to provide short-term support to vulnerable households in times of personal or more general crisis. While they are permanent programmes, households are to receive benefits only for a short, specified period of time. Human capital development, by definition, is long-term, ideally covering over a generation. This human capital development can be considered part of a long-term safety net or social insurance that protects chronically vulnerable households. But in any case the short-term and the long-term nature of these programmes come into contradiction most importantly on the question of exit rules.
Basically, the issue comes down to whether households should be removed from the programme when they are no longer poor, in terms of current well-being status (usually measured by income or consumption). Or, instead, should they be removed when they have completed with the human capital-building objectives (such as children finishing school), regardless of their status of well-being? Most programmes promise the benefits for a specified period of time, for example three years, after which participation may or may not be curtailed. This pragmatic response skirts the dilemma, of course, as a family may still be poor, and certainly will not have developed sufficient human capital, after three years. Ultimately, these considerations raise questions as to the sustainability of programme impacts and behavioural changes after the programme ends, as well as how to maximize the long-term impact of the programme, given cost constraints. One possible solution, employed in Brazil, is to combine the fixed time-limited PETI with a complementary programme, PRONAGER, which focuses on supporting income-generating opportunities for the household.
Further, measurement of poverty in these circumstances is problematic, as income and consumption are difficult to measure without error, are costly to collect and create incentives to cheat. A certain level of error is acceptable when analysing averages in a household survey, but not when deciding inclusion in a welfare programme. Using human capital-building criteria may be easier to verify and conceptually pleasing, but leakage may become a problem, and, as families may stay on longer, might be more expensive. As a number of the programmes come to maturity and enter a second round of implementation, a comparison of exit experiences across countries would be very revealing.
Are safety net and “productive” capital accumulation functions compatible?
Most conditional cash transfer programmes are linked to the accumulation of human capital in both mind and body, through education, health and nutrition. This type of productive capital takes years - sometimes a generation - to develop. Concerns have been raised regarding the relationship between conditional cash transfer programmes and the accumulation of productive capital for the here and now; that is, capital, such as land or non-agricultural assets, which lead to increased income in the short term. Can and should consumption goals and capital accumulation goals coexist within the same programme? In the current policy debate in Nicaragua it is argued that only in such a fashion can current transfers be made sustainable. While it is hoped that children will be better prepared for the labour market when they are older, productive investment of the transfer will allow the family to sustain the impact of the cash transfers, which cannot continue indefinitely.
The results from recent research (Davis et al., 2002 and a review in Peppiatt, Mitchell and Holzmann, 2001) show that even the extreme poor receiving PROGRESA and or PROCAMPO transfers may spend some part of their transfer on self-employment activities. This is an area that merits further analytical work as well as design conceptualization: how transfers are currently being used for investment and how the productive effect of these transfers can be maximized under different conditions. Such research would be of particular interest to the current debate on cash transfers in Brazil, where the new Fome Zero programme restricts spending of the transfer to food, thus precluding direct investment.
The debate between supply-driven and demand-driven interventions
A common debate in countries implementing conditional cash transfer programmes is over-providing incentives to increase household demand for public services (like schooling) versus providing more (and better-quality) supply of public services as a means of inducing greater participation. While most conditional cash transfer programmes are based on inducing demand directly via cash, they all recognize the importance of the latter. It does little to increase school attendance if the quality is mediocre. Often, in fact, supply and quality of supply suffer as a result of emphasis on demand.
Part of the problem stems from the fact that supply and demand are often controlled by different ministries, leading to problems of coordination and competition over spheres of influence. Some programmes assure funding by providing institutions with resources for each beneficiary; in some cases, such as the RPS health bono, the family chooses the health provider. The evaluation of PRAF-II is the only one that specifically looks at the issue of demand-side versus supply-side intervention in the design of the randomized experiment. Another way to look at this issue is that of the synergy between different components of a programme, such as health and education. This is a topic that merits attention both from an institutional as well as behavioural perspective.
What is the impact on social capital?
Given the characteristics of beneficiaries of these programmes (mostly poor, small landholders, often indigenous), it is very relevant to analyse the impact on social capital. In a number of the programmes, fomenting social capital formation is an explicit part of the intervention design. But does it work? Do the interventions - none of which involve pre-existing community political, social or cultural institutions - foment or reduce social capital?
Social participation varies from negligible to a requirement for benefits. But in most cases, targeted conditional-transfer programmes have not been successful in terms of fomenting local participation and social capital and in some places have been counter-productive. For example, PROGRESA targeting of individual households has in some cases created social conflict in communities. Households rarely understand - and often cannot discern - why some households are included and others not. This differentiation may lead to resentment and jealousy. Such separating out of beneficiaries is particularly egregious in more traditional indigenous communities, which have a strong communal tradition. Social capital in some cases has been weakened: where previously a whole community would participate in public works, now only PROGRESA households will do so. This cleavage can undermine the social fabric of a rural community. What is the overall impact of these different conditional cash transfer programmes on social capital and the social relations of a community?
What are the political economy considerations of rural policy reform?
The political economy of rural development policy constitutes a very interesting area of research. Political viability - or the political economy of policy making - has important consequences for feasibility, design and implementation of cash transfers programmes. The shift from universal subsidies to targeted transfers typically represents a hard political sell in most countries. Funding new programmes of this type, particularly under the jealous gaze of traditional health and education ministries, is an enormous challenge. Conditional cash transfers have come under increasing attack in a number of countries. Criticisms range from the political to the institutional, and without organized constituencies, the future of these programmes may be in doubt. Did political realities force design changes in these programmes, and if so, have these changes negatively affected the impact of the intervention? What has been the impact of political considerations on the functioning of these programmes? For example, political constraints required that PROCAMPO cover subsistence as well as modernized farmers and required beneficiaries to grow something (even if not profitable) on their land. Thus PROCAMPO has some poverty impact, but is not an optimal anti-poverty programme, and has some compensatory impact, but is not an optimal NAFTA compensation programme. Spelling out linkages between design, political economy and impact, as well as describing the political debate and history behind each of these programmes, can provide valuable lessons for the design and implementation of current and future programmes. A few efforts have been made along these lines, including Scott’s (2003) history of the development of PROGRESA, a forthcoming paper on PROGRESA by Santiago Levy and Evelyne Rodriguez, Aguiar and Araujo’s (2002) history of Bolsa Escola and Arcia’s (2002) interpretation of the fiscal and political challenges faced by the RPS programme in Nicaragua.
What has been the role and impact of gender considerations?
One salient feature of conditional cash transfer programmes is the channelling of transfers to women. As mentioned in the introduction, recent literature shows that when women are given money they are more likely to spend it on good consumption then men are. While this conventional wisdom provides a logical justification for such an arrangement, it is not a forgone conclusion that it is the gender of the beneficiary - and not the conditionality - that have brought about all the good impacts shown in the evaluations presented above. The design of interventions thus far has made it very difficult to separate out analytically the gender from the conditionality effect, because very few men receive the transfer, and those that do may be fundamentally different from males in general (e.g. they may be widowers). One comparison of male and female PROCAMPO beneficiaries with female PROGRESA beneficiaries shows that conditionality may be as important, or more so, than gender. This is a very promising area of research.
A second line of research involves looking at the change in power relationships within the household, as well as within communities, as a result of providing the transfer to women. Such an analysis was undertaken in the PROGRESA evaluation (see Adato et al., 2000, for example), but this topic cries out for comparative analysis across countries.
These types of analysis are important because different gender objectives may require different intervention mechanisms. For example, if transfers are conditional, and the primary objective is reducing poverty, then it does not appear to matter to whom the transfer is given. If transfers are not conditional, then the literature says to give to women. Further, if the objective is to maximize the household income-generating potential of the transfer, then the money should be given to whoever controls assets within the household. If the objective is to empower women, then obviously the transfer should go to them. But if the objective is really to change the status of women within the household in a more structural sense, perhaps women should be given access to productive assets (beyond their labour power) as well.
Many possibilities for empirical analysis
Finally, the wealth of data being produced by the evaluations of conditional cash transfer programmes leads to almost limitless possibilities for empirical analysis relevant to policy. The PROGRESA evaluation data has been heavily worked from many angles, both in terms of quantitative and qualitative impact analysis, evaluation design, simulation of programme design changes and targeting and cost-effectiveness. Many of the same issues can be researched with the new programmes, a number of which are quite liberal in their policies regarding access to data. However, while we are beginning to learn quite a lot about average impact over all the poor or extreme poor, we still know very little about whether it is specific kinds of households in particular that benefit more, and under what circumstances. And as repeated often above, we are reaching the point where comparative analysis across programmes would be quite revealing of how impact changes with country context and design differences.
Adato, M. 2000. The impact of PROGRESA on community social relationships. September. Washington, DC, IFPRI.
Adato, M., de la Brière, B., Mindek, D. & Quisumbing, A. 2000. Final report: The impact of PROGRESA on women’s status and intrahousehold relations. July. Washington, DC, IFPRI.
Aguiar, M. & Araujo, C. 2002. Bolsa-Escola. Education to confront poverty. Brasilia, UNESCO Brazil Publications.
Arcia, G. 1999. Proyecto de la red de protección social. Focalización de la fase piloto. Washington, DC, Interamerican Development Bank (IDB), May. (Consultant report)
Arcia, G. 2002. Impacto fiscal de la red de protección social - requerimientos y perspectivas financieras para su expansión. Managua, Government of Nicaragua, June. (Consultant report)
Baker, J. 2000. Evaluating the impact of development projects on poverty: a handbook for practitioners. Washington, DC, Directions in Development, World Bank (also available at http://www.worldbank.org/poverty/library/impact.htm).
Behrman, J. & Todd, P. 1999. Randomness in the experimental samples of PROGRESA (Education, Health, and Nutrition Programme). March. Washington, DC, IFPRI (available at http://www.ifpri.org/themes/progresa/methodology.htm).
Behrman, J., Sengupta, P. & Todd, P. 2001. Progressing through PROGRESA: An impact assessment of a school subsidy experiment. April. Washington, DC, University of Pennsylvania and IFPRI (available at http://www.ifpri.org/themes/progresa/education.htm).
Bolsa Alimentação. 2003. Webpage, January (http://portal.saude.gov.br/alimentacao/bolsa_alimentacao/index.cfm).
Bolsa Escola. 2002. Bolsa Escola Federal. The first year. Report on Activities, 2001 (available at http://www.mec.gov.br/bolsaescola/estrut/serv/resultado/default.asp).
Coady, D., Grosh, M. & Hoddinott, J. 2002. Targeting outcomes redux. FCND Discussion Paper No. 144. Washington, DC, IFPRI.
Colegio de Ingenieros Agrónomos de México. 2001. Estudio sobre la observancia de las normas de operación de PROCAMPO, los beneficios económicos y sociales y su costo efectividad. México, DF, CIAM.
Davis, B., Handa, S. & Soto, H. 2001. Crisis, poverty, and long-term development: Examining the Mexican case. FAO, IDB and PROGRESA, August. (mimeo)
Davis, B., Handa, S., Ruiz, M., Stampini, M. & Winters, P. 2002. Conditionality and the impact of program design on household welfare: Comparing two diverse cash transfer programs in rural Mexico. ESA-FAO Working Paper Series No. 7 (available at http://www.fao.org/es/ESA/work-e.htm).
de Janvry, A. & Sadoulet, E. 2002. Targeting and calibrating educational grants: Focus on poverty or on non-enrollment risk? Berkeley, University of California Berkeley (mimeo, available at http://are.Berkeley.EDU/~sadoulet/).
Dubois, P., de Janvry, A. & Sadoulet, E. 2002. Effect on school enrolment and performance of a conditional transfers program in Mexico. (mimeo, available at http://www.toulouse.inra.fr/centre/esr/CV/dubois/duboisen.htm).
FAO. 2001. Acuerdo para la ejecución del proyecto evaluación de Alianza para el Campo entre el Gobierno de Mexico y la Organización de las Naciones Unidas para la Agricultura y la Alimentación (FAO). FAO Evaluation of Alianza para el Campo, UTF/050/MEX/050. Rome.
Fome Zero. 2003. Web site (http://www.fomezero.gov.br/).
Fox, V. 2002. Informe de gobierno, 2002. September (available at http://informe.presidencia.gob.mx/Informes/2002Fox2/website/cfm/index.cfm).
Gonzalez, A. & Davis, B. 2002. Diseño quasi experimental de evaluación de impactos. FAO Evaluation of Alianza para el Campo, UTF/050/MEX/050. Rome, FAO.
Gonzalez, A., Konig, G. & Wodon, Q. 2002. Do cash transfers to farmers reduce migration? PROCAMPO in Mexico. Washington, DC, World Bank, July. (mimeo)
Government of Nicaragua. 2001. Estrategia reforzada de crecimiento económico y reducción de pobreza. July (available at http://www.setec.gob.ni).
Gutiérrez, J., Bertozzi, S. & Gertler, P. 2002. Evaluación de la identificación de familias beneficiarias en el medio urbano. Report of the Evaluación de Resultados de Impacto del Programa de Desarrollo Humano Oportunidades. Mexico City, Instituto Nacional de Salud Pública and CIDE.
Haddad, L., Hoddinott, J. & Alderman, H., eds. 1997. Intrahousehold resource allocation in developing countries. Baltimore, USA, Johns Hopkins University Press for IFPRI.
Handa, S., Huerta, M., Perez, R. & Straffon, B. 2001. Poverty, inequality, and spillover in Mexico’s education, health, and nutrition program. FCND Discussion Paper No. 101. Washington, DC, IFPRI.
Heckman, J., Ichimura, H. & Todd, P. 1997. Matching as an econometric evaluation estimator: Evidence from evaluating a job training programme. Review of Econ. Studies, 64: 605-654.
Heckman, J., Ichimura, H. & Todd, P. 1998. Matching as an econometric evaluation estimator. Review of Econ. Studies, 65: 261-294.
Heckman, J., Lalonde, R. & Smith, J. 1999. The economics and econometrics of active labor market programs. In A. Ashenfelter & D. Card, eds. Handbook of Labor Economics. Volume 3. Amsterdam, Elsevier Science.
IDB (Inter-American Development Bank). 2001. Family Allowance Program - Phase II. Washington, DC.
IFPRI (International Food Policy Research Institute). 2001a. PRAF: The Family Allowance Program. Washington, DC, September (available at http://www.ifpri.org/themes/praf.htm).
IFPRI. 2001b. Final Report. Consultancy to support the design and evaluation of Bolsa Alimentação. Washington, DC, September.
IFPRI. 2002. Informe final. Sistema de evaluación de la fase piloto de la red de protección social de Nicaragua: Evaluación del impacto. Washington, DC, July.
Ilahi, N., Sedlacek, G. & Gustafsson-Wright, E. 2000. Targeted conditional transfer programs in Latin America: An early survey. Paper prepared for the Regional Study on Economic Insecurity. Washington, DC, Office of the Chief Economist, Latin America and Caribbean Region, World Bank.
Jalan, J. & Ravallion, M. 1999. Income gains to the poor from workfare: Estimates for Argentina’s Trabajar program. Washington, DC, World Bank. (mimeo)
MNPTSG (Mesoamerica Nutrition Program Targeting Study Group). 2001. Targeting performance of three large-scale, nutrition-oriented, social programs in Central America and Mexico. Food and Nutrition Bulletin, 23: 162-174.
Newman, J., Rawlings, L. & Gertler, P. 1994. Using randomized control designs in evaluating social sector programs in developing countries. The World Bank Research Observer, 9(2): 181-201, July.
Parker, S. & Scott, J. 2001. Evaluación del Programa de Educación, Salud y Alimentación (PROGRESA) a partir de: Indicadores de seguimiento, evaluación y gestión 1998-2001. Encuestas de evaluación 2000. Mexico City, CIDE, October.
Peppiatt, D., Mitchell, J. & Holzmann, P. 2001. Cash transfers in emergencies: evaluating benefits and assessing risks. Humanitarian Practice Network Paper No. 35. London, ODI.
Presidencia da Republica, Government of Brazil. 2002. Programa de Erradicacao do Trabalho Infantil - PETI: Resultado consolidado nacional. Brasilia, September (available at http://www.presidencia.gov.br/cgu/ProgErrad.PDF).
Presidencia da Republica, Government of Brazil. 2003. Decreto No. 4 675, de Abril de 2003. Brasilia, April (available at http://www.presidencia.gov.br/ccivil_03/decreto/2003/D4675.htm).
PROGRESA. 1998. Metodología para la identificación de los hogares beneficiarios del PROGRESA. Mexico City (available at http://www.progresa.gob.mx/e_oportunidades/evaluacion_impacto/IFPRI/IDEAPE01.PDF).
Ravallion, M. 1999. The mystery of the vanishing benefits: M.s Speedy Analyst’s introduction to evaluation. Policy Research Working Paper No. 2153. Washington, DC, Development Economics Research Group, World Bank (available at http://www.worldbank.org/html/dec/Publications/Workpapers/wps2000series/wps2153/wps2153.pdf).
Ravallion, M. 2002. Target transfers in poor countries: Revisiting the trade-offs and policy options. Washington, DC, World Bank. (mimeo)
Rawlings, L. & Rubio, G. 2003. Evaluating the impact of conditional cash transfer programs: Lessons from Latin America. May. (Draft mimeo)
Rawlings, L., Sherburne-Benz, L. & Van Domelen, J. 2001. Letting communities take the lead: A cross-country evaluation of social fund performance. Washington, DC, World Bank, September.
Raymond, M. & Sadoulet, E. 2002. Educational grants: Closing the gap in schooling attainment between poor and non-poor. University of California at Berkeley. (mimeo, available at http://are.Berkeley.EDU/~sadoulet/).
Ruiz, M., Stampini, M., Davis, B., Winters, P. & Handa, S. 2002. More calories or food diversity: The impact of PROGRESA and PROCAMPO on food security in rural Mexico. ESA-FAO Working Paper Series No. 6 (available at http://www.fao.org/es/ESA/work-e.htm).
Sadoulet, E., de Janvry, A. & Davis, B. 2001. Cash transfer programs with income multipliers: PROCAMPO in Mexico. World Dev., 29(6): 1043-1056.
SAGAR (Ministry of Agriculture, Livestock and Rural Development). 1998. PROCAMPO, 1994-1998. Claridades Agropecuarias, 64: 1-40, December.
Scott, J. 2003. PROGRESA: Contexto y relevancia. Rome, FAO, January. (mimeo)
Skoufias, E., Davis, B. & de la Vega, S. 2001. Targeting the poor in Mexico: An evaluation of the selection of households into PROGRESA. World Dev., 29(10): 1769-1784.
Skoufias, E. & McClafferty, B. 2001. Is PROGRESA working? Summary of the results of an evaluation by IFPRI. FCND Discussion Paper No. 118. Washington, DC, IFPRI.
Skoufias, E. & Coady, D. 2002. Are the welfare losses from imperfect targeting important? FCND Discussion Paper No. 125. Washington, DC, IFPRI.
Tabor, S. 2002. Direct cash transfers. Social Safety Net Primer Series. Washington, DC, World Bank Institute.
Thomas, D. 1990. Intra-household resource allocation. An inferential approach. Journal of Hum. Res., XXV(4): 635-664.
World Bank. 2000. Nicaragua agricultural technology project. Project appraisal document. Washington, DC, Latin American and Caribbean Region Office.
World Bank. 2001a. Eradicating child labor in Brazil. Report No. 21858-BR, Washington, DC.
World Bank. 2001b. Brazil: An assessment of the Bolsa Escola programs. Report No. 20208-BR, Washington, DC.
Yap, Y., Sedlacek, G. & Orazem, P. 2002. Limiting child labor through behavior-based income transfers: An experimental evaluation of the PETI program in rural Brazil. June. (mimeo)
|
[23] The author is an
economist with the Agricultural and Development Economics Division of the FAO.
Thanks go to Saul Morris, Luis Gomez Oliver, Maria Grazia Quieti and Paul Levin
for detailed comments, with all errors and omissions the responsibility of the
author. [24] For a review of the experience of social investment funds, see Rawlings, Sherburne-Benz and Van Domelen (2001). For other recent papers on conditional transfers see Rawlings and Rubio (2003) and Ilahi, Sedlacek and Gustafsson-Wright (2000). [25] Coady, Grosh and Hoddincott (2002) provide a comprehensive review of targeting mechanisms. [26] Tabor (2002) and Peppiatt, Mitchell and Holzmann (2001) review the experience of cash transfers in non-emergency and emergency situations. [27] See, for example, Haddad, Hoddinott and Alderman (1997) and Thomas (1997). [28] The FAO Regional Office in Latin America and the Caribbean has recently begun a working group on incorporating impact evaluation into rural development projects in the region. [29] In practice, however, less then one percent of all PROCAMPO land forms part of an official environmental management programme. [30] A history of the development of the Bolsa Escola concept and the implementation of the programme can be found in Aguiar and Araujo (2002), as well as the Bolsa Escola Web page (http://www.mec.gov.br/bolsaescola/estrut/serv/historico/default.asp). [31] Here we focus only on evaluation, and do not discuss programme monitoring, which covers the continuous feedback on the status of programme implementation. [32] Good descriptions of impact evaluations can be found in Baker (2000), Rawlings, Sherburne-Benz and Van Domelen (2001) and Ravallion (1999). See also the World Bank website on impact evaluation, http://www.worldbank.org/poverty/impact/index/htm. [33] Newman, Rawlings and Gertler (1994) discuss some historical examples of randomized design along with a discussion of practical considerations to be taken into consideration when considering such an evaluation. [34] See Heckman, Lalonde and Smith (1999) for a thorough discussion of this topic. [35] Behrman and Todd (1999) evaluate whether the randomization was done correctly. [36] All the reports written for the IFPRI evaluation are available at the IFPRI Web site, http://www.ifpri.org/themes/progresa/progresa_report.htm. [37] This can also be looked at from the perspective of the social assistance versus social insurance functions of a safety net. The assistance function seeks to relieve short-term deprivation, regardless of the long-term impact on physical, human or social capital and thus on development. By contrast, the insurance function is designed with a long-term impact in mind and aims to protect people against the poverty and food security risks associated with income shocks. |
