8.5 Formal surveys: value, design, and implementation

Contents - Previous - Next

8.5.1 Efficient Use of Formal Surveys

Informal surveys can make an important contribution to the design of the more expensive, time consuming, formal surveys. However, these formal surveys are still cheaper and less time consuming than direct measurement. Formal surveys involving the use of questionnaires provide a systematic, ordered way of obtaining information from respondents and enable precise and statistically analyzable data to be obtained. Earlier (see Figure 7.1 ), it was indicated that formal surveys can be divided into single (i.e., one-shot) or few- and multiple- (i.e., frequent) visit surveys. The two major issues concerning the use of formal surveys are to ensure that:

• The benefits from such surveys justify the costs involved.
• Turnaround is reasonably quick in teens of incurring the costs and reaping the benefits.

Multiple-visit surveys with multiple objectives can yield useful information, but the research resources required -in terms of level and time -and the slow turnaround time in terms of the results not being available until much later' means that they have limited value in feeding, in a timely manner, into new research initiatives.

Therefore, the above issues have been addressed increasingly, in the following ways:

• Emphasizing special subject surveys with limited objectives rather than more general multi-purpose surveys.
• Implementing single-visit rather than multiple-visit surveys.
• Carefully considering how accurately individual variables need to be estimated in order to answer the objectives of the survey. Relative rankings may be sufficient, or if accurate measurements are required, a direct measurement rather than using a survey format may be more appropriate. Yet another alternative may be to use 'standard coefficients, obtained from other work for variables that are not critically important or are fairly predictable.
• Making sure the links between data collection and checking, followed by microcomputer data entry, processing, analysis, and writing are made as efficient as possible,

BOX 8.5: TAKE CARE WHO PROVIDES EVALUATION CRITERIA IN MATRIX SCORING

In assessing different varieties of sorghum in Southern Africa, the respondents indicated that, apart from palatability, the criteria farmers used to evaluate sorghum varieties related to yield stability. Yet sorghum breeders place a great deal of emphasis on yield level plus some degree of yield stability. When gender issues were introduced? females evaluated each of the criteria the group had selected as equally important, but the males evaluated palatability as most important and the need to do bird scaring (i.e., an activity done by women and children) as least important! There are obvious implications of using this method in helping breeders evaluate what criteria might be important to farmers in assessing varieties. The method also could be used in getting farmers to evaluate the criteria already in use in breeding programmes.

BOX 8.6: PRA TECHNIQUES CAN FACILITATE RECOMMENDATION DOMAIN IDENTIFICATION

A farm typology that is relevant to the key issues at hand is necessary to develop recommendation domains (Section 4.5) that are appropriate, PRA techniques, sometimes in combination with other methods. have been used to identify local people's own criteria for such important issues as wealth or poverty and their perceptions on how to measure differences between households within the community,

Informants were selected by an FSD team from among farmers of Kiponzelo and Kihanga farming communities in Tanzania to help with determining relevant criteria for measuring wealth as it relates to agriculture [Ravnborg, 1992], In consultation with village members, it was decided that the household would be the relevant unit of analysis for this determination.

Informants were asked first to map agro-ecological zones in the study areas, Next, names of households were written on pieces of paper, Each informant independently segregated names into different piles, If especially large piles appeared, the informant was asked to try and make divisions within just that pile. At each turn, the informant was asked to characterize the households of a pile: how they are similar among themselves and different from those of other piles. This exercise generated a set of qualitative criteria that might be used to distinguish households.

The FSD team then developed a questionnaire survey that addressed quantitative measures of the qualitative descriptive criteria that had been identified, Cluster analysis was applied to data from this questionnaire, and four categories of households were identified and characterized, Agriculturally wealthy households used improved seeds, applied chemical fertilizers, were middle aged, had large household labour pools, and consumed as well as sold relatively large quantities of maize and beans. These criteria were more important than the agro-ecological zone where the farms were located, The poorest households used fewer recommended inputs, tended to be younger or older, or were headed by single individuals. Two intermediate farm types also were characterized.

With a well organized system, it is possible to design, implement, process, analyze, and write up the results of a reasonably long, single-visit, special-subject survey of 100 households in a period of three months, half of which could be devoted to the design and implementation stages.

However, key pre-conditions for making this possible are the design of an efficient questionnaire and the ease with which a sample can be selected. Designing questionnaires, generating the sample frames, determining the required sample sizes, and selecting samples are all important ingredients in determining the value of formal surveys. These are discussed in the following sections,

8.5.2 Designing Formal Surveys

There is a logical sequence to producing a good questionnaire. The process of designing good questionnaires can be divided into six steps as follows:

• Determining Data Needs. It is first important to determine why the survey is required (i.e., justification) and, therefore, the objectives of the survey, Good definition of these then will help guide what topics need to be covered and, therefore, what the data needs are. Where information on both the technical and human environment are to be collected, determination of the data required and the design of the questionnaire itself should be a collaborative effort of technical and social scientists. Data needs should be viewed not only in terms of variables on which information is required but also in terms of the degree of precision with which they need to be measured. Making a decision at this planning stage on the type of analysis proposed can help in determining the answer to this and the cost of collecting the data.
• Determining Question Content. Three important issues to address with respect to question content are:

• Appropriate identification information and variables should be placed at the beginning of the questionnaire.
• Variables need to be included that enable the sample to be classified or stratified appropriately. Inclusion of possible classification variables in the questionnaire is important for meaningful comparative analysis to be carried out later.
• Obviously, questions must be developed for each of the variables in the hypotheses, and a determination must be made of the farmers, ability and likelihood of answering the questions, For example: questions must use terms and units of measure (e.g., for weight, area, distance, etc.) with which the farmers in the area are familiar; questions should not be used that require calculations (e.g., average bags of sorghum per hectare) by the farmers; and special care should be taken over questions that are sensitive.

• Determining Question Format. Four categories or classes of questions generally are used:

• Open-ended questions where the interviewer writes down the response in full.
• Close-ended (i.e., multiple choice) questions where the enumerator checks the appropriate response category.
• Dichotomous questions where only two responses are allowed (e.g., yes/no, sell/consume, etc.).
• Tabular questions where a question is asked after which rows and columns in a table are completed.

All have a role to play (e.g., dichotomous questions as a lead into close-ended questions, open-ended questions when it is difficult to develop a list of categories before the survey, tabular questions when there is a lot of comparable quantitative data, etc.).

• Determining the Wording of Questions. Good wording is critically important to avoid biased or inadequate responses to questions. Some examples are:

• Every question should focus on one point and have only one answer.
• Questions should be specific and not contain vague words such as many, often, and frequently.
• Every question should use terms the farmers commonly use rather than technical terms of the FSD workers.
• Every question should be phrased neutrally to avoid biasing the response.
• Questions should be phrased so that the respondent cannot determine which answer is preferred.
• Questions should specify the relevant time period for consideration.
• Some questions (e.g., on management practices) often can be clarified by increasing the similarity in the way a number of related questions are asked.
• Frequently, a pre-qualifying question may be necessary to verify that the question of interest applies to the respondent,
• Questions on management practices should be asked with respect to individual plots. A tabular question often works well in such cases.
• Questions that require accurate answers for aggregate values will be more reliable, if the amounts are first collected on an individual basis (e.g., weekly, monthly, or per parcel basis) and then summed by the FSD workers during the analytical stage,
• Each question should be numbered to aid in the processing of data. With respect to facilitating the collection-entry-analysis linkage, it is important to try to enter the data into the microcomputer straight from the survey form. This cuts down the possibility of errors in the collection-entry stage. To facilitate entry, it is good to record the name of the variable -- as it appears on the microcomputer -- on the survey form itself during the design stage and to provide a space for entering the figure that will appear in the computer (Box 8.7).

• Deciding on Question Sequence. Questions should be presented in a way that is logical from the farmer's standpoint, starting with the simple, more general questions and proceeding to the more specific, difficult, and sensitive areas, For some data, it may be advisable to check the validity of the farmers' responses by asking the question in more than one way.
• Physical Layout and Length. The beginning of the survey form, in addition to including identification and classification type information and data, also should have, where desirable, information that the interviewer can give the farmer on the purpose of the survey, who is being interviewed, etc. Any instructions on completing and coding the form for entry into the micro computer also should be given at the beginning of the survey form or at relevant points on later pages. Sufficient space for recording responses and any remarks should he provided on the survey form itself. In general, the length of interview should not exceed one hour. If it takes longer, the questionnaire should be divided and administered on separate days.
• Pretesting and Revision. After the initial design of the survey, it is worthwhile to ask interested and knowledgeable individuals for comments and suggestions for improvements. After making any changes that appear desirable, it is a good idea to pretest it with a limited number of farmers, after which changes can be made prior to its reproduction and implementation. These types of checks can help in:

• Determining whether questions are properly worded, understandable, sensitive, irrelevant, etc.
• Checking questions for adequacy of format and sequencing, adequacy of categories in close-ended questions, etc.
• Identifying missing information, estimating time requirements for the interview. testing the training and understanding of interviewers, etc.

BOX 8.7: IMPROVING THE EFFICIENCY AND ACCURACY OF DATA COLLECTION AND TRANSFER

Formal surveys often are implemented by relatively inexperienced enumerators, whereas transfer of the data to microcomputer databases commonly is done by fairly unskilled clerical staff. Close-ended questions can help to reduce ambiguity data collection, whereas mistakes in transferring data to the microcomputer can be reduced by having the acronym under which the variable appears in the database, entered on the data collection form itself. An example is as follows:

(6). What type of row planter do you own'?

1. Sebele Row Planter
2. Sebele Plough Planter TYRP^a________
3. Safim Row Planter
4. Other (Specify)____________

a. This is the acronym for the type of row planter that appears in the database.

8.5.3 Sampling for Formal Surveys

There would be no need to worry about a sampling procedure, if the characteristics of all members of a population were exactly the same. It would be necessary only to select one individual to identify the population characteristics. However, because of diversity in the technical and human environment, it is necessary to sample several members of the population before any conclusions can be drawn. Therefore, the purpose of sampling is to select a subset

BOX 8.8: WATCH FOR BIASES IN SAMPLING FRAMES

The problem with many sampling frames is that they usually are drawn up for particular purposes and thus may be biased, especially if they don't represent the whole population, It is important to be aware of the possible biases when sampling frames are used. Sutherland [1988], based on extensive experience in Zambia, has identified four common types of biases, These are as follows:

• Middleman bias. When an extension agent or local leader provides a list or recommends a group of farmers, there is often a bias in favour of more progressive farmers, male-headed households, and friends or family of the middleman.
• FSD team characteristics, FSD teams may have internal biases resulting in a gender bias (i.e., favouring either male or female participants), a language bias, or an innovator bias, Also, there may be a bias because teams have a preconception, inappropriate definition, or a lack of under standing about what constitutes a farmer and/or a household,
• Logistical factors. These may be the most difficult biases to avoid and involve a bias towards farms with cropped areas near roads and a bias towards working with progressive farmers in order to show results.
• Local circumstances. Other biases can arise from local circumstances depending on the ecology, geography, and social structure of the village. There may be a bias for 'home-centred farmers', when farming is a seasonal activity, and away from farmers who leave the area for certain periods of the year in order to find of the population that has the same characteristics as the whole group or is representative of the population. The term population refers to all of the elements -- such as farms, households, etc. -- from which the sample actually is selected, whereas a sample is a representative portion of the population under study.

The objective of sampling is to undertake statistical tests and, as a result, be able to say (i.e., predict) that the results of working with all the farms/households in the population would give the same results. The sampling process requires five activities:

• Specification of the Sampling Unit. Examples would be farming households, fields of a specific type, etc.).
• Preparation of an Adequate Sample Frame. This constitutes a list of the units from which to select the sample (e.g., lists of farmers kept by extension staff; a list of people receiving food at schools/ clinics during a drought relief programme, lists of farmers participating in government production campaigns, a list of households associated with a community development project, list derived from sources outside the village, for example, a census listing, etc. (Box 8.8). Preparation of sampling frames can be very demanding (i.e., in terms of time and money). An approach that has great promise and is relatively cheap is to use PRA techniques (see Section 8.4.4), particularly mapping, in which villagers themselves can give relevant information concerning the families whose dwelling units are given on the map they have drawn.
• Selection of the Sampling Method. These consist of two major types, namely probability (i.e., random) and non-probability (i.e., non-random). Probability sampling methods consist of simple random sampling, stratified random sampling, systematic sampling, cluster sampling, etc. When sampling frames are not available, then non-probability sampling methods have to be used such as purposive sampling and quota sampling. Probability sampling is preferable because statistical testing is then more valid. It is desirable in sampling that strata correspond to the tentative recommendation domains (see Section 4.5). Many references are available on sampling methods, and, consequently, there is no need to discuss them in detail in this manual (see, for example, Worman et al 1 1992: pp. 137-14()1 and Dillon and Hardaker [ 1993: pp. 47-501).
• Determination of the Sample Size. This is a complicated process and often requires the help of a statistician. A number of factors influence the sample sizes used in FSD. These include: variability of local farm conditions, degree of precision required, available time and research resources, type of data handling facility, details and complexity of the questionnaire, etc. It is important to understand that the appropriate sample size depends on the variability in the population and not on the size of the population. Therefore, the percentage of farming families that must be included may vary substantially between recommendation domains. It has been found that 3() to 5() farmers for each recommendation domain usually will reflect reasonably well the circumstances of farmers in that recommendation domain [Byerlee et al, 1980]. Others have suggested a minimum sample size of 20 from each sampling category [Yang, 1965: p. 9: Shaner et al, 1982: p. 471
• Selection of the Sample. Once the preceding considerations have been taken into account and related activities accomplished, the selection of the sample can proceed. As discussed earlier (see Section 5.6.2), to ensure the cooperation of the leadership, their involvement in the actual selection process may be desirable,

In conclusion, it is important to remember the following points when making decisions about sampling:

• Be practical.
• Sample design and logistics of field work are often complementary.
• Statistical desirability and practical feasibility often conflict.
• Knowledge of the area and the subjective judgement of FSD workers is crucial in selecting villages and samples.
• Use the simplest procedure -- in terms of costs and resources -- that will permit the achievement of the FSD goal.
• Remember, biased selection of farmers will give rise to biased answers and conclusions.
• Both FSD workers and extension staff can be guilty of selection biases.
• Use probability sampling whenever possible, but in any case, always be aware of the limitations of the method that is used.

8.6 Direct physical survey measurements

Discussion in this chapter has focused on surveys involving interaction with people, However, surveys do not necessarily have to involve people, For example, FSD teams often employ direct physical survey measurements to obtain baseline and progress information that contribute to other aspects of FSD work, Basically, such physical types of surveys fall under the 'direct measurement' -- and perhaps the 'observation' -- category indicated in Tables 7,1. and 7.2, because they usually require direct measurement techniques. Consequently perhaps, physical surveys should not be considered in this chapter, which concentrates on indirect measurements that usually are associated with surveys. However, because they do constitute a special type of survey, they are dealt with briefly at this point of the chapter.

• FSD tasks that often benefit from these physical types of measurements are:
• Making comparisons with scientific data from outside the FSD target area,
• Developing sample-frames for locating trials or other studies,
• Formulating the range and the severity of problems in the descriptive and diagnostic stage of FSD.
• Forecasting impacts that interventions might have.
• Contributing to the interpretation of subjective data obtained through farmer surveys and often providing a groundwork for dialogue with participating farmers,

Direct physical survey measurements can provide more precise quantitative information about certain aspects of the farming system that arise from indirect farmer surveys, However, this level of precision is not always necessary, and for each activity, the FSD team must determine the level of precision required, In general, direct measurements are most useful for describing objects and events of the farming systems' natural resource base: weather events, soil conditions, biological cover, and so forth. In addition, direct measurements often will be made for inputs and outputs when describing cropping and livestock management.

Many physical measurements are fixed to land and calendar coordinates and, thus, can be used to help define space-time frameworks for FSD analyses, Because of problems in managing large data sets, these types of frameworks have been under-utilized in FSD.

Techniques used in these measurements are frequently the same standards used by scientists in ecological and production research. Following are several suggestions for applying direct physical survey measurements in FSD:

• The FSD team, often in consultation with experts and with participating farmers, must determine which measurements are relevant and cost effective to the analyses planned. In making this decision, consideration should be given to both short-term and long-term planning of FSD work.
• The FSD team must be assured that staff members have the skills for making these measurements in a manner consistent with established standards. The FSD team must also ensure a consistency in the way any single measurement is made across project activities, Achieving this consistency may require training for staff in the procedures used (e.g., rating weed levels, determining soil texture by 'feel' method, etc.).
• Even with training, bias can occur in the way measurements are scheduled, and this will distort the objectivity of direct measurements. Remember, essentially all objective measurement techniques depend on some amount of human judgement. It would be inappropriate, for example, for one staff member to rate weed cover -- which can be somewhat subjective -- in one village area, and another staff member to make this rating in a second, and then plan to do statistical analyses comparing these village areas.
• FSD teams will find that many standard direct measurements are too time consuming and costly to be carried on the scale necessary for project work. FSD teams should constantly be seeking proxy variables that represent the variable(s) at issue but can be measured more quickly and at less cost. Proxy variables are only as good as their relationship with the variables that are important. For example, in some environments, earthworm counts have been found to be good indicators of soil health variables such as soil porosity, aggregation, and organic matter content. Developing or validating appropriate proxy variables is an excellent subject area for collaboration between FSD and scientists based on experiment stations. Another benefit in identifying good proxy variables, is that these sometimes serve later, as well, by helping farmers and dissemination agents monitor the impact changes are having on farms.
• The use of direct physical survey measurements should not be allowed to compete with indirect farmer surveys. FSD teams may find advantages in asking participants to help assess the need for various measurements. Results should be reported back to FSD participants. Exchanging notes in this way between what farmers observe and believe with relevant scientific measurements -- from both surveys and trials -- will bolster the opportunity for identifying problems and solutions.
• Even with this potential synergism, FSD teams can readily over-extend physical measurement work. Teams might experience some difficulties in completing data collection, but a more common problem is the management of data and integration of its analysis into the broader FSD picture. Developments in data management technology will help ease some of these problems in the future. Very powerful relational database management (RDBM) programmes are becoming increasingly available that combine analysis and graphics to present results in meaningful ways.
• Geographic Information Systems (GIS) technology, a further extension of RDBM, is also emerging as a tool with potential applications in FSD. GIS programmes and projects exist in many countries and zones where FSD teams operate. However' because work on developing initial layers of information for GIS is demanding and time consuming it probably should not be part of the work load of an FSD team. GIS programmes that are already developed and include relevant information for the target area on soils, weather, vegetation, demographic, etc., could be used and added to by FSD.

9. Trial methods (direct measurements)

9.1 Objectives

The objectives of this section are to discuss:

• The main types of trials used in FSD, the purposes of each, and some of the issues related to their design and planning,
• Some of the issues related to trials designed to address crops and livestock.
• Some of the issues involved in trial implementation, including selection and involvement of farmers and the specific use of farmer groups.

Contents - Previous - Next