Pre-test surveys and pilot censuses are a critical part of the census planning process; they provide the opportunity to test all aspects of the census programme in advance of the main census activities, thus ensuring the smooth operation and success of the census.
Pilot surveys should test the census methodology (including the sample design in the case of a sample census), the questionnaires, data collection methods, the training programme for field staff, instruction manuals, data editing and coding, data processing and data tabulation.
The results of the pilot census should be processed to test the programs in the data processing system. A complete set of tables should be produced using the data from the pilot census to evaluate the tabulation programme.
The effort put into conducting a pilot census is wasted if results are not available in time for the efficient planning of the main census. Critical issues and problems encountered in field testing must be corrected well in advance of the start of the main census.
Suggestions for changes in the material tested, and in procedures and methods followed in the pilot census, should be included in the report(s), and will form the basis of the census operation. They should, therefore, be examined and evaluated by all authorities concerned with conducting the census. The national agricultural census committee should take into account all these suggestions before finalizing the operation of the main census.
Introduction
13.1 An agricultural census is a complex and expensive operation consisting of a series of closely related steps which must be planned carefully in advance. When census field work starts, all procedures, and documents such as questionnaires, should already have been carefully checked and tested as mistakes or problems discovered at the time of enumeration cannot easily be corrected. It is necessary to assure that such mistakes or problems are discovered and corrected by carrying out a programme of pre-tests and a pilot census .
13.2 The first step is to make a systematic study of all activities that will be involved in an efficient census taking operation. If a census has been conducted in the past, and well-documented material relating to the census operation is available, this will help considerably in planning the present census. This is why it is important to prepare a good Technical Report at the end of every census (see Chapter 18). The experiences of personnel who worked on the previous census are also invaluable in planning the present census. Since the census is usually conducted after a long interval (5 or 10 years), many technological as well as socio-economic changes have taken place between the two censuses. Past experience alone may not be adequate for planning the current census and it is necessary to plan a programme of pre-tests and pilot censuses to study the various steps which are important in a census operation. The size of this programme may range from very small pre-test surveys intended for checking specific problems, to one or more large pilot censuses which are a final test and a rehearsal for the full census. While the terminology relate specifically to a full-scale census, the pre-testing and pilot exercises are equally important in any statistical survey.
Pre-test surveys
13.3 Before planning a pilot census, the conduct of a series of pre-test surveys is highly desirable. The objective of the pre-test surveys should be confined mainly to the formulation of concepts and definitions, census questionnaires, instruction manuals, etc., and the evaluation of alternative methodologies and data collection techniques. Pre-test surveys differ from the pilot census in that they are usually relatively small-scale exercises and the selection of respondents is often not on a random basis. Instead, in a pilot census a good cross-section of respondents is chosen in a systematic manner with consideration being given to ease of enumeration and quality of response. However, in large countries and where methodological considerations need to be fully evaluated, a pre-test survey may need to be conducted on a fairly wide-scale and rigorous basis.
13.4 The pre-test survey is particularly important for the formulation and wording of the questionnaires. This task is often entrusted to a group of agricultural census and survey experts. This group should be made responsible for testing the suitability of questionnaires in actual field conditions. Obviously, such tests should be conducted under varying socio-agro-economic conditions and the results of the tests should be submitted to the national agricultural census committee (see Chapter 2). The report should give concrete and constructive suggestions on the revision of questionnaires, etc. It should, in particular, emphasize the alternatives of the questionnaires which either need to be abandoned or revised. It should critically examine every question included in the questionnaire from the point of view of (i) the reaction of the respondents and quality of information furnished in the answers; (ii) the reaction of the interviewer and difficulties they faced in extracting the information; and (iii) usefulness of every question from the point of view of data obtained and tabulation planned.
The pilot census
13.5 The pilot census, unlike the pre-test surveys, is a 'dry run' for the main census but on a limited scale. It should evaluate all aspects of the census operation including the concepts and definitions, the adequacy of the questionnaires, the training of field enumerators and supervisory staff, field organization, census methodology, sampling design and estimation procedure, data processing and data tabulation. The results should be used when drawing up the final plans for the census and to provide a basis for the final calculations of resource requirements for the census.
13.6 The timely organization of the pilot census is strongly recommended. Through pilot censuses facts are obtained and experience gained. Without the pilot census, activities are not based on facts and are more or less qualified judgements. Pilot censuses represent a means for achieving rational designs through which the objectives of the census can be accomplished. The decision to take one or more pilot censuses will depend on the agro-climatic and socio-economic conditions of the country. An important principle of the pilot census is the wide application of the design. The design must be prepared in such a way that it is possible to derive a large variety of conclusions. If the census is being conducted for the first time and is on the basis of a sample, the pilot census should be organized in such a way that it offers the possibility to estimate sampling errors for a number of alternative designs as well as to evaluate their cost. Pilot censuses should be taken under realistic circumstances. In other words, all the possible conditions which are likely to be faced in the main census should be reflected in the pilot census. It must cover the divergent situations existing in the country. A well-organized pilot census will help to improve the efficiency of the main census. A certain portion of the total census budget should be earmarked for pilot studies. The pilot census should be large enough not only to finalize the questionnaires, concepts and definitions, but also to provide adequate information for determining the resource requirements (budget, personnel, transport, etc.), method and mode of tabulation, timetable, various types of biases and errors likely to occur in field data, etc.
13.7 A well-conducted pilot census must provide adequate technical inputs for improved planning of the main census. The effort put into conducting a pilot census is wasted if the results are not made available in time for the efficient planning of the main census. A critical report on the pilot census must be available well in advance of the start of the main census. The report should mention the main objectives, the sample design, and the various stages of planning and implementation of the project. Even though the results of the pilot census may not be meaningful, tabulations and derived tables should be produced to test the process through the final stage. Suggestions for changes in the material prepared for the pilot census and in procedures and methods followed should form part of the report. The suggestions given in the report will form the basis of the census operation for the country. They should, therefore, be examined and evaluated by all authorities concerned with conducting the census. The national agricultural census committee (see Chapter 2) should take all these suggestions into account before finalizing the operation of the main census.
Concepts and definitions
13.8 The first step in planning a census is to develop appropriate concepts and definitions which should be communicated to the census enumerators for the collection of data. Although the work done in the past by the census organization, or similar work done elsewhere, can be the basis for the preparation of a preliminary draft of concepts and definitions, its application to actual conditions will need verification. These should be tested in pre-test surveys by census enumerators responsible for collecting the data in the main census. For example, in the agricultural census, the definitions of agricultural holding, agricultural production, holder, total area of the holding, tenure, area rented, land utilization, net area sown, current fallow, uncultivated and waste land, etc., should be formulated in such a way that they can easily be collected by the census enumerators. Since the tenure system and agricultural practices vary considerably from province to province, it may not always be possible to adopt uniform definitions for the whole country. However, every attempt should be made to so, or develop these differences in such a way that they can be aggregated for national level estimates. Some explanation, which may vary from one province to the other, may be necessary. Any weaknesses in the concepts and definitions will affect the final results. Therefore, the census organization must include appropriate concepts and definitions developed through pre-tests (see Programme for the World Census of Agriculture 2000).
Estimating resource requirements
13.9 One of the main purposes of pre-test surveys and pilot censuses is to provide elements required for preparation of the census budget and work-plan (see Chapter 3). A fairly clear idea of what is needed should exist at a very early stage. Information is usually available from previous censuses. If the census is being organized for the first time, information from other surveys or other countries may be used. Such information should be improved based on actual field experience as soon as possible as precise data are required to prepare the work-plan and budget.
13.10 As enumerators constitute the major component of the census labour force, the pilot census should provide data for analyzing the time required for filling the various questionnaires by enumerators. The enumerators may be asked to record time taken to complete each field operation, such as preparation of list of holdings, contacting the holder, extracting the relevant information from the holder, etc. A critical analysis of time records will help census management staff to distribute the workload among field enumerators. It will also help in assessing the requirements of enumerators and supervisors for the main census and thereby an estimate of the cost of the enumeration phase of the census. Information on the cost of training enumerators for the pilot census will also be useful for budget planning.
13.11 Similarly, equipment requirements (transport equipment, equipment for objective measurements and cartography if applied, etc.) can be assessed in a pilot census. Testing data entry and data processing procedures with raw data will help assess the computer equipment and data entry requirements (see Chapter 17).
Frame
13.12 An appropriate frame of the census enumeration units is a major key to the success of the census operation and is essential, whether it is a sample census or a complete enumeration census.
13.13 Because of cost considerations, there may be a temptation to use some kinds of frames already available or prepared for other purposes. For example, there may be an attempt in the agricultural census to use the frame of households prepared for a population census. It is logical and economical to take advantage of the existence of such information. However, the frame needed in an agricultural census may not be readily available from information provided by a population census. Pre-test surveys conducted on an adequate scale, covering the different situations, should provide some help in evaluating how the population census frame can be used. If a new frame is to be prepared the problems become much more complicated. In practically all cases, a list of the agricultural holdings will have to be prepared for each enumeration area (or each sample block in the case of sample enumeration). The enumeration area may be a village or a segment of a compact geographical area. A clear identification of each enumeration area is essential to prepare an accurate list of agricultural holdings. Recognition of the boundaries of each enumeration area is often difficult. Enumerator mistakes usually create under-listing of units. Such under-recording will be more common in situations where the census is to be conducted on a sample basis with the primary sampling unit being the segment of area or a village. The demarcation of the boundary is more difficult in areas which have not been cadastrally surveyed. The problem may be similar in hilly and remote areas where households are scattered. Many such problems have been discussed in the chapter dealing with frames (see Chapter 6). All such difficulties should be solved through pre-test surveys or pilot censuses.
Questionnaire
13.14 A major source of error is improper wording of questions on the questionnaire due to persons responsible for designing the questionnaire not being sure of the meaning of various definitions and concepts used in the census programme. The questionnaire designers may assume that the holders know everything about the census. It is often not taken into account that holders belong to a different class or education level and may not be able to follow the vocabulary used in statistics. In cases where questions are ambiguous and are not explained to the respondent, errors can be made. A considerable amount of discussion and experimentation is essential before the questionnaire content and wording is finalized (see Chapter 8).
13.15 A series of pre-tests should be organized after the draft questionnaire is ready. Holders should be used in the pilot census to determine whether the enumerators use the concepts and definitions in a uniform manner, whether respondents understand the questions, whether the order of the questions is acceptable, and how long it takes to obtain the information. Questionnaire designers can also act as interviewers or can observe the interviews conducted by the staff dealing with data collection. The presence of an observer may influence the behaviour of both the holder and the enumerator, possibly distorting the results of the interview; however, this influence may be preferable to not having any observers as enumerators concentrating on recording data may miss some details. It is very important that specialized staff have an opportunity to observe how respondents react to the inquiry and how they and the enumerators are able to understand the various terms and concepts used in the census. The main objective of this test should be to finalize the concepts and definitions, the arrangement and sequence of the questions, the appropriateness of the language, format of the questionnaire, spacing between two questions, adequacy of space for writing answers, etc. It should be possible to finalize the various aspects of the questionnaire by interviewing a relatively small number of holders.
13.16 Further tests may be carried out in different agricultural zones of the country. Staff who are likely to be employed as enumerators or supervisors should be used as interviewers and the technical staff act as observers. Enumerators and observers should be asked, on the basis of their interviews to give their opinion on the questionnaire. The opinions of the interviewers and observers should be jointly discussed to finalize their comments on the questionnaire. Sometimes, instead of testing only one questionnaire, two or three alternative questionnaires which have different formats or which include different items or which formulate the questions in a different way are tested. On the basis of the results of the test it would be expected to find the most practical questionnaire or format.
13.17 Another part of testing questionnaires is ensuring their suitability for data processing. For this purpose, personnel responsible for data processing should be involved not only in preparation but also in testing the questionnaire in order to propose and evaluate different alternatives. Ideally, the instructions for data processing operations (data entry, coding, editing, etc.) should already be prepared in detail so that the suitability of the questionnaire for processing and the validity of a planned data processing operation can both be tested. It is also very important to measure the time required to enter data from an average questionnaire as this information is needed to plan the required number of data-entry stations and related staff. Often, data processing specialists indicate that they can process almost everything, but in practice it is very important to test their ability to organize timely and smooth processing.
13.18 As a final test of the questionnaire, a pilot census may be used to reproduce all conditions under which the census will be taken. In general, no special observers are used and only supervisors, as a part of their normal duties, will observe some interviews and give an assessment.
Training of enumerators and supervisors
13.19 Training will normally be in phases (see Chapter 12). In the first phase, the field supervisory staff responsible for the agricultural census operation and for conducting the training of the field enumerators will be trained. Since the supervisors are entrusted with the supervision of the field work of a large number of enumerators with different backgrounds, their training must be very intensive and thorough. The supervisors must be trained to become good enumerators. They must attain a thorough knowledge of agricultural census operations so that they are in a position to remove the doubts and difficulties of the enumerator. Pilot censuses are an excellent opportunity to provide on-the-job training to all supervisory staff. After the pilot census it can be determined if the supervisors' training was effective and what changes have to be made before training enumerators.
Methods of data collection
13.20 An agricultural census operation involves the collection of data on a large number of items. Most of the characteristics on which data are needed are of a quantitative nature. In most developing countries holders do not keep records of their holding. They may not even know in definable units of measure the amount of land they operate, particularly in areas where cadastral records do not exist. Collecting data through an interview will have many limitations under such circumstances. Often, the holders have no quantitative concepts, and even if they do, many of the agricultural operations are such that the holders fail to recall accurate information and consequently errors are introduced into the census data. For example, agricultural labour is an item on which reliable information usually cannot be obtained with a single interview. Similarly, data on production of agricultural commodities may not be readily obtained in one visit, particularly if the holder cultivates several crops spread over the entire production year. It is not easy to obtain data on the number of trees, or on age and species of livestock numbers, etc., through simple oral inquiries. In fact, different methodologies of procuring the data from holders will have to be developed and this will depend on understanding the socio-economic status of the holder. Pre-tests surveys should provide guidelines on the methodology to be adopted to collect data on different specific items.
13.21 Pre-tests should provide the information necessary to identify those items in the agricultural census for which data can be obtained through interview with specific reference dates. There may be some items for which there cannot be a reference date but only a reference year. There may also be items on which accurate information can only be obtained by using objective methods of measurement. Some data items, such as crop areas, are difficult to collect on a complete enumeration basis and cost considerations may not favour such an approach. In these situations it will be necessary to conduct sample surveys. One of the main objectives of pre-test surveys should therefore be to find appropriate measurement techniques and define the types of enquiries that can be used in the main census.
13.22 When the agricultural census is based on a sample, the results are subject to sampling errors. These errors can be minimized by employing an adequate sample size and a suitable sampling design. There are, however, other errors called non-sampling errors which affect all census results whether they are based on a complete enumeration or a sample. Many studies have shown that non-sampling errors can be quite serious, and can affect the results of the census to such an extent as to distort the picture of the agricultural situation presented by the agricultural census. Therefore, in planning and conducting the agricultural census, maximum attention needs to be paid to devise procedures for keeping non-sampling errors to a minimum and the additional cost incurred should be balanced against the increased reliability of the census results and their acceptance by users.
13.23 During collection of census data, both enumerators and holders contribute to errors. It is necessary to know, through pilot studies, the types of errors that these two groups of individuals make. Once the weaknesses of the enumerators are identified, it should be possible to remedy them through careful preparation and efficiently structuring the questionnaire and instruction manuals, by training enumerators, providing adequate facilities and incentives for their work, and by exercising close supervision.
13.24 It is more difficult to control the respondent's bias. In most developing countries holders are frequently illiterate, often lack a quantitative understanding of their agricultural operations and cannot interpret the questions correctly. In the absence of any record-keeping, they are not able to give reliable information for operations spread over the whole year from memory only. There may be other serious reasons why holders are unwilling to report correct information. They may show apparent cooperation with the census enumerator but have an ingrained suspicion and fear about the inquiry and the use to be made of the information given. In countries engaged in land reform programmes, fear in the minds of the holders is more intense. They may also be superstitious and fear that disclosure of precise information about their assets may bring bad luck. Through pre-test surveys such biases should be carefully studied and appropriate solutions obtained.
Sampling design and sample size
13.25 When an agricultural census is to be carried out on a sample basis, the pilot census can test the efficiency of the sample design and provide guidance on the optimum sample size.
13.26 The determination of sample size is a complex problem (see Chapter 7). The census is basically a multi-purpose inquiry. Data are collected on a large number of characteristics of the agricultural holding. The size of the sample will largely depend on the objectives of the census and the accuracy of the estimates of the different desired characteristics as well as the level of geographic detail required. It is sometimes possible to make some rough estimates of the variability among statistical units, and therefore of the sample size, from past surveys and censuses. When past data are not available the pilot census should provide information on variability and cost of enumeration, and will provide appropriate guidelines for planning the main inquiry.
13.27 The sampling design often adopted for an agricultural survey is a stratified two-stage design with villages or area segments as primary sampling units and agricultural holdings as secondary sampling units. Field data from pre-tests or pilot surveys may be very useful to test possible stratification and decide the number of primary and secondary sampling units to be included in the sample. Based on field data, it is also possible to experiment with different methods of estimation, such as a ratio or regression method, to arrive at the best possible estimate with the available data.
Data processing
13.28 Census data are summarized in the form of tables which provide a descriptive picture of agriculture. The pilot census data should also help in determining whether necessary tabulations can be produced easily. The tabulation plan of the pilot census data should be a miniature of the main census. A careful tabulation of the pilot census data would also point out the deficiency of the questionnaire with respect to coverage of items needed.
13.29 If the census is based on a sample, a proper estimation procedure will have to be adopted. There are several improved estimation procedures which can be used to derive the estimates of total population for each variable. Estimation procedures, such as a ratio method and/or regression method, depend on supplementary data. At the pilot stage it should be possible to examine the kind of supplementary information needed to improve the method of estimation, and once the appropriate supplementary variable has been determined, information on it can be collected as an integral part of the main census operation.
13.30 Some preliminary tabulation of individual segments or at village-level can be done by field enumerators and supervisors. The pilot census can determine what kinds of tabulations can be entrusted to field enumerators and supervisors. Often, with proper training, the work of evaluating, minor editing and coding of data may be decentralized and entrusted to field supervisors.
13.31 Processing pilot census data provides an opportunity to test various commercial software packages against computer programs prepared in the house. Also, the tests can evaluate the efficient use of microcomputers against mainframe computers and the suitability of decentralized processing in provincial offices or by private contractors. Various procedures for checking data, data entry, manual and computer data corrections, etc., must be checked with raw data. Time is a factor which should be examined while tabulating the pilot census data. If the census results are to be useful, these must be made available to the users in a timely manner. To achieve this objective, a timetable of various phases of processing census data can be formulated with the help of the pilot census. A rational decision about the mode of tabulation and the requirements of manpower and equipment can be taken on the basis of tabulation of pilot census data. At the pilot census stage it is important to consider alternative processing methods and all their implications, including speed, efficiency and cost, by preparing all tables by different methods.
