Connecting with the Unconnected: Proposing an Evaluation of the Impacts of the Internet on Unconnected Rural Stakeholders

Scott McConnell
University School of Rural Planning and Development
University of Guelph
Guelph, Ontario
Canada

Introduction

This framework for evaluating the impacts of non-governmental organizations (NGOs) with Internet connectivity on rural communities represents an attempt to address the shortcomings of previous ICT evaluations. It is not meant to be a definitive study of how to examine rural communities' information sharing with NGOs possessing Internet connectivity, but it does propose a framework from which to build upon, improve, and make as relevant as possible an understanding of the benefits and impacts of the Internet for those who lack connectivity in rural areas of developing countries. Only through application in the field can such frameworks be tested, assessed, and improved upon. It is hoped that this framework represents a fresh approach to understanding the complex issues and networks involved in information sharing and ICT use.

Background

Much of the literature on the use of information and communication technologies (ICT) concerns the potential that such investments offer for the developing world. Yet despite the rhetoric of how ICT can bridge the gap between the information haves and the have-nots, there is little research available that demonstrates that the use of ICT are in fact able to bridge that gap. Moreover, the focus of ICT research is currently fixated on the institutional level, which is composed of Internet service providers (ISPs), policy makers and governments. While the value of such research is not challenged here, the fact remains that very little investigation has been conducted concerning the impacts of ICT on those who have the most to gain through the expansion of ICT in the developing world: the rural stakeholder communities.

The benefits of expanding ICT to major urban cities has been well-documented, and the results of such urban investment can be seen in both the expansion of ISPs and the increasing connectivity throughout the cities of the developing world. The impacts on rural stakeholders and rural communities, however, are less understood (Richardson, 1996; Sirimanne, 1996; Wresch 1996; Zijp 1994). Northern development agencies have virtually ignored the development of information technology in rural areas in the South (Zijp, 1994: 14). Furthermore, the economics involved in development agencies' and corporations' willingness to expand Internet connectivity into rural areas in both Northern and Southern hemispheres has proven to be a major stumbling block. In fact, rather than expansion being based upon the desire to meet the communication needs of rural communities, rural ICT expansion is primarily based on the degree to which it will be a profitable and worthwhile venture for the communication companies involved (Richardson, 1996). Such sentiment is supported by Michel Menou in his book, Measuring the impact of information on development:

Information development support seeks a balance between the development of infrastructure and the satisfaction of the information needs of the poorest segments of the population. The latter, like rural populations, are mainly reached by indirect methods, usually through extension services. When resources are scarce, choices between infrastructure and serving the poor are often in favour of the former.

(Menou, 1993: 32).

The exclusion of rural communities and stakeholders from ICT expansion occurs in spite of the benefits that ICT is intended to bring to the isolated areas of the world. In What about the developing countries?, Shamrika Sirimane writes that ICT is a tool which can be used to narrow the information and communication gap between rural communities and urban centres through, "giving rural people access to valuable information; and by transmitting indigenous information and locally produced knowledge" (Sirimane, 1996: 6). Richardson writes that the Internet "offers a means for bridging the gaps between development professionals and rural people through initiating interaction and dialogue, new alliances, inter-personal networks, and cross-sectoral links between organizations" (Richardson, 1996:1).

Existing ICT evaluations and the position of rural stakeholders

The exclusion of rural stakeholders from ICT expansion mirrors the dramatic lack of research being undertaken to measure the benefits that such investments are able to provide for rural communities. A notable exception to this situation is Kingo Mchombu's 1996 report, Impact of information on rural development: background, methodology, and progress. In it, Mchombu outlines the mid-project results of INFORD 1 and 2, a two-phased project funded by the International Development Research Centre (IDRC) that monitored six rural communities throughout Southern Africa to investigate "the provision of information to support rural development and the impact of such information on development" (Mchombu, 1996: 2). This project assessed the general and specific information needs of the individual communities, and then examined the extent to which these needs were being met through various rural development activities employed in and around those communities. The evaluation's intention was to establish "if the information provided has led to development gains in the community"; a follow-up report has yet to be published (Ibid.: 11).

Although the Mchombu report researches the origins and impacts of information on rural communities, it does not specifically address the use of the newest form of ICT: the Internet. The irony is that recent reports that are more specifically focused on the Internet offer very little in the way of addressing the impacts of such technologies on rural communities. Ricardo Gomez's report, Electronic Agora or Disneyland democracy? (1997) examines the utilization of the Internet by Colombian non-governmental organizations connected to the Association for Progressive Communication (APC) network, and deals solely with the perceptions those NGOs have about their use of the Internet; no references are made at all to rural stakeholders in this report (Gomez 1997).

Both the Canadian International Development Agency (CIDA) and IDRC published reports in 1997 which were intended to provide indicators and results of the use of ICT in projects throughout the developing world. However, while data are available on impacts and results of ICT on groups that have connectivity, neither of these reports address the issues and needs of the rural stakeholders who have ties to organizations with access to the Internet. In short, those without connectivity have been as excluded from ICT research as they have from ICT expansion.

The Canadian International Development Agency's (CIDA) 1997 report, ICT and development: testing a framework for evaluation, volume 1, set out to address four core characteristics related to ICT use in CIDA-sponsored projects throughout Asia and Africa: information, borderless connections, timelines, and improving costs and benefits (Young et al. 1997). The report uses a framework which provides much attention to how ICT have been adopted within its projects, but places little emphasis on what the impacts of such connectivity means for those rural stakeholders without direct connectivity to the Internet. Indeed, of 30 key questions identified in the CIDA Evaluation Framework, only two address the extent to which stakeholders were in-volved in the projects. Unfortunately, CIDA's definition of stakeholder appears not to include those without Internet connectivity (Ibid.:10).

Even with the inclusion of the two questions relating to local stakeholders' acceptance, support, and ability to assume ownership of ICT, the CIDA report offers vague, insufficient analysis of the rural impacts of ICT. This is evidenced in the presentation of a case study of a Peruvian rural telephone expansion project; the report states that the project's immediate impact is an increase in telecommunications access for 200 000 people. Beyond this statistic, however, the report was unable to identify any of the direct and indirect impacts of this project at the community level (Ibid.: 19). That CIDA's report is deficient in its approach to rural communities and stakeholders is further demonstrated by the fact that the evaluation research was completed without the involvement of any stakeholder representatives; CIDA desk staff and NGO partners were contacted by phone, fax, and Email, with no research done in the field (Ibid.: 27).

The second of the two ICT evaluation reports to emerge in 1997, Use of information and communication technologies in IDRC projects: lessons learned, was written by Michael Graham and published by IDRC. This evaluation had the objective of identifying ways in which ICT might be better used to enhance people's lives. The report raises questions to address issues related to thirteen IDRC-funded projects, and, like its CIDA counterpart, discusses results only in terms of those stakeholders who have connectivity; no mention is made of rural stakeholders who are without ICT access. Of the eight evaluation questions posed in IDRC's report, only one concerns ICT in relation to its use and adoption by all stakeholders (Graham 1997: 8). Moreover, the IDRC report, like the CIDA report, was written without any field research being conducted and without the involvement of rural stakeholders (Ibid.: 11).

That both CIDA and IDRC, two of Canada's strongest players in terms of funding International Development and ICT research, should neglect the presence of rural stakeholders without connectivity in their evaluation reports is both perplexing and worrisome. What is the cost of ignoring the benefits and impacts reaped by rural stakeholders that are affiliated with NGOs with connectivity? Does this indicate that international development agencies such as IDRC and CIDA believe that an organization's potential with Internet connectivity ceases to have impacts beyond the organization itself? And if researchers overlook the links being made between rural communities and NGOs with connectivity, how will the powers that control and influence ICT infrastructure and investment be made aware of whether or not the technology is indeed assisting them? As Richardson states in a recent paper, "All stakeholders must encourage development agencies to engage in collaborative efforts that lead to activities and projects focused on rural and agricultural development" (Richardson 1997a: 3).

The need to focus on rural stakeholders

There is a need to refocus ICT research so that the communication processes occurring with the use of the Internet are examined, rather than the infrastructural components themselves. This is supported by Menou (1993), who states, "The whole communication cycle, from generation to assimilation, has not been studied in specific institutions or communities of developing communities" (Menou 1993: 68). Don Richardson, in his paper prepared for the Food and Agriculture Organization (FAO) The Internet and rural development: recommendations for strategy and activity, makes the case for increasing the focus on those who live on the fringe of ICT expansion, and against focusing all research on those with direct access to the connectivity. "Research on impacts," he writes, "cannot only focus on users and local applications, but also focus on people who do not participate directly in local Internet initiatives ... and suggest mechanisms for enabling them to benefit directly or indirectly from these initiatives" (Richardson 1996: 24).

Through learning how communities without Internet connectivity are able to benefit from organizations that have connectivity, and through learning how the two entities can work together to ensure that information is received, shared, and put to good use, those in development can move towards guaranteeing that stakeholders' needs are being met, and that adequate support is provided to those who need it most. If the benefits of the Internet are not made available to the rural peoples who can benefit from it the most, "then we are failing to fully leverage the large infrastructure investments involved, and we are failing to assist people in making appropriate decisions based on such valuable information" (Richardson 1997a: 5).

Participatory development communication and ICT

The concept of participatory development communication is central to examining the use of the Internet as it relates to rural community stakeholders. Its basic tenet is that, "the communication process is more important than the production of media products" (Richardson 1997b: 1). Participatory communication for development is a process that emphasizes a two-way communication process based on sharing ideas and information in order to improve the quality of life of the stakeholder communities (Bessette 1996). Unlike traditional com-munication methods of disseminating and transmitting information, development for communication "is about aiding different types of actors interested in understanding needs and assessing opportunities jointly; it is about providing them with the methods and media to reach common meaning, and about enabling them to negotiate with other actors with contrasting perceptions and interests" (Ramírez 1997: 3).

The Internet is the latest in a long line of communication tools that enable a community to receive information and assistance from outside organizations. Unlike technologies such as radio, newspaper, and video, however, the Internet, "is the first media tool that allows every user to be a sender, receiver, narrowcaster and broadcaster" (Richardson 1996:9). The tremendous opportunities that such technology can offer to rural stakeholder communities must not be missed. The challenge, however, is ensuring that the Internet is utilized for two-way communication processes such as communicating local knowledge and perceptions. Existing pilot projects in Chile, Mexico, and Southern Africa exemplify rural stakeholders' ability to successfully adapt to the opportunities offered by the Internet (Richardson 1997a: 2).

Röling (in Ramírez 1997) identifies three roles of communication which communication for development could play in natural resource management: making things visible through the creation of new perspectives, fostering policy acceptance, and facilitating platform processes involving all stakeholders (Ramírez 1997: 5). Ramírez sees this third role as "the less explored and more promising dimension of communication," where the focus is on "the orchestration of platforms of negotiation among multiple actors" (Ibid.). In working with these platforms of negotiation, "a true demand-capacity from the grassroots is able to match and select relevant information," thus ensuring that the rural communities' needs can be better served (Ibid.: 10).

Towards a new evaluation approach

As illustrated in the existing literature, current evaluations tend to ignore those rural stakeholders who are without connectivity. The need exists for an evaluation approach that will specifically examine the impacts of ICT on development at a grassroots level. Michel Menou writes of the importance of understanding the grassroots' impacts from the Internet when he states that "lower level data are needed to interpret higher trends" (Menou 1993: 37). He goes further in suggesting the importance of focusing on this segment of the population when discussing the issue of equality of access to information. Such access, "must also be considered in relation to its effectiveness for the worst-off segments of the population, such as handicapped people or geographically scattered rural populations. Appropriate information resources and telecommunication facilities made available to remote rural communities may contribute to improving their economic performances and well-being ..." (Ibid.: 47).

Richardson, too, endorses a refocusing of research and evaluation energies related to the local levels of development when he states: "Maybe it is better to use our resources to help analyse existing grassroots communication processes and knowledge networks, in conjunction with short-term, low-cost interventions that catalyse new participatory communication processes within these existing processes and networks" (Richardson 1997b: 3).

The time to focus on the processes that accompany technological tools such as the Internet is now. Without understanding how ICT such as the Internet are impacting those at the grassroots level, we risk remaining unaware of the effectiveness of technological investments that occur at the institutional level. Given the need for such an evaluation, this author proposes a new framework to look exclusively at the extent to which Internet use by local NGOs benefits the unconnected rural stakeholder communities that they serve. This evaluation will utilize a traditional process of measuring the effectiveness, efficiency, and impacts of the NGOs' use of the Internet in relation to the rural communities. The process can be considered traditional in the sense that without the "essential" interim measurements of effectiveness and efficiency, the measurement of impact might not be possible (Mchombu 1996: 4).

In measuring the impact of ICT on rural communities, this evaluation will focus on the process, rather than the end result, of information exchange between rural stakeholder communities and local non-governmental organizations. The author is currently undertaking research involving case studies of between two to four local, non-governmental organizations (NGOs) in Uganda, East Africa that are working on rural issues with rural communities. The methodology will include a combination of participant observation and key informant interviews with key actors in the stakeholder communities, NGO extension workers, and NGO staff to construct a realistic picture of how rural communities are able to benefit from Internet use. Using a model similar to that proposed by Tandon (in Mchombu 1996), this research hopes to follow a process of inquiry and intervention, in which both the researcher and the subjects not only learn from one another, but also learn from the very situation that they are a part of and engaged in studying (Mchombu 1996: 5). The research will collect both quantitative and qualitative data, though the very nature of evaluating the process of information sharing will place more of an emphasis on the latter.

Efficiency

The extent to which the NGO's use of the Internet allows it to provide the greatest amount of assistance, services, and information to stakeholder communities using the least amount of input factors.

Effectiveness

The extent to which the Internet enables the NGO to locate information that can be used in its programmes with rural stakeholder communities.

Issue: Impacts

The success with which the NGO's objectives for providing assistance, service, and information to rural, unconnected stakeholder communities are being met as a result of its own Internet connectivity.

The framework

Efficiency

This section will examine the extent to which the local NGO's use of the Internet allows it to provide the greatest amount of assistance, services, and information to stakeholder communities using the least amount of input factors.

Quality of Internet service: background

This section will examine the extent to which the organization finds its Internet system to be reliable and easily accessible, relative to the tech-nology that was in place prior to its arrival. The importance of this first section of the evaluation is evidenced by Michel Menou, who states that, "Lack of access to networks and databases is itself a measure of inequality ... because it prevents access to information and the chance to benefit from it" (Menou 1993:46). To provide further information on the degree to which the Internet is accessible to the NGO, frequency of Internet use within the organization will also be measured.

Indicators for the reliability of the Internet service include assessing whether the NGO's Internet service offers users any advantages to the staff over other means of communications such as post, fax, and telex. Advantages can be quantified in terms of less transmission time required for messages, reduced transmission costs, and faster turn-around time for replies. Frequency of Internet use can be quantified in general terms such as total number of bits transmitted and received, and total number of messages transmitted and received per day, per week, and per month.

Methodology: Open-ended interviews and participant observation can be used to gather the background data required for this section.

Capacity of NGO to acquire, disseminate, and respond to Internet information

This component will examine the number of individuals involved in disseminating information from the Internet to the community, and the length of time required for information to travel from the NGO to the stakeholder communities. These components of information capacity are directly linked to an organization's ability to manage and absorb information received on the Internet (Menou 1993, Zijp 1994). If the organization lacks the capacity to sort, identify and utilize the information that is available to it, then the organization's use of that system will be severely undermined.

To assess the efficiency of the dissemination process, the number of individuals involved in this process will be measured against the total number of individuals working in the organization. The length of time required for the dissemination process will be measured in terms of the elapsed time from the reception of Internet-acquired information at the organization level, to information dissemination at the community level.

Methodology: Open-ended interviews and focus groups can be used to understand the networks involved in the dissemination process. Techniques used for rural appraisal of agricultural knowledge systems (RAAKS), such as linkage matrices to determine which stakeholders have contact with whom, linkage diagrams to determine which of the key actors have influence over others, and linkage webs to represent levels of resource and decision control can also be used to conceptualize the links present in the dissemination process (Engel and Salomon 1986; Rogers and Kincaid 1981). The tremendous value of these techniques is stated in a 1995 FAO report on communication neworks among farmers in the Philippines: Mapping linkages in a knowledge system uncovers information exchange mechanisms (FAO 1995: 4).

Economic costs of the Internet

This component addresses both the real and actual costs of the Internet for the organization. Indicators for this component include assessing the infrastructure costs of computers, training, and Internet connectivity. These values can then be measured in terms of the organization's ability to meet the costs on their own, without external donor support. If the organization's connectivity is being subsidized by outside organizations or agencies, the real costs of the service can be used to determine the degree to which the use of the Internet is affordable for that organization, and for other organizations of a similar size. The annual cost of the NGO's Internet service can also be measured against the annual budget for the organization to determine whether such a service is indeed affordable.

Methodology: Open-ended interviews with key NGO staff, and inspection of annual budgets will be used to assist in the examination of the economic costs of Internet connectivity.

Effectiveness

This section will examine the extent to which the Internet enables the NGO to locate and deliver appropriate information that can be used in its programmes with rural stakeholder communities.

Appropriateness of information

This component examines whether information originating from the Internet and targeted for rural stakeholder communities is appropriate to the stakeholder's needs; whether the information acquired on the Internet is integrated into the programming of the organization, itself; and assesses the types of information which are received and sought after by the NGO. The perceptions of key actors within the rural community as to whether the information that is provided to them is useful will be used as an indicator of the degree of appropriateness. Indicators related to the integration of information into the organization's own programmes include examining whether changes which are implemented into the organization's programmes come about as a result of information acquired on the Internet. Types of information received by the organization will be measured in terms of whether the information being sought out by the NGO is process-oriented information that benefits the NGO in dealing with communities, such as information on Participatory Development techniques, or whether it is information that addresses the needs of the community, such as identifying which types of beans are suitable for growing in the region, or market prices for coffee.

Methodology: Open-ended interviews with both NGO staff and key informants within the community will identify communication patterns and linkages existing among the various actors in the information sharing process. Communication networks will be identified through the use of linkage diagrams, matrices and webs (refer to previous section "Economic costs of the Internet" - Methodology).

Information delivery, dissemination, and sharing process

This section of the Effectiveness component will examine the whether a two-way communication process is in place between the NGO and the community. This will include assessing whether information located on the Internet is in fact being shared and forwarded to the relevant key actors in the rural stakeholder community, and whether stakeholder knowledge is being disseminated and shared with other NGOs and rural communities over the Internet. Indicators related to the question of whether the information reaches the key actors in the stakeholder community include examining those communication networks which exist within the organization; between the organization and the community; and within the community itself. Indicators of stakeholder knowledge being shared and disseminated with the NGO, and then on to other organizations and communities over the Internet include assessing the degree to which there is evidence of information transmission from the stakeholder community to the NGO, and then on to other organizations via the Internet network.

Methodology: Open-ended interviews with both NGO staff and key actors in the rural stakeholder communities will be used to illuminate the communications environment that exists between the two bodies. The identification of communication networks as outlined in sections "Economic cost of the Internet" and Appropriateness of information" will serve to facilitate this examination.

User equity

Information Technologies (IT) can open opportunities for women and men alike by increasing their access to information, providing venues for the expression and sharing of knowledge, as well as providing training in a variety of areas (Rostagno 1997; Zijp 1996). This component will examine the issues of gender equity with respect to access to Internet services within the NGO, and the degree to which capacity building is a component of the NGO's Internet use. Indicators for these factors include the numbers of male and female staff who have access to and are able to operate the Internet in relation to total numbers of male and female staff in the organization, and numbers of male and female staff who have received Internet training.

Methodology: Open-ended interviews, focus groups, and participant observation can be used to identify the degree to which user equity exists within the organization.

Impacts

This section will assess the success with which the NGO's objectives for providing assistance, service, and information to rural, unconnected stakeholder communities are being met as a result of its own Internet connectivity.

Direct and indirect impacts

Two main categories of benefits result from access to information: Direct and Indirect (Menou 1993:44). Within these two categories, immediate and potential benefits are identifiable. Direct, Indirect, unexpected, and expected impacts resulting from the use of the Internet in the information sharing will be examined, as will the extent to which a multiplier effect is observable with respect to other organizations becoming aware of the Internet as a result of the organization's connectivity.

Indicators of direct and indirect impacts of the use of the Internet in delivering information to stakeholder communities will be dependent on each NGO's specific objectives in utilizing the Internet. Suggested direct impact indicators include the NGO's use of the Internet and its information to bring about the desired results for both the community and the organization. Indirect impact indicators may include strengthened links between the organization and the community, strengthened links within the community itself, strengthened links with other outside groups or communities, and the dissemination of information from the community to outside groups. Indicators for the multiplier effects may include the proliferation of other organizations' awareness and/or utilization of the Internet.

Methodology: Open-ended interviews with NGO staff, extension workers, and community members.

Appropriateness of information

This component will assess the degree to which the quality of information delivered to the stakeholder community has improved since the adoption of the Internet, and the degree to which the Internet has enabled the NGO to better meet the needs of the stakeholder community. Neelameghan (in Menou 1993) provides evidence of the need to look for improvements in the information being circulated when he states that, "...one benefit that should not be overlooked is an improvement in the appropriateness of the information, which information systems may achieve in transforming and transferring both indigenous and exogenous information" (Menou 1993: 48). This is further supported by Zijp (1994) who states: "It is essential to ensure that the information is of good quality, reliable, timely, and presented in a useful way. To do so, information system designers must clearly identify user's goals, the information that users need to achieve those goals, and the process by which that information will be gathered and entered into the system" (Zijp 1994:20).

Indicators related to quality of information include assessing whether the information provided to the stakeholder communities reflects their needs and objectives, as well as assessing the degree to which the com-munity has opportunities to give and receive feedback to the NGO. The opportunity for rural community feedback is an integral component in the communication for development process (Bessette 1996; Ramírez 1997; Richardson 1997b; Zijp 1994). As Zijp states, feedback oppor-tunities are important, "because rural people are not only recipients of information, but also creators of knowledge based on their own and other's experience" (Zijp 1994: 16). Indicators of the community having its needs better met include assessing whether the community's specific information needs are being satisfied by the organization.

Methodology: Open-ended interviews and focus groups with key informants from the stakeholder community, and interviews with NGO workers.

Sustainability of process

In discussing the issue of sustainability as it relates to communication for development, Richardson writes that, "it is the end result of catalysing empowerment and sustaining people's participation in their own development that ought to be the focus of our attention on 'sustainability'" (Richardson 1997b: 3). Thus, it is not so much the sustainability of the Internet service that should be the focus of this section, but rather the sustainability or replicability of the communications process between the NGO that has connectivity and the rural community that does not. Narayan (1993) defines replicability as, "the capacity to duplicate the processes and benefits of a set of development activities in new locations after their effectiveness has been demonstrated in limited geographic areas" (Narayan 1993: 95). In the context of this evaluation, replicability will refer to the duplication of processes or activities in new locations after they have been adapted in areas that receive information directly from NGOs with connectivity.

Secondary issues related to the sustainability of the process include whether the use of the Internet has become an accepted component of the NGO's work with stakeholder communities; and whether all members of the stakeholder communities benefit from the information disseminated from the Internet.

Indicators for the replicability of the information sharing process include the number of times the information sharing process has been repeated between the rural community and the NGO, and the degree to which such information sharing results in further examples of information adaptation throughout the community. Indicators related to the issue of the Internet becoming an accepted component in the information transfer process include assessing the degree to which the Internet is regularly used as an information sharing and gathering tool in its work with rural communities. Indicators measuring whether all stakeholder community members benefit from the information being shared include assessing which segments of the population are targeted in the information sharing process, and which of those targets actually receive the information being delivered.

Methodology: Open-ended interviews with key actors in the stakeholder communities, NGO extension workers, and NGO staff.

Conclusion

This report proposes a framework to evaluate the impacts of the Internet on rural communities without connectivity. Only through the process of testing frameworks such as this in the field, with real organizations and real, rural communities can the lessons learned be incorporated into a more improved framework with a stronger methodology. This evaluation framework is intended to be a stepping stone towards achieving a better understanding of what the existing research on ICT use in developing countries has been unable to tell us thus far. In discovering and understanding what the impacts of NGOs with connectivity are on rural communities that lack such technology, researchers, Internet technicians and development practitioners alike can work together to improve the networks and potential that Internet technology can offer to those who, for the time being, remain 'the disconnected'.

References

Bessette, Guy. 1996. Development communication in West and Central Africa: toward a research and intervention agenda. In G. Bessette & C.V. Rajasunderam, eds. Participatory development communication: a West African agenda. Ottawa: International Development Research Centre.

Engel, Paul, and Monique Salomon. 1996. Workshop: dare to share networking on ecologically sound agriculture. Draft Report of LEISA Working Group, Northern Ghana, March 26 - April 2, 1996. CEDEC Navrongo Workshop.

FAO. 1995. Understanding farmer's communication networks: an experience in the Philippines. Rome.

Graham, Michael. 1997. Use of information and communication technologies in IDRC projects: lessons learned. Ottawa, Canada: Evaluation Unit, Corporate Services Branch, International Development Research Centre.

Gomez, Ricardo. 1997. Electronic Agora or Disneyland democracy?. Partial results of a study on the uses of computer-mediated communication (CMC) among NGOs in Latin America.

Mchombu, Kingo. 1996. Impact of information on rural development: background, methodology, and progress. In P. McConnell, ed. Making a difference: measuring the impact of information on development. Proceedings of a workshop held in Ottawa, Canada 10-12 July 1995.

Menou, Michel. 1993. Measuring the impact of information on development. Ottawa, Canada: International Development Research Centre.

Narayan, Deepa. Participatory evaluation: tools for managing change in water and sanitation. World Bank Technical Paper Number 207. Washington, D.C.: the World Bank.

Ramirez, Ricardo. 1997. Communication: a meeting ground for sustainable development. Paper published on the World Wide Web and available at

http://www.fao.org/WAICENT/FAOINFO/SUSTDEV/CDdirect/CDanalys.htm

Richardson, Don. 1996. The Internet and rural development: recommendations for strategy and activity. Paper published on the World Wide Web and available at http://www.fao.org/WAICENT/FAOINFO/SUSTDEV/CDdirect/CDDO/contents.htm

Richardson, Don. 1997a. The Internet and rural and agricultural development. Paper published on the World Wide Web and available at

http://www.fao.org/WAICENT/FAOINFO/SUSTDEV/CDdirect/CDanalys.htm.

Richardson, Don. 1997b. The sustainability of participatory communication. Paper published on the World Wide Web and available at http://www.PanAsia.org.sg/confhome.htm

Rogers, Everett M., and D. Lawrence Kincaid. 1981. Communication networks: toward a new paradigm for research. London: Collier MacMillan Publishers.

Rostagno, Susan. 1997. Information and communication technologies for the empowerment of women. Paper published on the World Wide Web and available at http://www.PanAsia.org.sg/confhome.htm

Sirimanne, Shamrika. 1996. The information technology revolution: what about the developing countries? CIDA International Development Information Centre: Development Express, No. 04. Paper published on the World Wide Web and available at http://www.www.acdi_cida.gc.ca/xpress/dex/dex9401.htm.

Wresch, William. 1996. Disconnected: haves and havenots in the information age. New Brunswick, New Jersey: Rutgers University Press.

Young, Valerie, Grant Brown, and Jens Laursen. 1997. ICTS and development: testing a framework for evaluation. Volume 1: Final report. Ottawa, Canada: Performance Review Division, Canadian International Development Agency.

Zijp, Willem. 1994. Improving the transfer and use of agricultural information - A guide to information technology. Washington: World Bank.