Introduction and Question 1
Pressure is growing on all development agencies to report on their achievements in a credible way - yet progress on the ground is actually slow.
Communicating credible results in ICT4D projects is as important as achieving them!
One part of the solution, suggests Jim Tanburn, is that implementers can be much more clear about the logic of their work. In other words, if they could clarify exactly what they expect that the sequence of events will be (A leads to B, which then hopefully leads to C, and so to D) - then it would be much more feasible to check whether events are indeed unfolding in the expected way.
Logframes are fine as a summary of this logic, but strip out much of the detail that a programme manager needs; it is difficult to show the sequencing of outcomes, for example, or which things are likely to happen in parallel, in the logframe format.
Everyone has an idea in their head about the logic of their programme, but getting it down onto paper in sufficient detail is a particular skill that many of us need to develop. The Donor Committee for Enterprise Development (DCED) is working with many programmes that are developing that skill. You will find some case studies from DCED in the Resource section of this forum.
Once the logic or results chain is sufficiently clear, a monitoring system can be used to check whether it is valid, and also to estimate the results being achieved. The alternative, according to Jim, is that someone will come in from outside, and measure things that might not be relevant; even a clever evaluator or researcher needs to understand well the logic of a programme - yet it is often not made explicit.
In ICT4D there are additional challenges: its diffused nature; involvement of partners; attribution; scalability. This all make the field more interesting but at the same more demanding in how we go about measuring and capturing impact.
Let’s explore this line of thinking together now.
Dear Emefe,
I think what you mentioned in a simple manner is worth a lot of value - that 'access to information should be simplified'. I am not a subject matter expert in this case, but from my experience of dealing with software development and digital projects, I can positively confirm that 'user experience' and 'user journeys' are at the heart of modern day software development so that these can be programmed keeping the end user in mind and not the programmers preferences or the company's objectives.
I wonder to what extent impact assessment frameworks in use are being prepared keeping donor requirements and implementing agencies' objectives in mind - whereas assessing impacts from say a farmer's point of view may be very simple and away from a lot of jargon and anomalies.
It would be interesting to see what our Subject Matter Experts might have to say regarding your comment. Thanks.
Hi,
We need to be clear about the objective or purpose of delivering agriculture/ extension information to the farmers. The programmes or models through which this information is delivered, it is important for them to run their system efficiently and thus through a regular internal process they keep a close watch on if the information is being delived and how to improve the delivery. On the other hand it is also important to see if the information that farmers receive has any impact on their socio-economic conditions. Anectodal evidences from various earlier studies in India and South Asia (including mine ICRIER WP 246) has proven this fact. But the big question remains if actually a critical mass of farming community expecially the small farmers surviving on subsistence agriculture - mainly cereal based economy able to benefit from this information? How have they used this information? What mechanisms exist to see or to evaluste a continuous and rigorously acceptable (statistically) impact analaysis is something which I am also trying to find answer to. As an economist I have the options to do experimental trials or do an ae-ante and ex-post analysis. But then we work with a small sample of population. Is is possible to link the service provders with the researchers and through good data mangement infact using ICT itself, build up data to abe able to see the real impact in terms of income change, reduction in cost or yield improvements, reduced use of harmful chemicals, or even social changes. Important is also to observe and track the behaviour changes?
Cheers
Surabhi Mittal
Hi Surabhi
The challenges that you mention are very similar to what we face in our implementation. We are working in partnership with a mobile phone operator and helped established 500 telecenters across the country. These centers are self sustaining and provide a host of services include agriculture information. We hve done baseline and post intervention impact assessment and have found that among others better crop disease management (leading to "loss saved/not incurred') is the primary means of how this ICT initiatives impacts on the poor. However the challenge that present itself, is farrmers tend to share information with another and also that there is multiple information sources. Hence how does one attribute the extent of impact to an interventions. For us, we are trying different approaches, among others; first being to acknowledge all significant and relevant information source, second probing beyond our ICT sources to see whether other factors where also responsble and thirdly doing control and treatment groups. (although considering our scale, the third option sometimes is quite difficult to undertake)
Dear Surabhi,
You advised to observe and track behaviour changes in relation to tracking impact of ICT in agriculture. Curious to ask you if there exists any set group of performance indicators which are 'must have', should be tracked as a priority, hard indicators such as income etc.? Also may be another group of 'soft indicators' such as 'behaviour', 'lifestyle' etc.? Do you follow any such classification or prioritisation in your ICT impact assessment?
Thanks!
Shehzaad,
The indicators according to me as neccasary to track the change is
1. What % of messages do farmers act on?
2. With the new information received do qunatity/ cost of inputs used change?
3. What si the observed change in yields, income?
4. Do farmers change their behaviour over time by adopting to new technology or even trying to use new technology (seed technology, non- seed technology etc.)
I am trying to develop methodology on these lines and run some experimental trials, hopefully will be able to start some of these soon for India.
Surabhi
I am posting this on behalf of the undersigned who is not able to post his message here and has thus requested me to do so.
The recent 8th EFITA Conference in Prague http://www.efita2011.cz/index.
dedicated a Keynote to the Impact of Technological Innovations. In several sessions various
aspects of information-providing mobile systems were presented and discussed. An aspect
of particular interest included the impact Social Network information-exchanging (Facebook?)
have on Agricultural Productivity. This facet specifically under the title:
Collaborative platforms and Social Media for Agriculture.
The format was a query:
Are Social Networks A Technological Upgrade or a Paradigm Shift?
This questions and findings in this Indian study add universal clarity to the issues as
discussed in Prague - issues such as:
best
Ehud Gelb,
Center for Agricultural Economic Research, Israel
Dear Ehud,
We have just replied to your email. Please try to log in and post now.
We also noticed that you are an e-Agriculture member since 2007! that's great! It is very encouraging to see that members who joined our Community back in 2007 are still active in e-Agriculture forums and activities!
We look forward to hear more from you and future discussions,
Kind regards,
Lisa
e-Agriculture Team
You raised deep and probing questions. There is need to track down how ICT is affecting production and productivity, in relation to farmers' behaviour in terms of adoption rates, and what factors facilitate adoption of a particular technology - cost, availability, accessibility etc. coupled with farmers' access to other information sources. For emerging economies these are very important statistics we need especially as penetration of ICT is low.
Dear All,
It seems that the forum is warming up very well with great contributions from Surabhi, Shahroz, Arafat, Bulbul and John so far. Now building on Jim's suggestion as above, let me pose the following questions to the forum for further thoughts and opinions -
What is the most effective way to measure the impact of ICT for development (ICT4D) initiatives?
Please let us know what you think is the most effective way and why you think so.
Using logframes is one model.But i think relying on a logframe to develop an evaluation system is dangerous as in most cases logframes are designed at an early stage of planning of the project and also the logframes are rarely updated during course of project implementation and so it may not necessarily reflect real situations at the time of evaluation.
Developing a good Monitoring and evaluation systems to track progress, performance and to generate good outcomes following and impact assessment is good way to go.Also as part of Monitoring and evaluation, selecting indicators to measure progress and determine outcomes must be critically done.I work with an ICT based Organisation in Uganda called Women of Uganda Network (WOUGNET) and we implement ICT4Dev initiatives and would really be interested in understudying what OUR external evaluators will use to really help us comprehensively measure the impact of our ICT4Dev initiative in the near future !
Moses