Introduction and Question 1
Pressure is growing on all development agencies to report on their achievements in a credible way - yet progress on the ground is actually slow.
Communicating credible results in ICT4D projects is as important as achieving them!
One part of the solution, suggests Jim Tanburn, is that implementers can be much more clear about the logic of their work. In other words, if they could clarify exactly what they expect that the sequence of events will be (A leads to B, which then hopefully leads to C, and so to D) - then it would be much more feasible to check whether events are indeed unfolding in the expected way.
Logframes are fine as a summary of this logic, but strip out much of the detail that a programme manager needs; it is difficult to show the sequencing of outcomes, for example, or which things are likely to happen in parallel, in the logframe format.
Everyone has an idea in their head about the logic of their programme, but getting it down onto paper in sufficient detail is a particular skill that many of us need to develop. The Donor Committee for Enterprise Development (DCED) is working with many programmes that are developing that skill. You will find some case studies from DCED in the Resource section of this forum.
Once the logic or results chain is sufficiently clear, a monitoring system can be used to check whether it is valid, and also to estimate the results being achieved. The alternative, according to Jim, is that someone will come in from outside, and measure things that might not be relevant; even a clever evaluator or researcher needs to understand well the logic of a programme - yet it is often not made explicit.
In ICT4D there are additional challenges: its diffused nature; involvement of partners; attribution; scalability. This all make the field more interesting but at the same more demanding in how we go about measuring and capturing impact.
Let’s explore this line of thinking together now.
Dear participants,
With great pleasure and excitement, we can announce that our online forum on 'Challenges and Opportunities for Capturing Impact in ICT initiatives in Agriculture' is now open. I will be facilitating our conversation with the Subject Matter Experts and the other forum participants from all around the world.
So without further ado, I would like to ask you - In your experience,
What is the most effective way to measure the impact of ICT for development (ICT4D) initiatives?
Do you explicitly construct logical and linear paths for your programmes?
Do you carry out regular monitoring during projects, or do you prefer ex post facto studies?
Share your thoughts with us and discuss these critical issues with the Subject Matter Experts this week. Thank you for agreeing to take part and looking forward to a fruitful conversation.
Hello everyone! Just thought of sharing our/Katalyst experience of dealing with impact measurement for our ICT initiatives.
As discussed by Jim Tanburn, We are also using results chain for monitoring and capturing the impact for our Tele Center and Mobile Telephony based initiatives. And these kind of results chain also enable us to keep atrack of the out come of our activities on a continuous basis. We are not just doing a post assessment after the implementation work is done. Rather it is utilized as "True" monitoring tool for assessing the progress and impact of our activities. It also allow us to bring in changes at activity level based on the feedback we receive at different levels of the market system and end beneficiery.
In addition to that, I must say using impact chain for monitoring ICT4D initiatives is even more convenient and usefull becasue of the different "system generated" data for the usage of different ICT based solutions or services.
For an example: Our Agri help line project (Jigyasha 7676) with one of the largest Telecom operator in Bangladesh is generating good amount of usage data from the CRM that is being used in the call centre. It does not require us any extra cost for monitoring the impact at usage level. It also enabled us to design effective promotional campaigns, addition of new content and etc just by using these kind of system generated data.
Now, I would also like to know how other projects are dealing with this issue of impact measurement. Please share your experiences and thoughts!
Cheers!
Hello Arafat,
Welcome to the forum and thanks for sharing your experience. If I could ask you a few questions so that our participants could get some more insights on you have just shared -
1. When you say 'Rather it is utilized as "True" monitoring tool for assessing the progress and impact of our activities', how exactly do you perceive this as 'true' and what could be its relative strengths?
2. Do using 'impact chains' and 'system generated' data allow you to change any of your plans during the course of the interventions, say at regular intervals or you prefer not to be so flexible all throughout and prefer to wait unless the available data is sufficient to make an assessment?
Thanks for your time and hope to know more from you!
Among the several things that I have learned while implemeting interventions with ICT in agriculture, is that result measurment is something that starts off at the very get-go. Some have tendency to start thinking of capturing results only after interventions are proving to be successful. This outlook I believe make result measurement a very reactive tool. On the other hand when you have the clarity about results measurement from the very begining, in terms of what key factors are we trying to change, how would it change, how would we capture it, etc it will not only help you design better intevenions but also allow the result measurement processing to be a tool for proving and improving your work; hence a pro-active tool
Also when it comes to ICT initiatives, having such clarities beforehand, helps you, to an extent find the right areas of partnership and allows you to draw a much clearer picture of the overlapping areas of interest in capturing results between your organization and your partner organizations, if any.
Hi Shehzaad,
Thanks for further stimulating this discussion. Please find my response as below:
From the remarks of different forum participants of this on line conference, it is apparent that most of the development projects are following an impact measurement methodology that enables them to report the impact of a program or its activities designed beforehand. And following that kind of methodology does not allow the program team to reevaluate and bring in changes in program activities. In contrary to that, our program uses impact chain for each of the activities/intervention following logical consequences of the activities. Thus that enables us to tract whether the planned activities are resulting in intended outcome. If the planned activities are giving the program with an anticipated results well and good. But if the planned activities are not yielding anticipated results, then we redesign the planned activities based on the feedback from the intended beneficiary and/or market system. Thus we thrive for more result oriented activities for our intended beneficiaries following a non-linear process.
I believe flexibility is a key feature of our methodology and we also believe there is no alternate to this. We use impact chains for keeping a track on the changes facilitated in market system/dynamics and if things are not happening as expected, we take the liberty to bring in changes in the activity plan. And usually these kind of decisions are made based on system generated data. Where we primarily look at the impact scenario at the beneficiary level from the system generated data. Here, it is necessary to mention that for ICT interventions system generated data is quite readily available. For an example, our Agri help line project with Banglalink, we get monthly reports of service usage/service usage growth/service user retention and etc. And all these reports are generated from the CRM that is being used in the call center for the agri help line.
However, we are keen to improve our MRM system and looking forward learn about other good practices in monitoring and evaluation of ICT projects.
Cheers!
Dear All,
I would like to see contributions from the private sector on these issues which may give us a different dimension of this dicussion.
In general, when we (as a private initaitive) frame and develop the business plan as a commercial venture, it is must to see the RoI (Return on Investment) is most impotant factor. And this return basically measured from financial perspective where consideration goes to Opportunity cost, Net present value or time value of money. Here indicators are mostly in terms of revenue, time line and customer satisfaction.
All entities carry out regular monitoring regardless private or development organization. It has been observed that post facto studies are more often used in development initiatives.
Thanks,
Shahid
Agricultural information access is a right to farmers and ICT is a swift way to do that. But the ICT should use in correct way. The quality of information and quality of media should be well organised. Here the media should be closed with research.
Daer Bulbul,
Very interesting when you say that 'agricultural information access is a right to farmers' - Certianly worth thinking.
However I was wondering if you could elaborate a bit more when you say 'media should be closed with research'? Thanks and will wait to hear from you.
Dear Shehzaad,
Thanks for your asking. I get much pleasure having your quest. However Here I cited media word representing those organizations who are dealing with information dispersion. The time is changing. Attitude towards agricultural information, excepting ability of farmers, thinking powers all are changing as well as climatic environment. The theme that was highly accepted before, may be next not. The innovations are timely updating by research. But I think their is a unknown gap between the research organizations and extension. I am not claiming anybody or any system.
ok, then thanking you again and wishing next discussion.
cheers to all
Bulbul