Introduction and Question 1
Pressure is growing on all development agencies to report on their achievements in a credible way - yet progress on the ground is actually slow.
Communicating credible results in ICT4D projects is as important as achieving them!
One part of the solution, suggests Jim Tanburn, is that implementers can be much more clear about the logic of their work. In other words, if they could clarify exactly what they expect that the sequence of events will be (A leads to B, which then hopefully leads to C, and so to D) - then it would be much more feasible to check whether events are indeed unfolding in the expected way.
Logframes are fine as a summary of this logic, but strip out much of the detail that a programme manager needs; it is difficult to show the sequencing of outcomes, for example, or which things are likely to happen in parallel, in the logframe format.
Everyone has an idea in their head about the logic of their programme, but getting it down onto paper in sufficient detail is a particular skill that many of us need to develop. The Donor Committee for Enterprise Development (DCED) is working with many programmes that are developing that skill. You will find some case studies from DCED in the Resource section of this forum.
Once the logic or results chain is sufficiently clear, a monitoring system can be used to check whether it is valid, and also to estimate the results being achieved. The alternative, according to Jim, is that someone will come in from outside, and measure things that might not be relevant; even a clever evaluator or researcher needs to understand well the logic of a programme - yet it is often not made explicit.
In ICT4D there are additional challenges: its diffused nature; involvement of partners; attribution; scalability. This all make the field more interesting but at the same more demanding in how we go about measuring and capturing impact.
Let’s explore this line of thinking together now.
I agree with Jim that logframes can be a very useful tool for evaluating the impact of ICT for agriculture programs. While ICT is innovative, and offers new opportunities for outreach and engagement with program participants, at its heart, it is like a lot of other programs.
But there are some ICT-specific challenges that are necessary to consider while thinking through the program. These might seem evident to all of you -- if so, please forgive me for repeating them -- but these are lessons we learned with our IMAC program (a mobile phone-based price information program) in Niger:
1. Sometimes the ICT for ag projects involve developing new partners with the private sector. How can we bring these partners into our planning process and support?
2. Does the program manager have the necessary technical support within the organization (or outside of it) to ensure that the ICT interventions run smoothly?
3. If we observe that farmers are "doing better" after the ICT for agriculture program, does this mean that we should replace the "traditional" way of doing things with ICT? Or are they complements? In other words, what would have happened in the absence of the program? (This is the attribution issue).
We faced many of these challenges in our IMAC program in Niger. We learned that figuring out these partnerships and ensuring that the necessary technical support is available is something that must be decided early on, during the project planning stage. More is available here: http://sites.tufts.edu/projectabc
Hi Jenny,
I agree to all the three points that you have highlighted. To answer to the third point- ICT (mobile / internet based indformation delivery systems) are complementary to the all other existing extension networks. Traditional extension chanels help in strengthening the existence of modern ICT. We have observed this during our study as well as in present day field works in various low income states of India.
Cheers
Surabhi
Dear Jenny,
Drawing your attention to Moses' opinion that
'Using logframes is one model.But i think relying on a logframe to develop an evaluation system is dangerous as in most cases logframes are designed at an early stage of planning of the project and also the logframes are rarely updated during course of project implementation and so it may not necessarily reflect real situations at the time of evaluation'
What are your views on this?
Dear colleagues,
The discussions are quite interesting indeed. The issue is about how best we can provide evidence of impact of ICT4D to those who are sponsoring us. We mention the dreaded 'log frame' because that has been the traditionally accepted tool for planning linear interventions, people have been trained to use it and draw conclusions from it. However, it assumes that there is a logic flow or some sort of linearity in the activities carried out and the outcomes, with some degree of flexibility to allow for unexpected outcomes of the activities, which you could always explain with the use of the assumptions made and the risks mentioned upfront.
The ways in which we have adapted to the log frame approach has been to try to straighten what in general is a complex, interrelated set of factors that lead to a change in attitude, behaviour which may lead to socio-economic impact. For example, we have tried to target specific communities for which we have baseline data and then compared them during and after to see what changes have happened. By narrowing ourselves to a segment of the complex, we have been able to present a 'linear' and logical approach to our interventions and are able to satisfy the measurement criteria, to a certain extent. Unfortunately, we may also be missing the spill-over effects of our intervention by just looking through the log-frame lens.
So are we stuck with the log frame - or rather, are the sponsors and funding agencies stuck with the use of this tool to justify their investments? - the aim of evaluation from a 'sponsor' is that they are provided with evidence that their investment has done some 'good'. If it were a person you were explaining it to, perhaps you could address their feelings and make them 'feel good' that they have invested in a project. Unfortunately, we are also restrained by having to report in writing and it tends to be difficult to create the sensation of 'feeling good' through text, unless there are figures that express a sense of magnitude, which then is able to make us 'feel good'. So, part of the answer to me would be to explore new ways of reporting that can be acceptable to the 'sponsors' that represent the impact of a project. Thus, we find nowadays that there is a lot of focus on getting 'stories' from the target communities, short videos, or even field visits and 'immersions' in the target community etc. that can be used both to report on progress and at the same time 'touch' the feelings of the sponsor.
So, as innovators in this area of work, what are the means of reporting that we can include to complement what the 'log frame' indicators cannot capture? Is there also some advocacy to be made at the level of the 'sponsors' to enable them to quantify the 'feel good factor' in monetary terms?
Krishan Bheenick
Food and Agricultural Research Council, Mauritius
Daer Krishan,
I really liked your views on the need to capture issues that logframes can't capture. I wonder if you have seen views from Jim Tanburn in this relation?
Would love to hear your thoughts.
Shehzaad,
I could not agree more with Jim's views on the differences that exist between conceptualisation and implementation of the log frame. Definitely it seems that some donors are stuck by regulation to have a logframe in project proposals. How many of us have been through this cycle where a project proposal is being prepared: a consultant (No1) is recruited to develop a proposal as per the formats and requirements of the donor/sponsor. A lot of documentation on what should be done as part of the project is provided to the consultant by the stakeholders, the final output is a project proposal that summarises these proposed activities into clusters that are now linked to objectives and result areas. The ideas are present in the text in less details and the actual action (minus the spirit in which the action should be made) is represented in the logframe. If the stakeholders or beneficiaries are not well organised in formulating their views, they are generally happy that a document has been prepared by an expert which increases their chances of being funded.
Some time later (in some cases a year or more later) the project is launched. Consultant No2 is recruited to implement the project. Consultant No 1 is not able to be involved in implementation, so his/her views may not be considered. The new people involved in implementing may simply just start off with the project proposal and elaborate an activity plan based on that - unless, again the stakeholders are clear about their original proposals of what should be done as part of the project and bring these to the attention of the consultant. Consultant No2 may also adopt the attitude that they are here to implement the project so as to satisfy the indicators (again quantified, but with no indication of the spirit in which these have to be implemented), no more no less. This scenario is even more important when we are dealing with information and knowledge management because there are a lot of soft skills involved - hence my stress on the 'spirit of implementation' of an ICT4D project being as important as the number of publications printed and distributed for example.
I also support the idea of unpacking the process into steps (results chain exercise) so as to document each step of the process and how well it has worked. I will come back to this under the discussion on 'Do you construct logical and linear paths
Krishan