Devil in the detail - measuring sustainable development

September’s post-2015 development agenda summit of heads of state and government may well attract global headlines this year, but much of the really significant work is being executed behind the scenes.

As statisticians lock heads to come up with a workable indicator framework to measure and monitor the ambitious Agenda, Vikas Rawal, Associate Professor at the Centre for Economic Studies and Planning, at Jawaharlal Nehru University, New Delhi, describes the make-up and value of a sound indicator, bringing meaning to the complicated task of sizing up progress.

What role will indicators play in the post-2015 development agenda?

Indicators are a way for countries to assess progress on agreed targets. They are, therefore, extremely important. The MDG experience has demonstrated that to a large extent what gets measured gets done. That, I believe, is a serious problem, as you cannot measure everything, unless you invest all your money on measurement alone. Many crucial aspects of the development agenda may not be covered by indicators. And in some cases, indicators are extremely restrictive – measuring only extreme poverty and destitution, for example, may not correspond to the target’s ambition to end all forms of poverty.

What makes a good indicator?

A good indicator should be able to provide a robust measure of progress towards that target. If the target is multi-dimensional, a good indicator either provides a composite (compiled) measure of all the dimensions, or measures the most important dimension of the target. If the indicator does not measure all the target’s dimensions, then additional indicators may be required.

It’s important to understand that composite indicators have their own limitations. The methods for aggregating different dimensions tend to be arbitrary (or, at best, are based on a purely statistical reasoning). Also, composite indices are, by their nature, more abstract. Indicators that are less abstract - and based on concrete, direct measurement - are easier to comprehend and translate into clear policy action. Policymakers, the general public and other stakeholders should be able to easily grasp what the indicators is and its relationship to what is being observed.Finally, indicators should be based on data that is available, reliable and cost-effective to obtain in future.

There are 17 SDGs and 169 targets in the United Nation’s OWG proposal compared to 8 and 18 for the Millennium Development Goals.

The number of proposed SDGs and targets presents a formidable challenge. For the indicator framework to be manageable (economically), the number of indicators at international level should be kept to a minimum. The core set of indicators should directly measure the target’s intended outcome. They also have to be universal, meaning all countries need to report on them. Measurement of processes, drivers of outcomes and means of implementation should be kept out of this core set.

Given that we seem to be moving towards a development agenda that will have a considerably wider scope than the MDGs, not everything will be measured by core indicators. It is not possible to do so. A separate list of supplementary indicators is likely to measure processes and means of implementation. It may be that countries will be able to choose which of these they want to put their money into. After all, countries may already have to invest considerable resources in their statistical systems to generate information for core indicators.

What role will UN agencies play in developing indicators?

UN agencies have a critical role to play in developing indicators for those issues tackled by the new SDG framework that were not covered by the MDGs, especially with a view to capturing the three dimensions of sustainable development – social, economic and environmental. Agencies will need to define the methodology and establish standards for collection of data. They will need to organize technical cooperation activities to support countries’ capacity to produce the indicators on a regular basis. UN agencies would also have to set up international repositories of data to which member countries contribute information as per the agreed standards and methodologies. 

A key challenge will be developing indicators that can be measured regularly and cost-effectively by all countries. FAO is working on two fronts: improving the statistical measures for hunger, malnutrition, resilience and smallholder agriculture required for monitoring SDG 2; and strengthening the set of indicators that capture the sustainable use and management of natural resources for hunger and poverty eradication, across a number of other SDGs including 14 and 15.

Are we relying too much on data? Is everything important measurable?

Data is indispensable for measurement. Thanks to incredible developments in our technical capabilities to handle large volumes of data, we are now talking of a ‘data revolution’ to support the post-2015 development agenda.

I have three points to make in this respect.

First, there continues to be great disparity in the capacity of statistical systems across countries. The nature of economic and social relations in least developed countries often makes measurement of socio-economic changes costly and difficult. Statistical organisations of many of these countries have very limited capacity, and have to depend on donors and international agencies for data collection if the measurement is complex and demands large investment. So, the challenge is to develop standards and methodologies that are simple and cost-effective along with being robust and meaningful.

Secondly, it’s important not to let data and measurement tailor the agenda itself. The statistical systems of international agencies like FAO and of the member countries have to serve the development agenda that is evolving, and not the other way around.

Finally, the question is not whether everything can be measured. A more important problem is that if you measure everything that can potentially be measured, the cost of measurement will be too much. The challenge for statisticians in the UN agencies is to come up with measures that are meaningful, concrete and yet cost-effective. It is then for the decision-makers – the member countries – to choose and agree on those they think are the most important. While the UN agencies should provide technical advice on the matter, the final decision has to be political.