Review of the Evidence on Indicators, Metrics and Monitoring Systems
Abstract
Research and development stakeholders working on sustainable intensification of agro-ecosystems are striving to become more effective in achieving development outcomes. Key questions facing them are:
- How to evaluate alternative research and development strategies in terms of their potential impact on productivity, environmental services and welfare goals, including trade-offs among these goals?
- How to cost-effectively measure and monitor actual effectiveness of interventions and general progress towards achieving sustainable development objectives?
The purpose of this review was to identify lessons and opportunities for the derivation and use of data from monitoring initiatives in the sustainable intensification of agriculture. The ultimate goal is to provide decision-makers with tools that they can use to explore trade-offs between food security, environmental and socio-economic goals. The analysis is intended to inform the development of any future DFID research investments and engagement with stakeholders in this area.
The first step was to identify key initiatives in data monitoring systems relating to agriculture, paying particular attention to those that also acknowledge the impact on ecosystem health, and/or poverty and well-being. A total of 103 monitoring initiatives were identified. The second step was to review the identified initiatives with respect to their degree of achievement in meeting a set of 34 criteria that had been established from a general literature review. All initiatives were evaluated with respect to their conceptual framework and a subset of 24 initiatives was screened against the full set of criteria. Pertinent additional findings from the literature on monitoring of data in other fields were also considered. Based on this information a gap-analysis of the systems, indicators and metrics was conducted identifying strengths and weaknesses in methodology and use. Insights, lessons and recommendations were then drawn. Preliminary findings were shared with a group of experts and stakeholders identified in consultation with DFID and their feedback incorporated.
Common weaknesses were identified in a number of areas. Many initiatives lack a clear conceptual framework that could demonstrate an understanding of the system under study. In particular theories of change on how the monitoring results would affect behaviours and explicit linkage to specific decisions are lacking in most initiatives. There are problems in most initiatives in defining the target inference space (geography, population) and how that is being sampled. Very few initiatives have come to terms with how to integrate biophysical and socio-economic indicators and sampling frames. A common problem is the lack of well-defined sample. Use of statistically sound study designs to allow attribution of outcomes to interventions is still rare. A lack of consideration of uncertainties both conceptually and in communicating results is pervasive. Similarly few initiatives have tackled trade-offs among objectives. Data sharing agreements were found to be wanting. Evaluation of monitoring initiatives themselves is lacking and there is virtually no information on cost-effectiveness of the measurements or the impact of the initiatives. Few initiatives have been sustained over the long term pointing to inadequate consideration of institutional sustainability.
An over-riding lesson is the surprising lack of evidence for the impact of monitoring initiatives on decision-making and management. Useful insights were drawn from public health surveillance, systems thinking in industry and public services, and decision sciences. A set of recommendations for good practice in monitoring initiatives is given and opportunities for new thinking in monitoring design are identified, including a decision analytic conceptual framework.
As well as the review, some additional files are also appended:
Appendix 2 - Two Excel spreadsheets giving the scoring of the
individual monitoring initiatives against the criteria.
Appendix 5 - Applied Information Economics. Two pdfs - The Need for an
Intervention Decision Model, and Applied Information Economics Example.
Citation
Shepherd, K.D.; Farrow, A.; Ringler, C.; Gassner, A.; Jarvis, D. Review of the Evidence on Indicators, Metrics and Monitoring Systems. World Agroforestry Centre (ICRAF), Nairobi, Kenya (2013) 94 pp.