Review of Making Markets Work for The Poor (M4P): Evaluation Methods and Approaches. DFID Working Paper 41.
Abstract
This report presents the findings of a study undertaken by Itad for DFID to critically review the methods used to evaluate Making Markets Work for The Poor (M4P) programmes and thereby help guide the design and implementation of future evaluations.
The review compiled information about the scope and purpose of evaluations of M4P programmes and analysed the trends, strengths, and weaknesses of these. The reviewers analysed 32 M4P programme reviews and evaluations, monitoring and evaluation (M&E) frameworks and programme reports, including 14 evaluation reports, and consulted with 14 individuals and organisations in the field.
M4P programmes are defined as playing a facilitative, adaptive role in order to contribute to systemic, large scale and sustainable market changes that positively affect the poor. The nature of the approach and the complexity of the markets within which it operates present a number of challenges for evaluation. Evaluation approaches that address these challenges are needed to avoid inaccurate estimations of impact.
Key findings
The M4P evaluations reviewed were generally weak in terms of:
- consideration of systemic, sustainable changes in market systems
- data quality (small sample sizes with little consideration of sampling frames, statistical significance or bias)
- triangulation practices (particularly with regard to qualitative data collection)
- the use of theories of change (those used were often linear, not externally vetted, with assumptions not adequately tested)
- consistency in units for facilitating accurate aggregation
- consideration of unintended negative effects.
A number of recommendations are made with regard to the evaluation method (how), evaluation timing (when), evaluation responsibilities (who) and evaluation level (what).
Citation
Ruffer, T.; Wach, E. Review of Making Markets Work for The Poor (M4P): Evaluation Methods and Approaches. DFID Working Paper 41. DFID, London, UK (2013) iii + 57 pp.
Links