Corporate report

Annual Evaluation Report 2020 to 2021

Published 21 December 2021

This was published under the 2019 to 2022 Johnson Conservative government

1. Introduction

This Annual Report presents the achievements of evaluation in the Foreign, Commonwealth and Development Office (FCDO) from April 2020 to March 2021. This was a period affected by many challenges globally and within the organisation. FCDO staff in the UK worked almost entirely remotely during this period, and programmes (and any associated evaluations) had to be adapted quickly. In June 2020, the Prime Minister announced the merger of the Department for International Development (DFID) and the Foreign and Commonwealth Office (FCO) to form the new FCDO, which came into effect on 2 September 2020.

The global and organisational contexts have required changes in our approach to supporting and carrying out evaluations. In this report, we outline the work of the FCDO Evaluation Unit (section 2), including pivoting to address the emerging challenges of the pandemic, the development of a suite of central thematic evaluations and continuing the work of our evaluation programmes. In section 3, we provide an overview of evaluation activity across the business during this period and our ongoing efforts to ensure quality and rigour, and outline 3 evaluation case studies.

2. Central evaluation service

As of March 2021, the Evaluation Unit comprised 9 Evaluation Advisors supported by 3 programme managers. The Evaluation Unit is part of the Economics and Evaluation Directorate in FCDO, under the Chief Economist.

2.1 COVID-19 MEL Hub

In March 2020, the Evaluation Unit pivoted to respond to the challenge of COVID-19, recognising the need for a strong, coordinated monitoring, evaluation and learning (MEL) approach to ensure accountability and learning from evidence, and to improve current and future responses. Support over 2020 to 2021 has included:

• development of internal guidance notes on phone surveys, ethics and third party monitoring, as well as an overarching MEL FAQs
• seminars on e.g. adaptively managed programmes, remote data collection, using process evaluations in COVID-19, and using Big Data
• call down technical support, particularly for adaptive, nimble and innovative methods

We developed a layered, sequenced and proportionate approach to central evaluative activity. This has included ongoing mapping of evaluative activity within and beyond FCDO to ensure coordination and minimise duplication, and to help identify opportunities for learning and synthesis. FCDO is an active member of the OECD Evaluation Coalition on COVID-19 ^{[footnote 1]}, and the Vaccines Working Group within that. In response to demand, and emerging evidence gaps, we have identified a number of central strategic thematic evaluations related to COVID-19: see section 2.2.

2.2 Thematic evaluations

This year, we have stepped up our efforts to provide strategic and thematic portfolio evaluations on key areas, drawing together learning and evidence across sectors and geographies.

In 2020-21, the Evaluation Unit led and completed learning reviews, which synthesise existing evaluation evidence, on Beneficiary Engagement, Commercial Agriculture ^{[footnote 2]} and Girls’ Education. The learning was widely disseminated to policy and technical leads.

During the year, topics for thematic evaluations were identified that will meet evidence needs related to FCDO high level priorities on the COVID-19 response and climate change. A thematic evaluation examining evidence from FCDO programming on climate smart agriculture and small holder resilience commenced (report due in October 2021). Further thematic evaluations are planned, including evaluating humanitarian data collection and analysis options during COVID-19, and use and effectiveness of epidemiological modelling evidence to support policy development and response.

2.3 Measuring influence

We have adapted our work to meet the priorities of the new organisation, including developing and leading a cross-FCDO workstream on how to design, measure and evaluate FCDO’s diplomatic and development influencing work, working with colleagues across the organisation. This has included development of an internal toolkit, direct technical assistance, building capabilities and testing new monitoring approaches.

2.4 Central programmes

Through the Centre of Excellence for Development Impact and Learning (CEDIL) programme ^{[footnote 3]}, we have pioneered new rigorous methodologies for hard-to-measure problems, for example through mapping frontier work on using Big Data and providing guidance on how to use this in impact evaluations. CEDIL’s website has over 20 papers and 40 seminars available for all to use.

We are the largest donor to the World Bank Strategic Impact Evaluation Fund (SIEF) ^{[footnote 4]} and have worked with them to finance 6 COVID-19 emergency evaluations on keeping students engaged with learning, remote education and preparing for school return; and 6 research projects on COVID-19 response and recovery using educational technology.

We completed the Evidence in Governance and Politics (EGAP) programme ^{[footnote 5]} this year. It collected exciting new robust evidence on crime, community policing, community management of natural resources and incentivising citizens to access tax-funded public services. It also pioneered new, more rigorous methodology for testing whether an intervention is effective in multiple contexts.

Through our partnership with the Development Impact Evaluation (DIME) Department at the World Bank, the Fund for Impact Evaluation programme ^{[footnote 6]} continues to support a portfolio of 157 impact evaluations studies, of which 84 have now been concluded. Each of these studies generates new, policy-relevant and rigorous evidence on “what works” in under-evaluated areas of international development. DIME has pivoted a large part of its effort to produce robust evidence to inform the response to the pandemic on a global scale.

3. Evaluation across the organisation

3.1 FCDO evaluation in numbers

Sixty one evaluations were reported as completed in 2020 to 2021 ^{[footnote 7]}, of which 18 were partner-led and 13 were conducted under the Conflict, Stability and Security Fund (CSSF) ^{[footnote 8]}. The total spend was approximately £25 million.

The bulk of the evaluations were in Government and State Building (17), followed by humanitarian assistance (8) and agriculture (8).

Most evaluations were global (17), but there were a high number in Africa (15) and Middle East and North Africa (12). Figure 2 shows a breakdown by region.

An estimated 22 of the 61 evaluations were impact evaluations. The majority of others were thematic or process evaluations.

Figure 1: Number of evaluations by thematic area

Figure 2: Number of evaluations by region

Figure 3: Evaluation country coverage

3.2 Ensuring quality

FCDO is committed to ensuring rigour and quality in all our evaluations. For evaluations related to former DFID spend, it remained a requirement for all evaluations to go through our Evaluation Quality Assurance and Learning Service (EQUALS). This service, as well as offering technical assistance, provides independent quality assurance of all evaluation products. Each product is assessed against a template with criteria covering a range of areas we feel are important, such as independence, stakeholder engagement, ethics and safeguarding, as well as methodological rigour.

During the review period a total of 85 unique requests to quality assure evaluation products were raised from DFID/FCDO. Products are rated excellent, good, fair and unsatisfactory. Products are improved based on qualitative feedback before publication, with those rated unsatisfactory having to undergo re-review following improvements. Since 2016, 62% of products were rated good or excellent following their first review.

The service is also open to other government departments. Before the merger, the biggest user after DFID (about 88%) was FCO. The Department for Business, Energy & Industrial Strategy, the Home Office, the Department of Health and Social Care and the Department for Environment, Food and Rural Affairs have also made use of the service.

Figure 4 shows the percentage of each type of quality assurance product reviewed by EQUALS.

3.3 Evaluation impact: case studies

On the following pages, we look in more depth at examples of how evaluations have influenced decision making within and beyond FCDO. The case studies are:

• the Making Country Health Systems Stronger process evaluation provided timely findings on health system strengthening
• an impact evaluation on an agriculture intervention in Syria filled vital evidence gaps
• a portfolio evaluation of gender and inclusion in the Prosperity Fund is informing FCDO’s approach to Gender and Equality

Figure 4: EQUALS Quality Assurance Requests April 2020 to March 2021

Case study: Making Country Health Systems Stronger

Evaluation type: Performance Evaluation

Publication date: May 2020

Full report: https://iati.fcdo.gov.uk/iati_documents/55762647.pdf

Management response: https://iati.fcdo.gov.uk/iati_documents/56815723.pdf

The programme

The Making Country Health Systems Stronger (MCHSS) programme ran from October 2017 to March 2020, with the aim of contributing to faster progress towards universal health coverage, by supporting low-income countries (LICs) and lower middle-income countries (LMICs) to strengthen their health systems for all population groups. The programme was budgeted to provide £28 million, most of which was channelled through the World Health Organisation (WHO). It had 4 thematic components:

• Global coordination
• Health financing
• Access to medicines
• Health information systems

It supported activities in ten LICs and 17 LMICs, as well as at global, regional and multi-country levels.

The evaluation

The evaluation of DFID’s approach to the MCHSS was carried out from April 2019 to May 2020, using a theory-based approach, articulating the theory of change for the programme as a whole and for the 4 Output Areas, and testing the validity of the intervention logic and of key design assumptions that underlay the programme. It included a review of monitoring data, a literature review, a stakeholder perspectives study, country case studies and a programme management review. The objectives of the evaluation were to evaluate the performance against the goals and outputs, to assess complementarity with broader global health system investments, to assess how global investments enhanced broader health system strengthening (HSS) efforts at a country level, and to make recommendations to inform ongoing and future HSS investments.

The evaluation found that programme made a modest contribution to the overall MCHSS outcomes of strengthened and more resilient country health systems. The majority of outputs were achieved, although some deliverables were only partially achieved in some of the technical areas, particularly on access to medicines and health information systems. Its most successful component was the health financing work, where the programme helped produce a wide range of potentially significant deliverables or reforms at global, regional and country level.

The evaluation made a number of recommendations, all of which were accepted or partially accepted by FCDO. These were focused on FCDO’s future support to Health System Strengthening, included future support to the WHO, with recommendations around setting clear objectives for grant support; mechanisms for working with multiple partners; coordination and harmonisation on HSS; building resilience and preparedness; results, M&E and value for money, and contracting, staffing and resources.

The impact: how recommendations were used

The evaluation was shared widely through relevant networks and teams within FCDO, as well as with the WHO and with other donors. The results of the evaluation were used in discussions about FCDO’s approach to Health System Strengthening, including the development of the Business Case Core Voluntary Contribution funding to the World Health Organisation.

Progress can already be seen on some specific responses to recommendations. For example, on Health Information Systems (recommendation 10), FCDO is now actively engaging with the Health Data Collaborative, including taking on the role of representative for the donor constituency. In response to challenges posed by COVID-19, there is an increased focus on core systems as part of the HSS work, building resilience and preparedness (recommendation 11). The evaluation also recommended a results framework (recommendation 12): FCDO now has a theory of change and a results framework, based on the WHO’s existing framework.

Though the context and global health landscape has shifted considerably since this evaluation was conducted, the lessons and recommendations remain relevant, and will continue to influence how the FCDO and others work in the Health Systems space.

Case study: Supporting Emergency Needs, Early Recovery and Longer-term Resilience in Syria’s Agriculture Sector

Evaluation type: Quasi-experimental impact evaluation

Publication: forthcoming; baseline report available: https://isdc.org/wp-content/uploads/2020/11/Baseline-Report-FAO-Syria-2019-04-18.pdf

The programme

FCDO funded the Supporting emergency needs, early recovery and longer-term resilience in Syria’s agriculture sector programme, implemented by the UN Food and Agriculture Organisation (FAO) Syria. The programme sought to deliver emergency and recovery support to vulnerable smallholder farmers across Syria and increase food availability. The programme had 3 key objectives:

1. to increase food availability for vulnerable households through improved smallholder production
2. to build sustainable access to productive assets, income and food supply
3. to foster enabling environments for resilience building and recovery of the agricultural sector

The budget for the programme was £7.5m over 3.5 years.

The evaluation

The evaluation adopted a quasi-experimental approach, using household survey data from 880 beneficiary and non-beneficiary households before, during and after the intervention took place. The evaluation sought to measure changes in food security status, the use of harmful livelihood strategies to cope with shortages of food, agricultural crop and livestock production, and income generated from agricultural value chain activities. The evaluation found the following changes when comparing beneficiary and non-beneficiary groups:

1. the programme strengthened food security of vulnerable smallholders in Syria by 13% from baseline values, with stronger effects for households receiving support in vegetable production
2. female-headed households benefited considerably from the programme, increasing their food security status by 32% compared to female-headed households who did not receive support
3. targeted households were more likely to have reduced negative coping behaviours, especially with regard to child labour and sale of household assets
4. impact was also stronger for households with access to irrigation, who saw a notable improvement of 23% in their food security status due to the programme

Some activities were found to deliver stronger positive impacts on food security, income generation and resilience: vegetable kits for household production, livestock support and specific value chain interventions were all high impact activities. Beekeeping and emergency poultry kits delivered mixed, low or no impact.

The impact: how recommendations were used

Conducting an impact evaluation which requires panel survey data in a complex environment is extremely challenging, due to higher rates of attrition, fluctuations in programme design and beneficiary groups, protocols and permission requirements, security and safety during data collection. However, this study showed that it is possible to conduct a rigorous impact evaluation in Syria, filling a vital evidence gap. This will inform future monitoring and evaluation plans for FCDO programmes.

Other lessons have emerged on developing measures for such a programme, and the need to ensure they are directly linked to the causal analysis and theory of change. Outcome indicators should focus on the difference achieved for individuals’ resilience, rather than on quantitative changes in food production or income generation alone, and include measures of gender, disability and equity to better assess the differing impact on different groups.

This evaluation will inform future programme design and work with partners, drawing on the evidence base to shape how agriculture can be incorporated into resilience. Lessons learned from the evaluation suggest that future agriculture programmes in Syria should:

• be designed as multi-year in length, to build stronger and more sustainable impacts on resilience
• provide multiple lines of support simultaneously, even if this requires lowering the reach, to create stronger and more durable impacts
• include a Gender, Age and Disability analysis at the outset of future programming, to better understand and address social barriers and enablers to their access to livelihoods
• invest in independent evaluation, with quasi-experimental design, to support more confident attribution of results to interventions, with large baseline datasets to mitigate challenges in the Syrian context

Case study: Gender and inclusion evaluation of the Prosperity Fund

Evaluation type: Portfolio evaluation

Publication: Forthcoming

The programme

The Cross-Government Prosperity Fund (PF) ran from 2015-2021, seeking to address the challenges faced by middle-income developing countries, such as rapid urbanisation, climate change and high and persistent inequality (including gender inequality) which can lower long-term economic growth. The PF aimed to support the broad-based, sustainable and inclusive economic growth needed for poverty reduction, through improving the global business environment, strengthening institutions, improving infrastructure and services, building human capital, and encouraging greater global private investment. UK Government departments developed 27 multi-year programmes in over twenty countries. A strong focus on gender and inclusion (G&I) was recognised as essential for achieving the PF’s primary purpose of supporting inclusive growth and poverty reduction. The PF was officially closed in March 2021, with remaining prosperity programmes (at country, regional and global levels) transferred to the FCDO.

The evaluation

Independent evaluation was used throughout the life of the PF for both accountability and identifying lessons to improve delivery of the PF and inform the design of future initiatives. Evaluation for the PF took place at programme and portfolio level in cycles, usually on an annual basis, including on cross-cutting themes. This portfolio-level final evaluation focused specifically on the cross-cutting theme of G&I, with the overarching evaluation question: Is the PF promoting gender equality, women’s economic empowerment and social inclusion? The evaluation used a theory-based approach, synthesising findings from individual, independent programme evaluations and G&I scorecard assessments, annual reviews, monitoring data, and other programme documentation.

The evaluation found clear progress made by the PF on G&I, as a central part of achieving its primary purpose of promoting inclusive growth and poverty reduction. In particular, the evaluation noted that the PF and its programmes adopted approaches that align with widely accepted drivers of women’s economic empowerment (as identified by the UN Secretary General’s High-Level Panel on Women’s Economic Empowerment) and have also targeted interventions in ways likely to benefit other excluded groups. It identified examples of direct benefits for different target groups, such as: increasing incomes and access to markets for women farmers in Colombia, enabling poor farmers in China to access affordable insurance and make claims for weather-related crop damage, improving access to public transport for low-income women in Mexico, and helping secure the return of stolen assets to people living in poverty in Nigeria.

The evaluation also recognised that the PF G&I policy and programme guidance (2016), PF G&I Minimum Standards and Framework (2018), Cross-Government G&I Champions network and Social Development Advisers (SDAs from ex-DFID, FCO and Joint Funds Unit) have all played an important role in supporting integration of G&I considerations into programme design, implementation and capacity building. It confirmed that working towards the Fund’s clearly identified G&I minimum standards helped to ensure that PF programmes achieve G&I results and comply with the International Development (Gender Equality) Act. However, the evaluation considers that the scale of results is likely to be lower than originally envisaged (in part due to some programme delays and cuts), and the PF would have benefited from further consideration of G&I in programme design, particularly for earlier programmes; more internal PF and partner G&I capacity; more consistent senior leadership support; and further disaggregation of results. Nevertheless, despite the impact of COVID-19 and subsequent ODA reprioritisation, PF programmes evaluated during 2020/1 improved their performance against all G&I scorecard dimensions (design, capacity, accountability, financial resources and delivery partners). Evaluators found an increase in the proportion of programmes achieving the Fund’s minimum standards on design and internal capability from over 50% in 2018/19 to over 80% by 2020/21.

The impact: how recommendations were used

The evaluation was circulated to a cross-government network of programme teams who worked on the Prosperity Fund, with presentations of key findings and recommendations by PF’s G&I Evaluation and Learning Team. Relevant PF G&I Evaluation findings, lessons and recommendations are being used to inform next steps, Social Development advisory support and monitoring reviews for legacy prosperity programmes in different sectors, as well as wider policy and programme work in the FCDO’s Economic Co-operation and Growth Directorate. This PF evaluation usefully confirmed the value of tailored G&I guidance, adapting frameworks to different sectors (with minimum standards and increasing levels of ambition), and an active Community of Practice - which has informed approach to the development of future tools, good practice resources and capacity building work for promoting economic inclusion.

4. Looking forward: building an evaluation system for FCDO

Harnessing and developing expertise

The FCDO merger has brought together experts from DFID’s evaluation cadre and FCO M&E specialists, particularly those working on Joint Funds. FCDO is committed to nurturing that expertise whilst upskilling on evaluation across the organisation. In 2021 to 22, the Evaluation Unit will conduct a mapping exercise to better capture the expertise we have in the organisation and assess where further support is needed.

Developing an organisation wide strategy and policy

Our evaluation approach must be fit for the new organisation, covering our development and diplomacy ambitions and ensuring we invest in areas where learning is needed most. We will develop a strategic approach which allows us to maximise our range of monitoring and evaluation tools in a proportionate way. This will ensure we continue to learn from individual programmes, whilst providing more scope to evaluate beyond the programme level – across themes, portfolios or organisational trends.

An evaluation policy, outlining principles, minimum standards and basic processes and how they apply to evaluations in FCDO, will also be developed.

Annex 1: List of evaluation reports published 2020 to 2021

Note this is a list of reports published during this timeframe, not evaluations completed. There may be delays in the publication of some evaluations, meaning that the list below includes evaluations that were completely outwith 2020/21, and some which were completed during this period but are pending publication.

Project title	Department	Published date
Global Health Support Programme	China Global Programme Department	April 2020
Programme to pilot the use of Development Impact Bonds - a new payment by results tool - to achieve development outcomes (DIBs Pilot).	Private Sector Department	February 2021
Support to Uganda’s Response on Gender Equality (SURGE) Programme	Uganda	November 2020
Skills Development Programme	Pakistan	February 2020
UK Aid Match II Fund	Inclusive Societies Department	March 2021
Innovative Ventures & Technologies for Development (INVENT)	India	March 2021
CHURP Central African Republic Humanitarian Recovery Programme 2016 to 2019	Africa Regional/Africa (East & Central)/Humanitarian/WASH?	February 2021
Global Mine Action Programme 2	Stabilisation Unit	April 2020
Monitoring , Evaluation and Learning from the International Climate Fund	International Climate Change & Green Growth Dept	May 2020
Monitoring , Evaluation and Learning from the International Climate Fund	International Climate Change & Green Growth Dept	May 2020
Global Open Data for Agriculture and Nutrition (GODAN)	Global Statistics	July 2020
International Growth Centre (Phase 2)	Growth & Resilience Department	July 2020
Reducing Maternal and Newborn Deaths in Kenya	Kenya	July 2020
Making Country Health Systems Stronger	Human Development Dept	August 2020
Skills Development Programme	Pakistan	August 2020
Twende Mbele (“Going Forward”) – Strengthening African Monitoring and Evaluation Systems	Southern Africa	September 2020
Evaluation of the UK’s National Action Plan on Women, Peace and Security (2018-2022) : Strategic Outcome 7 - UK Capabilities	Stabilisation Unit	January 2021
Humanitarian Innovation and Evidence Programme: greater use of evidence and innovation in humanitarian responses	Research Department	September 2020*
Poorest States Inclusive Growth Programme	India	September 2020*
African Union Support Programme	Africa Regional Department	November 2020
Young Lives Survey- Improved understanding of the causes and effects of childhood poverty, inequality and education	Research Department	December 2020
Stamp Out Slavery in Nigeria (SOSIN) Programme	Nigeria	December 2020

*Date of the publication of the management response

1. The COVID-19 Global Evaluation Coalition is an independent collaboration made up of evaluation units from bilateral development co-operation providers, multilateral institutions, United Nations agencies and partner countries. ↩

2. Learning Review of Recent Agriculture Evaluations ↩

3. https://cedilprogramme.org/ ↩

4. https://www.worldbank.org/en/programs/sief-trust-fund ↩

5. https://egap.org/ ↩

6. https://www.worldbank.org/en/research/dime/overview#3 ↩

7. Numbers compiled from internal systems, including the Aid Management Platform, the Procurement Database, the EQUALS database, CSSF evaluation database. Includes evaluations recorded as ‘complete’ between April 2020 and March 2021. Data is largely input by users, and is subject to inaccuracies where the original data sets contained errors or outdated information. ↩

8. CSSF interventions are cross HMG, therefore the evaluation products will include aspects of interventions from other HMG departments. ↩