Research and analysis

Climate Smart Agriculture thematic review: management response to evaluation report

Updated 4 March 2022

The thematic review of Climate Smart Agriculture (CSA) aggregated and synthesised evidence from a portfolio of Foreign Commonwealth and Development Office (FCDO) programmes to draw out learning on reducing Smallholder Farmers’ (SHF) vulnerability to climate variability and shocks.

The review covers thirteen programmes seeking to reduce poverty by building resilience or raising productivity through diverse strategies, including agricultural development, market systems development, livelihoods, food security, landscape and environment focused approaches.

The thematic review was commissioned by the Evaluation Unit, in collaboration with Economic Co-operation and Growth Department, the Climate and Environment Directorate and Research and Evidence Directorate. It aimed to provide evidence from FCDO programming to inform the FCDO’s approach to events including the UN Food Systems Summit 2021, the UK’s COP26 Presidency campaigns on nature and adaptation, and the FCDO-led famine prevention campaign. It supported FCDO food systems and agricultural policy and programme development on food by delivering recommendations on ‘triple wins’ for nutrition, climate resilience and adaptation, and economic benefits. Its focus was on learning rather than accountability. The review is published, in line with FCDO commitments to transparency on GOV.UK, so that the findings are available to other donors, agencies and academic bodies working in this field, as well as to members of the public.

The scope of the review was to determine the relevance, effectiveness, impact and sustainability of different CSA interventions in the context of different geographies, climate variability and target groups. The review was conducted remotely using mainly qualitative data gathered through document review and key informant interviews, due to Covid limitations on travel. The review used a rigorous methodology to triangulate evidence sources supporting the findings. The evaluation team worked closely with FCDO advisors to ensure that the focus was relevant and useful to FCDO’s priorities.

Key findings

• CSA approaches are more likely to be adopted when they have been developed through participation with beneficiaries, where they are productive and profitable to farmers and businesses, and where the necessary resources and services to enable adoption are available
• Most programmes focused on potential synergies between CSA pillars (sustainable productivity, adaptation and mitigation) rather than being explicit about trade-offs between them (for example, mechanisation creating trade-offs between productivity and mitigation)
• Programmes largely focused on delivery of CSA interventions rather than measuring their secondary environmental co-benefits
• Sufficient synergies between production, adaptation and mitigation are critical to CSA. This may require the judicious use of temporary subsidies, long-term private sector investments, and governments that demonstrate their commitment to CSA with matching fiscal expenditure
• While most programmes had a goal of increasing resilience to climate variability and shocks, approaches to defining and measuring resilience were insufficient and inconsistent across the programmes. This made it challenging to identify common factors for success in building resilience
• Terms like CSA and Nature-based Solutions (NbS) were used generically by programmes. This obscured the linkage between CSA activities and the design and monitoring of productivity, adaptation and mitigation outcomes. This limited learning on what works, when and for whom
• Significant time-periods (10-20 years) may be required to bring about changes in resilience, though programme funding or activities often cover shorter periods

FCDO welcomes the findings of this thematic review, and accepts most of its recommendations. Some of these recommendations reinforce the need for processes and guidance that FCDO has already put in place to ensure high standards of programme delivery, such as the FCDO Programme Operating Framework (PrOF), launched in 2021, and the ‘Smart Rules’ used by the former Department for International Development.

This management response has been prepared by the Economic Co-operation and Growth Department, the Climate and Environment Directorate and the Evaluation Unit.

Recommendation a)

On resilience and sustainability: Design for climate resilience by identifying the priority threats and best bet opportunities relevant to different target groups within the specific context of the project. Consider opportunities to enhance anticipation, adaptation and absorption and the future capacity of participants to continue to adapt beyond the project to achieve transformational change. Make resources available for ex-post learning and monitoring and/or evaluation of sustainability. This will show whether the programme has achieved its objective of sustainability. It may also establish whether farmers have developed an ability to cope with climate variability and shocks.

Response: accepted

The FCDO has the processes described below already in place to support ongoing engagement, monitoring and learning on resilience.

The new FCDO Programme Operating Framework (PrOF), launched in 2021, supports embedding climate resilience within the design of programmes through PrOF rules 5 and 20:

Rule 5: All programmes (and the projects, interventions or events within them) must align with the Paris Agreement – an international treaty on climate change – and assess climate and environmental impact and risks, taking steps to ensure that no environmental harm is done. Any International Climate Finance (ICF) programmes must identify and record ICF spend and results.

Rule 20: All projects, programmes and portfolios must have sufficient monitoring in place to provide performance and financial oversight, manage risks and support decision-making at appropriate levels.

The portfolio-level indicators for climate programmes include ICF KPI 1 which measures the reach of adaptation programmes; ICF KPI 4 which measures their success in terms of the number of beneficiaries with improved resilience; and ICF KPI 15 which uses a scorecard approach to assess the potential of a programme to bring about transformational change. Climate programmes are required to include ICF KPIs in their monitoring framework relevant to the specific climate aims identified in the business case. The ICF KPI 4 methodology allows programme teams to choose an appropriate resilience model. The 3As model is an example. This tracks improvements over three dimensions of resilience: anticipatory capacity, adaptation capacity, and absorptive capacity. Programme managers are supported to apply the ICF KPI 4 and ICF KPI 15 methodologies through recorded webinars on a widely accessible MS Teams site.

The FCDO will continue to invest in climate resilience research and learn lessons for policy and programme design from this. For example, learning on resilience will be supported by the internal launch in early 2022 of the ‘Best Buys’ on nature, which will summarise rigorous research on the effectiveness of different nature-based interventions.

Decisions on programme monitoring and evaluation budgets are devolved to senior responsible owners and approved by development directors. Budget for a formal evaluation is included if needed to meet evidence gaps that are not met through regular monitoring activities. An evaluation funded in this way would run concurrently with the programme to inform decision-making such as programme scale-up or refocus. The FCDO’s new approach to evaluation (currently under development) is considering ways to support post-programme evaluation centrally, if this would meet strategic evidence gaps for the Department. Furthermore, there are plans to establish a new centrally managed programme to deliver experimental and quasi-experimental evaluations for high priority areas.

Recommendation b)

On adoption: Be clear about what is meant by adoption and set appropriate indicators. Build in time for participatory scoping of CSA options and allow for their iterative adaptation during implementation. Ensure inclusive access to resources and support services in a sustainable manner. Build in sufficient granularity in monitoring to be able to establish which groups adopt CSA practices and why/why not.

Response: partially accepted

FCDO accepts the recommendation that ‘adoption’ should be clearly defined in business cases and annual reviews, wherever the term is used and that time should be allowed for participatory scoping of CSA options and iterative adaptation. A principle of FCDO’s Programme Operating Framework is that our programmes should be context-specific, draw on available evidence and listen to views of stakeholders. The programme operating framework provides guidance to integrate engagement with beneficiaries and other programme constituents into decision-making, monitoring and evaluation and how to ensure that requirements to engage are included in contracting arrangements that FCDO has with its implementing partners. FCDO recognises the particular importance that engaging beneficiaries has in driving uptake of climate-smart agriculture approaches.

Furthermore, FCDO PrOF guidance encourages programme teams to provide disaggregated data and analysis by sex, age, disability, geography and other characteristics to understand how interventions impact upon gender equity and other potentially excluded groups.

FCDO also recognises that good programme design requires scoping of options for providing accessible services for climate-smart agriculture. However, the extent to which inclusive access to goods and services is incorporated in programme design depends on the scope of the programme and the market opportunities on offer. Thus this recommendation is partially accepted.

Recommendation c)

On designing and measuring relevant outcomes: For CSA (or sustainable agriculture intensification or NbS) programmes, break down the term to its specific components so as to be able to set outcomes that are relevant and measurable. In the case of CSA start by looking at what the programme aims to achieve in terms of productivity, adaptation and mitigation. Key outcomes like production, adaptation, CC mitigation and other environmental services need to be clearly defined, with targets set where possible in relation to objective needs and the priorities of diverse participants. The ToC should reflect the interaction between these diverse participant types, possible CSA approaches and required outcomes. This complexity is likely to require elements of participatory design on what is expected to work, for whom, when, where and why. Monitoring, reporting, implementation adjustment and evaluation should also reflect outcomes for different participant types, including representative participant voices.

Response: partially accepted

FCDO partially accepts the recommendation that business cases incorporating climate-smart agriculture and nature-based solutions should align with consistent definitions of these terms. However, given the terms are internationally contested, and not necessarily always used in applicable contexts, it is probably more important that FCDO programme documentation includes full information on specific practices being proposed/implemented in a particular programme.

A rule of FCDO’s Programme Operating Framework is that all projects, programmes and portfolios must have sufficient monitoring in place to provide performance and financial oversight, manage risks and support decision-making at appropriate levels. Programme monitoring is aligned with a theory of change in the business case, which articulates the causal chain from inputs, through outputs, to outcomes and impact. The focus of this approach to programme monitoring is very much on outcomes which measure the intended benefit of the programme. Given the confusion on definitions of CSA and NbS, FCDO will define our outcomes clearly in programme documentation. Guidance on ICF nature is being developed to support this. Since 2020, a new ICF key performance indicator for International Climate Finance (KPI 17) has been included in portfolio monitoring. This measures the area of land which has received sustainable land management practices. A clearly written methodology note defines terms and explains how to report against this indicator.

See also the response to (d) below in relation to participant voice.

Recommendation d)

On the enabling environment and subsidies: Create an enabling environment through wide-ranging consultations during programme design to ensure government commitment, private sector participation, legitimate farmer organisations and sufficient intrinsic benefits of CSA interventions to motivate adoption. During programme design, consider the type, level, timing, and need for subsidies – direct and indirect – bearing in mind their impact on sustainability and resilience to climate change when projects end.

Response: accepted

FCDO recognises the importance of wide-ranging consultation during design and implementation, including consultation with beneficiaries as well as stakeholders in the government and private sector. This is reflected in its programme operating framework.

FCDO has much experience working on private sector development and will continue to factor in sustainability considerations on the use of subsidies. For example, programmes using the ‘Making Markets Work for the Poor’ approach have aimed to create changes without the use of subsidies. These programmes have been an important part of the agriculture portfolio.

See also response to (e.) below.

Recommendation e)

On carbon finance and CSA/NbS. FCDO should investigate opportunities to increase CSA and NbS adoption and sustainability through blended sustainable carbon (or PES) finance, complementing development finance in projects like those found in the portfolio. Blended finance approaches can include climate or PES credits to reward public good outcomes and to enable longer-term sustainability. This finance may be most effective at the latter part of the project cycle after awareness raising, CSA demonstration, governance and community organisation capacity building is underway. There may also be a role for FCDO in building capacity in aggregator and other carbon credit and PES service providers, or empowering programme implementors to investigate these financing opportunities.

Response: accepted

FCDO recognises the importance of looking to alternative and longer-term sources of finance to sustain changes sought in farming practices. While FCDO agriculture programmes have not to date directly supported growth in carbon markets, this is actively being considered in the design of future programmes.

Recommendation f)

On addressing time-lag issues: Design CSA programmes with timeframes appropriate to the need and with mechanisms to overcome any incentive gap between adoption and farmer benefit. Finding ways to overcome the time-lag between participant effort and benefit from adopting some longer-term CSA/NbS approaches need to be considered in project design. Project timeframes for should be based on realistic estimates of the time needed to achieve not only immediate objectives but longer-term sustainability, resilience and transformation. This is likely to vary with the type of CSA being proposed and also the readiness of project participants. Where longer-term approaches look likely to deliver significant benefits, longer project timelines and/or innovative ways of bridging this gap may need to be incorporated. In some cases, carbon finance or other PES should be explored as an option.

Response: partially accepted

Although programmes are usually designed with longer time frames in mind, the duration of FCDO funding commitments is closely linked to the length of the government Spending Review cycle. This is determined by HM Treasury, and sometimes lasts only 1 year. Nevertheless, most programmes included in this review have implementation periods that cover several growing seasons. Programmes review sustainability throughout their lifetime in order to support high-potential longer-term approaches.

A robust theory of change within the business case helps to manage the risk of incentive gaps. FCDO’s Programme Operating Framework provides guidance on using theory of change tools and processes during programme design to articulate how the intervention will achieve its goals in its specific context and assumptions underpinning the expected changes.