Corporate report

Evaluation Task Force Output and Outcome Indicators (March 2023): Technical Annex

Published 30 March 2023

1. Background

The Evaluation Task Force (ETF) published its strategy in November 2022 and committed to publicly report on a series of output and outcome indicators. This was in response to recommendations featured in an NAO report and the Public Accounts Committee’s recommendation for the ETF to establish quantifiable improvements on the scale and quality of evaluation across government. This paper accompanies the ETF’s March 2023 publication and provides supporting information to demonstrate how the figures for March 2023 have been derived for each of the indicators.

The ETF will report publicly on each priority indicator on an annual basis. The indicators are not wholly controlled or ‘owned’ by the ETF. Driving progress towards the targets outlined in the ETF strategy will be dependent on cross-government partners working together to build an improved evaluation ecosystem which creates more and higher quality evaluation in government. For example, the ONS Analysis Function is responsible for professional standards in evaluation.

Across indicators that note ‘within the last 12 months’ this denotes financial years. For example, in this round of reporting dates are 1st April 2022 to 24th March 2023^{[footnote 1]}. Where cumulative figures are reported, these are taken from the ETF inception (April 2021).

2. Output and Outcome Indicators

2.1 Indicators 1.1, 1.2, 3.1 – ETF Advice

Indicator	Data Source
1.1 Number of evaluations ETF has advised on (cumulative)	ETF monitoring data
1.2 Value of programmes ETF has advised on (cumulative)	ETF monitoring data
3.1 Number of HMT business cases and spending review bids that the ETF has advised on (cumulative)	ETF monitoring data

The ETF systematically records information on the advice delivered across all three of its support ‘tiers’^{[footnote 2]}. This includes the value of programmes advised on across the Spending Review 2021 settlement period, as well as the channel the advice has been delivered through. The ETF advises on evaluation design and delivery through a variety of channels including:

• Complex Grants Advice Panels
• Business Case reviews
• Spending Reviews (e.g. SR21)
• Shared Outcomes Fund
• Evaluation Accelerator Fund
• Direct advice to government departments
• Infrastructure and Projects Authority/Government Major Projects Portfolio
• Evaluation and Trial Advice Panel

The figures included in this report reflect a total count of evaluations advised on by the ETF, the cumulative value of these programmes over the SR21 period, and a count of advice recorded as being delivered to HMT on spending review bids or business case reviews.

Indicator 1.1 Number of evaluations ETF has advised on (cumulative) is 211. This includes all programme evaluations where the ETF has delivered Tier 1, Tier 2 or Tier 3 support. This excludes advice provided on specific Spending Review business cases only.

Indicator 1.2 Value of programmes ETF has advised on (cumulative) is £115 billion. This includes the value of programmes over the SR21 period where the ETF has delivered Tier 1, Tier 2 or Tier 3 support. This excludes advice provided on specific Spending Review business cases only.

Indicator 3.1 Number of HMT business cases and spending review bids that the ETF has advised on (cumulative) is 97. This includes the number of outline and programme business cases; and Spending Review business cases shared by HM Treasury with the ETF for review on the quality of evidence used to inform the business case and the quality of evaluation plans.

2.2 Indicator 1.3 – Evaluation awareness training

Indicator	Data Source
1.3 Number of evaluation awareness-raising opportunities ETF has led in past 12 months	Speaking events, written pieces

The ETF systematically records when it has designed and delivered evaluation awareness-raising events to understand the scale and nature of its communications. These events are defined as: presentations, conferences, and teach-ins delivered beyond regular department engagement (e.g. regular catch-ups or meetings). This includes delivering a speech and/or presentation to an audience either within or outside of government. The total number of ETF led evaluation awareness raising events delivered between 1st April 2022 to 24th March 2023 is 31.

2.3 Indicator 1.4 – Evaluation Accelerator Fund

Indicator	Data Source
1.4 Proportion of Evaluation Accelerator Fund projects on track (RAG rated ‘Green’)	Quarterly monitoring

The Evaluation Accelerator Fund (EAF) is a £15m fund designed to accelerate evaluation activity and the creation of actionable evidence in HMG priority areas. EAF projects are supported by ETF technical leads who provide advice and support to projects on technical evaluation issues, and facilitate engagement with relevant policy and research teams across government. Each project is subject to a quarterly monitoring and review process, where technical leads meet with project teams to discuss the progress of projects and their RAG status for time, cost and scope (i.e. is the programme delivering to planned timelines, within planned costs, to the agreed scope). Any adjustments to programme delivery are discussed and agreed upon through this process, with issues escalated to the EAF SRO where necessary.

An overall RAG rating to assess if projects are on track or if there are any significant risks or issues is also agreed upon through this process. The RAG ratings included in this report reflect the status of projects at the end of the 2023 financial year (the first year of EAF funding) and are an assessment of if projects are being delivered to the agreed scope, on time and within budget - and of any identified risks to future delivery. 47% (7 of 15 projects) of EAF programmes were rated ‘Green’, 7 were rated ‘Amber’ and 1 ’Red’. Appropriate mitigations have been identified to address issues identified with those projects that are not currently rated ‘Green’. Many projects rated ‘Amber’ are in this category due to challenges with recruitment. The ‘Red’ project will be subject to closer monitoring and support from the ETF.

2.4 Indicator 1.6 – ETF priority projects

Indicator	Data Source
1.6 Proportion of ETF priority projects with robust evaluation plans (cumulative)	ETF monitoring data

Following SR21, the ETF selected a range of priority areas for more intense scrutiny and/ or practical support over a longer period. Further information is here.

Evaluation robustness assessment

The framework for assessing the robustness of evaluation plans is modelled on the approach used in the ETF’s wider work on Major Projects. Briefly, the framework includes the following assessment factors:

1. Quality of scoping phase. For example, existence of a Theory of Change, clear problem definition and/or articulation of key research questions; and understanding of the intervention design.

2. Appropriateness of method/s. Methods which align with the core principles set out in the Magenta book and are the most robust methods to use given the nature of the project. This may also include HMT Green Book supplementary guidance on economic evaluation as appropriate.

3. Resources allocated to the evaluation in proportion to the scale of the project. For example the evaluation budget and/or department analytical resource dedicated to the project.

A Red, Amber, Green (RAG) rating has been applied across each of the above areas to provide an overall RAG rating for each project. 73% (11 of 15 projects) were rated ‘Green’ in March 2023.

The ETF provides support and scrutiny throughout the lifecycle of the evaluation but it is the responsibility of the department to deliver a robust evaluation which can provide high-quality evidence of effectiveness.

2.5 Indicator 2.1 – Evaluation strategies

Indicator	Data Source
2.1 Number of departments with a departmental evaluation strategy published (one-time reporting)	ETF monitoring data

The ETF worked with HM Treasury to develop a series of settlement conditions at SR21, requiring ETF involvement in programmes as well as general conditions. The general conditions pushed the evaluation agenda across HM Government, for example, by asking all departments to commit to and publish departmental evaluation strategies outlining a long-term approach to monitoring and evaluating their programmes and policies.

The ETF maintains a list of published department evaluation strategies on its website. 12 departments had published an evaluation strategy by 24th March 2023.

2.6 Indicator 2.4 – Published evaluation reports

Indicator	Data Source
2.4 Number of evaluations with published reports on GOV.UK in last 12 months	Evaluation Registry, department data

The Evaluation Task Force, alongside i.AI (the Cabinet Office’s Incubator for Automation and Innovation), has developed an Evaluation Registry, a unified location to amass evaluation evidence and increase the visibility of evaluations across government. For the year to March 2023, the number of evaluations published has been established via a search of gov.uk that was conducted in order to pre-populate the Evaluation Registry. Following the launch of the Evaluation Registry later in 2023, future statistics on the numbers of evaluations published will be accessed directly from the Registry. 108 evaluations were published on gov.uk between 1st April 2022 to 19th March 2023.

2.7 Indicator 3.2 – Evaluation settlement conditions

Indicator	Data Source
3.2 Proportion of departments who are compliant with department-level evaluation settlement conditions (cumulative)	ETF monitoring data

The ETF monitors and oversees departments’ compliance with evaluation settlement conditions. These conditions, set at Spending Review 2021, were put into place to advance the evaluation agenda across HM Government. Departments were asked to share materials that satisfied the conditions with the ETF. Original due dates for returning the settlement conditions were staggered throughout 2022.

The three department-level settlement conditions set for each of the ministerial departments were:

• Provide the Evaluation Taskforce (ETF) with an outline plan to address evidence gaps ahead of the next spending review
• Publish an evaluation strategy which outlines a department-wide approach to identifying and priotising areas for evaluation activity
• Provide ETF and HM Treasury with an assessment of how much evidence you have to support the 20 highest cost policy areas/programmes, using the Nesta Standards of Evidence

As of 24th March 2023, 66% (12 out of 18) of departments were compliant with all of the core department-level evaluation settlement conditions.

2.8 Indicators 3.3, 3.4, 3.5 – ETF annual survey of senior stakeholders

Indicator	Data Source
3.3 Proportion of senior evaluation stakeholders who agree evaluation evidence informs the design and delivery of policies	ETF annual survey of senior evaluation stakeholders
3.4 Proportion of senior evaluation stakeholders who agree their department needs to do more to ensure evaluation evidence informs spending decisions	ETF annual survey of senior evaluation stakeholders
3.5 Proportion of senior evaluation stakeholders who agree that SCS support evaluation	ETF annual survey of senior evaluation stakeholders

The figures for indicators 3.3-3.5 included in this report are derived from an online survey conducted by the ETF which is designed to gain insights from senior evaluation stakeholders on how evaluation is prioritised across government. The ETF will conduct the survey on an annual basis.

Indicator 3.3 Proportion of senior evaluation stakeholders who agree evaluation evidence informs the design and delivery of policies is 80%.

Indicator 3.4 Proportion of senior evaluation stakeholders who agree their department needs to do more to ensure evaluation evidence informs spending decisions is 83%.

Indicator 3.5 Proportion of senior evaluation stakeholders who agree that SCS support evaluation is 80%.

Sampling

The survey sample includes the following target groups:

• Directors of Analysis and/or Chief Analysts
• Heads of Policy Profession
• GSR Heads of Profession
• Heads of Evaluation
• Chief Scientific Advisers (CSAs)

The ETF’s directory of stakeholders across government was used as the sample frame for the survey. This provided a total sample of 116 individuals, or individual email addresses.

Survey

The ETF carried out an online survey in January and February 2023. An email inviting the groups outlined above to take part was sent on 12th January 2023. The survey was open between 12th January and 10th February.

Response

The table below outlines response rates across each of the target groups included in the sample by number and by percentage of the total response. There were a total of 31 responses, representing a response rate of 27% from the sample of 116.

In four cases, respondents selected the ‘Other’ option for their job role. This group includes:

• a consolidated response from a Chief Analyst, GSR HoP, and Team Lead for Evidence and Evaluation;
• an Evaluation Lead;
• a Public health consultant and UK NSC lead officer; and
• a joint reply from two Heads of Evaluation

Role	Number of responses	Percentage (%)
Director of Analysis or Chief Analyst	8	25.8
Head of Evaluation	7	22.6
GSR Head of Profession	6	19.4
Head of Policy Profession	3	12.9
Chief Scientific Adviser	3	9.7
Other	4	9.7
Total	31	100

Data analysis

Responses were collected anonymously via an online platform and downloaded for analysis in Excel. Figures were cross tabulated to analyse responses disaggregated by role. However, given the low response rate and small number of total responses, it has not been possible to report on findings at this level. The small number of total responses also means that caution should be exercised in the interpretation of the findings. For example, it may be inappropriate to make inferences surrounding the degree to which the response is representative of the views of the target groups included in the sample.

Responses to the questions outlined in the table below provided the basis for the figures included in this report.

Indicator	Question: Thinking about evaluation in your department, to what extent do you agree with the following statements?
3.3 Proportion of senior evaluation stakeholders who agree evaluation evidence informs the design and delivery of policies	In my department, evaluation evidence informs policy design and delivery.
3.4 Proportion of senior evaluation stakeholders who agree their department needs to do more to ensure evaluation evidence informs spending decisions	My department needs to do more to ensure evaluation evidence informs spending decisions.
3.5 Proportion of senior evaluation stakeholders who agree that SCS support evaluation	Senior civil servants support evaluation in my department.

3. Appendix: Full Questionnaire

3.1 Background

Please select your role(s) from the list below:

• Chief Analyst / Director of Analysis
• Head of Policy Profession
• GSR Head of Profession
• Head of Evaluation
• Chief Scientific Adviser
• Other - please specify

Please select your grade:

• Grade 7
• Grade 6
• SCS 1
• SCS 2
• SCS 3
• Other - please specify

3.2 Evaluation in your department

Thinking about evaluation in your department, to what extent do you agree with the following statements?

1= strongly disagree, and 7= strongly agree.

[Select from scale for each statement]

Evaluation design and delivery

• My department has sufficient resources to deliver evaluation in all priority areas
• It is difficult to get evaluation reports published within 12 weeks in my department
• We have the right skills to undertake high quality evaluation in my department
• There are enough people with the right skills across government professions to deliver on priority evaluations
• Short term delivery pressures usually trump evaluation planning in my department
• We have access to the data we need to undertake high quality evaluation in my department

Supporting evaluation

• Evaluation is actively and consistently promoted in my department
• Senior civil servants support evaluation in my department
• There is a clear commitment to transparency in my department
• In my department, senior civil servants have a high level of evaluation awareness e.g. they know how to evaluate or are aware of the benefits
• There is a lack of ministerial support for evaluation in my department

Using evaluation evidence

• In my department, evaluation evidence informs policy design and delivery
• My department needs to do more to ensure evaluation evidence informs spending decisions
• Policies with both a strong evidence base and a good evaluation plan are most likely to get implemented in my department
• My department takes a strategic approach to ensure we have the evidence we need to inform decision making

3.3 Using evaluation evidence

How satisfied are you with the degree to which evaluation evidence is used to inform policy development and spending decisions in your department?

Very dissatisfied

Somewhat dissatisfied

Neither satisfied nor dissatisfied

Somewhat satisfied

Very satisfied

Please use the space below to provide further information on your answer. You may wish to reflect on key facilitators and barriers to using evidence in your department.

[Free text]

On a scale of 1-7, where 1 = not at all challenging, and 7 = extremely challenging,

How challenging is it to get evaluation evidence used in decision making in your department?

[Select from scale]

3.4 Evaluation Task Force

The Evaluation Task Force is a joint Cabinet Office-HM Treasury unit set up following the 2020 Spending Review to ensure evidence sits at the heart of government decision-making.

The ETF aims to drive continuous improvements in the way government programmes are evaluated in order to inform decisions on whether they should be continued, expanded, modified or stopped. These decisions should improve public sector outcomes.

Before today, were you aware of the Evaluation Task Force (ETF) and its strategic aims?

• Yes, aware of ETF and strategic aims
• Yes aware of ETF, but not of strategic aims
• Not aware of ETF or strategic aims

The ETF also supports the network of What Works Centres to ensure departments are taking advantage of their evidence and policy expertise. These centres have a shared aim of improving the way government and other public sector organisations create, share and use high-quality evidence in decision-making and are often well-placed to provide support, resources and evaluation advice on specific policy issues.

Before today, were you aware of any of the What Works Centres (WWCs) and their aims?

• Yes, aware of one or more WWCs and their aims
• Yes aware of one or more WWCs, but not of their aims
• Not aware of any WWCs or their aims

What more could be done cross-government to ensure evaluation evidence is at the heart of decision-making?

[Free text]

1. In line with PAC recommendations, the ETF must publish these indicators by the end of each financial year. As such, the last date data was collected was 24th March 2023. ↩

2. ETF support falls into three categories, or tiers: 1) ETF-identified priority areas and programmes; 2) Programmes with evaluation funding conditions and/or of interest to Spending Teams and Ministers; 3) All other programmes seeking one-off evaluation support. ↩