Policy paper

Driving our performance: DBT’s monitoring and evaluation strategy 2023 to 2026 (web version)

Updated 17 November 2023

This was published under the 2019 to 2022 Johnson Conservative government

Forewords

Gareth Davies – Permanent Secretary

The Department for Business and Trade (DBT) is a newly formed department that brings together Britain’s world-class business expertise with trade policy, negotiations and promotion. We are the department for economic growth. We do this by:

• redrawing our rules to ensure businesses thrive, markets are competitive, and consumers are protected
• securing investment from UK and international businesses
• advising, supporting, and promoting British businesses to grow and export
• opening up new markets for businesses by removing barriers and striking trade deals
• promoting free trade, economic security and resilient supply chains

This is a refreshed strategy, following the Machinery of Government changes and the creation of DBT in 2023. It articulates the collective approach for the newly formed department and most importantly, how we will implement the strategy over the next three years.

DBT’s analytical functions and analytical skill sets are critical to its mission and vision. Being accountable for taxpayers’ money and having evidence-based policies is essential to achieve our mission.

Policy should be guided by analysis, monitoring and evaluation. DBT is a learning organisation. Monitoring and evaluation are fundamental activities for any learning organisation that is keen to improve its performance. They help to foster a culture of innovation and learning.

Monitoring and evaluation have a crucial role to play at 3 stages within the policy cycle. These include programme design stage to inform and clarify objectives, at implementation stage to flag risks and enable adjustments, and at exit stage to gather lessons learned. This strategy sets out how we intend to achieve this.

Ceri Smith – Director General, Strategy and Investment

This is a joint strategy that brings together our policy profession, operational delivery and analytical communities. Ultimately using evaluation to drive our performance is about making better policies and delivery strategies. Evaluation evidence informs policy design, helps set clear objectives to measure progress and builds in feedback loops and robust evaluation plans to address evidence gaps.

Our department’s mission is to enable economic growth and to make a difference to people’s lives across the country. We combine excellence with pace to maximise our impact. As the flagship economic growth department, we will support economic growth across the country and continue to represent the UK on the world stage, support British businesses and champion free trade.

It is more important than ever to embed monitoring and evaluation and measure our impact. It will help us capture the impact of our regulation on businesses, consumers and trade flows. We have export and investment transformation programmes and need a coherent approach to understand their value to the UK economy, consumers, regions and trading partners. This evaluation strategy is a fantastic opportunity to understand more about our impact on growth for UK businesses and economy.

Gavin Lambert and Caleb Deeks – Director General (Jobshare), Competition, Markets and Regulatory Reform Group, Joint Heads of the Policy Profession

This strategy is for everyone in DBT. Accountability and learning are critical areas not only across all analytical professions but also crucial for the policy and project delivery professions.

As DBT’s joint Heads of the Policy Profession, we are keen that the DBT policy community understands how monitoring and evaluation can make policies better – to do policy well, we need to know if it has worked. Monitoring and evaluation helps comprise the key skills of all policy practice which form the Policy Profession Standards framework.

This joint strategy is a great opportunity to support the professional development of policy colleagues across DBT. It reminds us that monitoring and evaluation fits throughout the policy cycle and the Policy Profession has a key role to ensure policies deliver the intended impacts. It complements the role of our existing structures that manage performance and ensure quality programme delivery.

We want to create a one-DBT environment, which promotes an inclusive culture of effective networking and knowledge sharing between business and trade colleagues in DBT. Our department has great opportunities for sharing learning and retaining longer-term institutional knowledge of what has and hasn’t worked in the past. Playing this learning back into policy development and future programmes and projects is vital.

Executive summary

The why

Holding ourselves to account, reflecting on our experiences and learning from our mistakes are core values of DBT. DBT believes monitoring and evaluation will help achieve our vision: to drive economic growth, create and support jobs and back British businesses at home and abroad, and helping to level up the country.

Timely performance data and insights on what works enable us to help more businesses grow, export, import and attract inward and outward investment, as well as protect consumers. It is also essential to help us secure investment, help business grow and export and redraw our regulations so that businesses can thrive and consumers are protected.

We see having a compelling evidence base as helping to avoid the risk of spending public money on activities that do not work as intended. We will use monitoring and evaluation to support policy and operational delivery, inform our interventions and guide decisions on whether to continue, stop or amend our interventions. For example, the UK is negotiating new free trade agreements as an independent trading nation.

The insights from planned evaluations of the free trade agreements that we have already conducted and agreed will inform our negotiations with other countries. We will evaluate the extent to which our regulations remain fit for purpose and have enabled the department to identify areas of improvement to support UK businesses. This will ensure DBT is focusing its resources where they add most value, maximising the impact and cost-effectiveness of our investment of taxpayers’ money.

The how

To deliver on this vision DBT will draw on its greatest asset – our people. Championed by senior leaders, we will use our expertise across policy and analytical professions, and data scientist, project delivery and data, digital and technology, and science and engineering communities.

We will embed monitoring and evaluation into DBT processes, making it a fundamental part of the policy/programme cycle. This will ensure DBT is focusing its resources where they add most value. Our analysts will make sure that evaluations in DBT are designed to the highest standards. This will ensure we provide the best evidence we can use to make a difference – in terms of timeliness, relevance and accessibility.

The what

Our strategy relies on effective use of budgets, resources, governance processes and operational tools to achieve our objectives. DBT is committed to resourcing evaluation staffing and budgets adequately to undertake high quality evaluations and proactively use results to inform our work.

We are committed to ensuring our programmes and activities allocate appropriate and proportionate resources to evaluation – whether it is funding to outsource independent evaluations, or securing analytical resource to undertake evaluation internally. We will also ensure sophisticated and accessible presentation tools are in place to enable this work to be both usable and impactful.

How we will measure success

We will increase the volume of high-quality monitoring and evaluation (M&E) plans and demonstrate robust planning, commissioning and delivery and use of M&E activity. Creating enhanced recognition and understanding of monitoring and evaluation across the DBT community will enable us to demonstrate learning from our evaluation evidence base. It will also offer increased ability to account for our impact which we will share through publishing our evaluation findings. Figure 1 shows our guiding principles for monitoring and evaluation in DBT which are described in Section 2 of this document.

Figure 1: our guiding principles for monitoring and evaluation in DBT

Figure 1 includes Principle 1 - use of monitoring and evaluation evidence to make decisions and Principle 2 – strong governance for M&E. Principle 3 covers capability building in M&E for all staff and Principle 4 covers transparency and accessibility of evaluation evidence. The final elements are Principle 5 - proportionate coverage for M&E and Principle 6 – high quality standards for M&E.

1. Introduction

Understanding performance is the foundation from which any organisation can aim to improve its services. Monitoring and evaluation support this by collecting systematic evidence of what has been achieved and the impact of our activities. This then illustrates the added value of our services, what works and what doesn’t work, how and why. Creating this evidence base can be supported by good quality management information as an important source of data to enable performance tracking over time.

a. Purpose of the strategy

This is a refreshed strategy, following the Machinery of Government changes and the creation of DBT through merging 2 government departments in 2023. It articulates the collective approach for the newly formed department and how we will implement the strategy over the next 3 years.

This strategy is for everyone in DBT and aims to embed the use of evidence to understand what works at the heart of DBT’s activities. It will touch on the work of many colleagues across all professions including policy, project delivery, and analysts. Our commitments will help raise understanding of performance across the department by outlining how M&E will support our policy making and service delivery.

The outcome indicators used in M&E frameworks will be aligned to our approach for departmental performance. Evaluation priorities will feed into DBT’s Areas of Research Interest publication which communicates our evidence and research needs to external stakeholders.

b. Why using monitoring and evaluation to understand performance is important for DBT

Monitoring and evaluation are essential to help us deliver on DBT’s vision and mission and keep us on track. Our mission is to facilitate economic growth, support businesses and industries to grow and thrive, secure domestic and foreign investment, leading to, better jobs, higher wages and improved UK and global living standards. To achieve this, DBT aims to help businesses export and grow nationally and in global markets, as well as helping overseas companies to grow in the UK. Evidence gathered via monitoring and performance reports and evaluations is therefore essential to prioritise our resources and efforts towards this. By understanding the performance of our policies and programmes, DBT can achieve its mission faster and better. We will be even more effective in helping UK businesses grow and facilitating economic growth and prosperity through redefining regulation and increased trade and investment.

Evaluation can demonstrate the impact of what we have achieved and show value for money of our work to the public. As stated in our export strategy, DBT tries not to offer services that the market is better placed to provide. Rather we focus on where markets fail and where we can add value^{[footnote 1]}. We also have to deal with matters that are essentially intergovernmental. To do that well, we need timely evidence of our progress and examples of where we add value to fill evidence gaps and inform our future work. To provide the right mix of services to businesses we need to measure their relative effectiveness, return on investment and value added.

So, as a department, we are keen to build an agile evaluative approach to inform our progress towards our goals. This could involve new ways of using existing data. For example, using export, investment and automotive transformation data can reduce the lag between insight and action to proactively guide intervention design.

We will use monitoring and evaluation to support policy delivery, inform our interventions and guide decisions on whether to continue, stop or amend our interventions. The UK is negotiating new free trade agreements as an independent trading nation and insights from these evaluations will inform negotiations with other countries. We will also evaluate the extent to which regulations remain fit for purpose following EU exit and have enabled the Department to identify areas of improvement to support UK businesses. This will ensure DBT is focusing its resources where they add most value, maximising the impact and cost-effectiveness of our investment of taxpayers’ money. Strong M&E evidence also gives assurance to continuing initiatives, interventions and policies that are proven to work while informing decisions to deprioritise investments. To support these decisions, we aim to gather insights and evidence by using a combination of established and innovative approaches.

c. Monitoring and evaluation planning for statutory commitments to review regulation

Our department is required to produce impact assessments (IAs) assessing the costs and benefits of regulatory changes prior to consultation, enactment and implementation. The evidence and analysis used within IAs are scrutinised by the independent Regulatory Policy Committee (RPC).

Post implementation reviews (PIRs) of these regulatory changes are a key element of the policy-making cycle and provide an evidence-based evaluation of the effectiveness of a measure after it has been implemented and operational (after an appropriate period of time).

A PIR will review: the original policy objectives, the extent to which the measure is achieving its intended effects/meeting its objectives, whether there have been any unintended consequences, how well it is working, and the reasons why. It will also assess whether the objectives could be achieved with a system that imposes less regulation.

Evidence from PIRs will support decisions about the next steps for a measure, which are:

• renewal - measure continues without change
• amendment - measure remains but changes are made to improve it
• removal - measure is removed without replacement
• replacement - measure is replaced or redesigned substantially

DBT has a legal requirement to complete PIRs for regulatory measures with expected significant impact on businesses^{[footnote 2]}, and which include a statutory review clause. This allows us to review whether the regulation has met the intended objectives of the legislation. Having in place an effective monitoring and evaluation plan will be crucial when the time comes for DBT to complete a PIR and/or measure.

Statutory guidance sets out key considerations for the appropriateness of a review, with the first and most significant factor in the consideration being whether the measure has an impact greater or less than £10 million (net annualised) on business. Previously the relevant figure was £5 million. Having in place an effective monitoring and evaluation plan will be crucial when the time comes for DBT to complete their reviews of regulatory measures.

Regardless of the presence, or otherwise, of a review provision in secondary legislation, the PIR statutory guidance under section 31 of the Small Businesses, Enterprise and Employment (SBEE) Act provides that all Regulatory Provisions should be subject to proportionate monitoring, evaluation, and non-statutory review, though not all measures require a PIR (for example, measures with negligible impacts).

You should take a proportionate approach to evaluation to ensure that resources assigned to monitoring and evaluation are deployed effectively from the outset. This is important to ensure that departments are allocating their analytical resource to best achieve ministerial priorities and commitments.

A PIR should be scrutinised by the Regulatory Policy Committee (RPC) if the original regulatory provision required scrutiny. Where there is no formal requirement for RPC scrutiny, departments may wish to voluntarily submit a PIR for scrutiny. Further information is set out in the reformed Better Regulation Framework.

d. Scope

DBT’s monitoring and evaluation strategy covers all our programmes and operations globally and is informed by HM Government (HMG)’s Magenta Book guidance on evaluation. It excludes independent arm’s length bodies, as their day-to-day decision-making is independent of government. We will consider any commonalities between their work and DBT’s as needed in evaluating DBT activities and services. This strategy applies to post implementation reviews of regulation. It excludes non-statutory commitments for post implementation reviews made by the previous departments.

The strategy covers externally commissioned and published evaluations, but not unpublished internal assessments with commercially sensitive content. The guiding principles also apply to internal evaluations with the caveat that publication must be considered on a case-by-case basis.

The core principles and vision set out in this strategy will guide DBT’s approach to understanding our performance via monitoring and evaluation over the years to come. But we also recognise that departmental needs evolve, so we will review and update our detailed evaluation plans in 3 years’ time. This strategy covers the period financial year (FY) 2023/24 to 2025/26.

What counts as evaluation in DBT

Focus on relevance, effectiveness, efficiency, value for money or impact: our evaluations will meet the minimum standards and use methods and evaluation design approaches recommended in the Magenta Book. These may include impact evaluation, economic, theory-based, or process evaluation and mixed designs where needed. We may also include identifying unintended consequences.

Independent: delivered by individuals outside the team making the intervention to ensure findings are objective and impartial.

Peer reviewed: independent quality assurance by external expert(s) where appropriate.

External or internal: either externally commissioned or conducted internally. For internal evaluations, results should be externally peer reviewed where appropriate to ensure independence.

Published: commissioned evaluation reports will be published by default on GOV.UK. For internal evaluations we will consider publications as appropriate.

2. Our vision for monitoring and evaluation that drives performance

We will achieve DBT’s vision through our 6 guiding principles for using monitoring and evaluation to drive performance. These principles support DBT values^{[footnote 3]} of ‘‘Excellence’ via focusing on quality M&E evidence and being “Connected” through making findings transparent and accessible. They support being ‘‘Connected” and “Collaborative” via our ambition to use innovative techniques and drive learning through understanding performance.

To ensure we deliver on our commitments, DBT will resource evaluation adequately. We will ensure the department has the financial and staff resources needed to undertake quality evaluations and use them proactively to inform our work. Depending on the intervention’s size, evidence gaps, and priority level, funding for evaluation can vary. As a rule of thumb, spending 1%-5% of the total intervention’s cost on evaluation is seen as standard, but establishing a proportionate approach to evaluation means spending for each evaluation will be tailored to each policy or intervention.

Principle 1: use of evidence and uptake of learning to design and implement policies and programmes

Why

Many of our customers and stakeholders value the insight we provide to their programmes to improve delivery. But if we do not implement learning from monitoring and evaluation systematically, we will not reap the benefits of our investment. Since excellence is one of DBT’s core values, we look for ways to continuously improve our processes, systems and products, to ensure we are always making a difference.

How

DBT is committed to fostering a culture of experimenting, learning and adapting. We will prioritise learning from pilots and monitoring and evaluation to inform all our programme and policy decisions and future interventions. We will view any negative findings as just as useful for learning as championing our work through success stories.

What

Senior Responsible Officers and leaders will be expected to show how they are using learning from research, monitoring and evaluation to make decisions about their programmes. Learning from performance reports, monitoring dashboards or evaluations will feed into policy decisions on whether interventions should start, continue, be adapted, paused or stopped. We will use this learning to build a strong narrative of our successes and understand lessons from initiatives that failed or need improvement.

DBT will provide training and support to colleagues in using evidence to make decisions as needed via the policy profession network and the Central Analysis Team.

We will seek opportunities to apply principles of ‘test, learn, adapt’^{[footnote 4]} through pilots, trials and experimentation. This will help us to learn iteratively about what works through the policy design and delivery cycle.

We aim to understand the impact of our regulation on businesses, consumers and our economy as a whole. To that end, DBT is committed to producing proportionate, high-quality and robust post implementation reviews for all our statutory requirements by our previous departments in line with the Better Regulation framework.

Under the new Better Regulation Framework, post implementation reviews are to be scrutinised by the Regulatory Policy Committee, where the original Regulatory Provision required scrutiny.

In practice this means: * earlier and more holistic scrutiny of regulatory proposals through consideration of wider impacts beyond direct costs to business * stronger focus on evaluation to assess whether implemented regulations are achieving their aims

Principle 2: strong governance for focus on learning and results from M&E

Why

Being confident is a key part of DBT’s core values. We are willing to take risks and hold ourselves accountable to learn from our mistakes. To do this properly, we need to embed monitoring and evaluation into DBT’s DNA. This means having supporting mechanisms in place which enable us to track our performance and accountability and exert control. Without this we may not be able to ensure that all parts of DBT are fully reaping the benefits of investing in M&E.

How

DBT will embed proportionate M&E arrangements (see Principle 5) into existing departmental processes to ensure we learn from our evidence base, even in challenging circumstances. We will build opportunities for new programmes to learn from evidence at the design, implementation and closure stages.

Monitoring and evaluation have a central role to play at multiple stages within the policy cycle (see also Figure 2).

In the policy cycle:

• at programme design stage they inform interventions and help clarify objectives
• at implementation stage, timely monitoring data and evaluation can flag risks and prompt adjustments
• at exit stage after a policy has been implemented, findings from evaluation activities can offer an opportunity to gather lessons learned for future interventions

What

To ensure seamless integration of evidence on performance into departmental structures, DBT will further embed M&E into existing decision-making boards and processes. This will include delivery, investment and management boards at the business case development and delivery stages for regular monitoring.

Our central committees will provide a challenge function and will oversee progress in implementing these commitments every six months to ensure M&E planning, resourcing and delivery is adequate across all our areas of delivery. Through DBT’s existing processes for developing business cases we will ensure that proportionate monitoring and evaluation arrangements are planned for every new area of spend.

DBT’s Investment Committee will have oversight of how evaluation has been used and has fed into policy making. DBT’s Performance and Risk Committee will oversee resourcing and delivery of M&E activities across the departmental key spending areas.

Following the new Better Regulation Framework, there is now a government requirement to ensure that regulatory provisions have robust and effective monitoring and evaluation plans. The Regulatory Policy Committee will be scrutinising these at the options assessment stage as part of their new remit.

Principle 3: capability building across the department

Why

The need to grow evaluation expertise is common across government and DBT would benefit from a broader base of evaluation capability across our community. A recent report on evaluation in government found more effort is needed to increase capability in using evaluation among policy developers and decision makers.^{[footnote 5]}

How

DBT is already upskilling staff across the department including analysts, policy and decision-makers through externally commissioned and cross-government led training by the Evaluation Taskforce. DBT is committed to further building capability in using M&E for policy and project delivery colleagues by working with our policy profession network. We will use our governance mechanisms (see Principle 2 to ensure analytical monitoring and evaluation capacity and coverage across DBT is proportionate and prioritised.

What

DBT already helps staff from all professions learn about how M&E can inform our understanding of programme/service performance. We will continue to support and expand these activities.

Monitoring and evaluation training is part of the core curriculum for all analysts in the department. We use it to build the right skills for M&E plan development, managing evaluations, quality assuring results, or supporting the design of new interventions. The Central Analysis Team will expand the support we provide to reach more colleagues in our overseas network.

DBT’s Central Analysis Team coordinates capability building activities and produces guidance on monitoring and evaluation for the department in line with HMG guidance.^{[footnote 6]} It supports a network of M&E practitioners who peer review central guidance and provide practical support for ongoing M&E activities. We will work with this group to define further M&E skills needs and broker suitable solutions. This will ensure we are well placed to build understanding of the impact of our services.

The team has established an M&E Community, open to everyone in DBT across all professions. Its goal is to provide a platform for sharing learning across the department and encourage more people to champion the use of learning from evaluations. We will seek to maximise engagement across all areas of DBT.

DBT will continue to draw on insights and inspiration for good practice from across government and wider specialist networks such as the UK Evaluation Society to improve our capability. This will include supportive challenge from the Evaluation Taskforce and the Cross-Government Evaluation Group.

Principle 4: transparency and accessibility of evidence on performance, monitoring and evaluation

Why

Having easy access at the right time and in the right format to insights from monitoring and evaluation is critical. Without this the most important customers may not reap the benefits of learning from evidence. This applies to external as well as internal customers so everyone can learn and benefit from DBT’s monitoring and evaluation.

How

DBT is committed to making evidence about performance and evaluation easy to access including publishing externally commissioned evaluations by default on GOV.UK. There is a route to exemption if methodological or other sensitivities apply. The department will publish details of how recommendations and learning from published evaluations have been taken on board within 6 to 12 months.

What

Project managers of evaluations will distil and tailor their findings into the most appropriate formats to meet the needs of internal and external audiences. They will share and promote results widely across the department through a bespoke internal communications strategy. DBT will commit to meet HMT’s expectations by following the Government Social Research Publication Protocol and will support publication of externally commissioned evaluations by default on GOV.UK with an agreed process for considering exceptions on a case-by-case basis.

Evaluations will be published promptly, within 12 weeks following agreement of the final output. Protocols and analysis plans should usually be developed and published in advance of any study being started.

The Central Analysis Team will support colleagues through capability building to ensure core messages from evaluations are communicated clearly. We will work constructively with the Cabinet Office’s Evaluation Taskforce to maximise transparency of our evidence base.

DBT will not typically publish detailed process evaluations using for example internal management data of uncertain quality though these will be undertaken for continuous improvement.

We will use networks and online resources to share monitoring and evaluation evidence across the department ensuring it is easy to find and accessible for all teams.

Principle 5: proportionate coverage for learning and accountability

Why

Project and programme interventions vary hugely in characteristics such as size, novelty and risk. This means they require different levels of intensity and complexity in analysis to understand how successful they are.

How: setting up clear criteria to identify and prioritise evidence generation for evaluation across DBT

At a minimum DBT will collect monitoring data for all programmes and services we deliver to businesses and investors.

Proportionate approach to evaluation and post implementation reviews: As outlined in the Magenta Book, not all activities will require the same effort and resources for evaluation. Large-scale, high profile, untested, complex or costly policies/programmes tend to require external evaluation so we can be confident in drawing conclusions about their success.

Smaller, routine policies which have been already tested may only need light touch assessment. To assess the impact of regulation, the scale of impact on businesses, consumers and the UK economy will determine the level of effort in assessing regulatory activity.

To ensure DBT’s approach is pragmatic and proportionate, teams designing and analysing programmes will decide what is proportionate for every programme or area of spend. The criteria to be applied for every DBT policy area or service will make up a basket of measures to judge the level of effort that is needed.

They include:

• the scale of investment
• small programmes require as a minimum monitoring through SMART indicators or combining monitoring with internal evaluation by DBT analysts
• medium sized programmes would normally consider external evaluation on top of monitoring
• large programmes require robust monitoring and independent assessment which by default would include external evaluation
• strategic imperative: high profile policies of strategic importance, or of high media or public interest would normally require external evaluation
• ability to influence delivery of the policy and ongoing roll-out is likely to increase the need for evaluation. Pilots for smaller spend interventions can require more resources for evaluation
• delivering statutory obligations may require evaluation even for smaller scale programmes
• degree of risk: innovative policies/programmes or activities delivered in highly uncertain contexts carry more risk. This requires using evaluation to understand how successful they are to justify continuation and use opportunities to adapt to maximise their impact
• need for evidence on what works: untested policies and services which are not yet supported by evidence of what works require evaluation. DBT will pilot these programmes where feasible and use monitoring and evaluation evidence to inform their development before scaling them up

Much of DBT funding is used to deliver services that support businesses domestically, as well as services across the world to promote trade and investment with partner countries. Activities such as supporting the large-scale industrialisation of supply chains or influencing trading partners contribute to wider government objectives to increase the UK’s influence, economic growth and prosperity.

These objectives are shared with other departments such as the Foreign and Commonwealth Development Office, the Department for Transport or the Department for Levelling Up, Housing and Communities. We will aim to understand DBT’s contribution to wider goals and where appropriate our teams will collaborate with other departments. This may include sharing data and undertaking joint analysis to identify what works in different contexts.

What

For every project, programme or service funded by DBT, project managers will work with analysts to help develop a proportionate M&E framework as part of their design. The quality of the evaluation plans is assessed by an internal independent analyst. DBT’s Investment Committee will require an M&E plan and proportionate resource to implement it to approve business cases. The framework or plan will aim to address evidence gaps and include an outline of research, evaluation and monitoring arrangements to plug them.

Our funding commitment: DBT is committed to ensuring it has sufficient staffing and allocated funds to undertake high quality evaluations and use their results to inform our work. For the 2023/24 to 2025/26 period DBT will aim allocate sufficient funding and resource to assess the impact of our work.

Our approach will be tailored to the size and specificity of each programme and informed by our analysts. We will be guided by industry standards 1%-5% of the total intervention’s cost on evaluation, but this will be adapted to each policy or intervention, to ensure a proportionate allocation of resources.

Analysts will have a key role in establishing what proportionate arrangements look like for their interventions. For example, for pilots cost for evaluation may exceed these thresholds, even for smaller scale interventions. We will ensure these funds are used to deliver the evaluations we need using the scrutiny function of our central committees (see Principle 2).

In line with the statutory guidance under the Small Business Enterprise and Employment Act we will review regulation at timely intervals for regulatory measures with significant regulatory impact (over +/- £10 million net impacts annualised) on business only where:

• they are deemed necessary
• they are having the intended effect
• any associated costs to business are appropriate

Principle 6: high quality monitoring and evaluation

Why

Drawing sound conclusions and making good decisions is dependent on having a compelling evidence base. Without ensuring high quality standards in all our monitoring and evaluation activities, we may make choices which are suboptimal or just plain wrong.

How

DBT’s approach to evaluation is informed by HM Treasury’s Green Book guidance on appraisal and evaluation and its Magenta Book guidance on evaluation.

Our research and analysis is informed by government social research guidelines^{[footnote 7]}, the Aqua Book and wider good practice.

We will follow the Government Statistical Service Code of Practice and project delivery standards in generating and reporting on monitoring data. DBT is interested in understanding its performance and the added value of its work. We will aim to establish baseline data, performance metrics and points of comparison so we can understand the impact of our policies and programmes.

What

To generate evidence on programme and policy performance, we will choose the most appropriate evaluation methods for each project/programme.

All evaluations will have clear aims, be informed by a bespoke theory of change for each area and be designed to meet stakeholders’ needs. We will use a collaborative approach and will involve all stakeholders to agree an optimal set of questions to investigate.

Focus on impact and causality. For every programme we want to collect data on participating and non-participating organisations to understand how far we are making a difference. In line with the Magenta Book, we will aim to use experimental and quasi experimental approaches. Where this is not feasible, we will seek to understand how far DBT plausibly contributed to programme results, using theory-based approaches.

Every programme funded by DBT is expected to record activities and to assess outcomes and, where possible, impacts. Where assessing impact is not possible or appropriate, programmes will need to explain and justify this. When it comes to assessing the impact of regulations, we acknowledge the limitations of undertaking robust control group methods compared to spend interventions. We will rely on appropriate alternative approaches such as systems methods and process evaluation to understand why and how outcomes arise and how process can be adjusted for maximum impact.

Across our work, we will aim to understand DBT’s contribution towards government priorities. In practice this means better recording and breaking down data to allow detailed spatial or sub-group analysis where possible. This also means identifying relevant cross-cutting priorities and planning thematic evaluations aligned to strategic priorities.

Every intervention will have clear requirements to generate good quality data to monitor performance and to understand performance outcomes. We will drive up the quality and coverage of routinely collected data by developing consistent protocols and draw on departmental performance metrics. In line with DBT’s Data Strategy, our data scientists will use our digital infrastructure to gather information enabling us to monitor the progress of our work.

We are reviewing our approach to collecting and reporting data on customer interactions to ensure it will help us systematically understand our performance. We will invest in data storage, ingestion, analysis and visualisation. We will supplement primary research with secondary data and additional sources as necessary. We are collaborating with other government departments to expand the features of various data sets and will seek to avoid duplication and make use of data matching where possible.

In line with our funding commitment, DBT will allocate adequate analytic and project delivery resource to support the generation, collection and quality assurance of reliable performance monitoring data.

We will harness impartial perspectives through using external contractors where appropriate but due to the sensitivity of some of DBT’s work we will undertake some evaluation internally. For example, we need to treat UK negotiating positions for trade agreements confidentially.

Build and maintain impartial and transparent quality assurance. This will ensure the quality and reliability of evaluations and will provide DBT and its partners with assurance in the findings. Quality standards and independence of internal evaluations will be maintained through quality assurance using HMG analysis profession standards.

We expect every evaluation conducted in DBT to be quality assured by:

A steering group^{[footnote 8]}: harnessing all principle stakeholders to ensure delivery and provide critical friend challenge. Stakeholders can provide valuable input throughout the evaluation from design stage to initial findings and final report prior to publication.

Independent peer review: we will take a proportionate approach to using external and internal peer review of evaluations in DBT depending on size, scale, complexity and degree of risk. This will make best use of independent experts to support high quality evaluations at crucial stages of their development and delivery.

Internal analytical quality assurance processes built into monitoring reports: DBT will embed quality assurance of relevant monitoring and performance data. This will enable progress tracking over time across programmes and priority areas and ensure data meets appropriate quality standards.

3. Rolling out our strategy in practice

Having integrated business and trade in a single department, DBT is positioned to balance the opportunities of international trade with the interests of UK businesses, maximising the benefits of business and trade for households all over the UK while increasing consumer and business confidence in times of economic uncertainty.

This evaluation strategy is an exciting opportunity for DBT to understand more about our potentially powerful impact on domestic growth and trade and investment opportunities for businesses. These may benefit firms in the UK and the wider global community.

We will communicate the strategy widely with our staff, partners and stakeholders and implement our commitments for evidence-based programmes through robust monitoring and evaluation plans. We will embed monitoring and evaluation in our policy and programming cycle to inform not just policy design but also drive delivery throughout implementation.

Figure 2 shows how monitoring and evaluation can feed in throughout the whole of project and programme life cycles. It shows that evaluation evidence is expected in developing the business case for intervention and stresses that objective setting should include considering what success looks like. It invites use of evidence about what works in considering options and using data to assess progress in delivery over time against anticipated results. Lastly, evaluation considers whether and why the policy or project achieved the desired impact and uses learning to improve future policy and service delivery.

Figure 2: Embedding monitoring and evaluation in DBT’s policy and programme life cycle

Figure 2 illustrates the ROAMEF (Rationale, Objectives, Appraisal, Monitoring, Evaluation and Feedback) policy and project cycle and how monitoring and evaluation supports this.

The Central Analysis Team will review our progress against the actions we have committed to take in this strategy every 2 years.

This will include:

• establishing how far evidence about delivery and performance is being used well to reshape our services to better meet customer needs
• tracking how our plans to monitor accountability for M&E delivery through our governance structures are influencing plans across programmes, use of resources and investment decisions
• monitoring the take-up, coverage and application of learning and development support to build evaluation capability
• ensuring internal accessibility of learning from evaluations and tracking delivery of our commitment to external publication
• tracking external spend and staff resourcing against evaluation investment commitments
• mapping how well we are applying review and assurance processes to maximise evaluation quality

DBT’s mission is to support businesses to grow and create jobs and opportunities around the country. To do this we need to redraw our rules to protect businesses, and consumers, secure investment, support British businesses to grow and export, remove trade barriers to new markets and help maintain fair trading systems across the world. Monitoring and evaluation is a powerful approach in our toolkit to drive our performance and through implementing this strategy, we will ensure the insights it can create are accessible to and useful for all DBT’s teams.

1. DBT ‘Made in the UK sold to the world, UK Export strategy’ 2021 ↩

2. Significant impact means greater than +/- £10 million net annualised), Small Businesses, Enterprise and Employment Act 2015 ↩

3. DBT. ‘Annual Report and Accounts 2020 to 2021’ 2021, page 69 ↩

4. Cabinet Office. ‘Test, Learn, Adapt: Developing Public Policy with Randomised Controlled Trials’ 2012 ↩

5. National Audit Office. ‘Evaluating government spending’ 2021 ↩

6. HM Treasury. ‘The Magenta Book’ 2011 ↩

7. Government Social research Profession. ‘GSR code: products 2018 and Government Social Research Profession. ‘GSR Ethical Assurance for Social and Behavioural Research’ 2011 ↩

8. HM Treasury. ‘The Magenta Book’ 2011, page 71 ↩