Guidance

Guidance on assessing performance and value of Public Sector Research Establishments

Published 25 January 2022

Background and Purpose

Public Sector Research Establishments (PSREs) play an important role in the UK’s science, research, development, and innovation (RDI) landscape. Principally, they support government by providing science advice to policymakers, by acting as a strategic capability in policy delivery and by delivering critical science services for government, business, and society. Given the wide-ranging policy and operational needs of government departments, PSREs themselves vary greatly in function and mission. Some are highly research intensive (for example, The Defence Science and Technology Laboratory), seeking to develop new technologies, whilst others play critical operational and regulatory roles necessary for running the country (for example, the Environment Agency).

The Science Capability Review (2019) recognised that PSREs represent a significant public asset that is currently under-utilised and not well understood across government (see Science Capability Review Recommendations 3 and 4). The purpose of this document is to provide a broad framework and a set of common principles to support departments in assessing the value and performance of the PSREs they sponsor, and to think critically about how they could be better utilised. This framework should be used during the Cabinet Office’s Arms-Length Body (ALB) review process to ensure a consistent standard of reviews of PSREs across departments, but it is also recommended that future reviews of PSREs conducted separately to the ALB reviews, should make use of this framework. In addition, departments that sponsor PSREs that are not considered to be ALBs should also make use of this framework.

This document is not intended as a single, complete treatment for planning a review of any individual PSRE. Development of specific PSRE performance measures - and the conduct of reviews more broadly - should be co-developed in consultation with key stakeholders. Critically, the PSRE’s Chief Scientist (or equivalent) and the sponsoring department’s Chief Scientific Adviser (CSA) should be involved throughout the review process. The performance measures and methodology identified throughout the course of the initial PSRE review should continue to be monitored and used for future reviews, to ensure consistency of methodology over time and tracking of progress.

It must also be recognised that whilst this document focuses on the guidance of reviews of individual PSREs, reviews should acknowledge the wider departmental context. Specifically, there should be alignment between reviews of different PSREs conducted simultaneously by the department. The department’s CSA has an important role to play in ensuring reviews of the department’s sponsored PSREs are conducted with coherence and a focus on ensuring the portfolio of PSREs are delivering the capability needed by the department and wider government within their respective RDI areas.

Scope

The guidance contained within this document is applicable to a range of ALBs: while this guidance will predominantly be applied to PSREs, it can be applied to other bodies that fit within the guidance’s scope. At present, the following definition of PSREs from the Royal Society^{[footnote 1]} is being used. PSREs that fall under this definition are listed in the accompanying Royal Society document.^{[footnote 2]}

PSREs are a diverse collection of public bodies carrying out research. This research supports a wide range of government objectives, including informing policy making, statutory and regulatory functions and providing a national strategic resource in key areas of scientific research. They can also provide emergency response services. They interact with businesses around a wide array of innovation-related functions.

Within this, we use the definition of research outlined in the Frascati Manual, namely that this covers work undertaken on a systematic basis to increase the stock of knowledge, and the use of this stock of knowledge to devise new applications. As such, this includes all activities along the spectrum of discovery research, translational research, mission-driven and applied research and the development, demonstration and diffusion of new approaches, methods, and technologies.

It is strongly recommended that departments use this Value Framework for any ALBs matching the below three key criteria:

• the ALB is owned, partly owned, or sponsored by a central government department, and receives a substantial part of its total funding from public sources

• the organisation has a clear remit to conduct research itself (as opposed to contracting external organisations to conduct research)

• the ALB plays a key role in the UK science and research community, and is already, or has the potential to be, well aligned with strategic governmental priorities

Responsibilities

The Lead Reviewer. The Cabinet Office’s “Guidance for Undertaking Reviews of Public Bodies” recommends that departments appoint an independent lead reviewer responsible for ensuring that a proportionate and effective review is conducted. The Lead Reviewer is ultimately accountable for the recommendations of the reviews they lead. In addition to requirements set out by the Cabinet Office, this person should have a relevant scientific background for the PSRE that is being reviewed. The lead reviewer must also be sufficiently independent to enable an objective review. The Cabinet Office’s “Guidance for Undertaking Reviews of Public Bodies” provides further details of the roles of the Lead Reviewer.

The Review Team are responsible for supporting the lead reviewer and will be made up of individuals from the department undertaking the review. It is the responsibility of departments to adequately resource their reviews depending on the nature of the PSRE under review. The departmental Review Team should be independent of the PSRE, and its sponsoring team. Other responsibilities of the review team include shortlisting Lead Reviewers, building a trusting and effective relationship with the PSRE, ensuring appropriate engagement between the Lead Reviewer and the PSRE and ensuring appropriate engagement with the department’s CSA. The Cabinet Office’s “Guidance for Undertaking Reviews of Public Bodies” provides further details of the roles of the Review Team.

The Department’s Chief Scientific Adviser (CSA) should take an active and supporting role in the review process to ensure the reviews are being conducted to an appropriate standard. If the department sponsors multiple PSREs, the CSA must also ensure coherence across the various reviews being undertaken by the department. This will include ensuring Lead Reviewers and Review Teams work together during the review process and ensuring that a portfolio view is taken so that the relevant sponsored PSREs are able to deliver the capabilities needed by the department and wider government. They are also responsible for disseminating the results of the review to key stakeholders within the department.

PSRE Value Framework Overview

This Value Framework aims to help departments develop performance measures, assess performance and value and identify areas of further opportunity through four assessment steps.

1. Evaluate Mission: To identify the PSRE’s roles within the RDI landscape and assess its relevance to the department, to broader government priorities and to the wider RDI landscape.

2. Assess Impact: To develop appropriate PSRE performance measures relevant for the PSRE’s role and assess the performance of the PSRE with respect to its mission.

3. Assess additional value: To identify the wider value created by the PSRE beyond its core functions (for example, skill development, resilience and infrastructure, operational standards).

4. Assess further opportunity: To think critically about how the PSRE can be better utilised by the department and across government.

By gathering evidence to inform these steps, the review team should develop an evidence report and, if necessary, convene an expert panel to support an assessment of the evidence and any recommendations developed. The Lead Reviewer and Review Team should also consider the option of consulting with the PSRE’s Scientific Advisory Committee (if one exists) on the findings and recommendations of the review. Evidence from previous reviews can, and should, be leveraged where possible if the original evidence remains relevant and applicable. The evidence report can then be used for a variety of outcomes:

• the report will be submitted to the Cabinet Office as part of the ALB review process and can be disseminated further (including to Permanent Secretaries), to enhance visibility of PSREs across government

• if necessary, the report can inform an independent expert panel assessment of the impact of the PSRE and its scope for further improvements

• the report should include, or be used to develop, recommendations to stakeholders within the host department, including Chief Scientific Advisers or sponsor committees, around enhancing the impact and opportunities of a PSRE

The outcome of this review process will play a crucial role in boosting the impact that PSREs have, and in enhancing awareness of the role that PSREs play in the RDI landscape. This will be especially important in the context of the wider governmental push to increase spending on R&D, with an aim to spend £22 billion annually by 2026 or 2027 and hit an overall R&D spending target of 2.4 percent of GDP by 2027.

PSRE Value Framework – Evaluation Steps

This section outlines how to gather evidence and carry out an evaluation of a PSRE. It is separated into the four assessment strands. Departments should collect the evidence outlined in this section into the form of an evidence report.

Identify the PSRE’s missions and assess its relevance

Having a clearly defined mission is vital to set the overarching direction and scope of a PSRE. Mission statements should, in a paragraph or less, accurately define why the PSRE exists and what it hopes to achieve over the mid- to long-term. With a clearly outlined mission statement, an evaluation of the impact of a PSRE and its potential scope for improvement can then be carried out in relation to the mission.

Given that there are a very wide range of PSREs with varied scopes and providing different services, three broad categories are outlined within which mission statements can fall. Any individual PSRE may have multiple objectives that cover one or more categories, and the categories are defined as follows:

1. Development and exploitation of new technologies and knowledge assets: The PSRE plays a role in research and development, enhancing existing or developing new technologies, products, services, and other knowledge assets (models, methods, procedures).

2. Policy-making and regulatory support to government: The PSRE provides crucial research, information, monitoring/surveillance capabilities, data, and scientific advice to inform government policy.

3. Operational science services to government, business, and society: The PSRE provides operational science services (for example, forecasting, measurement, collection custodians, horizon scanning) to government, society, and businesses.

As an example, the Defence, Science and Technology Laboratory (Dstl) delivers new technology that enhances military capability across land, sea, air, cyber and space (technology development), and provides technical advice to support counterterrorism (policy-making support). In these cases, it is important to be clear on the different mission types as the assessment tools will likely differ (see below).

Table 1

Technology Development	Policy-making and regulatory support	Operational Science Services
Example: (UK Atomic Energy Authority) To lead the development of fusion power and related technology; (Met Office) Develop cutting-edge weather and climate forecasting models	Example: (UK Health Security Agency) Promote public health objectives by advising government on policy; (Dstl) Provided expert input on re-opening areas of Salisbury after Novichok attack	Examples*: (National Physical Laboratory) Provides measurement services to businesses developing new products; (Natural History Museum) Custodian of world-leading collection of more than 80 million specimens

*Examples are illustrative and not exhaustive of PSRE objectives.

While PSREs’ missions often align with the policy and operational needs of its sponsor department, a PSRE’s mission may be more cross-cutting to align with wider government objectives. In turn, appropriate strategic leadership is required to ensure a PSRE’s mission maximises value to its sponsor department or wider government priorities. To understand whether a PSRE is carrying out the best and most relevant research, development and innovation, judgement is required from a range of stakeholders. A non-exhaustive list of specific review questions is provided below, to help inform this judgement.

Review questions for assessing the PSRE’s mission
PSRE mission statements	What is the PSRE’s mission statement and in which mission categories does the mission fall? Is this mission still relevant to the department, wider government objectives and the rest of the RDI landscape? How does the PSRE’s mission and capabilities fit within the context of the wider PSRE landscape?
Departmental perception of the PSRE’s mission	Is the PSRE’s mission well-understood and communicated across the department? Do relevant policy officials understand how they can use the PSRE?
Evidence of specific, targeted roles within the RDI ecosystem	Does the mission statement fill a unique gap in the RDI landscape?
PSRE alignment with departmental and governmental objectives and activities	How well do the PSRE’s objectives align with the departmental objectives? Does the PSRE impact upon wider government priorities?

Recommendation: The first step of the PSRE Value Framework is to identify the PSRE mission statements and understand within which mission categories these fall, before reviewing this mission in the context of the PSRE’s sponsor and wider RDI landscape.

Assess impact of PSRE

The second evaluation section is to understand a PSRE’s progress towards its mission. To carry out a comprehensive review that reflects all the objectives within a PSRE’s mission, a variety of quantitative and qualitative tools can be used – as outlined in the table below. Each evaluation measure is evidence that can be used to enable a full, context-dependent evaluation of a PSRE.

If assessments of the impact of a PSRE have already been carried out, the Review Team should refer to these and include their findings in the evidence reports that they compile, if the original assessment remains relevant and applicable. Examples of such reports could include looking at societal impacts, assessing returns on investment, or commissioned economic cost-benefit analyses.

For the impact assessment, there are two sets of measures that offer different advantages:

1. Broad, mission-relevant narrative: an overarching narrative, whilst difficult to quantify, is a useful measure of the overall health of the PSRE and demonstrates the degree to which the PSRE is progressing towards its mission statements.

2. Narrow, mission-relevant metrics: these indicators can be used to provide factual evidence of specific aspects of PSRE activity - they are useful in understanding the exact role and impact of a PSRE, within its RDI context.

In deciding which evaluation tools to use, both categories should be made use of fully.

• Broad, mission-specific measures, such as PSRE self-reporting on their progress, provide a good qualitative narrative as to what an individual PSRE views as their achievements and provides apt framing for more specific indicators of progress later in the review process.

• Narrow metrics are able to shine light on specific PSRE strengths and weaknesses, and provide evidence in support or disagreement of the overall PSRE narrative. These indicators are specific for the different mission categories, and must not be interpreted in the wrong context. (For example, it would be inappropriate to evaluate the commercialisation activity of a PSRE that has a purely regulatory role.) Care must therefore be taken when selecting narrow indicators to ensure they measure the right aspects of PSRE activity relevant to a PSRE’s objectives.

A wide, non-exhaustive range of suggested indicators across these two categories can be viewed in the table below. PSREs may already have specific indicators relevant to their functions and these indicators should be considered in the ‘narrow indicators’ assessment. In addition, review teams should consult the PSRE on the most relevant narrow indicators and these should be agreed in advance of compiling the final report. Finally, the results from these indicators should be considered in the context of the PSRE and comparisons between other similar PSREs or ALBs should be used as a starting point only in understanding how a PSRE might be improved.

Table 2: Broad and Narrow indicators, oriented around a PSRE’s mission, to guide evaluation of a PSRE’s impact

Mission category	Evaluation tools
Broad mission-relevant measures
All	PSRE self-report on progress
	Departmental sponsor report on progress of PSRE
	Report on international reputation
Narrow mission-relevant metrics
Technology and knowledge asset development	Number and value of active R&D and demonstration projects (inc. growth over time)
	Percentage of projects meeting key technical objectives (see Annex B for examples)
	Number of technologies reaching commercial production or number of spin out companies
	Number of knowledge asset disclosures (an extensive list can be found in Annex A)
	Success stories and case studies
Policy-making and regulatory support	Number and size (length of time, number of researchers) of policy-related engagements or research projects
	Percentage of research projects or policy engagements meeting key milestones
	Success stories and case studies of policy support, data and information provided
	Departmental “customer” feedback on policy engagements
Operational science services	Number and value of customer-related engagements (e.g., revenue earned on a commercial basis)
	Diversity of “customers” in government, business, academia, wider society.
	Number of active R&D projects (inc. growth over time)
	Percentage of projects meeting key technical milestones
	Size and value of critical specimen collections (if applicable)
	Success stories and case studies

Recommendation: Agree between Lead Reviewer, Review Team, PSRE Chief Scientist (or equivalent) and departmental CSA the most appropriate performance measures for the PSRE in the context of its mission and carry out a comprehensive assessment of impact based on those performance measures.

Assess additional value of PSRE

As well as the value generated from directly delivering on their missions, PSREs add value to society in several more indirect ways. These can include developing skills of employees, enhancing local economic productivity and supporting the UK’s RDI landscape.

An assessment of these impacts, which are mission-agnostic, can be carried out by selecting additional value indicators. These enable review of aspects of PSRE value outside of direct delivery of mission-oriented objectives:

• Indirect value drivers: PSREs impact society and add value indirectly driving positive spill-over effects, supporting the local economy, impacting education, or supporting key aspects of the research, development, and innovation landscape. These indirect value indicators aim to assess additional value sources that are not necessarily closely aligned with the mission of a PSRE.

Table 3: Indirect value drivers, used to assess wider, non-mission specific impacts of a PSRE.

Measures of indirect value
Science Capability and Resilience	Case studies of roles of PSRE in times of emergency response
S&T Research Practises	Percentage of open access publications; Number of collaborations leading to publication; Number of other RDI actors making use of PSRE facilities
Economic Impact	Number of jobs created and retained; Gross Value Added to local and national economy; Number of partner companies and institutions (inc. size, growth, retention, and investment leveraged from partners); Level and quality of co-investment from non-government sources (for example, private R&D leveraged); Intellectual Property
Enhancing Skills, education, and workforce development	Number of PSRE skill programs/training schemes internally at all levels; Number of PSRE-supported education programs to the wider community (with universities, partner organisations and schools); Employee satisfaction levels and employee treatment; Staff diversity and inclusion metrics

Recommendation: Agree between Lead Reviewer, Review Team, PSRE Chief Scientist (or equivalent) and departmental CSA the most appropriate indirect value drivers to measure and use this to achieve an understanding of the wider impact and added value of the PSRE.

Assess further opportunities and potential scope for improvement of PSRE

The fourth key evaluation section is to assist departments in understanding the areas of potentially improved use of a PSRE. Identifying areas for PSRE improvement can be separated into three main sections. Key review questions for each section can be seen in the table below.

• First, critical to PSRE improvement are effective internal mechanisms for monitoring and evaluating organisational progress and creating feedback loops to enable learning from activities. The performance measures identified through this Value Framework should be revisited at appropriate time frames and reported to the department’s CSA (the time frames, formats, and routes for reporting should be agreed between the PSRE and the CSA in accordance with the governance framework agreement in place). This will provide a consistent approach to monitoring and evaluation and, with the right data collection infrastructure in place, prevent excessive future reviews and review fatigue.

• Second, identifying what current barriers and enablers exist that hinder or drive PSRE performance can shape decisions around to better utilise it. Key barriers that might hinder PSREs include governance structure, funding gaps and low visibility across government. Key enablers may include investment in existing or new capabilities expected to drive future performance and productivity.

• Third, review questions can be used to identify specific areas for further PSRE alignment with government priorities, either through current pipeline projects or PSRE capabilities.

Table 4: Areas to assess for improved use of PSRE.

Areas of further opportunity	Review questions	Evidence sources
Organisational monitoring and evaluation	How will the PSRE, department and CSA monitor progress towards objectives?	Interviews with PSRE stakeholders; Interviews with departmental sponsors; PSRE project pipeline overview; PSRE capability overview
	What mechanisms are there for PSRE/departmental learning?
Current barriers and enablers to performance	How does the current governance structure impact mission progress?
	Are there critical funding gaps?
	Are there barriers to local skills or access to business/academia?
	Are relationships between the PSRE and sponsor department operating effectively?
	Are there technical or PSRE-specific barriers to progress?
	Are there barriers that prevent the PSRE from working effectively with other PSREs, research institutes, universities, and industry?
Improved utilisation of existing PSRE capabilities	How could the department better use the PSRE’s existing capabilities or enable improved performance?
	How could other government departments better utilise the PSRE’s existing capabilities?
	How could the PSRE further orient its activities towards industry and academic engagement?
Identifying options for new capability investment	What investments are needed in the PSRE’s capabilities to drive future performance?
	Are there capabilities are there that the department needs and the PSRE could deliver with the right investment?
PSRE alignment with government priorities	How do projects in the pipeline align with departmental or HMG priorities?
	How could the PSREs capabilities be further aligned with departmental activities?

Recommendation: Evaluate the scope for improvement in performance of the PSRE, potential further opportunities, and scope for enhancing its impact and visibility across government.

Compilation of final report and outcomes

The above evaluation tools and evidence questions provide a strong foundation to develop an evidence report. The evidence report must cover the four evaluation sections, with an emphasis on understanding the impact of the PSRE (Question 2) and its scope for further opportunity (Question 4).

As a minimum requirement, the final report should be reviewed by the ‘independent reviewer’ in charge of the ALB process within a department, and then submitted to the Cabinet Office, as part of the departmental submission to the wider ALB review process.

In situations where a PSRE is found to either be having a minimal impact relative to its mission(s), or where there are substantial opportunities for improving the performance of the PSRE, this evidence report should then be submitted for review by an expert panel. The panel can provide judgement on the four assessment questions, and should be composed of experts in the field of the PSRE, or with significant relevant background:

• Diverse stakeholders should be considered, including experts in the field alongside the independent ALB reviewer, departmental Chief Scientific Advisor, and senior departmental officials.

• The Lead Reviewer and Review Teams should also consider the option of consulting the PSRE’s Science Advisory Committee (if one exists) on the findings and recommendations of the review.

Finally, based on the evidence report, the PSRE together with the lead independent reviewer (or the expert panel) can make recommendations on potential routes for increasing the impact of the PSRE, removing barriers to its success, and enhancing its visibility and role. These recommendations should then be passed on to relevant departmental and cross-government stakeholders, including GO-Science where necessary, to ensure they are implemented.

Recommendation: Collect the evidence required for the review in the form of an evidence report and submit to Cabinet Office. Decide whether a further review by an expert panel is required and pass on to relevant stakeholders any recommendations stemming from the evidence report.

Annex A: Extensive list of disclosures

Category	Types of knowledge asset	Relevant IP rights and other methods of protection
Information	Data and other information	Database rights; copyright; Crown copyright; law of confidence; contract; trade secrets.
Innovation	Inventions and designs	Patents; designs; plant varieties.
Creative	Texts, videos, graphics, software, and source code	Copyright, performers’ rights, and designs; Crown copyright.
Reputational	Brands and services	Trademarks and other badges of origin; goodwill.
Knowhow	Expertise such as in project management, process efficiency, ways of working	Knowhow could be associated with any or all of the IP rights, but may not be protectable in itself, other than by trade secrecy and the law of confidence.

Annex B: Percentage of projects meeting key milestones examples

• Information – research or acquiring data.

• Experiments – conducting experiments or tests.

• Prototypes – Building out elements of a solution to gather information such as user feedback or to confirm feasibility.

• Algorithms – developing and implementing steps that solve a problem.

• Infrastructure – foundational elements that support things like facilities, applications, systems, and business processes.

• Tools – for example, technology tools such as a business application are viewed as the objectives of a business.

• Automation – Systems and equipment that automate work.

• Quality – improving the quality of technology services, e.g., improving the availability of an application.

• Measurement and calculations – developing measurements, calculations, metrics, reports, and benchmarks.

1. Explainer, https://royalsociety.org/topics-policy/publications/2020/uk-research-organisations/ ↩

2. List of public and non-profit research organisations, https://royalsociety.org/topics-policy/publications/2020/uk-research-organisations/ ↩