Research and analysis

Tranche 1: Findings from the Regulatory Horizons Council evaluation interviews and retrospective (accessible webpage)

Published 11 March 2022

Introduction

Following the publication of the Regulatory Horizons Council’s (RHC) final report for its first tranche of work (report on the regulation of drones), the RHC wanted to complete work to evaluate and reflect on this first set of deep dives.

Monitoring and evaluation are critical elements for the effective delivery of the RHC’s aims and objectives, and ensures that it is sourcing the views of its key stakeholders and making change based on key lessons learned.

The following slides outline the central findings from this first set of evaluative work. These efforts form part of a larger ongoing set of evaluative work for the RHC to continually learn and improve.

Furthermore, we want to instil a growth mindset in the team where we continually reflect on our processes, with a view to reflecting and improving and this is part of that wider effort.

The RHC conducted 14 one hour interviews with key stakeholders that were involved with the RHC deep dives, including policy officials from relevant government departments, policy officials from regulators, a partner organisation that the RHC worked with on one of its deep dives, and council leads.

The interviews were largely structured, with some room for appropriate probing questions.

The fieldwork was conducted over the course of around 2 months between 27 September 2021 and 24 November 2021.

Findings were discussed across 2 workshops in December 2021:

• Workshop 1 - presentation of findings from evaluation interviews (see Findings from evaluation interviews).
• Workshop 2 - retrospective with RHC secretariat to reflect and discuss next steps (see Retrospective with RHC secretariat)

Findings from evaluation interviews

What went well

Independence:

• participants stressed that the RHC being an independent body was, and still is, very valuable. Whereas other parties may have directly vested interests in the particular policy space, the RHC can come into an area with a more neutral presence that is mostly separate from government and which was noted as adding credibility to its reports and recommendations
• many respondents also noted that the RHC’s independence made it better suited for conducting this type of work compared to other similar groups

Effective stakeholder management:

• interviews highlighted that the RHC stakeholder engagement was incredibly effective. The RHC were generally seen to have spoken and engaged with the right groups and the right people, and critically, balanced these engagements well
• the resources the RHC has directed to these activities was seen to be very significant
• this is also reflected in the monitoring data for Tranche 1, which shows that the RHC engaged with 127 stakeholders across the four areas (with 17 of those being international) 35 interviews, 6 workshops, 5 roundtables, and one bespoke survey

Strong synthesis of information:

• there was a consistent feeling among interviewees that the deep dive reports were very good syntheses of information and stakeholder views
• there was consistent mention of their utility and insightfulness, and in some instances, their ability to highlight to experienced policy officials some of the issues or concerns with the relevant field
• in some cases for instance with the RHC fusion report there was a wider international utility of the report

Excellent secretariat work:

• the strong work of the RHC secretariat was perhaps the most resounding and consistent theme across all of the interviews. Interviewees consistently noted the hard work and effort put in by all of the RHC secretariat across the 4 deep dives in Tranche 1
• even when there were some procedural difficulties during the initiation of 2 of the deep dives, the secretariat were noted to have handled these matters professionally

Stimulate thinking:

• participants discussed how the reports have been able to stimulate a lot of thinking on the areas of emerging technology; often bringing forward timetables for government consultations, taking bold steps within the recommendations, and/or providing balanced views that are not afraid to go against the grain of existing work and thinking in the wider discourse

What went less well

Early to fully understand impact:

• participants across the interviews noted that it was quite soon after publication to determine whether the RHC had been successful in enacting impact on the regulatory system
• this finding was expected, as the interviews were conducted at a relatively early point. This timing was, however, necessary to avoid problems with recall and potential staff changes that would occur if the interviews had been done further in the future
• critically, these interviews are part of a larger evaluation effort from the RHC.

Information not always new:

• a common theme across some of the interviews conducted, was that the deep dive reports did not always present new information about the regulatory landscape of the respective technology area
• participants discussed how the RHC’s reports collated evidence effectively and presented and findings and views from the field in an accessible manner
• the reports were referred to as effective means of driving issues forward and drawing attention from key stakeholders

Resourcing difficulties:

• within some of the interviews, it was noted that resourcing challenges were encountered on 2 of the RHC’s deep dives
• most interviewees appreciated this was unavoidable as there was a wider diversion of resources across government to focus on COVID-19

The clarity of the RHC’s role:

• for some interviewees, there was discussion around the RHC being an independent body, but with a secretariat within BEIS. For some stakeholders, this created confusion
• additionally with the RHC being a new body there were some participants who described difficulties in getting buy in from certain stakeholders early on in the process of its deep dives
• these findings were somewhat unsurprising, considering that the RHC is a relatively new body, and the deep dives in question were its first set of work

What could be done in the future

Improved timing:

• across the interviews, there were suggestions of how the RHC could improve the timings for parts of its processes
• in part, future RHC deep dives should try to ensure that momentum is maintained throughout it is, however, noted that changes to resourcing can sometimes unavoidably impact this
• the RHC has already begun work to incorporate these changes in its second tranche of deep dives; making greater consideration for the timings of the processes, and monitoring the momentum of deep dive projects

Greater collaboration with other council members:

• it was indicated within the interviews that stakeholders generally had contact with one member of the Regulatory Horizons Council the lead for that particular deep dive
• moving forward, the RHC will aim to make the role of its primary and secondary deep dive leads clearer

Directly involve externals:

• while the deep dives extensively engage with a variety of stakeholders from each of the respective fields, in 2 of the sets of interviews, there was discussion around the value of external assistance
• interviewees noted that the partnership with Birmingham Health Partners for the medical devices deep dive was really beneficial and allowed for a large amount of high quality stakeholder engagement to be completed. There was some suggestion that more of the same collaboration would be useful for future deep dives
• other interviewees suggested the possibility of ‘technical advisors’ to support the deep dives and provided expert advice on a more regular basis
• the RHC is now exploring the viability of both options for future tranches

Follow up work:

• within the majority of the interviews, there were suggestions of how the RHC could follow up its deep dive reports
• these suggestions ranged from blogs to update the issues and keep the information current, additional work on public engagement, additional stakeholder engagement post consultation to see if groups got what they wanted, widening the lens to look at the international perspectives more substantially, and exploring the secondary exam questions in more detail
• the RHC is currently revising its communications strategy and exploring the feasibility of different follow up actions for its deep dive reports

Retrospective with RHC secretariat

Background

For the second workshop, the RHC secretariat participated in a retrospective, facilitated by its analyst.

Retrospectives are a tool typically found within Agile software projects and involve a team looking back on its work to understand how to improve its processes and delivery in the future.

Prior to this second workshop, the RHC secretariat were instructed to reflect on the findings from the evaluation interviews, and come prepared to consider the following questions:

• what worked well?
• what have we learned?
• what still puzzles us?
• what actions can we take?
• the aim of the session was to provide a space for the RHC secretariat to consider their own views on how the first set of deep dives went, and produce a set of actions that could be taken forward for future tranches of work
• the final slides outline the actions that the RHC is considering and aiming to take in its future deep dives

Actions for the future

1. Recognise current strengths and maintain them.

2. Conduct ‘pre mortems’:
- Exercises that are run at the beginning of a project and consider everything that could possibly go wrong

3. Maintain flexibility in processes:
- Be willing to change an approach, even if it has worked previously

4. Continue to nurture an atmosphere of support and constructive challenge:
- encourage flat hierarchy and value opinions from the entire team.

5. Improve information management:
- implement more robust structures/databases for information and ensure that these are maintained over time.

6. Build in more technical expertise:
- engage with more academics, explore the possibility of ‘technical advisors’.

7. Refine communications strategy:
- particularly steps taken after the publication process.

8. Develop processes for trialling new approaches:
- build in defined time to project timelines for testing and reviewing.

9. Refine our process for engagement post publication.