Research and analysis

Evaluation of the Regulators’ Pioneer Fund (round 1) - findings from interim and follow up case studies, July to October 2019 and May to June 2020 (May to June 20) (web)

Published 15 March 2021

1. The high level objectives of the Regulators Pioneer Fund

The Regulators Pioneer Fund (RPF) is an initiative set up by the Better Regulation Executive (BRE), part of the Department for Business, Energy and Industrial Strategy (BEIS), to help create a regulatory environment that gives innovative businesses the confidence to invest, innovate and deploy emerging technologies for the benefit of consumers and the wider economy.

Key aims of the funding, which will invest up to £10 million over 2 years in 15 regulator-led projects across 12 sectors are:

• Enabling economic growth
• Boosting value for consumers
• Projecting a pro-innovation image internationally
• Fostering a pro-innovation business culture

1.1 Aims of the RPF programme evaluation

• Assess the extent to which RPF grants have enabled regulators to become more innovation-enabling
• Identify the effects on business innovation in the sectors impacted by the projects
• Enable BEIS to improve the RPF design and competition process for any future funding rounds
• Contribute to BEIS work to create best practice guidance on innovation friendly regulation and inform future policy decisions

1.2 The policy logic model for the RPF

The policy logic model is in four stages:

1. Outputs
2. Outcomes
3. Impacts
4. Goals

1.3 Outputs

RPF regulator(s):

• permits new business innovation (e.g. products, services, processes, business models), for example through new licensing or sandbox regime
• stimulates new business innovation (e.g. products, services, processes, business models), for example by setting challenging outcomes or releasing new information
• reduces time or cost of introducing business innovation (e.g. products, services, processes, business models), for example by providing better advice or simplifying processes
• improves business or investor confidence in how business innovation (e.g. products, services, processes, business models) will be regulated, for example through comms
• improves consumer confidence in business innovation (e.g. products, services, processes, business models), for example through improving protections or enhancing comms
• influences other UK regulators to take a pro-innovation regulatory approach, for example through forming partnerships or disseminating findings
• influences other administrations to align with its regulatory approach.

1.4 Outcomes

• Increased generation of ideas (e.g. products, services, processes, business models) by businesses operating in the UK.
• Increased investment in innovation in the UK.
• Increased competition (i.e. increased entry and exit to UK markets).
• Increased consumption of innovation (e.g. through greater consumer confidence) – i.e. increased UK market size.
• Lower regulatory barriers to trade (e.g. through global regulatory influence), with increased overseas market size.

1.5 Impacts

Increased quality/quantity of business innovation (e.g. products, services, processes, business models) that benefits the economy, society and the environment.

1.6 Goals (outside of scope of the evaluation)

• Faster, more sustained growth in productivity and wages.
• Increased resilience to economic shocks.
• Faster delivery of wider benefits of economic development (e.g. wider consumer choice, advances in medicine, transport, etc.).
• Better outcomes for consumers, workers, citizens and the environment.

2. Case studies: selection and methodology

Three case studies were selected to show early lessons from the programme.

Qualitative interviews were conducted with regulators and their external stakeholders to explore their perspectives on progress, impact and lessons learnt. These were:

2.1 Solicitors Regulation Authority

• The project team involved in the bid and delivery
• Three external stakeholders involved in the project in different capacities

2.2 MHRA

• The project team involved in the bid and delivery
• Two external stakeholders involved in the project in different capacities

2.3 Civil Aviation Authority

• The project team involved in the bid and delivery
• Three external stakeholders involved in the project in different capacities

For each case study, early lessons have been captured from the following phases and themes:

• Project set up
• Internal engagement
• External engagement
• Enabling innovation through regulation
• Value of the RPF

2.4 Case study methodology – interim case studies

Participants’ selection and data collection

Regulators interviewed for the case studies were selected by BEIS and Kantar to reflect emerging lessons from their projects.

For each project, the research team conducted: 1x 60 mins semi structured qualitative interview with key members of the project team (e.g. project and programme manager or bid manager, depending on team composition); 1x 60 mins semi structured qualitative interview with a strategic lead overseeing the project; 3x 30 mins interviews stakeholders (e.g. delivery partners, businesses or other organisations operating in the sectors, other regulators) identified by the regulators and selected by BEIS. 2x 30 mins stakeholder interviews were conducted for the MHRA case studies because of limited stakeholder availability.

Fieldwork was conducted between July and August 2019 (9 months into regulators’ projects). These interim case studies follow two rounds of a quarterly information management questionnaire delivered by Kantar. This questionnaire was used to understand how far projects were achieving the outputs and outcomes in the RPF logic model (see slide 3). Topics explored are included below.

Project team / strategic lead - the same topic guide was used flexibly to explore different perspectives:

• Regulators’ understanding of the RPF and motivations to apply for funding
• Vision for the project and connection with organisational mission
• Experiences of main stages of work (e.g. set up, internal and external engagement, delivery of project specific activities)
• Expected and unexpected challenges encountered
• Lessons they learnt about enabling innovation in their sector, engaging with businesses, regulators and other stakeholders
• Any emerging outcomes of their project on innovation in their sector
• Perceptions of RPF support, any impact the Fund had on their ability to enable innovation in their sector, and ways the RPF could be improved for the future

Stakeholders

• Stakeholders’ exposure to and perspective on regulators’ work
• Nature of their involvement and views on their engagement with regulators
• Ways in which regulators can more effectively engage with stakeholders to enable innovation in their sectors
• Any perceived initial outcomes and benefits deriving from regulators’ projects
• Views on future outcomes and how regulatory activity could be improved to encourage innovation

Analysis of the information collected through the case studies interviews

Material collected in interviews (e.g. audio files, notes) were organised through a thematic framework developed in Excel, informed by evaluation objectives.

Individual and joint brainstorming sessions were carried out by researchers in the Kantar team to review and consolidate insight, and draw key overarching themes.

2.5 Case study methodology – follow-up case studies

The following slides include updated findings from follow up qualitative interviews carried out with SRA between May and June 2020 (once their project concluded). These slides outline further progress, achievements and lessons learnt.

For each project the project team conducted 1x60 mins semi structured qualitative interview with the project team. Interviews covered additional activities carried out since the previous interview with Kantar: challenges faced; additional achievements and outcomes; lessons learnt about enabling innovations in their sectors through regulatory activity; perceptions of RPF support and plans to continue their innovation work past the RPF lifetime. 

Material collected in interviews (e.g. audio files, notes) were organised through a thematic framework developed in Excel and analysed in individual and joint brainstorming sessions carried out by researchers in the Kantar team.

3. Cross cutting lessons (October 2019)

3.1 Interim lessons from the case studies (October 2019)

The regulators interviewed noted that their experience of the Regulators’ Pioneer Fund programme had been very positive and valuable. They felt that the RPF had provided them with:

Time and resources to work on issues that may otherwise fall outside the scope of their day to day activities and operating models;

Opportunities to engage and collaborate with stakeholders (e.g. regulated businesses, other regulators (within and outside their sector), academia, tech innovators) to learn and understand the issues relating to innovation;

Impetus to seed cultural change within their own organisation and an opportunity to create networks and partnerships to continue and deliver impact in the sector after RPF funding has terminated. 

Beyond this, regulators also highlighted some areas in which they felt lessons (about the RPF design) could be drawn for future engagement:

Regulators explained that they didn’t feel fully prepared to start activities on their project when they were awarded funding and would have welcomed a longer time window between notification of a successful bid and the start of funding to set up their project;

Regulators also thought they would have benefitted from additional clarity on the administrative responsibilities connected with being awarded RPF funding. In particular, SRA and MHRA struggled with some of Innovate UK’s audit requirements, which delayed them in receiving payment; 

Additionally, both MHRA and CAA explained that they found funders’ decision to split the project budget over the two financial years of the project seemingly without flexibility to reallocate challenging. This issue was exacerbated by the short setup timescales as detailed above, as both CAA and MHRA faced delays at the beginning of their projects.

Regulators were of the view that it was too early to evaluate the impact of their projects and provide conclusive lessons on how the programme can best enable innovation in their sector. However, they felt that the programme had already allowed them to achieve positive results. They explained that their engagement with innovators and other stakeholders within their sector had allowed them to understand their sector better, to embrace different ways of working and to create fruitful collaborations both within and beyond their sector.

Thinking about the future, regulators highlighted elements of their projects that they would want to continue beyond the end of the programme, for example:

MHRA’s funded work covers a feasibility study for the development of synthetic data, and they hope to conduct future work to understand how to integrate this into their regulatory activities. CAA’s Innovation Hub has experienced high industry demand for continued support via their sandbox and other international engagement activities beyond the RPF funding end date of March 2020 but have not so far had confirmation that they would secure internal funding and are assessing access to external funding opportunities.

SRA explained that they found the Regulators Innovation Network, created for the RPF, extremely valuable as a way to share their thinking with other regulators. They thought the network should continue to exist past the end of this edition of the RPF so that regulators can continue to collaborate and share their experiences to foster innovation across sectors. They reported they are starting to look into how to develop their networks from the project to make sure they have an ongoing impact.

4. Case study 1 – interim findings (Oct 2019), Solicitors Regulators Authority (SRA)

4.1 Project at a glance – data-driven innovation in legal services

Role as the regulator

The SRA is the regulator of solicitors and law firms in England and Wales. They regulate more than 180,000 solicitors in England and Wales.

The SRA’s purpose is to protect the public by ensuring that solicitors meet high standards, and by acting when risks are identified.

The SRA’s policy team explains it has been active in supporting innovation in delivery of legal services.

Vision for RPF funded project

The SRA project consists of the Legal Access Challenge, a competition designed to enable the development of innovative digital technologies to make obtaining legal help affordable, understandable and convenient for users, particularly those under-served at present.

Short term, the SRA said their vision is to bring “real new solutions” to the sector to improve access to justice. Long term, their aim is to upskill themselves and modernise their regulation, engaging with strategic questions about how they regulate in a changing sector.

“Embracing all the new ways of working in the world around, encouraging innovation. Firms often think there’s a barrier to innovation when there’s not.” - SRA

Main activities involved

Key components of the project (in partnership with Nesta):

• Research phase and consumer / stakeholders consultations
• Legal Access Challenge design and set up
• Challenge launch and promotion
• Selecting and awarding winners
• Support to new technology development
• Evaluating outcomes

Motivations to apply for RPF funding

The project team explained that RPF was seen as an opportunity to catalyse efforts in an area the SRA had long been interested in, providing focus and resources to think differently.

The project team conveyed that SRA’s application to the fund fit within an existing stream of work on policy innovation, and was aligned with the regulator’s goal to be “ahead of the curve in identifying the challenges of how to regulate tech innovation”.

“We saw a really good opportunity to build on our direction of travel, do things that we don’t necessarily have the resource to do.” - SRA

4.2 Project set up

Understanding of the problem and RPF purpose

SRA said they started off with a very good grasp on what the problem was in terms of access to justice issues and lack of legal services for the general population and vulnerable audiences. Their understanding was refined through consultation with consumers and small businesses in the legal and advice sector. They explained their understanding and ability to support technologies addressing this improved over the course of the project through their collaboration with Nesta and discussion with tech companies and innovators.

SRA understood that the RPF “builds on the government’s industrial strategy, and its part of a whole set of initiatives to try and ensure that the UK remains a global leader, and one of key things to that is to stay at the forefront of the technical revolution”. Kantar felt they showed they were committed to the RPF’s mission and to their role in enabling innovation in the legal sector, as part of the programme.

“Regulators quite often don’t lead technical revolutions, but if the regulation’s not right, and regulators aren’t doing what business needs them to do to encourage support and remove barriers from innovation, it won’t happen.” - SRA

Challenges in setting up the project

The first challenge was defining the scope of the project, as SRA explained: “We tried to make sure that the Challenge was set in a way that was manageable and had some focus, but we didn’t want to narrow down too much, because the whole point of innovation is we don’t know what’s out there” (SRA). To address this, SRA partnered with Nesta, because of their expertise in designing challenges to promote innovation.

SRA also mentioned they found “setting up the logistics and the administration” for the project challenging, as they felt the turnaround between being awarded the funding and having to start activities was quite quick.

Also the project team felt they were not prepared for Innovate UK’s mechanisms on auditing and reporting (e.g. in terms of timings and resource needed to comply) and that they would have benefitted from additional clarity, as issues with audits also resulted in delays with receiving payments. Despite recognising that relevant information was provided online, the team thought they would have benefitted from more upfront guidance at set up: “You almost need a manual, somewhere that explained ‘now you’ve got the funding, these are our expectations in terms of what our governance requirements are, and that wasn’t really there.” - SRA

Lessons learnt on project set up and planning

Key lessons the team identified from the set up stage included better understanding the mechanisms of the funding and building the capability for a quick start (e.g. they hired a project manager to look after the Challenge alone). The SRA had never applied for government funding before, and described the experience as a positive learning curve that will inform how they approach future funding opportunities.

“We have experience now, we know what questions to ask, up front, having spent those four quarters going through challenges each time, we’ve built up a wealth of experience now.” (SRA).

SRA also noted the importance of early brainstorms with Nesta and collaboration with their network within the legal sector (i.e. consumers, advice centres and members of law society), which they thought were critical in setting up and positioning the Challenge. “I think our relationship with [Nesta] means that we’re clear on the strategy, we have clear project plans and we follow through which have really helped progress.” - SRA

4.3 Internal engagement

• Internally, SRA engaged:
• Internal partner teams
• Board and senior management
• Wider organisation

Nature of internal engagement and response from the organisation

The policy team (responsible for the Challenge) engaged a few other teams with the project directly. The SRA’s research and analysis team provided support with the initial stages of bid and project development; the executive director for finance was involved to make sure they would promptly sign off payments and support with administrative challenges (such as addressing issues in getting payment claims approved by Innovate UK).

The SRA have noted the importance of engaging their Executive Team and Board, getting a clear commitment from them from the bid stage. They thought that senior buy-in and engagement with the Challenge has been fundamental to ensure speed of decision-making and a clear direction on the project: “The senior management has really helped because of that hold up and down to say this is where we are, this is what we’re doing, this is where the focus has been really important.” - SRA

In order to ensure that their leadership team is consistently updated on the project’s progress and that expectations are aligned from the top down, the project sponsor (an executive director) sits on weekly catch ups with Nesta, and another executive director (part of the Board) is involved in fortnightly calls on the project. Furthermore, Nesta held a workshop with the SRA Board, to present findings from their scoping research on current developments in technologies, related practices in other jurisdictions, barriers to take up in the UK legal sector. These findings are currently being considered to inform the development of the regulator’s 2020-2023 strategy. “All of that is feeding into our strategy for the next three years. It hasn’t been signed off yet, but I think it’s highly likely that we’ll have innovation in technology as one of our core objectives.” - SRA

In terms of wider internal engagement, the SRA explained they have several internal communication mechanisms in place (e.g. all staff meetings, management meetings). While some updates on the Challenge have been provided in those occasions, the team said they plan to send out regular communications on the Challenge once finalists have been selected, so that the wider organisation can see some concrete outcomes of their work.

The project team said that overall the response they have seen from their organisation so far has been very positive, and felt this was because the project aligned with an ongoing internal interest in innovating practices. “Innovation is something that we’ve focused on prior to the challenge, so it’s been on the organisation’s agenda for a while.” - SRA

4.4 External engagement

Externally, SRA engaged:

• Delivery partner (Nesta)
• Tech companies & others active in technology promotion
• Other regulators and government (e.g. legal sector regulators, FCA, ICO, CAA, HMCTS)
• Law firms and sector professionals
• Consumers and perspective users
• Advice sector (e.g. charities, Law Centres Network)

External stakeholder engagement

As part of the project, SRA engaged with a number of stakeholders, spanning the legal sector, advice sector (e.g. charities, Law Centres Network) and technology sector.

The partnership with Nesta led to a close collaboration, which started with bid development and will continue into impact evaluation of the Challenge once the winning technology services are launched. “We have a very open and regular contact. [SRA] are very responsive and helpful.” - Nesta

As part of the set up phase, SRA and Nesta said they also engaged with stakeholders in the advice sector and consumers to gain their views on current challenges in accessing legal advice, both for vulnerable people and the general public.

As the project developed, SRA mentioned being active in involving members of the judging panel (including the Law Centres Network and HM Courts & Tribunal Service) and organisations that will provide support for winners, forming direct collaborations with Hogan Lovells (who will provide legal advice to winners) and the ICO (supporting with any data safety and information sharing queries). “[Our collaboration] is well organised, it’s all happening according to plan.” - Law Centres Network

The launch of the Challenge was also a good opportunity to engage with tech companies, who have since reached out to talk about their technologies and get more information about the Challenge. The SRA also took advantage of the Regulators’ Innovation Network created by the Better Regulation Executive after the launch of the RPF. “We’ve been learning loads from the network. We started a conversation with FCA and CAA about sandboxes and addressing similar issues across sectors.” – SRA

Lessons learnt on stakeholder engagement

SRA felt they learned a valuable lesson about the importance of agreeing a clear framework for delivering timely communication when working in a partnership with Nesta.

The project team explained that it has been critical for all parties across both organisations to be involved in the sign-off of communications. This included SRA senior leadership, which the team now knows need to be engaged as early and consistently as possible. “I think the lesson learnt is being clear around responsibilities and accountabilities.” - SRA

SRA reflected on the importance of sharing thinking with legal actors and tech start ups and companies, as well as other regulators. Both the project team and Nesta felt they learned valuable lessons on what the barriers are for innovators (e.g. access to funding and the development of a sustainable business model) and on how to bridge the gap between tech companies’ agile way of working and the legal sector’s reactive, conservative style of thinking. As the project lead from Nesta explained: “Attitudes in the legal sector have come up in quite a lot of conversations. Around risk-averseness and willingness to change.” - Nesta

SRA felt that the consistent promotion of their efforts was also instrumental in enabling progress to date, and a key lesson in this area. “At every opportunity we could we promoted the Challenge, and what the Challenge is trying to achieve. I think engagement, engagement, engagement is why we are where we are.” - SRA.

Other stakeholders interviewed explained it was too early for them to derive lessons on engagement, as they will be more meaningfully involved as the project develops.

4.5 Enabling innovation

Initial positive outcomes and achievements

Because at the time of the interviews the technological services funded by the Challenge had not yet been developed, both SRA and stakeholders interviewed were able to identify limited final project outcomes. However, they mentioned there were some early achievements, particularly in relation to opening the sector up to innovation.

Both SRA and stakeholders interviewed believe that the project started a valuable and fruitful conversation on innovation in the legal sector, sending the message that “it is possible to go to the regulator” to discuss innovation. This contributed to promoting the SRA’s efforts to tech companies and other relevant stakeholders, both in the UK and abroad, raising the profile of the Challenge. It also provided an opportunity to share ideas for those involved. Nesta highlighted that 117 applications were received for the Challenge, showing considerable interest and willingness to take part in the SRA’s innovation effort. “Saying ‘we’re open for business’ was a big achievement after lots of work.” - SRA

SRA also explained using opportunities for networking as well as contact with innovators and other regulators to advance their understanding of technological innovation. They also took these opportunities to share knowledge on how it is possible to tackle cross sector issues in a data-driven economy and develop effective services fit for the future. “We’re trying to leverage as much as we can out of the relationships we’re building.” – SRA

Overall, the team thought their work as part of the project helped the SRA refine their understanding of the role technology, whether digital or more advanced applications, can play in the legal sector, and how to enact more innovative and proactive regulation in the future.

“They have a lot of people thinking about [access to justice] that wouldn’t have otherwise. For instance I got someone from France contacting me to talk about what would be possible, which was surprising… that knowledge of [the Challenge] extended that far.” - Law Centres Network

“Nesta has done an excellent job raising the profile of the challenge, reaching out to the right people. The launch event was excellent. It was clear what they wanted to achieve.” - Hogan Lovells

We will have more tech companies that are thinking about this area and engaging with us on this issue, and through the network working with other partners.” - SRA

Potential beneficiaries of the project

SRA will benefit from a refined understanding of how technology can support the legal sector and how to promote it.

The general public, particularly vulnerable consumers, could benefit from improved access to justice thanks to the technologies developed.

Innovators may benefit from a direct link with the regulator and the knowledge that they are open to supporting innovation.

The legal sector might benefit from more proactive regulation and the introduction of digital technologies and tools.

Lessons learnt from effort to enable innovation in the legal sector

The SRA highlighted that it is still early to identify the impact of the project as it is in progress. The SRA noted the following lessons learnt so far:

Better understanding of how technology could help resolve problems of access to justice. SRA explained that through their project they learnt more about technology, and developed their understanding of its market and of the consumer needs tech solutions need to address. “[The project] has put us in a much more informed position about the market.“ (SRA) The Challenge also allowed them to gauge the appetite for innovation in the sector.

Appreciation of barriers to the adoption of technology in the legal sector. While there can be resistance to tech based services from the legal sector, due to lack of trust or understanding, both the SRA team and Nesta thought that progressing in the project will help to better frame these challenges. The SRA said they plan to build on the lessons they will learn from the Challenge’s later stages (e.g. tech development and implementation) to progress their work in overcoming barriers to innovation in the sector. “We’ll get a much better understanding of what the barriers to access to justice related technology are, and what role we can play in overcoming them. We will take the learning [from the Challenge] and we will build on the barriers and the learnings that we get from the Challenge to progress our work going forwards.” (SRA)

Need to adapt intellectual property clauses address innovators concerns. SRA described having to adapt intellectual property clauses to address tech developers wish to maintain rights to their innovations when taking part in the Challenge. By amending these clauses, they were able to reassure innovators and ensure their participation.

Importance of having a regulators’ network and ongoing collaboration with stakeholders. SRA recognised that they heavily relied on the Regulators’ Innovation Network, which allowed them to share ideas and build collaborations with other regulators, helping them progress their innovation efforts. They said they intend to continue contributing to this network which allowed them to find cross sector areas for possible collaboration. “The [Regulators Innovation Network] events have resulted in at least one direct collaboration with the ICO on our project. That’s a really good example of the benefit of the Fund and networks that goes with the Fund.” (SRA). The project team also learnt the value of collaborating with partners (e.g. Nesta, ICO, other regulators in the legal sector) with different areas of expertise, which allowed them to share and update their knowledge, finding solutions to issues they wouldn’t otherwise had thought of.

Plans to disseminate learnings

The SRA shared the following ideas to publicise learnings from their project:

A communications strategy to announce winners and share achievements to date is currently being developed.

The policy team explained they will ask to present to other regulators within the Regulators Innovation Network on findings from implementation of their project. They will also share learnings on any cross-cutting issues and the impact on other sectors so other regulators can benefit from that knowledge.

SRA said they have started conversations about innovation with organisations within the sector (e.g. other legal regulators and law firms), and while they did not specify the format, they plan to feed lessons from their project back to their networks.

SRA also said they will recast their innovation offering to the sector on their website and share their work on innovation through a marketing campaign.

Internally, they will apply lessons to any new projects or opportunities following the project. They also have plans for an internal newsletter, aside from including learnings on innovation in their next three year strategy.

4.6 Value of the RPF

The RPF supported the SRA effort to enable innovation in their sector, providing value in a range of ways:

• Providing focus for innovation efforts and necessary resources
• Enabling upskilling and learning to inform the regulatory approach
• Creating a network with regulators to collaborate on shared challenges (e.g. information security and access to advice)
• Incentivising collaboration with partners across different sectors

“[Innovation] is within our regulatory remit, but [RPF] has acted as a catalyst. Ultimately we’ve always been interested in putting resources into this area, but it’s that focus.” - SRA

“[RPF] has given us the opportunity to develop our direct involvement in innovation and to leverage that for our wider regulatory learning, which at the time was too far from our day-to-day and normal operating models. So it gave us the funding to do that and the incentive to collaborate with someone like Nesta and use their experience.” - SRA

Thoughts on how the RPF could be improved and better support regulators if the programme was repeated

Overall, the SRA explained that that have found their engagement with the RPF very valuable and found the BRE team available and helpful. The following suggestions for improvement were provided:

Longer application window and set up process – SRA found that timescales between being awarded the fund and starting with project activities were quite tight, making it challenging for them to get organised in time.

Upfront directions on participation commitment - SRA thought that clearer guidelines on the anticipated project management commitment would have helped with better understanding on project administration reporting expectations. This includes more clarity on number and timings for audits and when regulators can expect audits and to receive payments.

Adapting intellectual property clauses – the SRA thought that adapting the contract terms set by Innovate UK on intellectual property, so that terms and conditions clearly delineate the regulators’ rights and specific purposes for using innovations, would facilitate arrangements with innovators.

Permanent regulators network – SRA thought that “the innovators’ network has been fantastic” and felt that they benefited greatly from it. As it is currently tied to this edition of the Fund, the project team thought it should be recast and made permanent once the current projects are over, not to lose its value to regulators and cross sector innovation.

4.7 SRA project – follow-up findings (July 2020)

Update on data-driven Innovation in Legal Services

Update since July / August 2019

Since their interim interviews with the Kantar team, SRA has been active in completing Legal Access Challenge activities, building further links across the sector and evaluating their work.

Once applications closed, the Challenge’s judging panel selected eight finalists who after six months developing solutions were narrowed down to two winners (awarded additional time and funding to develop their solution further). SRA explained they worked to provide regulatory assistance to the finalists and facilitated connections with other bodies providing specialist advice (e.g. ICO on data protection, Hogan Lovells on legal matters, Law Society for access to specialist committees). Since the Challenge finished SRA have been working to gather feedback from their finalists and compile their final report on the project (now published: https://www.sra.org.uk/sra/news/press/legal-access-challenge-final-reports/).

SRA developed their partnership with Nesta over the course of the project. The project team saw Nesta’s contribution as complementary to their work because of their expertise in Challenges process and their connections with innovators: “We are the regulator and can open some doors, Nesta being the innovation organisation that it is also has their ways and approaches and sometimes it’s better they made the initial approach.” - SRA.

The partnership arrangement worked well bringing together Nesta’s wider perspective on innovation and tech organisations and the SRA’s expertise in the regulatory landscape and legal services market. This has helped the team look beyond their own sector and laid good foundations for a focus going forward on tech and innovation in the legal services market by the SRA.

SRA maintained regular contact with legal regulators and established long term connections with other ones they didn’t previously have a lot of interaction with, as the team explained they took part in “350 across types of engagement” (e.g. meetings, committees, events) over the course of the project. SRA said they found these interactions helpful for their work, allowing them to navigate cross sector issues raised as solutions were developed (e.g. how to develop datasets for testing with the ICO and managing the overlap between financial and legal regulation on claims management with the FCA).

After Challenge winners were announced the SRA formally continued to provide support to them, but they also stayed in contact with the other six finalists to help them with issues and maintain a conversation with them. SRA explained they continued to be open to requests and to support innovators who took part in the Challenge. The team were keen to progress their work on removing barriers to introducing innovative technologies in the legal sector, and to promote a more proactive and enabling approach to regulation in the legal sector.

Once their organisational profile was raised through promotion and events, SRA also proactively sought opportunities to expand their network and become involved with innovators and other organisations working on legal tech. They explained that they actively reached out to some, but in a few instances they had innovators contacting them because of their work: “as you put feelers out to speak to other people they suggest things. So your organisational profile is raised, so others become aware of you and also proactively we went and contacted people.” - SRA

The team explained that alongside engaging with some colleagues directly, they were active in promoting their work within the SRA at key milestones (e.g. selecting winners, events and press releases) through their internal bulletin and podcasts: “We raised the publicity about what we were doing in the Challenge partly just as an effort to make people aware of what the SRA was doing but partly because it was quite useful just so we could approach people and say: ‘you probably heard about the Challenge, but actually in this instance we need your help on something’.” - SRA

Additional outcomes and achievements of the project

SRA explained they had a range of technologies coming out of the Challenge and that “seven out of eight finalists actually said that the process of being in the Challenge accelerated the development of their solutions, so being part of the Challenge helped.” - SRA. The team thought this was partly due to the support they offered to innovators in navigating the regulatory landscape, and partly to the opportunity tech organisations got to meet with others at working events during that time. This contributes to the RPF goal of enabling economic growth, connected with the RPF outputs (see logic model on slide 3) working towards this.

The team also felt that their work opened up avenues for collaboration with innovators and other relevant stakeholders in the sector: “From the regulators point of view, we were able to meet access to justice organisations that normally we wouldn’t come across. For us that was a hugely expanding experience, because they got to meet us and realise that we are a regulator that’s quite open and we got to meet a lot of them which actually helped them to realise that the regulator for solicitors [i.e. SRA] was quite interested in a dialogue with them in trying to understand their language, their issues.” - SRA. SRA also created new connections with organisations involved in developing or promoting legal technology (e.g. City of London Corporation Legal Tech Group, Tech Nation) and saw this as the “continuation process” of the work they started with the Challenge.

SRA noted that their project was helpful in increasing their general knowledge of the sector and understanding of the problems, and learning about technology and access to justice issues. For instance, they discovered that the regulator itself is not an obstacle, and that innovators actually can hold back because of challenges in navigating through the regulatory regime. Through the Challenge they were also able to gain in house capability, learning from innovators and Nesta’s work. This along with the point above contributes to the RPF goal of fostering a pro-innovation business culture, and connected outputs such as outputs 4 and 6).

In the context of COVID-19, with technology coming to the fore for most people, the team explained that “us being involved with the Challenge at this time has been extremely helpful because we’ve had a laboratory to look at the impact of the imposition of tech on legal services.” - SRA

SRA explained that their project also informed the development of their upcoming corporate strategy, also raising general awareness of implications of legal tech and “what need to think about, what need to try and anticipate” within the organisation: “There is now a second objective which is all about developing innovation and technology, partly was general thinking, but a lot of that has come out of the challenge and what we learnt from the challenge on which direction we should go on.” (SRA) This gives a positive indication of a mindset shift within the regulators, potentially leading to longstanding changes in their approach to innovation.

“[The project has been helpful in] accelerating our own sense of knowledge and understanding about it, but also being able to understand what else the issues are going to be. Part of it is being anticipatory in one sense, but also if we then get approached by say a legal start up we’ll have a bit more familiarity of where they fit in and if they need any other assistance we’ll be much better grounded in being able to offer guidance or to know who they should speak to.” - SRA

“The challenge has helped tremendously in just increasing the organisational knowledge and awareness of issues and things that need to be addressed.” - SRA

Additional lessons learnt and value of the RPF

SRA reiterated the value of cross sector collaboration, as they realised throughout their work that “some issues that we raised by some of the finalists crossed over regulatory boundaries.” – SRA. They explained that bringing in expertise from other regulators was not only beneficial for regulators themselves, but also for innovators who could realise the positive impact of regulation as an enabling force, rather than a barrier to innovation.

“It has multiple benefits. Us speaking together actually benefits us in terms of increasing our knowledge, it benefits the party involved, so the innovator, and it also helps the FCA, the ICO or whichever regulator it is to actually understand the issue from another angle.” - SRA

When it came to establishing contacts and collaboration, SRA learnt the benefits of “going to see [stakeholders] where they are, not expecting them to come to you” - SRA, adopting a more personal approach to interactions to maximise engagement.

The team felt they learnt the importance of being open and receptive to the industry and innovators to break down perceived barriers linked to regulation. They started an ongoing dialogue with their sector with the Challenge, which they felt not only allows regulators to understand language and issues faced by innovators, but also to create a positive conversation about how innovation can be developed.

SRA also reflected they learnt to be “creative and a bit lateral [in our thinking]” in trying to understand and proactively tackle problems in their sector. They felt it was important to “question everything and looking at areas where you’re operating and see what you can do to help development.”

Additional feedback on the RPF

SRA felt that extending the funding period from 18 to 24 months would be helpful for regulators to develop to understand issues, develop their projects, and detect outcomes: “Some things just take a while to get sorted out and sometimes people just find that they need a skill or a piece of tech that they don’t immediately have.” - SRA

They also thought that it would be beneficial to agree an escalation procedure for issues that require decisions made at higher level within BEIS or IUK. This would allow regulators to quickly get to the people who can give advice and resolve problems.

SRA valued the informal meetings with other RPF regulators set up by the CQC to resolve issues and keep updated on others’ work. So they explained that establishing formal communication mechanisms for Fund winners to exchange experiences and challenges would be very positive in the future.

Future plans

SRA plans to maintain contact and form a community with Challenge’s applicants to develop their work on innovation: “One of the things that we want to develop and we want to take on is about developing a community. We don’t see the Challenge as the end of our process, we just see it as part of the work that we’re doing. We want to be able to develop it.” - SRA

With legal tech innovation being part of their new corporate strategy, SRA also plans to continue their work to help innovators navigate the regulatory context in their sector and put them in contact with others to resolve other regulatory obstacles: “Looking forward that would be an aspiration for us. Creating an environment where we could bring together a number of organisations to say you go off to create a collective response, but actually [in that way] we can see the problem from start to finish, not individual parts of it.” - SRA

5. Case study 2 - Interim Findings (Oct 2019), Medicines and Healthcare products Regulatory Agency (MHRA)

5.1 Project at a glance - Developing synthetic datasets for the validation of algorithms

Role as the regulator

The Medicines and Healthcare products Regulatory Agency regulates medicines, medical devices and blood components for transfusion in the UK.

MHRA is an executive agency of the Department of Health and Social Care and aims to play a leading role in protecting and improving public health, supporting innovation through scientific research and development. 

Vision for RPF funded project

One of the key challenges in both validating and developing innovative, new medical devices and software (such as those involving artificial intelligence) is having access to sample patient data to build and test new products. However, the need to protect patient privacy means that such data cannot be freely or cheaply shared. MHRA’s aim is to address this challenge by developing a first-of-its-kind synthetic dataset that mimics real patient data while protecting real patient confidentiality.

The RPF project serves as a proof-of-concept research project on which, should the concept be proved and further funding be secured, further work can be based.

Main activities involved and focus outputs

The main activities of the project are:

• Desk-based research (reading academic papers)
• Innovator and healthcare stakeholder engagement 
• Analysis of existing data
• Iterative generation and refinement of synthetic data
• Collaborative testing with innovators and academic challenge

Focus RPF programme outputs:

1. Innovation by changing outcomes/releasing new information

2. Other UK regulators influenced to take up pro-innovation regulatory approach

Motivations to apply for RPF funding

MHRA had identified synthetic data as a potential way of overcoming some of the challenges in regulating new products in their sector (see ‘Vision for RPF funded project’ box). However, they said, as an executive agency of the Department of Health and Social Care, MHRA has a statutory public health remit that steers its core activities. Conducting research of this nature would fall outside of their core-funded mandate. 

The team recognised the RPF as an opportunity to allow them to  explore a potentially valuable proposition through a research project that, at the time of bidding, they felt would serve as a new tool in their regulatory toolkit:

“It was really lovely to have a grant that allows us to embark on a proof of concept project, which one could fail on.” – MHRA

5.2 Project set up

Understanding of the problem and RPF purpose

Kantar judged that MHRA had a strong understanding of their specific problem – “Regulation of machine learning algorithms has been a hot topic, it’s been a concern for some years now. And there’s a concern that there’s not enough approaches or […] enough tools in the toolbox to be able to do this.” - MHRA

Synthetic data, as a potential option for validating such algorithms, had been an idea that MHRA had had approximately one year before the RPF bid, but nobody knew how to go about developing it. Previous work showed that synthetic data could mimic the structure of real datasets but didn’t stand up to analysis.

The team understood the RPF aims as follows: “to allow regulators to develop methodologies or innovations which will ultimately help promote people in industry or other innovators to get their products through regulation [into markets] in a more streamlined way.” - MHRA

Challenges in setting up the project

Three challenges were mentioned:

The RPF funding structure: The fund was broken into two parts (as it spanned two financial years), leading to problems for MHRA: “The initial part of a project is always difficult, because you have to get memorandums of understanding, get people in place and all those sorts of things. And so because you had to spend your first chunk within the first financial year, we didn’t, so we lost some money simply because of that rule that we couldn’t carry over [unused budget] into the next year. So that was a bit of a pain.” - MHRA

Lack of administrative awareness: MHRA said they were required by Innovate UK to complete an audit after the first month that MHRA had not anticipated. This led to delays in having funding released to them.

Perceived high level of bureaucracy: MHRA felt that the level of reporting and administration for the project has been high, taking up a lot of the project manager’s time since the start of the project.

Lessons learnt on project set up and planning

Need for a dedicated project manager: Although they have found the project administration burdensome, MHRA have a dedicated project manager who has handled this. This has allowed the rest of the team to focus on project delivery: “I think what I’ve learned is that it is so, so important to have someone who can coordinate the project and be a project manager because otherwise, if all our time went on that project management and bureaucracy, we wouldn’t have time to do this stuff.” - MHRA

Project computational load: The work that the team are doing requires the processing of large quantities of data and has required them to find hardware capable of handling their needs – something MHRA would address earlier if they were to do it again.

Project managing innovation: MHRA found that they have learned lessons about setting up projects of this nature (such as the time taken to ensure memoranda of understanding, non-disclosure agreements and other legal agreements were in place) that may mean they balance the timelines of future projects differently. MHRA have also developed technical skills about working with data that will support future projects to get set up more smoothly.

5.3 Internal engagement

Internally, MHRA engaged:

• Internal teams
• Corporate Executive Team

Nature of internal engagement and response from the organisation

The project had to be signed off by MHRA’s corporate executive team. However, the executive team were aware of the RPF from the bid stage and encouraged the project team to bid for the funding in the first place so have given their ongoing support and endorsement for the project. 

MHRA were able to anticipate and put in place key personnel and internal technical resources in some areas before the project started. For example, they lined up their academic supervisor before they were awarded the funding and freed up their key researcher in advance “so we were able to get off to a fairly flying start.” - MHRA

Due to the research nature of the project, there had been no organisational change at the time of the interviews. However, there had been some reallocation of resources to deliver the project (i.e. team members moved to work on the project). There had been no formal communication within the organisation about the project at the time of interview as MHRA note that they were too early in the project, but should the concept be proved viable, they anticipated more internal engagement, particularly in any follow-on work. 

One internal challenge involved overcoming internal IT restrictions, limiting MHRA’s ability to use specialist software on internal machines – this required the team to develop technical workarounds. 

5.4 External engagement

Externally, MHRA engaged:

• Project Partner (NHS Digital)
• Academic Partner (Brunel University)
• Industry stakeholders (long list of 20 narrowed down to 3)
• Academic journals and wider academic community
• Steering Committee

External stakeholder engagement

MHRA approached approximately 20 industry stakeholders who MHRA felt may be interested in their synthetic data work. The team then explained they carried out interviews with these stakeholders before narrowing down to three industrial partners.

Working with these partners involved discussions with innovators to understand the sorts of data needs the innovators had and what sort of structure a synthetic dataset would need to have to be useful to validate and license their product. Practically this involved sharing of data (both real and synthetic) between organisations. For example, Sensyne provided MHRA with data used to develop a Sensyne medical device to manage gestational diabetes. MHRA used this data to create synthetic data that can be trialled by Sensyne and other similar organisations.

This engagement proved critical to MHRA as it highlighted unexpected use-cases for their synthetic data: “…there are things we couldn’t have imagined […] and it’s again emphasising the importance of having stakeholder involvement because there’s no point in us doing all this and then someone at the end saying ‘oh, it doesn’t work for us.” - MHRA

Lessons learnt on stakeholder engagement

MHRA were reminded of the time that it takes to set up relationships where commercial sensitivities are a factor (e.g. setting up Memoranda of Understanding (MoUs), data sharing agreements, etc). 

They also learned that interviewing innovators privately rather than holding public discussions (e.g. in a group) was a more effective approach at encouraging innovators to be open with discussing their ideas.

The project steering committee of industry and academic partners proved essential because “things that would never have crossed our minds have really been highlighted and shaped the project quite a lot.” (Brunel – Academic Lead)

From the perspective of the industrial partner, MHRA’s engagement was highly valued: “Having someone at government who is truly interested in doing something to better a situation or to fill a gap in the regulation, it just feels good. It means that they are listening to what the industry has always been shouting about, i.e. help us out doing things better.” - Sensyne – Industrial Partner.

5.5 Enabling innovation

Initial positive outcomes and achievements

The key positive outcome of the MHRA project has been the unanticipated value of their synthetic data work. At the time of bidding, their focus was limited to developing a new regulatory tool. However, their feedback from industry and academia revealed that MHRA’s methodology had much broader commercial value and myriad applications. The methodology they have developed has been academically validated and represents a world-first approach in data science. At the time of interview, however, MHRA were cautious to downplay the potential impacts as their concept had not yet been proved and they had no guarantee of future work exploring potential commercial applications and exploitation. However, the team all felt positive about their progress and all signs pointed towards them achieving their proof-of-concept. As such, the key anticipated positive outcome was the potential to move on to follow-on work bringing the synthetic dataset into their regulatory activity and potentially sharing it with industry. Other outcomes and potential impacts include:

The team (including the academic and industrial partners) has enjoyed working on the project and the freedom that it has afforded them to work. The fact that the fund doesn’t dictate requirements for exploitable outcomes has provided “true freedom of exploration” (MHRA) and allowed them to evolve the project as they go.

The learning the team is gaining is supporting the advancement of synthetic data science through their dissemination, but also allowing them to provide insight to the NHS and other data custodians about existing issues with datasets such as bias, missing data, etc.

“By generating synthetic data it’s opened up all sorts of interesting questions that have branched off from the original aim.” - Brunel – Academic Lead

“This is proper science getting done” - Sensyne – Industrial Partner

“There’s also big benefits to worldwide thinking on this [due to the dissemination - anticipated five publications] and it is a world-first, I think. Nobody else has done anything like this.” – MHRA

Potential beneficiaries of the project (should the project lead to wider application of synthetic data):

MHRA may benefit through being able to add a novel and unique approach to their regulatory toolkit for validating medical innovations

Patients may benefit through increased safety (ensuring devices/software are fit for purpose) as well as privacy

Health services may benefit through new tools that improve the ability to diagnose and manage patient health, and save costs through efficiency

Innovators may benefit from synthetic data making product development easier and cheaper, smoothing the regulatory pathway and safely accelerating their route to market

Lessons learnt from effort to enable innovation in the health sector

MHRA were keen to point out that their current project was focused on developing a proof-of-concept and that understanding wider impacts on innovation in the sector will be examined in a planned follow-up project. However, some learnings have emerged already.

One lesson they spoke about is that although the project conceived of synthetic data being used in a regulatory context (i.e. for testing and validating new products and services), working with industry has highlighted additional potential for using synthetic data to enable innovation by supplying it to developers to build their software around the synthetic data. 

Another critical lesson they identified was that developing synthetic data can’t be “one-size-fits-all” and will need to be tailored to the needs of different innovators. MHRA are keen to share their methods for developing synthetic data through publishing their approach in academic journals, allowing other organisations to develop their own synthetic data. However, initial feedback from some of their stakeholders has been that they would rather pay the MHRA to do this for them. This suggests a potential new source of income for MHRA that was not anticipated. 

The project team recognised that the potential applications of synthetic data extend beyond the health sector. Their academic lead noted that learning from this project could be applied on projects in the environmental sector (as an example) too. 

The industry partner commented that this project demonstrates a positive shift in the way that the regulator is working – moving from a reactive to a proactive approach. This is seen as a key enabler for innovation in the sector. 

Plans to disseminate learnings

MHRA, with the University of Brunel, have already published a paper in an academic journal that has gained positive feedback and international interest.

A further two journal papers were in progress, at the time of interview, with an additional two planned by the end of the project – a total of five academic papers.

MHRA have published their methodology for the creation of synthetic data, so others can use and develop this for their own goals and in their own sectors. Furthermore, because their dataset has been created with sensitive data (i.e. health data), similar applications with other sensitive data types can use the same methodology.

They have presented their initial progress at a conference and intend to present more in future

5.6 Value of the RPF

The RPF supported the MHRA effort to enable innovation in their sector providing value in a range of ways:

• Enabling a project that would normally be out of scope for the regulator
• Generated new knowledge with potential multi-sector applications
• Developed new skills in MHRA that can be applied to other projects
• Supported an exploratory research project, that did not place commercial success ahead of scientific inquiry

“None of this would have happened without the RPF, so it’s incredibly valuable. If there were another round, we would probably be asking for a bit more money for the next phase.” - MHRA

Thoughts on how the RPF could be improved and better support regulators if the programme was repeated

MHRA were keen to convey that they were extremely grateful for and satisfied with the funding and that their suggestions for improvements were minor:

Dedicated resource for project management – MHRA have benefited greatly from having a project manager (PM) role in their team who removes administrative burden from those tasked with technical delivery. They would recommend that dedicated PM funding be an essential component of similar funding in future.

Greater flexibility over funding phases – MHRA experienced issues in the early part of the project due to external factors, resulting in a need to delay some elements of the project. However, the phasing of the funding prevented them from carrying funding from one financial year to the next. This forced them to spend money or lose it, meaning some work was rushed or done in a way that didn’t follow their ideal plan. 

More time for bid and project setup – the timescales for preparing the bid were challenging for MHRA. Similarly the time between being awarded funds and needing to start work was short. More time to prepare the bid and more time from notification of award before start date would have helped them be better prepared.

Follow-on funding opportunities – MHRA were confident that their project will be a success but were unsure about how to fund the next phase of it, which will begin to explore how to bring synthetic data into regulation and wider industry applications. MHRA are keen to avoid seeking private investment as they want to share their knowledge. They suggested that the RPF could be extended or that, otherwise, Innovate UK could support them to access other funding opportunities of which MHRA are not currently aware. 

6. Case study 3 - Interim Findings (Oct 2019), Civil Aviation Authority (CAA)

6.1 Project at a glance - Innovation in Aviation Engagement Capability

Role as the regulator

The Civil Aviation Authority (CAA) regulates the aviation industry in the UK. They “make sure that the aviation industry is safe and [they] protect consumers - those who choose to fly and those who don’t choose to fly [i.e. members of the general public that might otherwise be impacted by aviation activity] […] That covers helicopters, planes, pilots, cabin crew, right through to dangerous goods, airports and then the people who manage the skies.” - CAA 

The CAA additionally have some responsibility for environmental standards and cyber security for the aviation and travel sector. They also help set regulations and influence aviation standards internationally. 

Vision for RPF funded project

The project addresses the problem that technology in the aviation sector is changing rapidly compared to the incremental change of the past. Regulation was struggling to keep up with new changes, responding reactively, rather than looking to the future. 

The project established a dedicated innovation team to bridge the gap between innovators and the CAA. Engaging with innovators simultaneously supports these innovators to develop their proposals in a way that will make approvals more likely and gains insights (for the CAA’s benefit) to support the development of new regulatory approaches.

Main activities involved and focus outputs

The team supports stakeholders through three key services:

Gateway – the team’s ‘shop front window to innovators’ making it easier for Innovators to access CAA expertise, guidance, and viewpoints on regulations by providing a focal point of contact and information

Sandbox – where the CAA can trial innovations in a safe space, help innovators overcome any hurdles or help them consider what needs to be done to get regulatory approval

Regulatory Lab –accelerates the development of new policies and regulations by anticipating regulatory challenges in areas of innovation and by defining the requirements for new policies and regulations.

Focus RPF programme outputs:

1. New business innovation through new licencing or sandbox regimes

2. Reduced time or cost of introducing business innovation

3. Improved business/investor confidence in business innovation regulation

4. Influencing other international administrations to align with regulatory approach

Motivations to apply for RPF funding

The CAA recognised a need for a change in the way they address innovative technologies in their sector but had no way of funding a team to address this through their existing funding structure. This structure depends on charges levied on the organisations regulated by the CAA. This means that their activities had often been limited to things with direct benefits for major aviation stakeholders.

The CAA said that the RPF represented an opportunity to address their innovation challenges without needing to overcome this hurdle.

“[The CAA] have been starved of resource, for some while, to look to future regulation…” - Cranfield University – Sandbox Participant

6.2 Project set up

Understanding of the problem and RPF purpose

Kantar judged that the CAA had a clear understanding of their problem and how their team would solve that issue: “It’s about engaging better with innovators and emerging technologies to ensure that they are in the best possible state of regulatory readiness to go through regulatory approval when that is appropriate, but also to ensure that our teams have the greatest state of readiness and awareness for when they are asked to look at something that’s new.” - CAA

The CAA felt that the purpose of the RPF is to prompt regulators to take a more anticipatory approach to innovation and new technologies. They felt that the RPF is also there to explore opportunities that could flex or adapt existing regulatory boundaries to meet a changing sector without compromising regulatory standards or consumer protections. 

Challenges in setting up the project

Three key challenges related to setting up the project were mentioned by the CAA – all related to the nature of the RPF:

Short timescales: The CAA were notified that their RPF bid was successful in September 2018 and they were expected, by Innovate UK, to start spending shortly after. However, due to their project requiring the creation of a new team, by recruiting new staff or reallocating internal staff, they took several months to get the full team in place. They felt that the timescales between the notification of their award and the project start-date were tight for their project as they couldn’t begin recruitment/reallocation until they knew their bid had been successful. 

Phasing of the fund: Related to the above, the RPF funding was split across two financial years, requiring the CAA to spend a large amount of their allocation within the first six months, which they found a significant challenge.

Follow-on funding: From the beginning of the project, the CAA were aware that the RPF funding only lasted until March 2020 and that they needed to find a way to keep the new team going beyond that date. This issue has taken up a lot of time and focus of the team since the start of the project. 

Lessons learnt on project set up and planning

A key lesson identified by the CAA about setting up a project like this was around the value of early stakeholder engagement to steer the project. The CAA included a phase of industry research at the beginning of the project. This was done in order to validate their assumptions around the problem and whether their offering was of value to their industry stakeholders. This process supported and informed the shaping of their service propositions and gave valuable insight into how stakeholders viewed the CAA.

The CAA felt that this early industry research was essential to the way they shaped their offering to ensure that their team could gain industry buy-in and have the biggest impact.

6.3 Internal engagement

Internally, the CAA engaged:

• Internal teams
• Directorial board
• Project control board

“We directly have an impact on helping the core regulatory parts of the [CAA’s] business have the space and the time to think about these issues [new challenges in regulating emerging technology and supporting innovation] by providing the framework and leadership to have those discussions.” – CAA

Nature of internal engagement and response from the organisation

The CAA’s bid for the RPF was supported by the CAA Board following a paper presented to them in summer 2018. The Board understood the need to address the problem described previously.

The Innovation Team provides regular reports to the Board, including a progress report to the Board (summer 2019) that was received positively. “They think we’ve done a great job and it’s better than they ever expected this time last year.” - CAA 

They also have a project control board comprising key people from around the CAA to help steer the project.

The Innovation Team have delivered presentations to at least 20 different teams across the CAA. They have also invited other CAA teams to present to the Innovation Team. The Team has also led an internal task force on Unified Traffic Management, bringing together specialists from across the CAA that were previously working in isolation. They also have weekly meetings with their drones team.

The Innovation Team said that although they have had positive recognition and interest from the Board, they currently could not confirm continuation of the Team beyond the RPF funding timeframe without further external funding. The Team are putting a case together to put to the Board to fund them through the ‘Scheme of Charges’ (as the CAA are not directly funded through Government, their funding comes from a Scheme of Charges levied on the aviation sector) now that they have demonstrated value. They hope that industry will accept this uplift in their charges even though some may not directly benefit. 

Engaging with the large number of teams across the CAA, with diverse and complex areas of focus, was initially a challenge for the Team. Knowing who was working on issues relevant to innovative technologies was not always clear and ensuring that everyone across CAA knew about the Innovation Team was also difficult. However, the Innovation Team felt that their activity overcame this challenge through engagement with internal news and communications as well as carrying out drop-in presentations across the business (see above). 

No regulations have been changed by the CAA yet as a result of their RPF project, but this is as expected – The CAA noted that regulatory change is slow in their sector and the project is still in progress with much work to be done.

6.4 External engagement

Externally, the CAA engaged:

• Aviation innovators
• Sandbox participants (including Amazon and Cranfield University)
• DfT
• EASA
• ICO
• Other international stakeholders

External stakeholder engagement

The Innovation Team has focused their external engagement through their dedicated website or “Gateway”. In addition to this, they have made use of social media, videos, blogs, press releases and a podcast to reach out to stakeholders. 

Their Gateway attracted 738 relevant enquiries between April and August 2019, leading to the team supporting 128 requests, primarily through signposting (i.e. responding to user queries by directing users to information or individuals that can help).

The CAA Sandbox has helped the Innovation Team work closely with six participants (including Amazon and Cranfield University) with plans to work with further cohorts of sandbox participants in the near future.

The Team has also engaged with international stakeholders, including the European Union Aviation Safety Agency (EASA), and taken the lead at the European level on urban air mobility by chairing the sub-working group on safety standards for new urban aircraft. Other engagements have included work with China, Canada and New Zealand. 

Lessons learnt on stakeholder engagement

The RPF allowed the CAA to try new ways to engage stakeholders (e.g. podcasts, videos on YouTube, etc.). The Innovation Team are unsure how effective this has been, but it has allowed them to explore these new ways to engage and collaborate. Using social media seemed, to the Team, to have worked well for communicating in a more engaging way, based on the initial feedback they have received. 

Another lesson learnt by the CAA is that it is better to be open about the questions that they don’t yet have answers to when engaging with industry. They felt that this was a different mindset for the CAA as the regulator typically felt the need to have all the answers and, if they didn’t know something, to say nothing. However, the Innovation Team recognised that this wasn’t helpful to innovators as these innovators then often made their own assumptions without regulatory guidance and proceeded along lines that may present future challenges to regulatory approvals. 

Working on the RPF and speaking with other regulators looking at innovation has also been helpful for the CAA (e.g. engaging with the FCA). They noted that learning transferrable lessons was useful as it has avoided duplication. They felt that there are many areas where technology is crossing sectors (e.g. machine learning and automation), which makes such cross-regulator engagement mutually beneficial and necessary.

Specific stakeholder feedback, Amazon

Amazon have been very satisfied with their experience working with the CAA’s Innovation Team on developing their drone delivery service. Amazon identify as a forward-thinking, innovative company and believe that the Innovation Team are equally enthusiastic.

“The efficiency in the administration side of [the regulatory process] is now staying apace with the enthusiasm and the very business-like approach that CAA is taking right now, so I’m quite pleased. […] The other thing I think is really helpful is the Innovation Hub folks are not acting as the regulator; they’re acting as the intermediary with the regulator. I don’t think a lot of people understand that fully: that yes, they have that background, but they are not the approvers. They’re the translators. They’re the ones that help us to understand what the process requirements are. So, in that sense it gives them a little bit more flexibility and agility to support us.”  - Amazon – Sandbox Participant 

Amazon felt that the choice to develop their technology in the UK was influenced by the existence of the CAA Innovation Team (who they say are well-respected, knowledgeable and forward-leaning). They felt that the CAA Team were a great fit with Amazon’s existing innovation hub in Cambridge. 

Specific stakeholder feedback, Cranfield University

Cranfield University have also had a very positive experience working with the CAA on their plans to develop the Digital Aviation Research and Technology Centre (DARTeC) and saw the value in the existence of the CAA Innovation Team: “It’s hard to understate it: they [CAA] are basically enabling the redefinition of our UK airspace at low-level and will help to establish commercial and social implementation of a whole new form of logistics, supply and transport.” - Cranfield University – Sandbox Participant

One concern that Cranfield raised was the need for clarity on the future existence of the Innovation Team beyond March 2020: “We absolutely want further engagement [from the CAA] from March 2020. That question mark and that date is getting in the way. We would really like a commitment [from the CAA] of ongoing support beyond March 2020 and we want that commitment as soon as possible as we need to build them into having an advisory role in Airspace for the Future [DARTeC project] and other projects.” - Cranfield University – Sandbox Participant

Specific stakeholder feedback, Information Commissioner’s Office

The Information Commissioner’s Office (ICO) is delivering their own RFP project, supporting other regulators to support stakeholders with concerns around data management, privacy and security. The CAA’s work is of particular interest to the ICO due to the potential for drones to carry camera equipment and the associated privacy concerns that follow from this.

The ICO have had a positive experience working with the CAA’s Innovation Team although the ICO would like to see more efforts to promote the ICO’s services to industry stakeholders who do not seem to see the value and benefits in collaborating with the ICO.

Otherwise, the ICO were able to learn and benefit from the CAA through this collaboration: the CAA shared some terms and conditions that they were using on part of their work for the ICO to review. On review, the ICO’s lawyers found that some of the wording used by the CAA in their terms and conditions was preferable to their own and have updated their ICO wording accordingly. 

6.5 Enabling innovation

Initial positive outcomes and achievements

The CAA felt that their biggest achievement was that they successfully navigated a fast turnaround to build a team of nine members of staff at the beginning of the project without significant delay.

The CAA felt that their Innovation Gateway has allowed them to engage with industry in a more consistent and structured way and generate case studies on how to support innovators. For example, a North American start-up developing a hydrogen fuel cell technology for civil aviation had an interest in bringing manufacturing to the UK from the US. The stakeholder sent an email to the Innovation Gateway, leading to an initial discussion between the team and the start-up. The CAA then convened all the relevant people within their organisation in a longer meeting with the stakeholder to support them in getting the regulatory approvals they needed to meet their ambition of bringing their technology to the UK.

The CAA has supported six participants in the sandbox with project plans in place. The CAA report that a second call for sandbox applicants has attracted a huge amount of interest for beyond-visual-line-of-sight operation of drones. The team received dozens of enquiries in the two days since they put out the call – they report that they have been “absolutely swamped.” - CAA 

The CAA are now looking at a third sandbox call on future air mobility based on their success with the previous calls. 

“I would put the CAA folks, especially the team that we’re working with, amongst the very, very top tier of regulatory groups that we’ve worked with.” - Amazon – Sandbox Participant

“[Our approach has been to] try things and fail fast and keep moving.” - CAA

Beneficiaries of the project:

The CAA will benefit through improving their ability to engage directly with industry and better meet the regulatory needs of their sector

Innovators may benefit through having a direct contact with the CAA through the hub and a clearer path to regulatory approval, allowing them to bring their products to market more easily

The UK economy may benefit through increased investment from international aviation businesses and innovators, facilitated by an innovation-friendly regulatory climate

The general public may benefit through gaining access to safe new aviation products and services enabled by better regulation

Lessons learnt from effort to enable innovation in the aviation sector

The CAA felt that a dedicated innovation team is necessary to do things like bring the Amazon project to the UK. Without such a team, the CAA would compromise their regulatory decision-making role and they wouldn’t have time to respond to the Amazon proposal without removing resource from existing applications.

The CAA learnt that lead times are important, both in terms of getting up-and-running for the team, but also in working with innovators. The CAA said that building collaborations takes time and even large, established industrial partners may struggle to move as quickly as they would like - “maturity doesn’t always match ambition.” - CAA

The CAA have found that experimenting with new ways of working and then pivoting to something else if it doesn’t work allowed their team to rapidly develop, build their profile and learn how to engage with and support stakeholders. The CAA felt that this isn’t typical of the broader regulatory sector, which they believe tends to be “set in its ways.” - CAA

Stakeholders like Amazon have found that their experience of the Innovation Team provides greater confidence that they can commit resource into the UK (both capital and operational expenditure). Without the existence of the CAA’s Innovation Team, Amazon said that the experience would have been frustrating – having to deal with multiple different individuals at the regulator with time wasted going back and forth between these people.  

Plans to disseminate learnings

At the time of interview, the CAA were about to publish a document called ‘Novel operations in specific trial areas’ – a paper that supports a particular Sandbox user, looking at how existing regulation might be applied to this case study, but with transferable learning for similar innovators.

The CAA were also about to publish guidance on urban air mobility (e.g. taxis and electric VTOL aircraft).

All learnings gained from working with participants in the Sandbox will be published (without compromising IP), sharing the CAA’s view of the challenges they are facing and how to overcome them.

The website (Gateway) will be the primary vehicle for dissemination of guidance, reports and industry communication along with MailChimp mailing lists.

The CAA are considering running workshops but need to secure future funding first before setting these up. They have presented at conferences and are on the lookout for relevant opportunities for future presentations on what they have learned from their project.

6.6 Value of the RPF

The RPF supported the CAA effort to enable innovation in their sector, providing value in a range of ways:

• Enabling a project that the CAA had no other way of funding through existing channels
• Creating a dedicated team to support innovators to prepare their products for regulation
• Learning from innovators to inform regulatory change to meet changing sector needs
• Raising the international profile of the UK aviation sector, attracting innovation investment into the UK

“What was attractive about the RPF was that it was funds that could be spent on anything to do with innovation. You didn’t have to be focused on specific deliverables. There was quite a lot of freedom to be open to what industry would come and bring to us.” - CAA

Thoughts on how the RPF could be improved and better support regulators if the programme were repeated

The CAA have found the interactions with their RPF funding officer very positive and generally had a positive experience. The following suggestions for improvement were offered:

Greater flexibility over funding phases – The CAA felt that the split of funds between the two financial years was “totally arbitrary” and that reallocating budget was a difficult process. The result of this was that they felt under pressure to spend at the beginning of the project: “We had to run at 1000mph just to get everyone up and running by April.” - CAA. They would have appreciated more flexibility and dialogue in how to spend the budget across financial years and an easier process to reallocate funding if necessary.

Longer project duration – The CAA felt that the 18-month duration of the RPF was too short for them to generate the greatest impact. As a project relying on recruiting a team, this process can take six months to complete. It is then important to have a plan in place for the team after the funding stops, at least six months in advance. Gaining approval for any ongoing support requires a demonstration of value and “it would be much easier to make that argument when you had a full year of work already under your belt.” - CAA. The CAA therefore recommend that such future funds run for a minimum of two years.

Follow-on funding opportunities – The CAA would welcome opportunities to gain further funding to extend their project. They felt that there should be a mechanism to continue to fund similar projects that demonstrate value, particularly if there are difficulties in funding through core, internal channels.

6.7 CAA project - follow-up findings (July 2020)

Update on Innovation in Aviation Engagement Capability

Update since July / August 2019

Since the previous interviews with Kantar, CAA have expanded their Innovation Team further with an innovation analyst to help gather intelligence and engage with industry. They have also expanded their networks and developed new partnerships (see right)

Significantly, CAA have now adopted the Innovation Team as a core part of their business, being funded through the scheme of charges levied on the aviation sector following the end of RPF funding. The Innovation Team presented a financial case to the CAA Board, which was then sent out for industry consultation. This received broad support and only minor queries, leading to the permanent establishment of the team in March 2020.

While CAA noted that the aviation sector has been particularly impacted by COVID-19 (as well as other issues such as the collapse of major airlines and growing public concern for the environmental impacts of flying), CAA felt that innovation in aviation was still working well. This was largely due to innovation relying less on commercial income and the ability for many research and development activities to proceed during lockdown. They also noted that the pandemic has led to a growth in demand for unmanned aviation technologies and a surge in appetite for innovation in this field.

CAA have built new partnerships with start-up accelerators and incubators to enhance their visibility to new innovators and build links at the early stages of innovation for these companies. This supports the RPF goal of enabling business growth. They have adopted a 30-minute consultation model that allows them to pick up more issues from more companies: “It makes us much more permeable than we’ve ever been before.” - CAA

CAA have partnered with UK Research and Innovation (UKRI) to support the delivery of UKRI’s COVID-19 response and Future Flight programmes. The latter is a four-year, £300m programme to support technical demonstrators of drones, urban flying passenger vehicles, etc. and through their commercial arrangement with UKRI, CAA plan to employ an additional six full-time employees. CAA noted that this partnership arose due to the recognition by UKRI that CAA’s RPF-funded Innovation Team is a critical enabler for their project: “That’s where our sandbox and our Reg Lab have come in and they’ve (UKRI) looked at those and gone ‘Wow, that’s exactly what we needed.’” - CAA

CAA have expanded their international networks through two successful missions by the Innovation Team in partnership with the UK Department for International Trade (DIT) and other government departments. The first mission, to China and Singapore, shared knowledge on future air mobility. The second, to the USA, involved meetings with innovators focused on artificial intelligence and other future aviation technology.

CAA have pivoted away from the cohort approach of their innovation sandbox towards a model that focuses on specific challenges that they have identified from their horizon-scanning work. Their first challenge to industry on regulatory barriers to beyond-visual-line-of-sight drone operation was heavily oversubscribed. This model has supported them in building networks with even more innovators.

Additional outcomes and achievements of the project

In their international outreach work, CAA have built additional linkages with UK governmental departments. Through this they have learned that when working on building international linkages, the most helpful outcome has been developing a relationship with the UK DIT representatives and other internationally-focused civil servants. This is because these individuals are regularly encountering new opportunities of relevance to CAA and, with knowledge of the Innovation Team, are passing these on to CAA. CAA feels that this expands their intelligence-gathering capability to a much larger scale. This international work contributes towards the RPF goal of projecting a pro-innovation image internationally and supports RPF Focus Output 7.

Internally, CAA has noted a significant positive impact brought about by their work on the wider CAA response to innovation: “You can tangibly feel the shift in focus in readiness.” (CAA). This has been supported by more formal engagement (e.g. collaborative policy development) requested by other CAA teams after seeing the value of the Innovation Team: “This has only really been possible as we’ve demonstrated success, that people are willing to bring you into their formal policy-making and work processes. […] What we had to do was demonstrate that we were bringing value and that we weren’t just a talking shop.” - CAA. This demonstrates CAA’s contribution towards the RPF goal of fostering a pro-innovation business culture.

CAA noted that their experience may not be reflected in other sectors as their perception is that the aviation sector is inherently pro-innovation. This has helped them gain support and interest from their external and internal stakeholders, particularly at the CAA Board “You need to have a board that is progressive and is focused on the future.” - CAA

CAA also felt that they had a deeper understanding of the problems surrounding innovation in their sector by the end of the project. They understand the regulatory barriers and the technological readiness of industry. CAA noted that there are many instances where their understanding of the problems have surpassed industry understanding. It is important to CAA, therefore, that they communicate effectively and share this knowledge with industry to ensure that the regulator is seen as an enabler and not a potential scapegoat for lack of progress in innovation.

Through CAA’s sandbox (Focus Output 1), lab and gateway, CAA have now supported over 300 different entities and are continuing to enhance their content to best respond to the needs of their stakeholders. They feel that this has built stakeholder confidence in the regulator (Focus Output 4). Furthermore, the combination of horizon-scanning, working with start-ups and close interaction with industry has supported CAA in closing the gap between innovators and regulators supporting a more responsive regulatory approach. This has the potential to reduce time/cost of innovation (Focus Output 3) and ultimately support the RPF goals of enabling business growth and boosting value for consumers.

“We understand better where the regulatory barriers are and what you need to do to overcome them in the sense of how to build an evidence base so that our regulatory teams can understand them as problems.” - CAA

Additional lessons learnt and value of the RPF

CAA have concluded that the achievements of the Innovation Team would not have been possible with a smaller team or a team not wholly dedicated to innovation. A smaller team, or a non-dedicated team wouldn’t have been able to build the relationships they have whilst simultaneously gathering intelligence, developing effective communication and supporting existing innovators “we would have just nibbled around the edges.” (CAA) The RPF was a critical enabler of scaling, as despite general support from the CAA Board, they would not have been able to commit to a team of nine people (particularly given the scheme-of-charges funding structure) without the demonstration of value that the RPF project has facilitated.

CAA felt that an important part of communicating with innovators on innovation is to openly acknowledge uncertainty and communicate a willingness to find answers. “Silence is the killer […]. People are very happy or very comfortable that we might say that we don’t know the answer yet, but we’ll tell you what our thinking is and the process we’re going to go through to try and come up with a better answer for you. And that gives people a lot of confidence.” - CAA

CAA felt that the role of the regulator in supporting innovation is to let industry lead but follow closely behind. They note that regulators don’t own the technology or the risk and are not best placed to create the solutions, so they shouldn’t dictate the direction of innovation. They felt that regulators can best support innovation by maintaining close links with industry regulators and cutting down the lag between innovation and regulatory response. CAA felt that it is important that as innovators develop their solution they know that “when they do get there, we won’t be that far behind.” - CAA

Additional feedback on the RPF

CAA felt very positive about their experience with the RPF. “I can’t speak highly enough of the BEIS team, both in terms of the behind-the-scenes team that runs the scheme and also the monitoring officer.” - CAA They felt that the RPF administrators balanced a need for discipline and significant reporting requirements (which CAA perceive as a necessary feature of a publicly funded project) with support that was pragmatic and understanding.

CAA stated that the RPF is an excellent use of public money and the return on investment has been very high. CAA have worked with hundreds of companies and achieved major benefits for them as a regulator as well as built a world-leading platform to enable and attract innovation in aviation in the UK. While they felt that it is difficult to put a financial value on this benefit, they felt that it is clear to see that this has made life better for innovators.

6.8 Future plans

As a core part of CAA now, the Innovation Team are continuing to build their offering and enhance their value to the sector to support innovation. Through their partnership with UKRI they are expected to take on new staff and play an important role in the UK’s future research and development landscape.

Through their revised sandbox model, designing challenges based on their horizon-scanning work, they intend to address the most pressing challenges to regulating innovation in the aviation sector to ensure a rapid regulatory response to new developments.

They will continue to apply an agile approach to their activity, pivoting and amending what they do to learn and deliver the best impact for stakeholders as things continue to change. “The methodologies that we use pivot constantly. It is very iterative, we have to look at the nature of the problem and the maturity of the technology solutions.” - CAA