Research and analysis

Evaluation of the Cyber AI Hub programme

Published 8 January 2025

1. Introduction  

In November 2023, The Department for Science, Innovation and Technology’s (DSIT’s) Cyber Security Programme Delivery Team commissioned Steer Economic Development (Steer-ED) to conduct a comprehensive evaluation of the Cyber-AI Hub Programme (the Programme), which is being delivered in Northern Ireland by Queen’s University Belfast (QUB) between 2023 and 2027. 

The evaluation is determining how the Programme and its four components– the Cyber-AI Technologies Hub, the Centre for Secure Information Technologies (CSIT) Doctoral Training Programme, NICYBER2025 Masters Bursaries, and The Cyber AI Snapshot – are performing over the period of programme delivery. The study comprises three reinforcing strands: 

• an ongoing review of programme delivery to provide learning to help shape and develop future iterations of the intervention; this first process evaluation is part of this strand 

• the development and implementation of a data collection plan, to monitor outputs/outcomes delivered by the programme (including post-evaluation) to understand the ongoing benefits of the Programme for beneficiaries and participating companies 

• an assessment of the extent to which the Programme has achieved its objectives; the impact of the Programme, and any unintended outcomes realised; and the Value for Money achieved. 

1.1 Overview of the Cyber AI Hub Programme 

1.1.1 Background and context 

Prior to the COVID-19 pandemic, Northern Ireland’s (NI) Gross Value Added (GVA) per head lagged behind the UK average by around a quarter. The Total Early-stage Entrepreneurial Activity (TEA) rate in NI was 6.5% (well below the UK rate of 7.9%), and the number of firms that were ‘innovation active’ in NI was decreasing, from 45% in 2014 to 32% in 2018 (BEIS, UK Innovation Survey, 2019). 

However, the cyber security sector is seen as having a key contribution to improving Northern Ireland’s innovation levels and productivity. The 2024 NI Cyber Security Snapshot  reveals significant growth in Northern Ireland’s cyber security sector. The industry now supports over 2,750 jobs across about 130 firms, with an average salary of £54,000, contributing more than £237 million in direct Gross Value Added (GVA) to the Northern Ireland economy. Notable points include: 

• Belfast remains one of the world’s most concentrated cyber security hubs, with over 100 cyber security businesses and teams located within a 3-mile radius of the city centre  

• Northern Ireland is the leading global destination for US FDI, and 66% of firms are headquartered overseas. Almost 1,800 cyber security jobs in NI are backed by US FDI 

• 33% of cyber security firms are locally-founded.

Since the 2023 Snapshot , NI-headquartered cyber security firms have seen a 67% increase in Venture Capital (VC) investment, raising £72 million. This is driven by substantial investment in companies such as Cloudsmith and B-Secur. 

The previous NI Executive’s 10X Economic Strategy  (published in 2021) recognised NI’s strengths in the cyber security sector. It outlined that NI has the potential to become a major cyber hub but being able to scale-up and create a critical mass of activity is key. In line with this, there is a need to position NI as the internationally recognised go-to place for innovation in the Cyber-AI sector, by developing world-leading capability.  

To do this, NI needs to stimulate this high-value and growing sector by investing in commercialisation of R&D and in developing a pipeline of professional cyber skills, as outlined in Steer-ED’s ‘Understanding Northern Ireland’s Software Capability and Future ‘Windows of Opportunity’’ report . In addition, by focusing on promoting the growth of R&D intensive companies there is potential to reverse the trend of decreasing innovation across NI. 

The cyber security sector has enormous global growth potential  and, reflecting its existing strengths, NI has the potential to claim some of that growth, which could be transformational in terms of skilled jobs and investment. This is recognised in a report published by the UK and Irish Government (‘New Decade, New Approach’ ) which includes a target to deliver 5,000 Cyber jobs in NI by 2030; equating to a net increase of approximately 300 jobs per annum. To achieve this, CSIT’s Northern Ireland Cyber Security Snapshot  identifies four key barriers that need to be addressed: 

• ensuring a sufficient pipeline of talent: Support in the development of a sustainable pipeline of talent, including the education and entry of talented high value cyber security professionals; the DCMS Cyber Recruitment Pool research  recommends that “alternative and innovative routes to a cyber security career need to be clear, funded and accelerated” 

• maintaining and enriching partnerships and collaboration: Increasing the strength of relationships between academia and the private sector 

• knowledge sharing: Promotion of knowledge sharing in Northern Ireland’s cyber security ecosystem, including embedding academic staff in industry 

• promoting Northern Ireland as a leading centre for commercial R&D in cyber security 

1.1.2 Programme overview and objectives 

The Programme is being delivered by QUB over three and a half years, from 2023 to 2027. ‘New Deal for NI’ funding is contributing £10.4 million, and a further £4.5 million is being provided by commercial partners involved with the Programme . 

The overall goal of the Programme is to grow the NI cyber security sector and increase significantly the talent pool of Cyber-AI professionals in Northern Ireland by meeting the following high-level objectives: 

Figure 1: High level objectives for the Cyber AI Hub Programme, 2023-2027 

 

Source: Steer-ED, 2024, drawing on Cyber AI Hub Business Case  

To achieve these objectives, the Programme is delivering four interrelated components (see further detail below): 

Figure 2: Overview of The Cyber AI Hub Programme and its component parts 

 

Source: Steer-ED 2024, drawing on CSIT programme documentation 

With regard to Programme management and delivery, a Memorandum of Understanding (MoU) between DSIT, the Northern Ireland Office (NIO), and Innovate UK (IUK) was prepared to set out respective roles, expectations and, joint governance and oversight arrangements. CSIT is responsible for the day-to-day management of the Programme and provides DSIT and NIO with monitoring reports each quarter. There is a quarterly governance meeting to track progress against deliverables and milestones. This is attended by DSIT, IUK and NIO. As part of its standard management processes, Innovate UK produces quarterly monitoring reports which sets out overall progress and scores the programme against key measures, including: scope, time, cost, exploitation, risk management and project planning. Note that this does not include an individual assessment of the four components. 

Each R&D project with industry under the Cyber-AI Hub component, has two ‘co-investigators’ who are academics from QUB; they provide supervision of the research component, alongside technical advice and input to the businesses engaging with the Programme and the Cyber-AI Hub engineering team. The Cyber-AI Hub Manager acts as the orchestrator across the industrial projects – currently eight in number - and is supported by a ‘Scrum Master Mentor’. Each R&D project also has a dedicated Lead Engineer, post-doctoral Research Fellow, and Reviewer. The PhD and Masters components of the Programme operate in the same way as QUB’s other courses.  

1.1.3 Programme components 

Cyber-AI Technologies Hub (the ‘Technologies Hub’)

The Technologies Hub is supporting eight collaborative projects between academia (QUB) and NI-based cyber industry businesses, with a particular focus on the use of AI in cyber security. The seven agreed projects and their aims are set out in Table 1; note that one further project is currently in development. Further information about each of the projects is at Appendix A. The Technologies Hub’s objectives include creating jobs, improving competitiveness for beneficiaries, and improving the long-term capacity of the NI cyber security ecosystem. 

Table 1: Overview of projects currently funded through the Cyber-AI Technologies Hub 

Firm	Project	Aim
Ampliphae	Application of AI to cyber-security of 5G/6G mobile telecoms networks	Researching the cyber security of the application of AI technology to Open RAN 5G (O-RAN) mobile telecoms networks. Intention is to develop techniques and/or applications that improve network security, addressing AI-related cyber security capabilities, vulnerabilities and potential mitigations in O-RAN.
Angoka	Device ID generation and entropy analysis using cryptographic techniques	Identification of optimal combination of device characteristics to achieve a unique, immutable fingerprint.
Industry partner to be confirmed	ICS malware	Using AI/ML to develop techniques for the detection of industrial control system (ICS) malware.
Nvidia	Threat prediction and prevention	Reduce computational costs (not using a whole data centre for AI just to monitor one server, next generation of DPU will likely be at 400Gb/s); and to address the false/miss alerts problem for firewall/IDS using traditional ML-based approach (false negatives and false positives).
Pytilia	Context Aware Vulnerability Detection and Mitigation	Creating an automated context aware vulnerability detection and mitigation solution. This solution should provide priority-based alerts on vulnerabilities considering the context of an organization and their tech stack. To develop a means of delivering the AI component of this system.
Rapid 7	AI for Cloud Security	Detection of sensitive file content in publicly accessible buckets.
Thales	Project “STEADY (Secure & Trustworthy Ensemble AI systems for real-world DeploYment)”	Providing tools and techniques to enable verification of machine learning (ML) models’ robustness. The overarching goal is to ensure the trustworthiness of ML models by addressing vulnerabilities and uncertainties associated with their deployment in real-world scenarios.

Source: CSIT, 2024 

CSIT Doctoral Training Programme (DTP) in cyber security 

The DTP is supporting a cohort of 15 industry-facing PhD students, including through an enhanced stipend of £27.5k per annum. The CSIT DTP has four strategic objectives, namely to: 

• produce a cohort of industry-conscious thinkers and leaders who will influence the roadmaps of cyber security technologies and their applications 

• produce world leading scholarly output and impact through supervision from the academic team at CSIT  

• address challenges that are strongly relevant to industry and society through partnerships and close interaction between students, industry and government stakeholders 

• conduct responsible research and effective innovation in the field by encouraging ethical and society-focused exploration of emerging cyber security technologies and challenges 

It is intended that there will be collaboration and knowledge sharing between the PhD students and the Cyber-AI Technologies Hub, some of whom may work on the Technologies Hub projects. 

The NICYBER2025 Masters Bursaries

This third component of the Programme is providing a bursary to 40 students to undertake QUB master’s degrees in Applied Cyber Security or Artificial Intelligence. The objective of the Master’s bursaries is to help ensure a sustainable pipeline of talent to support continued sector growth, including education of talented high value cyber security professionals, as well as opportunities for career retraining and apprenticeships for those employed in sectors with similar skill sets. 

The Northern Ireland Cyber Security Snapshot

The final component of the Cyber AI Hub Programme involves CSIT producing an annual NI Cyber Security Snapshot (the ‘Snapshot’). The Snapshot maps the growth and development of the NI’s cyber security industry through to 2025. This information will be used to track progress in the sector over the programme’s delivery period and will also provide firm contextual evidence for the evaluation. 

1.2 Overview of the programme’s evaluation 

The Programme’s evaluation is being be delivered over five key stages (as set out in Figure 3 and aims to assess (i) the extent to which the Programme has achieved its objectives; (ii) the impact of the Programme for beneficiaries, stakeholders and the wider sector in NI; and (iii) any unintended outcomes arising. 

The process evaluations, which will take place annually from 2024 to 2026, aim to provide learning in real time to help shape and develop the Programme as it progresses. These studies will provide insights at key stages to ensure the Programme is on track to deliver against its anticipated outcomes and impacts. 

The overarching evaluation of the Programme will help DSIT understand the impact of the Cyber-AI Hub programme as a whole, how Government intervention can help grow the cyber security ecosystem, and any policy considerations around this. The work will provide DSIT with options and a robust, independent evaluation of existing mechanisms in place; to ensure that best practice is identified, boosting cyber security growth and innovation in a sustainable, effective way and avoiding duplicate interventions from other government departments and across industry. 

Figure 3: Overview of Cyber AI Hub Programme Evaluation 

 

Source: Steer-ED, 2024 

1.2.1 Overview of the process evaluation 

This report forms the first of three process evaluations and aims to: 

• generate learning to inform the ongoing delivery of the Programme, and any future, similar initiatives 

• assess the extent to which the Programme is being delivered as intended, and to identify key challenges and enablers to effective delivery 

• assess the suitability of the three programme strands at this early stage in delivering the anticipated programme outcomes 

An overview of the process evaluation, including the current stage, is set out below.   

Figure 4: Cyber-AI Hub Process Evaluation Cycle (with current stage highlighted in red) 

 

Source: Steer-ED, 2024 

Three evaluation reports will be produced which will present the process evaluation findings and key recommendations for the programme. 

1.2.2 Methodology 

At this initial stage, the process evaluation seeks to respond to six key questions, as outlined in the Cyber AI Hub Evaluation Plan: 

Table 2: Process Evaluation Question for Cyber-AI Hub 

#	Process Evaluation Questions
P1	Is the Programme being delivered as intended?
P2	What is working well, or less well, for whom and why?
P5	Have there been any unexpected or unintended issues in the delivery of the Programme?
P6	How could delivery of the Programme be improved?
P7	How have external factors influenced the delivery and functioning of the Programme?
P8	Is the Programme design and activity set appropriate in delivering the anticipated outcomes?

Source: Cyber AI Hub Evaluation Plan, Steer-ED, 2024 

To answer these questions, three strands of research were undertaken from May to August 2024: 

• In-depth analysis of Programme data and documents: detailed analysis of internal programme data, including IUK Quarterly Monitoring Reports, the notes from the Quarterly Programme meeting, programme financials, and other relevant documents, such as project overviews. This aims to build an understanding of overall delivery progress. 

• Interviews with Cyber-AI Hub Programme delivery teams: six semi-structured interviews with representatives from the Programme Delivery Team, including at QUB, IUK, and DSIT. The interviews covered themes such as Programme design, delivery progress, and management and governance. Where appropriate, the questions were tailored to the four programme components. 

• Interviews with Cyber-AI Technologies Hub beneficiaries: three semi-structured interviews were undertaken with businesses delivering the collaborative R&D projects. The interviews sought to gather perspectives on programme delivery, including project set-up, collaboration arrangements with academics, engagement with CSIT and wider stakeholders, and management and governance processes. 

The evidence generated from the research was analysed and triangulated to inform the findings and conclusions of this report. An internal workshop was also held by the Evaluation Team to support the synthesis of findings across the three research strands. 

Methodological Considerations 

Due to delays in programme delivery (outlined in Section 2), fewer interviews were undertaken than originally anticipated at this stage, in particular, with the businesses participating in the Technologies Hub. As such, while perspectives are presented from the business beneficiaries, this may not be representative of the community of seven funded projects. 

In addition, Steer-ED experienced issues in accessing relevant programme information to inform this report (e.g., financial data) which delayed the overall timing of the report and created difficulties in gaining a detailed understanding of programme delivery. In particular, having the opportunity to discuss and query programme information with the appropriate person. Further information regarding this is set out in Section 3 below. 

1.3 Purpose of this report and structure 

This report is the second of five outputs from the evaluation and forms the first Evaluation Monitoring Report. The remainder of this report is structured as follows: 

• Section 2: Overview of Programme progress 

• Section 3: Process reflections 

• Section 4: Summary and recommendations 

Subsequent process evaluations, scheduled for 2025 and 2026, will add to this first report by focusing on: the extent to which the Programme is being delivered as anticipated (reflecting the initial delays), the effectiveness of the Programme design in realising emerging outcomes, and any key barriers or enablers in achieving outcomes. 

2. Overview of programme progress  

2.1 Programme progress 

An overview of progress across the Programme components is set out in Table 3. In summary, two of the four components - namely, the CSIT DTP in Cyber Security and the NICYBER 2025 Masters Bursaries - are progressing well and as anticipated. However, there have been delays to the second Cyber AI Snapshot, which was published six months later than expected in September 2024; and significant delays have been experienced to the delivery of the Cyber-AI Technologies component, due to contractual issues and issues with staff retention (see below). The Technologies Hub is approximately seven to eight months behind schedule. Further information regarding each of the components is set out below. 

Table 3: Overview of Cyber AI Hub Programme’s progress 

Programme component	Output / KPI	Target	Actual	Notes of progress	Overall status
Cyber-AI Technologies Hub	Number of projects contracted	8	5	Expected in Q3 2023/24; contracts for three projects were signed in Q1 2024/25 and one project contract was signed in Q2 2024/25. The project with partner Nvidia is ongoing though it does not require a funding agreement as Nvidia is not accepting funding through the Hub. Two contracts are not yet signed but are in progress. One firm went into administration and alternative options are being explored for this project	Delayed (seven to eight months)
CSIT DTP in Cyber Security	PhDs recruited/ completed	15	15 candidates enrolled: 13 students have passed their Differentiation.	2 (January starts) passed initial review	On track
NICYBER 2025 Masters Bursaries	Masters recruited/ completed	40 (20 in cohort 1 and 20 in cohort 2)	20	20 students enrolled in cohort 1: 11 have graduated; 5 part-time students are progressing into the second year; 1 PG Certificate and 1 PG Diploma have completed; and 2 students have withdrawn. Recruitment for cohort 2 took place in September 2024: 14 full-time and 1 part-time students were enrolled. Six bursaries are remaining and will be awarded to commence in autumn 2025.	On track
NI Cyber Security Snapshot	Annual NI Cyber Security Snapshot	3	1	2023 Snapshot published 2024 Snapshot due in Q4 2023/24, published in Q2 2024/25	Delayed (six months)

Source: Cyber AI Hub Quarterly Meeting Slides, July 2024 

Due to the aforementioned delays, actual Programme expenditure is also behind forecast at this stage (-17%), see Table 4. This is due primarily to staff retention issues and because the Technologies Hub projects have not been able to claim for costs incurred. A breakdown of expenditure by Programme component was not available to the Evaluation Team at the time of writing; therefore, it is unclear whether expenditure is on track for certain aspects of the Programme. 

Table 4: Overview of Cyber AI Hub Programme’s expenditure 

Programme Component	Forecast spend to date	Actual spend to date	Variance (actual vs forecast)	Variance % (actual vs forecast)
Cyber AI Hub Programme	£3,453,928	£2,490, 723	-£963,205	-28%

Source: QUB 

At the time of reporting, DSIT was working with HM Treasury to agree a revised profile of spend for the Programme, due to the initial delays in commencing the initiative. This will be reflected in future reporting. 

2.2 The Cyber-AI Technologies Hub  

As outlined above, there have been notable delays to the Cyber-AI Technologies Hub, primarily relating to Subsidy Control issues. Three considerations are important here: 

• at the time of the Programme’s application, there was an initial lack of clarity regarding beneficiary firms’ existing de minimis levels.  As part of its review of the project finances of proposed industry partners, QUB identified that in a number of cases firms might be in excess of Minimum Financial Assistance and/or de minimis limits, meaning that their intended projects might be unable to proceed at all, or only proceed at reduced scale, speed and/or quality to that initially intended. Understandably, this became an issue of significant concern for the industry partners during the Programme’s application phase, and work was done by CSIT to ensure that firms remained engaged at this stage 

• in the subsequent grant offer letter, the subsidy mechanism was formalised around MFA and/or De Minimis. In looking pragmatically at possible alternatives, QUB sought legal opinion regarding the use of a Trade and Co-operation Agreement model. IUK offered two other viable options, namely the use of (i) the Research, Development and Innovation Streamlined Route and (ii) General Block Exemption Regulations 2014 (as amended), neither of which were subsequently taken up 

• as a direct consequence of the application of Article 10 of the Windsor Framework in June 2023, (which replaced Article 10 of the former Northern Ireland Protocol), there proved to be limited legal expertise (and no case history) in Northern Ireland, and legal advice on the best subsidy mechanism had to be sought further afield. After significant back-and-forth between legal teams, it was agreed that the Programme should adhere to the original subsidy rules – i.e. the MFA and de minimis mechanism set out in the original letter of grant offer 

Subsequently, changes were made to the UK Subsidy Control Regime, which took effect at the start of Programme delivery (December 2023) . 

This issue has had significant implications for QUB and the delivery of the Technologies Hub and its projects, as follows: 

• it was necessary for QUB to consult with expert legal counsel on the matter, which incurred significant time and cost 

• this delayed effective project scoping on various projects and projects had to review their original scope and activities with a view to reducing this to align with the smaller subsidy controls, however, for the majority of projects, this did not ultimately change 

• it was not possible for QUB to contract with the Technologies Hub projects until the subsidy position was agreed with IUK, however, some projects started ‘at risk’ 

• due to delays in contracting, the timescales for project delivery are now condensed and it may be challenging for projects to deliver their original scope within a shorter timeframe 

• in addition, the delays impacted on QUB’s reputation with industry partners, which has required significant efforts from QUB to minimise 

As a result of the delays, four out of seven project collaboration agreements have been signed with companies, between seven and eight months later than scheduled (note, that an agreement is not required for Nvidia as they will not receive Technologies Hub funding); two agreements are in the process of being signed; and for the final project (Skurio project), the company went into administration in March 2024 (due to reasons unrelated to the Technologies Hub). In response, the QUB project team have continued elements of the Skurio project with Rapid7 on a short-term basis (six weeks). Upon completion, Rapid7 will decide whether to engage in a more substantial project. 

While many of the projects have made progress, they are behind schedule. Detailed monitoring information in regard to progress against anticipated project delivery milestones has not been available for this evaluation. Therefore, it is not possible to comment on the extent to which projects are behind schedule at this stage. 

Note that the reach of the consultations in the initial process evaluation was limited due to challenges regarding engagement (see Methodological Considerations above). A broader range of consultations are proposed as part of the next process evaluation. These consultations will focus on key issues, including the Technologies Hub preparation and project management/ monitoring.  

2.3 The CSIT Doctoral Training Programme (DTP) 

The PhD students were fully recruited in 2023/24 as expected. The titles for each of the PhD projects are as follows: 

Table 6: Overview of funded-PhD projects 

No.	Project title
1	Cyber DALL-E for adversarial dataset generation
2	Efficient RISC-V Chips with future-proof security
3	DPU-Centric Distributed Network and Datacentre Security
4	Integrated quantum-resilient cryptography
5	Side-channel Analysis Supports in Physical Unclonable Function Modelling
6	Attacking is the best form of defence, also in machine learning
7	Cyber security for digital twins
8	Highly distributed learning for privacy preserving analytics
9	Security of 6G O-RAN
10	Side-channel Analysis and Countermeasure on Post Quantum Cryptographic
11	Using reinforcement learning to prepare cyber-attack responses for cyber physical systems
12	Malware Vulnerabilities as a Heuristic for Layering Defence Mechanism in Detection Technology
13	Deep-Learning Based Hardware Trojan Detection
14	Quantum machine learning for securing 6G networks
15	A Policy-Based Security Approach using Programmable Network for Delay Tolerant Networking

Source: QUB, 2024 

2.3.1 Overview of PhD candidates 

Data were provided by QUB on the number of CSIT PhD starts and graduations per annum, from 2019/20 to 2023/24 (see Figure 5). This shows that, prior to the enhanced stipend offered through the Programme, the number of starts remained relatively low (<6), with a slight increase each year from 2020/21 to 2023/24. In 2023/24, the number of starts increased substantially, due to the 15 programme PhDs (although, excluding these PhDs, the number showed a slight increase to six). 

Figure 5: PhD starts and graduations, 2019/20 to 2023/24 

 

Source: QUB, May 2024 – NB, 2024 graduations not yet listed 

Analysis of PhD students’ demographic data shows that all candidates are male, and primarily of British nationality (53%); 20% are Vietnamese nationals, and the remainder are Turkish, Greek, Pakistan or Bangladesh nationals. However, the domicile of most the students (prior to the course start date) was the UK (67%).  

2.4 The NICYBER2025 Masters Bursaries 

To date, 20 students have received the NICYBER2025 Masters Bursary (Cohort 1): 10 to undertake the MSc in Applied Cyber Security, and 10 to undertake the MSc in Artificial Intelligence. The students are selected by the Programme Team who rank all the applicants and select the top ten who are then offered the bursary. Recruitment for cohort 2 took place in September 2024: 14 full-time and 1 part-time students were enrolled. Six bursaries are remaining and will be awarded to commence in autumn 2025. 

2.4.1 Overview of master’s candidates 

Data was provided by QUB in relation to the characteristics of bursary students. Analysis of the data shows: 

MSc Applied Cyber Security:  

• all ten are Northern Irish nationals; 
• 50% are completing the course full-time, and 50% are part-time; 
• the gender breakdown shows that 80% of students are male and 20% are female.  

MSc Artificial Intelligence:  

• nine are Northern Irish nationals;  
• 50% are completing the course full-time, and 50% are part-time;  
• the gender breakdown shows 80% of candidates are male and 20% are female. 

Overall, the demographic data shows an equal distribution of full-time and part-time bursary candidates, compared to the total MSc candidate population, which has a significantly higher proportion of full-time students (89%). However, the gender split amongst bursary recipients was more in line with that of the total MSc candidate population in which, on average, 75% of students are male and 25% are female. 

In terms of interest in the MSc Applied Cyber Security course, there has been a general increase in applicants over the last three years: 

• 450 applicants in 2021/22 

• 623 applicants in 2022/23 

• 659 applicants in 2023/24 

As such, it is unclear at this stage whether demand has changed as a result of the master’s bursary. The MSc Artificial Intelligence course was introduced in 2023/24 and therefore, information on historic applicant demand is not available. 

2.5 NI Cyber Security Snapshot 

The inaugural Cyber Security Snapshot was published in May 2021 , and set out a thorough baseline, ambitions, and challenges for the Northern Ireland Cyber Security ecosystem. The Cyber AI Programme’s Strategic Outline Case presented a need to continue this research on an ongoing basis, including a full list of companies in the sector and their performance data, economic statistics on Cyber Security, and a labour market overview. 

Since the start of the Programme, two NI Cyber Security Snapshots (‘Snapshot’) have been published; the first in April 2023 and the second in September 2024, six months later than expected. One consultee explained the reason for the delay was twofold: 

• There was limited change in the sector from the previous iteration of the Snapshot and, as a result, the Snapshot provider (Perspective Economics Ltd) suggested to incorporate additional elements to add greater value; this was agreed by QUB, but required additional time for delivery.   

• QUB was working with Perspective Economics Ltd on another project and, reflecting the limited capacity of the small firm, agreed the other project should be prioritised ahead of the Snapshot; one consultee described that this was beneficial as it allowed the provider to include the data published by the Higher Education Statistics Agency (HESA), which was released in September. 

The 2024 Snapshot is currently high-level and has been presented in a ‘short version’ which does not contain the same level of detail set out in the inaugural Snapshot. In particular, information related to skills supply and recruitment flows (a key focus of the Programme) is currently missing, and sectoral trends over time are also not captured. However, a detailed version of the 2024 Snapshot has been developed and will be published imminently. This will be reviewed as part of the next process evaluation. 

3. Process reflections 

This section presents the findings from the qualitative research against the process evaluation questions covered at this stage. 

3.1 Findings against key research questions 

3.1.1 P8: Is the Programme’s design and activity set appropriate in delivering the anticipated outcomes?  

Feedback from consultees suggests that the rationale for the Cyber AI Hub programme remains valid, with a continued focus on supporting the growth of the Cyber and AI sectors in Northern Ireland. The Programme has two main objectives: first, to increase the talent pool and number of experts in Northern Ireland; and second, to grow the cyber cluster in Northern Ireland while fostering collaboration between academia and businesses, and between businesses. The Programme builds on existing strengths and relationships in the sector (formed partly through a previous Strength in Places bid) and seeks to address market failures, particularly in terms of skills supply constraints. The Department for Science, Innovation and Technology (DSIT) noted that, despite the absence of a Northern Ireland Executive when the Programme was established, the current Executive and the Department for the Economy regard Cyber and AI as crucial growth sectors, with the Programme being key to this development. 

---

If anything, [the Programme] has become more relevant. When the bid was being put together, it [Cyber AI] was in a new and emerging area – but now, it’s a really core thing that everyone is talking about. 

- Programme Delivery Team (QUB) 

---

Beneficiaries of the Cyber AI Technologies Hub noted the continued rationale for the programme as follows: 

• the UK Government and industry increasingly need to understand how AI can be used to improve cyber-security 

• to drive long-term growth in talent, NI needs to bring together and further invest in skills pipelines 

• there is added value for QUB (QUB) by connecting researchers to industry to produce higher quality research outcomes 

Programme delivery consultees also noted this was to improve productivity through increased R&D spend in businesses, and to deliver impact through high quality research outcomes, including four-star Research Excellence Framework (REF)  impact cases. 

Since launch, there has been no changes to the programme’s overall design, although the scope of the Technologies Hub’s R&D projects has been altered, in some cases, due to the aforementioned delivery challenges.  

Beneficiaries and wider stakeholders highlighted four key successes in relation to the overall programme design: 

• PhD recruitment: One consultee noted that a recruitment campaign delivered by QUB had been particularly effective in securing interest and filling the places. 

• flexibility of the Technologies Hub: One beneficiary welcomed the ability to ‘leave their project fairly open-ended in terms of goals of [the exploratory, academic] research’. They attributed this approach to creating a technology test-bed environment that is adaptable to change, with no strict emphasis on specific outcomes or rigid timelines. 

• collaboration between industry and academia: Particularly the role of early academic research in shaping industry project activities. 

• engaging with other firms and international firms: Leading to early signs of expanding networks and market opportunities. 

There were no specific improvements to the programme design suggested by consultees. 

Alignment between programme components 

There is clear alignment between the four programme components and the primary objectives of the programme, highlighted by consultees as follows: 

• collaboration between the Technologies Hub and PhD programme: Several hub projects have been identified as potentially relevant to PhD projects, and PhD students are being encouraged to collaborate with projects. PhD students have also been invited to networking events with funded projects to present a brief overview of their doctoral research 

• collaboration between the Technologies Hub and MSc programme: There is an opportunity for MSc students to be offered both placements and projects by supported firms, allowing them to gain market experience alongside their academic learning 

• co-location of project beneficiaries: A workspace has been created within QUB for Technologies Hub businesses and students to use, with a view to fostering networking, collaboration and knowledge transfer 

---

The idea here is to try and create a category of about 100 highly specialised domain experts applying AI to cyber security, who will ultimately go out into the Northern Ireland cyber cluster and companies and build teams around them. 

- Programme Delivery Team (DSIT) 

We now want to bring [the firms] into the building… we want to change their status, not just visitors, but as tenants while in the building. We see the Hub as an innovation space… the co-location thing is key, it’s the basis for innovation, if firms are not co-locating it will be hard to achieve the objectives. 

- Programme Delivery Team (QUB) 

---

3.1.2 P1: Is the Programme being delivered as intended? 

Two of the three academic components are progressing as planned in terms of scope, timing, and budget – the NICYBER2025 Masters Bursaries and the Doctoral Training Programme in Cyber Security. By contrast, the delivery of the Technologies Hub, a new endeavour for QUB, has proven more challenging. One consultee noted, ‘the Hub is the most complex component of the Programme but there has been no divergence so far’. Eight companies have been identified and approved by IUK; however, the signing of the contracts remains the largest hurdle for programme delivery, which has caused a seven-to-eight-month delay. This has been acknowledged by QUB consultees as a key challenge in implementing a new and innovative model and learning has been taken from this to inform future, similar initiatives. 

In relation to the Master’s and PhD component, consultees explained that QUB’s extensive experience in managing academic programmes has contributed to the successful rollout of these two elements. This said, it was noted that the competitive cyber labour market had made PhD recruitment challenging, particularly as there was high level of demand from international candidates but only UK students were eligible for programme support. 

Finally, the publication of the second NI Cyber Security Snapshot was delayed by six months, due to a complication in contracting arrangements with Perspective Economics. 

Programme Management and Governance 

Consultees reported that robust management forums and procedures are in place at both the programme and component levels. Consultees said that the processes in place are efficient in creating open lines of communication and a platform to escalate issues. The forums are detailed in Table 7. 

Table 7: Programme and component management forums 

Programme level	Component level
CSIT industry and stakeholder advisory board: This board includes industry partners, academics, and government representatives who provide strategic advice and resolve issues at a higher governance level.	QUB governance process: The PhD programme works through the standard University governance process of progression checks in terms of gateways, which is a well-established structure.
Two quarterly review meetings of the programme: This involves: i) an internal review of the companies’ progress, followed by ii) a review by QUB, IUK and DSIT.	Frequent review and sprint planning meetings: Engineers and Postdoc research fellows meet frequently with product owners to ensure progress and address any challenges. Researchers and engineers use a SCRUM methodology, meaning they build up their own workload and outputs come every two-to-three weeks.
Weekly CSIT operations team meeting: This covers budget needs, project change requests, and other issues, providing opportunities to escalate concerns.	Weekly Project catch up: Each project holds a weekly catch-up to discuss progress.
	Every day team huddle: Eight lead engineers meet daily to review project progress and challenges, for example training gaps.

Source: Steer-ED, based on analysis of consultation data, 2024 

Consultees reported five key successes in Programme management and governance: 

• collaborative approach: a range of meeting formats and attendees encourage collaboration and networking 

• triangular relationships: the triangular relationship between the engineers, academics and businesses is positive and is gaining momentum 

• varied management forums: providing a standard process but tailored and flexible support to each project, remaining sensitive to the needs of each business 

• effective decision-making: decisions are made at the strategic and operational level 

• clear and timely communication: between QUB staff and beneficiaries, facilitated by weekly reviews 

Despite positive feedback overall, four key challenges were highlighted by consultees and the wider fieldwork: 

• lack of clarity: in the early stage of Programme delivery, there was some confusion in terms of the management of the Programme components; in particular, whether IUK/QUB should manage subcontractors, which led to delays 

• issues with staff retention: five QUB staff members have moved to positions elsewhere, including at companies funded through the Technologies Hub; while this is a sign of progress, it also presents challenges for Programme delivery and continuity 

• unclear progress monitoring: currently, a structured, standardised framework for monitoring the progress of the eight funded Technologies Hub projects is not in place. While updates are provided on a quarterly basis, the extent to which projects are delivering against milestones and budget (as expected) is unclear 

• financial reporting: there have been difficulties experienced in accessing quarterly financial data for the respective Programme components, including forecast and actuals; this has raised some concerns among consultees about the transparency of the financial management of the Programme 

In addition, while the IUK Quarterly Monitoring Reports provide a useful update on progress, it is unclear how these reports are being used to inform Programme management. In particular, the scores and comments in regard to ‘time’ and ‘cost’ in the IUK’s reporting frame appear to be out of sync with the findings of this evaluation report. For example, the Period 4 report states that the ‘The project is meeting the planned timetable’, and that, while financial data was not available, ‘it is anticipated that costs will be in line with the forecast’. 

3.1.3 P2: What is working well, or less well, for whom and why? 

Various aspects of the Programme were highlighted by consultees as working well, including: 

• positive engagement: There has been strong interest from cyber security businesses and active participation from stakeholders, including ministers 

• recruitment of PhD and master’s students: this has exceeded expectations, with higher-than-expected levels of interest 

• quality academic outcomes: PhD students and post-doctorates have begun publishing academic papers, some of which have won awards 

• technologies Hub management and engagement of businesses: Hub beneficiaries were positive about their engagement with QUB, particularly how the subsidy control issues were handled and managed; this gave firms confidence to progress with their projects at risk 

• collaboration between industry and academia: This was cited as a key strength of the Programme by one consultee. In particular, the initial research conducted by academia was considered crucial for identifying opportunities and challenges in the Cyber AI market; this helped to shape project activities and has ensured their continued relevance  

---

We were not aware of the expertise and capacity available within CSIT, which has been a real strength [for our project]. 

- Technologies Hub Beneficiary 

---

Five areas of the programme which are reportedly working less well included: 

• delays to contracting with the Technologies Hub projects due to subsidy control issues: as outlined in detail above (see The Cyber-AI Technologies Hub), this has been the primary challenge in delivery to date 

• delays in submitting financial claims to DSIT: QUB uses a third-party financial auditor which has created delays in submitting financial claims for the Programme; although, this is not expected to affect timelines and overall spend in the long-term 

• collaboration and knowledge sharing between firms: Consultees reported there has been limited networking and collaboration between firms at this stage and one consultee noted that some companies are reluctant to share knowledge due to competitive concerns. This limitation is claimed to have arisen because QUB’s primary focus has been on contracting and resolving the wrangles around Subsidy Control. Two QUB consultees said that this will be a key focus moving forward 

• inadequate computing resources and access to datasets: Smaller businesses have reportedly struggled with a lack of infrastructure required to deliver their projects; namely computing capacity, requiring large models with high computing capabilities. This challenge has been addressed by businesses reallocating funds to equipment and finding additional resources. In addition, some smaller companies have had difficulties with access to datasets required for the project, which has required further investment from QUB in test-bed facilities 

• project withdrawal from the Technologies Hub: One of the eight approved firms went into administration in March 2024. A waiting list of replacement businesses was prepared, and an alternative firm has been identified to take this forward 

---

We are not seeing [company to company collaboration] … we need to start to stimulate that. It hasn’t happened because it has taken so long to get [contracts] signed. 

- Programme Delivery Team (QUB) 

---

In relation to the contractual delays, the Technologies Hub firms interviewed outlined the implications for projects as follows: 

• Companies had to proceed ‘at risk’ without a signed contract, delaying invoicing and causing underspend – this was a decision projects’ felt necessary due to having researchers already in place 

• The delay may have reduced the projects’ impacts, although it is too early to determine for certain. While the projects were considered to be ‘ahead of the curve’, their market timing advantages may have been diminished due to the fast-moving nature of the Cyber AI field 

• The uncertainty surrounding the contracting prevented firms from fully committing to the project in the early stages. Due to this uncertainty and the reduced workload, one consultee explained they had lost key members of staff  

Despite these unforeseen challenges, beneficiaries commented positively on QUB’s handling of the situation, for keeping them updated on progress, and doing all they can to mitigate the situation. Firms also took confidence that QUB had delivery teams in place and were progressing with other elements of the Programme. 

3.1.4 P6: How could delivery of the Programme be improved? 

Consultees identified at this stage three main areas in which Programme delivery could be improved: 

• subsidy control management: A robust process for managing subsidy control (and similar knotty issues) should be established by QUB prior to Programme/grant setup. If similar programmes are to be delivered in the future, QUB delivery teams should increase their knowledge and awareness of subsidy control requirements and/or ensure this is resolved during programme design 

• programme monitoring: A systematic and consistent approach should be implemented to capture data and report on Technologies Hub project progress. This should include identifying key milestones, outputs, and expenditures in line with delivery timescales. This would provide greater insight in regard to project progress and increase confidence in project outcomes 

• tailored support for Hub projects: The Technologies Hub is supporting a range of firms to deliver innovative projects, and each has different capabilities, depending on its size and access to resources. Smaller firms, in particular, are likely to require more support in upskilling and access to equipment in order to successfully deliver project. This should be a key consideration in future Programme design 

In line with comments regarding Programme monitoring, it is recommended by the evaluators that quarterly financial data are reported for each component of the Programme, against forecast spend. This will enable a better understanding of Programme delivery progress. 

3.1.5 P7: How have external factors influenced the delivery and functioning of the Programme? 

The most significant external influence on Programme delivery has been the changes to the UK Subsidy Control Regime, as outlined above. Consultees did not mention any other external factors that had influenced the functioning of the programme at this point. 

4. Summary and recommendations  

4.1 Summary of report 2’s findings 

While the political contexts in NI and the wider UK for the Programme have changed in its short delivery period, the cyber and AI sectors continue to be a priority for both nations on the island of Ireland. The Programme’s rationale for public intervention – namely to address information/ knowledge asymmetries and negative externalities - continues to be current and valid, and the consequent objectives remain well-aligned with broader UK economic policies in AI and cyber security. The Programme’s design, including the four components, remains relevant to achieving the overall objectives of increasing the supply of cyber-AI skills and growing the cyber cluster in Northern Ireland. The development of new Industrial Strategy/Strategies (out for consultation at the time of writing) by the UK Government is an important likely contextual development, which the Programme’s Management Team will want to track proactively.  

Overall, Programme delivery at this point has been mixed, with some aspects progressing well, alongside major issues being experienced with others. Specifically, the academic components of the Programme, such as the PhD and MSc components, have progressed well against forecast delivery and budget. This is to be expected as most of these courses were being delivered by QUB prior to the Programme, and as such are broadly known entities and models. 

In contrast, the Cyber-AI Technologies Hub has faced significant set-up challenges, primarily due to the complexities of implementing new delivery processes for an intervention which QUB has not previously delivered, compounded with external complexities relating to Subsidy Control. This component has proven to be more complex than anticipated and these issues have slowed progress, caused underspending, and created uncertainties for participating businesses. Despite these unforeseen challenges, the situation has been handled effectively by the Programme Delivery Team, which has ensured continued engagement from the Technologies Hub projects and provided confidence to projects to begin delivery at risk. The interface between the Technologies Hub component and the PhD and MSc components is developing some virtuous connections, more of which can and should be encouraged, and there is clear scope to develop significant links and crossovers amongst the businesses undertaking the Technologies Hub’s R&D projects. 

For the fourth component, the NI Cyber Security Snapshot, its role, alignment and integration with the other Programme components is not yet clear to the Evaluators, and there are issues around the delivery timing and content of the Snapshot (relative to the intended model) which need to be considered. Increasingly, the Snapshot needs to be evidently leading and shaping the Programme rather than following it, as is the sense at present. 

Feedback from consultees suggests that management and governance structures are generally functioning well; however, Programme delivery is behind schedule and budget. In addition, for this phase of the evaluation, there have been various difficulties in accessing fundamental management and monitoring data in relation to Programme expenditures and project progress. In the absence of these data, the Evaluators are constrained in offering early qualitative observations on the Programme’s economy, efficiency, and/or effectiveness. The Evaluators are on hand to help the Programme Delivery Team address and strengthen these data-related issues as the Programme and its evaluation advance. 

Looking forward, the focus for the Cyber AI Hub Programme should be on expediting delivery and achieving anticipated milestones to stay ‘ahead of the curve’; this will help ensure that NI is at the forefront of technological development, and to prioritise activities that deliver the added value of the programme components to achieve cluster growth (e.g., collaboration, network formation and knowledge transfer both between firms and between firms and academia). 

4.2 Key learning and recommendations 

Reflecting on Programme delivery to date, various points of learning were raised by consultees: 

• for future initiatives that involve providing financial assistant to businesses, checks should be undertaken prior to agreeing the programme scope and application in relation to the existing public finance likely beneficiary businesses have (or are expected to receive) and therefore, the funding amounts that can be awarded 

• while the Subsidy Control issues were unprecedented, for future programmes that involve several stakeholders and multiple layers of governance, additional time should be incorporated in the Implementation Phase to provide flexibility to manage unforeseen issues 

• the complexities around Subsidy Control have required significant effort from QUB staff and their legal advisors, and underscore the need for clear guidelines from Innovate UK and more efficient processes for managing public subsidy issues in future programmes 

• there have been various issues experienced in relation to accessing Programme management information from QUB. More structured and transparent communication of Programme progress and project updates will support improved partnership working and allow issues to be identified and addressed promptly 

• collaboration and knowledge sharing in the Technologies Hub has been limited to date. An increased focus on activities to encourage synergies between projects and cross-company knowledge exchange as the Programme matures is a significant opportunity 

4.2.1 Recommendations for programme delivery 

This closing sub-section draws on the findings from this initial process evaluation and presents five recommendations for delivery of the Cyber AI Hub Programme as it goes forward: 

Table 8: Recommendations for programme delivery 

	Recommendation	Rationale
1	Report financial data (budget, actual and forecast) quarterly by programme component	It has been challenging to access detailed financial information from QUB, which is fundamental to a robust process (and later impact) evaluation. This information should be collated and reported on regularly to ensure it is easily accessible, both in terms of programme management and for evaluation purposes.
2	Improve the timing of and provide greater detail in the NI Cyber Security Snapshot including in relation to skills supply and recruitment flows, and sector changes over time.	The two Snapshots published to date have been summary reports (in contrast to the inaugural Snapshot and its then expectation of subsequent Snapshots). This is sufficient at this early stage in programme delivery. However, looking forward, the next Snapshot should include further detail on skills supply and recruitment flows, as well as commentary on changes in the sector observed over time. This will support increased understanding of the sector challenges and the outcomes to be achieved realised by the Programme.
3	Establish clear milestones, deliverables and forecast expenditure for the Technologies Hub projects to report against on a quarterly basis	From the progress updates currently provided by projects, the extent to which projects are on track against anticipated objectives and deliverables is unclear. While the implicit nature of R&D projects is exploratory and flexibility in delivery is required, clarity on a set of anticipated milestones will support effective management of the Technologies Hub component and ensure progress can be reviewed as part of the next process evaluation.
4	Enhance collaboration between the Technologies Hub projects.	To date, there has been limited focus on activities to encourage greater collaboration between the eight Hub projects. This should be a key priority in the next stage of delivery, including structured opportunities for collaboration and knowledge transfer; this could include regular networking events, shared workspaces and platforms for knowledge exchange, and collaborative workshops. This will boost synergies, foster innovation, and support agglomeration/cluster growth.
5	Document and apply learning on subsidy control	The challenges and solutions related to subsidy control should be thoroughly documented, as these insights can be used to develop best practices for future programmes, to reduce risk of delays and ensure a smoother set-up process.

Source: Steer-ED, 2024 

4.3 Next steps with the evaluation 

The next phase of the evaluation will deliver the third evaluation report: Interim Process and Emerging Outcomes Evaluation’ (see Figure 6). This stage was originally scheduled to commence in January 2025, with the report due in August 2025. However, reflecting the delays to Programme delivery (and consequently to this evaluation report and subsequent evaluation deliverables), these timescales will need to be revisited and agreed with DSIT. 

Figure 6: Overview of Evaluation Framework (with the next stage highlighted in orange) 

 

Source: Steer-ED, 2024 

Ahead of Stage 3, the Cyber AI Hub Evaluation Plan will be reviewed and, where required, refined in light of the early process findings identified in this report. In addition, a counterfactual approach will be further developed for the Masters component of the Programme and (if required) additional data collection mechanisms will be implemented. This is scheduled for completion in Q4 2024/25.