Guidance

Test and Evaluation (T&E): Future Advantage Through Evaluation (FATE)

Published 9 September 2024

This document sets out why MOD’s approach to T&E is changing, the ideas shaping its future and how defence responds to them.

Test and Evaluation in defence

T&E is the disciplined collection and analysis of evidence about the behaviour and performance of systems used to understand, improve, and assure that they are safe and fit for purpose.

We test military equipment and evaluate the results to evidence how equipment performs under operationally-representative conditions. We do this to help ensure our equipment is fit for purpose, developed and operated safely, and capable of reliably contributing to the success of military operations.

The scale and diversity of the MOD’s equipment programme means the defence enterprise must conduct a wide variety of T&E. This has made improving the efficiency and efficacy of T&E a source of constant debate.

By the 2030s, risks will materialise that will undermine our ability to conduct T&E as we do today. Organisations across defence are responding with forward-thinking initiatives, which will need to be coordinated to maximise their benefit.

FATE provides the foundation upon which to coordinate a defence response to this challenge, and to accelerate the joint-MOD enablers to make this happen.

Drivers of change

The UK has always had the geography, science and engineering talent, and complex military capabilities to sustain world class military T&E facilities.

While these facilities have evolved to meet changing defence needs, the speed and scope of this evolution is no longer enough to address the materialising risks to T&E and the strategic context which it exists to support.

Some of the long-term themes within that strategic context are:

• increasing the pace of acquisition
• more rapid technology exploitation
• adapting faster to battlefield lessons
• normalising spiral development
• embracing digital engineering
• increasing platform availability
• clearer focus on the latest threats
• increased use of automated systems

Threats to T&E

In addition to the changing strategic context, there are risks specific to T&E that are starting to materialise:

• Surveillance from satellites now persistently undermines the security of real-world testing.
• Advanced adversary systems cannot be usefully represented in live or isolated virtual testing.
• Electromagnetic environments are likely to be contested, complex and not easy to reproduce on UK ranges.
• System integration complexity will make routine live testing of Systems of Systems (SoS) prohibitively expensive.
• Limited asset availability for live testing limits operational realism, innovation, and test utility.
• Higher levels of autonomy, AI and software-defined systems challenge established approaches.
• Novel systems cannot be tested within the constraints of established T&E capabilities in many cases.
• Adversaries will tolerate safety and ethical risks within their T&E to a greater extent than the UK.
• Rapid capability development and constant spiral development could exceed accessible T&E capacity.
• Siloed and fragmented T&E data is preventing exploitation of insight.
• High demand for specialist skills in some niche areas of expertise.
• Industrial-age culture that can be averse to risk and collaboration, particularly around the use of data.

Opportunities for T&E

• Virtual proving is already used in defence, and could scale-up to improve the impact of T&E outputs.
• Increasing levels of automation and digital integration could enhance the speed at which T&E delivers meaningful outcomes.
• Digital engineering is being used more and more across defence, and will increase the availability of data for simulation-based T&E.
• Technology for advanced live T&E is maturing and could improve the efficiency and utility of live tests.
• Commonality with our allies and partners who share the same risks and are open to collaboration.

Why our approach to Test and Evaluation needs to change

• It is not practical or affordable to conduct operationally-realistic, real-world testing of increasingly capable, highly-integrated and adaptive military systems within the UK.
• The pace at which we now need to develop equipment, exploit new technology and adapt to battlefield learning exceeds what our current approach to T&E can support.
• The impact of COVID-19 highlighted the need for the UK to further develop resilience in its ability to test and evaluate critical military equipment independently and with partners.
• Mature digital technologies could improve the speed and quality of T&E and transform its utility for defence, reducing costs, time and risk in both acquisition and operations.

How we respond

Our response starts by recognising that T&E has significant potential to equip the UK with battle-winning insight. This is why we believe in a future of advantage through evaluation, where T&E will help us explore the boundaries of battlefield performance.

We will respond to risks by:

1. Shifting mindsets: T&E as help, not hindrance.
2. Adopting collaborative, lifelong, efficient, actionable and realistic (CLEAR) T&E plans.
3. Enabling blended testing in virtual, range and deployed environments.
4. Setting findable, accessible, interoperable and reusable (FAIR) data rules for T&E.
5. Supporting T&E innovators, regulators, users, and suppliers across the enterprise.
6. Upgrading MOD’s T&E services, adopting 7 key principles (see ‘Principles for T&E services’ below) to embed change.

Purpose of Future Advantage Through Evaluation (FATE)

FATE will act as a foundation for consultation across the defence enterprise on the opportunities and challenges of FATE, the enablers required to deliver it, and how collaboratively the T&E enterprise can accelerate change.

Many organisations and programmes are already working across the defence enterprise to deliver or enable FATE, which includes:

• designing a new T&E enterprise operating model
• investing in infrastructure
• developing capability
• specifying common enablers
• setting standards and supporting early adopters through innovation funding open to MOD and industry

Thinking differently about T&E

FATE goes beyond the conventional view of T&E: that it is a segregated assurance activity to check a standard has been met. Assuring that systems are safe and satisfy requirements is an important application of T&E, but isn’t the only one, and increasingly doesn’t need to be a segregated gateway activity.

The MOD is acquiring advanced, adaptive military systems which will be deployed into complex and contested environments. Delivering and operating these systems will require access to continuous feedback on the impact of design and deployment decisions on battlefield performance.

Change will be built into the DNA of these systems and the enterprise that develops, acquires, and operates them. Systems which are software-defined or highly-automated may need to be updated in minutes and hours, rather than weeks, months, or years.

Creating this level of agility demands a new, non-linear way of thinking about T&E, going beyond the limits of evaluating for acceptance to pursue the potential of evaluation for advantage: a shift from testing performance to coaching it.

To support this shift, we will:

• support pilots that de-risk adoption of a whole-life approach to T&E
• deliver campaigns that promote the benefits of evaluation for advantage

Making T&E plans CLEAR

T&E has always delivered data that makes a difference, but in future that difference will be determined by how CLEAR we can make T&E planning:

1. Collaborative – growing understanding over time by building up the data.
2. Lifelong – how can it inform decision making from day one, through life?
3. Efficient – making best use of the tools, data and support available.
4. Actionable – time to inform decisions, not just validate them.
5. Realistic – reflecting the reality of where/how systems will be used.

The need to enable CLEAR T&E plans will guide our approach to implementing new T&E technology, so that we drive a focus on achieving better capability outcomes and not just delivering better T&E.

A blended approach to testing

To achieve CLEAR T&E, we will advocate for the use of a blend of synthetic and live approaches to testing, maximising the benefits of both while recognising the limitations of each.

Testing in synthetic environments

A mature capability in specific fields of application and within some defence organisations. The use of modelling and simulation is widespread, although only well-integrated across the defence enterprise in the context of training.

To achieve CLEAR T&E, a common approach to the use of synthetic testing is required, and the capability to do so at both system and SoS level, where organisational and technical complexity is prohibitive today.

To accelerate the adoption of common and collaborative approaches to synthetic T&E across the defence enterprise and with partners, we aim to:

• define interoperability standards
• develop interoperability architecture
• improve enabling infrastructure
• develop enabling capabilities, including across industrial and international partners

Testing on dedicated ranges

A mature activity in the UK, both in the public and private sector. Test ranges however are often geographically and digitally isolated from the defence enterprise, and haven’t been equipped or configured to capture and easily share data in support of model validation and synthetic testing.

For range-based testing, we aim to:

• improve connectivity to MOD ranges
• explore and implement range-based data capabilities
• pursue a ‘frictionless’ interface between virtual and physical testing
• establish an innovation pipeline for UK test ranges so they keep pace with MOD requirements and the systems they are required to test

Testing systems in the field

Testing systems in the field, either while they are being used on training, exercises or operations, is another currently underutilised enabler of CLEAR T&E.

Combining data already collected by onboard systems with objective evidence from deployed T&E capabilities could improve our understanding of performance, add value to exercises and training, and in some cases allow longer or more effective sustainability on operations. To support deployable testing, we plan to develop deployable T&E capability and enablers for mobile ranges.

Making T&E data FAIR

As well as collecting new data through a blend of testing, we must liberate the value of existing data to support T&E.

The defence enterprise already holds much of the data it needs to fuel better decisions, but a combination of circumstances can often mean that data is beyond reach of T&E exploitation.

Many digital initiatives are already underway in MOD and across the defence supply chain to address these issues. T&E must make itself an early, willing, and consequential adopter of this change, and address specific T&E challenges to establish the digital foundations and data fabric on which new T&E practices, standards, initiatives, and capability can be built.

There is significant potential for the application of automation and machine learning to rapidly improve the efficiency and quality of insight that can be developed from performance data. To accrue these benefits, data must be made accessible to those with the skills to identify opportunities, innovate software and apply these technologies.

For these reasons, data will be critical to CLEAR T&E. We plan to:

• make T&E data FAIR to those who need it at several classifications
• introduce new policy and standards where required to prevent siloing
• implement new common data handling capability where required
• amplify, augment and focus digital skills initiatives for T&E audiences

Supporting those involved in T&E

Our response focuses on enabling new ways of conducting T&E. Working in new ways requires those using, developing, delivering, supporting, sponsoring and regulating T&E to take the lead.

To give them the tools to do so, we plan to:

• deliver a T&E innovation fund
• lower the knowledge barrier to T&E
• help regulators with digital evidence
• invest in training and support
• champion T&E and reward success

For UK industry, we propose to:

• improve visibility and the interface between T&E users and suppliers
• provide innovation funding and improve visibility of UK T&E small and medium enterprises (SMEs)
• support the development of the UK T&E community and workforce
• consult on a T&E operating model and establish enterprise governance

Principles for T&E services

To anchor these elements within the T&E services and support MOD invests in, we are proposing 7 guiding principles for T&E services. These principles are intended to help service sponsors and requirements managers to embed the ideas within FATE into the services MOD relies on to deliver its T&E activities. They are:

1. Fit for the fight - able to test the way the UK fights, incorporating the latest tactics and doctrine, intelligence-based threats and realistic operating environments.
2. Cutting-edge capable - able to respond to the technologies being introduced into capability over the life of the service, by being aware of and responsive to R&D.
3. Designed for data - able to automatically capture, process and transfer all relevant data, making it FAIR to those who need it in support of live and synthetic testing.
4. Integrated for insight - able to integrate with other relevant elements of the T&E enterprise to enable federated testing, data transfer and real-time simulation as required.
5. Primed for people - able to be easily and efficiently be fully exploited and managed by MOD personnel, without the need to develop specialist skills and expertise.
6. Innovative by instinct - able to constantly change by accessing and conducting R&D to identify and incorporate innovation which MOD identifies as beneficial.
7. Consistent and commercially coherent - able to be accessed and engaged through the MOD’s centralised T&E support services, and interoperable with MOD’s centralised T&E services and relevant international and industrial T&E capabilities.

How we deliver

Achieving FATE will not be the product of one programme or organisation, but the result of collaborative efforts across the defence enterprise to enable the potential of data and technology for T&E.

FATE, and the industry consultation that follows it, will guide the work of the following initiatives, supporting them to contribute to and converge on a shared vision of the future for T&E. It will also help us identify where concerted joint effort is required to address the barriers to this future experienced today.

The T&E Transformation Programme

A long-term change programme being delivered by MOD Head Office, Defence Science and Technology Laboratories (Dstl) and Defence Equipment & Services (DE&S) working closely with the Front Line Commands (FLCs).

This programme is developing and investing in the infrastructure and enablers of FATE and supporting T&E innovators in MOD and industry through dedicated innovation funding. Visit Defence Ideas or Defence and Security Accelerator (DASA) for more information.

The Strategic T&E Portfolio (STEP)

The portfolio of centrally-managed T&E support available to MOD users covers centralised T&E service contracts, such as the Long-Term Partnering Agreement with QinetiQ for MOD range services; Radar Signature Measurement services with Thales; and the Combined Aerial Targets Service with QTSL, as well as international agreements.

This portfolio will be refreshed over the next decade to better enable FATE and the future T&E operating model, with STEP evolving to accommodate any new centralised enablers. This future portfolio will be known as ‘STEP2’.

Defence Simulation Network

Otherwise known as the Joint and Multi-National Interoperability Assurance Network (JMNIAN), the Defence Simulation Network provides a pan-government, multi-national networking capability, underwriting interoperable T&E amongst other capabilities without risk to any live operational networks.

Requirements for the next upgrade of the JMNIAN network will consider the increasing needs of T&E, both at live test ranges and between T&E practitioners across MOD.

Defence Synthetic Environment Programme (DSEP)

To improve the efficiency and effectiveness of MOD’s use of M&S technology, Defence Digital is developing DSEP to provide MOD users with a coordinated M&S ecosystem to enhance discovery, composition and execution of common and customised M&S activities, including T&E.

DSEP requirements will take into account FATE and the outputs of the T&E Target Operating Model.

Defence Simulation Centre Catalogue (DSCC)

Defence-wide access to common M&S resources across all M&S applications, including T&E, through the DSC, to support coherence and enable reuse, interoperability and value for money.

Joint Service Publication (JSP) 939

JSP 939 provides direction and guidance for the acquisition, development and usage of modelling and simulation across UK defence. It contains the policy and direction on M&S from a defence enterprise perspective, guidance on the processes involved and best practice to apply to any M&S development.

This JSP will be updated to align with FATE and the T&E standards and best practice that emerge through consultation.

Defence M&S Standards Profile (DMSP)

As covered by Defence Standard 03-050 for contractual purposes, the DMSP lists mandated and preferred M&S standards. It is aligned to the NATO equivalent to promote interoperability with allies.

Assessment and authorisation duties against JSP 939 will ensure that listed standards are selected, unless a waiver is issued by the appropriate Service Command Technical Authorities (SCTA).

The DMSP is maintained by the DMSO technical authority and reviewed annually. This will include accounting for standards which emerge for T&E through the implementation of FATE.

DE&S DEC

A key component of DE&S Digital Transformation - the DEC was created to increase the speed of capability delivery to the user, through the up-scaling and exploitation of digital engineering.

The DEC will drive the adoption of MBSE at the enterprise level, which will act as an enabler for the increased use of M&S in T&E, particularly at early stages of the life cycle.

Front Line Commands (FLCs)

The individual initiatives listed are all working closely with the FLCs, their T&E and innovation communities, as well as other MOD organisations.

In many cases FLCs are leading the development and application of new T&E technologies, and these initiatives often seek to enable this work and remove barriers to the wider adoption of their findings.

Other partners

We will reach out to industrial and international stakeholders, allies and partners to understand how best to engage them in FATE and the programmes listed here.

Comments from T&E experts

Andrew Caldwell (Head of Research, Development, Test & Evaluation - Strategic Programmes, MOD):

As head of both R&D and T&E, I am keenly aware of the complex challenges involved in ensuring our science and technology investments reach the front line at the speed of relevance.

It is not only the cutting edge where this is an issue. The performance of our military capabilities relies on our ability to understand, assure and enhance the performance of our military systems in operationally-representative conditions.

The complexity of these systems and the environments they now face is beyond what we can reproduce through live testing alone, and the established norm that T&E is a discrete acceptance activity undervalues its potential. Indeed, through-life T&E is critical to the development of future capability and pursuit of operational advantage.

FATE intends to disrupt the status quo by re-framing T&E as a critical pathfinder of performance and enabler of advantage, and how we ensure our investments in R&D, technology and equipment are destined to pay off on the battlefield, when it matters the most.

Brigadier Chris Coton (Head of Concepts at the Development, Concepts and Doctrine Centre, MOD):

We are in an era of global power competition, and the technical advantage we have enjoyed over our adversaries is diffusing. In response, defence must be able to adapt routinely and rapidly to the threats faced. 

Defence’s T&E effort must become more agile and capable of assisting decision-making across the capability life cycle, helping new ideas develop safely, securely, and quickly into real-world advantages. T&E must drive, not delay, technical evolution. 

Our adversaries will seek to understand and undermine our technical capabilities. So, we must increasingly evaluate in more secure environments and maximise the T&E insight derived from the real-world use of systems, from training, to trials, to operations.

An improved T&E capability will also strengthen our international partnerships, enhancing our ability to interoperate with allies. In short, the changes envisaged in FATE are extremely important as they will have a direct impact on the operational effectiveness of the UK and our allies.

Bijal Mistry (Head of Defence Modelling & Simulation (M&S) Office, DMSO):

M&S has always been a powerful way to drive more efficient and effective defence outcomes. Training - an early adopter of M&S - has clearly demonstrated its benefits, from being able to do more flight training without the cost of operating an aircraft, to keeping our people safe while training them for situations where they will be in harm’s way. Situations that are often impossible to stage for real can be made real through simulation: a great benefit to complex T&E.

As interest in the benefits and application of M&S continues to expand beyond training and across MOD, so does the need to ensure that the many common enablers and assets involved are procured, managed, and used efficiently. This is the role of DMSO: to act as the cohering force for M&S in defence, centred around the policies set out in JSP 939 and the enablers that support it.

FATE sets out a vision for blended testing that draws much more heavily on collaborative M&S than happens today. Leveraging the enablers, DMSO has already created and is shaping those still to come, to enable a future of FATE that both benefits from and contributes to an enriched and efficient M&S ecosystem.

Paul Belcher (Head of Engineering Group, Directorate of Engineering and Safety, DE&S):

FATE describes a world in which defence must be able to utilise and exploit digital capabilities to provide the edge to our armed forces at speed. Digital ways of working, digital tools and digital processes are key to unlocking this potential and moving us from ‘months to minutes’. Digital gives us the ability to confidently make evidence-based decisions at the speed of relevance.

The Digital Engineering Centre (DEC) hosted by DE&S on behalf of UK defence is a crucial enabler that highlights step change opportunities, raising skills, knowledge and awareness, and brings MOD and industry closer together through culture change, adoption of standards, mutual recognition and integration.

Carrying out digital design, testing and evaluation of capabilities with models, simulations and emulators means we can run immense numbers of tests, scenarios and permutations. This enables us to respond rapidly to operational conditions and modify our capabilities accordingly. Model-Based Systems Engineering (MBSE) and digital verification tools coupled with a digital backbone give us the opportunity to agree models with regulators that we can use through life to improve speed, safety and performance.

Get involved

If you’d like to know more about FATE, apply for innovation funding, or be involved in creating the future vision, please contact the T&E Transformation Team: FMC-FutureTE@mod.gov.uk.