Get started: evaluating digital health products
How to begin with evaluating a digital health product, including when to evaluate, budgeting and getting support.
This page is part of a collection of guidance on evaluating digital health products.
What evaluation is
Evaluation is essential to finding out what works and what does not work, and why. It involves collecting data that enables you to assess the effectiveness of your product.
Challenges of digital evaluation
Non-digital healthcare products and services are routinely evaluated to make sure they are safe, effective and a good use of public money. For example, pharmaceuticals are rigorously tested.
Digital health products should also be evaluated, for the same reasons. Digital health poses particular challenges for evaluation but also offers opportunities. Traditional approaches to evaluation in healthcare may not work well. For example:
-
digital health products often change rapidly, which can be difficult to align with the time it takes to do some evaluations, but the speed of digital can also mean evaluations can be done more quickly
-
most digital health products are complex, with many different functionalities and features, so evaluations may focus on specific features or take a more global approach
-
the regulatory framework for digital health products is evolving: new methods of evaluation are also being developed and tested and it is not always clear what current best practice is – this guide will help you to navigate your options
Choosing your evaluation approach
Evaluation is a broad term covering a range of different methods. Some are cheaper and quicker; some are more expensive and involved. There is no one correct waya of carrying evaluation out. However, some methods will give more useful results and some will be more approximate.
You will need to decide the scope of your evaluation. This will depend on:
- what the purpose is
- who your audience is
- what stage you are at in creating your product
Also consider practicalities, such as:
- who will carry it out
- what resources you can give to it – time, money, people
Data collection or evaluation
Collecting data is not evaluation, although it is an important step in the process. Evaluation means taking your data and using it to assess the impact of your product and how that compares to your stated goals.
Consider when to evaluate
You can evaluate your product at different times in its lifecycle. You might want to evaluate it:
- as you’re developing it, to work out how to make it better (formative evaluation)
- when you’re launching it, or soon after, to find out whether it achieves its aims (summative evaluation)
- when you’re are scaling up use or using the product in new contexts (implementation evaluation)
- once your product is in use, to check it still works in practice and has no unintended consequences (post-implementation or ex-post evaluation) – in some contexts, this means monitoring its safety for the public (post-market surveillance)
Digital products are often developed iteratively – testing products repeatedly and improving them throughout the development. Evaluation of one version will act as a formative evaluation for designing the next version.
You may need to use different evaluation methods at different times, but evaluation is much easier if you start planning for it during the development of your product (in the discovery phase if you are using an agile delivery process).
Evaluation scope
All digital health products should be evaluated. However, the scope of the evaluation can vary depending on the situation. If your product or service does something for which evidence already exists, a lower-level evaluation may be enough.
For example, if you have developed a game and already demonstrated in a thorough evaluation that it helps increase physical activity, then a light-touch evaluation would be enough to test new levels you add to the game.
What evaluation is needed
In some circumstances, you might want your evaluation to meet certain standards.
The National Institute for Health and Care Excellence (NICE) has developed the NICE evidence standards framework for digital health technologies. This suggests what evidence you should gather to demonstrate the value of different types of intervention. This is based on:
- risk of harm to users
- economic risk
Apps linked to from the NHS website will need to meet the standards set out in Digital Technology Assessment Criteria (DTAC).
BSI PAS 277 sets out quality criteria for developing, testing and releasing health and wellness apps.
If your product is classed as a medical device, it must meet the Medicines and Healthcare products Regulatory Agency (MHRA) guidelines. For some devices, you will need to follow the ISO 14155 standard for clinical investigation of medical devices for human subjects.
Get support
You may need help to carry out the evaluation. Sources of support will depend on the context you’re working in.
Knowledge-sharing services
Academic Health Science Networks (AHSNs) are regional hubs that encourage the NHS, higher education institutes, private sector organisations and other partners to work together and share knowledge. They promote health innovations. You can access support from the AHSNs and other organisations through the NHS Innovation Service.
If you work in the NHS, the Global Digital Exemplars may provide guidance.
Other regional research alliances can provide support. For example:
- MedCity (London and Greater South East England)
- Northern Health Science Alliance (North England)
- Health Innovation Research Alliance Northern Ireland
- MediWales
- Life Sciences Scotland
Partnerships
You might want to carry out an evaluation together with another organisation, such as a university or healthcare provider. AHSNs can support you to do this.
If you want to work with a university, look for departments with digital expertise as well as expertise in the subject you’re evaluating.
There might be groups in your organisation with specific evaluation expertise who you could work with.
Paid support
Paid external consultants provide a range of services, including help to develop your evaluation, collect evidence and analyse your data. These services might be supplied by contract research organisations (CROs), higher education institutes or freelancers.
You could also pay an external organisation to conduct the evaluation. If the evaluation of your digital health product is carried out by a group who are independent of you, this can make the evaluation more impartial.
Training
This will depend on the organisation you’re working for. Higher education institutes often offer training. You may be able to buy training from them, including bespoke training.
If you are working in the NHS, there is relevant training at various levels:
- relevant streams in the NHS Scientist Training Programme for new graduates
- the NHS Digital Academy, open to chief information officers (CIOs), chief clinical information officers (CCIOs) and prospective CIOs and CCIOs
Funding
You might be able to apply for government grants from the National Institute for Health Research (NIHR) or UK Research and Innovation (UKRI). Some charities also fund studies.
The NIHR Research Design Service provides advice and guidance to researchers on all aspects of preparing grant applications. This includes support with study design, research methods and identifying suitable sources of applied health and social care research funding.
If you are working in the private sector, there is government support for research and development – for example, Innovate UK provides grants. There may be tax exemptions. There are also sources of private funding for start-ups – for example, angel groups, venture capital firms and crowdfunding platforms.
App development
The NHS website links to a range of recommended apps, selected by clinical policy teams. If you’re developing an app, read the Digital Technology Assessment Criteria (DTAC) published on NHS.UK to find out what is expected of digital health technologies entering the NHS.
Budget for evaluation
As a rough rule, you should allocate 10% of your total project budget to evaluation. However, this may not be possible, or it may not be adequate for your project. Your evaluation budget should be proportionate to what you need to learn from your evaluation.
Evaluation costs can range from around £500 to several million pounds. Typical costs include:
- staff salaries
- equipment and materials
- travel
If you are carrying out research in the NHS, there may be interventional costs. The NIHR website has has an interactive costing tool for planning commercial trials in the NHS.
Example of an evaluation with a smaller budget
This is the budget for a micro-randomised trial of an app to see whether reminder messages to encourage engagement could be improved. In the trial, certain users of the app are randomised each day to receive a reminder message in the late morning or early evening.
| Budget item | Amount |
|---|---|
| Developer salary: Making changes to the app to get participant consent, carry out the randomisation and send messages at different times | £2,500 |
| Staff salaries: 1 staff member leading the evaluation (50% full-time equivalent (FTE)); 5 staff members contributing to the work over about 6 months | No budget – absorbed into usual business costs |
| Total | £2,500 |
Based on Bell L, Potts H, Williamson E (2019), ‘Improving engagement in a health app: Considerations in designing a micro-randomised trial.’ Trials, 20(Suppl 1): P-17.
Example of an evaluation with a larger budget
This is the budget for a 4-year randomised controlled trial, including some initial work to check the feasibility of doing a trial. It will evaluate the effectiveness of an app to promote physical activity among people diagnosed with cancer. It is a standalone trial using an existing app.
| Budget item | Amount |
|---|---|
| Lead researcher (100% FTE throughout) | £206,771.88 |
| Trial manager (40% FTE throughout) | £82,434.55 |
| Statistician (100% FTE for 4 months) | £20,519.50 |
| Research assistant (100% FTE for 30 months) working on participant recruitment and data collection | £108,947.39 |
| Research assistant (100% FTE for 18 months) working on participant recruitment and data collection | £66,359.64 |
| Research associate (50% FTE for 6 months) providing health economic input | £13,212.00 |
| Consultant nurse (5% FTE throughout) providing clinical input | £14,227.27 |
| Travel | £23,400 |
| Equipment (mainly wearables for collecting data from individuals) | £47,504.13 |
| User involvement | £1783.20 |
| Transcription | £1,170 |
| Data entry | £5,220 |
| Postage | £28,486.80 |
| Printing | £383.04 |
| Total | £620,419.40 |
Acknowledgements
Content written by Henry Potts (University College London), Flora Death (Public Health England) (PHE), Paulina Bondaronek (University College London), Manuel Gomes (University College London) and James Raftery (University of Southampton) with additional input from PHE.
Project advisory group: Ann Blandford, Jamie Brown, Tim Chadborn, Darren Curry, Kenny Deighton, Pall Johnson, Indra Joshi, Scott Mahony, Elizabeth Murray, Felix Naughton, Claudia Pagliari, Neelam Patel, Colin Pattinson, Quynh Pham, John Powell, Susannah Robinson, Ian Roddis, Mark Salmon, Tero Väänänen, Matthew Walmsley.
We would like to acknowledge the PHE team behind ‘Evaluating Digital Health Products’: Hasan Ali, Rachel Clark, Iain Cooper, Simon Dixon, Charlotte Fountaine, Felix Greaves, Pouria Hadjibagheri, Bernie Hannigan, Kassandra Karpathakis, Vicki Litherland, Ros-Mari Mitova, Fionnuala O’Toole, Claire Rackstraw, Bobak Saadat, Anya Zeitlin.
We would like to acknowledge Livework Studio’s contribution to the earlier phases of the project: Anton Drachuk, Lilli Graf, Claire Henderson (King’s College London), Gene Libow, Nick Manson, Ben Reason, Paul Sims. This work was solely funded by PHE and is now maintained by the Office for Health Improvement and Disparities.