Analyse your data: evaluating digital health products
How to prepare and analyse the data you collected for your evaluation.
This page is part of a collection of guidance on evaluating digital health products.
Analyse qualitative data
Most approaches to analysing qualitative data involve some sort of coding. This means associating meaningful ideas (codes) with the data through a process of understanding and finding meaning in it. The most common approach is thematic analysis.
Thematic analysis
Thematic analysis involves identifying and developing common themes in the data you have collected. It can help you to make sense of large volumes of data.
A theme is a topic or pattern that you see repeatedly in your data. It is common to divide these into themes and subthemes. Themes can contain both positive and negative ideas about a topic – for example, in an interview study about a chatbot for sexual health, one participant says that they prefer talking to a computer program because of the intimate nature of the problem. Another participant says they don’t like the service because they want to talk to a real person. Both comments might fall under the same theme, although one is positive about the service and one is negative.
When to conduct thematic analysis
This type of analysis can be started while the data is still being collected. The questions you ask might change based on what the analysis shows. For example, you might find that your first 2 interviewees mentioned unprompted an aspect of the digital product you hadn’t asked about. You might then add a question about that aspect to your interview guide.
You can use this method to evaluate data collected through methods including:
- interviews
- focus groups
- observations
- diaries completed by participants
- user reviews
The source of data is important to consider. For example, researchers can control the questions in an interview, but that is less likely with online reviews.
Developing your themes
There are 2 steps:
- Create the themes.
- Code the data in line with the themes.
You will need to move forwards and backwards between these steps.
You should not just count the number of times particular words or phrases occur – thematic analysis involves making connections between related words and concepts. You will probably need to read through the data several times, comparing responses from different participants, and comparing responses each individual gave at different points in the interview.
You will also need to develop a coding scheme. This will tell you how to categorise the data according to the themes. This should be seen as part of the analysis. Once you are familiar with the data, you can start to develop initial codes and then develop themes from these.
You may have an existing code scheme you are working to (for instance, this study of a digital weight management intervention used a COM-B model of behaviour to analyse the qualitative data), or you may use an approach that allows you to be more responsive to the data. You can also combine these approaches, starting with an existing code scheme and adding to it.
Checking your analysis
How will you know whether you have spoken to enough participants for a good thematic analysis? You might reach a point where extra data you collect does not give you any new information (theoretical saturation). For example, you are not identifying any new themes or subthemes. In practice, it can be difficult to know whether you have reached this point.
Another common approach is to check your analysis with participants. You could share the interview transcripts and themes with participants to find out whether they agree with them. You could also involve participants in developing the themes.
Coding your data
Data can be coded by hand or using software that allows you to mark up text and find all passages that belong to a particular theme.
More information and resources
Braun and Clarke (2006): Using thematic analysis in psychology. This paper outlines how to do thematic analysis.
Fugard and Potts (2019): Thematic analysis. In Atkinson PA, Delamont S, Williams R, Cernat A (eds.), ‘SAGE Research Methods Foundations: An Encyclopedia’, SAGE.
Gale and others (2013): Using the framework method for the analysis of qualitative data in multi-disciplinary health research. This paper describes the framework method, a form of thematic analysis.
Graneheim and Lundman (2004): Qualitative content analysis in nursing research: concepts, procedures and measures to achieve trustworthiness.
Prepare quantitative data
The raw data you have from an evaluation may need preparation before you can analyse it. This is particularly likely if you use routinely collected data rather than data collected specifically for the evaluation.
Anonymise data
You will often want to analyse anonymised data, so the first step may be to anonymise it. Make sure you only extract data without personal identifiers or delete any personally identifiable information. Follow General Data Protection Regulation (GDPR) requirements. Read more about what personal identifiers are on the Information Commissioner’s Office website.
Anonymising data can be more complicated than just removing people’s names or addresses. For example, if you know age, gender and full postcode, individuals can probably be identified from this, even if your data does not include a name. Detailed global positioning system (GPS) data from a phone can also identify a person.
You may want to use pseudonymisation, where personally identifiable data is replaced by artificial identifiers (pseudonyms). You could use data that is less easy to identify individuals from, for example, by swapping from a date of birth to just the year of birth, or from a full postcode to just the first half of the postcode.
Check for anomalies
Check your data looks correct. Ask yourself:
- do you have the data you expect?
- is the number of participants correct?
- are there any repeats in the data?
For example, in an evaluation of app usage, the data showed a lot more users than expected. The team found that identification numbers which were meant to be unique were repeated several times, creating additional rows in the dataset. They concluded that this was caused by problems users encountered when they first registered. They removed additional entries for the same person.
Outliers
Are data values within ranges you expect? Data values that are not are called outliers.
For example, in an evaluation of an app for heart disease, one age was recorded as 6, but the app was intended only for adults, and it seemed unlikely that it was being used by a 6 year old. The data item was removed.
If you can find out where an outlier came from, for example by looking back through records to see if it was a data entry mistake, this can help you to decide what to do about it.
You shouldn’t remove an outlier just because it is a big or small number. You may want to conduct the analysis with the outliers and without them to see if this produces substantially different results. This is a type of sensitivity analysis (see ‘Sensitivity analysis’ section below).
Missing data
A big challenge, particularly with routinely collected data, is missing data. Small amounts of missing data are not usually a problem, but more missing data can be.
You should work out why the data is missing. This will help you decide what to do about it.
Example
In one study, data on individuals’ physical activity each week was sometimes recorded as ‘NULL’ in a database. What had led to this?
The software team explained that these entries meant the app had recorded no physical activity in the period. This could be because the user did not participate in any physical activity above a set threshold, so the true value should be recorded as 0. But it could also mean that the user was not engaging with the app and had turned off automatic data collection, so the app was not recording data. These 2 interpretations have different implications for the evaluation.
The team decided that individuals with NULL for the entire 6-week period of the evaluation were probably not using the app at all. They were excluded from the analysis as non-users.
If individuals had physical activity recorded for some weeks, but NULL for others, it was more likely that weeks with NULL represented no significant physical activity in that week, so these were recorded as 0.
What to do about missing data
There are several techniques for dealing with missing data, but some are more useful than others. Imputation means replacing missing data with a value you have calculated. If an individual has some missing data but other data is present, imputation can allow you to make use of their partial data. There are several variations on this technique. We recommend multiple imputations, as it reduces bias and inaccuracy.
In some situations, you can make conservative assumptions about missing data – assumptions that are deliberately more cautious. For example, with a behaviour-change app, if users are not recording data any more, you might assume that they have not made the behaviour change. If an evaluation still shows the app is effective when you have made this conservative assumption, you can be more confident that it works.
Sensitivity analysis
If you need to make choices about how to prepare data or handle missing data, there can be concerns about how reliable the results are. Sensitivity analysis is an approach that can help. If you have 2 options, A and B, you repeat the analysis for each option. If your overall conclusion is the same for both options, you can be more confident that your result does not depend on the choice made.
If you get varying conclusions with your sensitivity analysis, this is a problem, but you will know that the choice you make is important.
Analyse quantitative data
If you have collected quantitative data, you usually start with descriptive statistics. For example:
- how many people downloaded an app
- the average amount of physical activity done by people using an app
- the proportion of users who said they were satisfied with a digital service
Data visualisation is one approach to making sense of descriptive statistics. It involves showing data as a graphic, such as a graph or a map.
Descriptive statistics may be all you need, but you will often want to take the data from your sample and use it to draw conclusions about a whole population (inferential statistics).
For example, you collected data from 500 users (your sample) of a smoking cessation app, but you want to draw conclusions about all the users (the population) from this data. Answers about the population are estimates. How confident you are in these estimates depends on several factors, including:
- the size of the sample
- how representative the sample is of the population being evaluated
- how the sample was selected
- the statistical methods used
Inferential statistics work on the assumption that you have selected a random sample from the population.
The quality of your estimates
Usually, the best estimate of a statistic for the whole population is the value of that statistic in the sample. Confidence intervals, also known as margins of error, say how good that estimate is.
The confidence interval is the range that is very likely to include the true answer, usually with 95% probability. For example, your sample estimate is that users smoked 10 fewer cigarettes after using your app. A 95% confidence interval of 5 to 15 cigarettes means that the true answer is very likely to be between 5 and 15.
A wide confidence interval means the sample statistic is less precise as an estimate of the population value. The answer could be any one of many possibilities.
Hypothesis testing
A hypothesis is a proposed answer to a question. You do not yet know if it is true or false. For example, ‘This app helps users to smoke fewer cigarettes.’ Hypothesis testing is a statistical method to discover if the hypothesis is true or false.
Use hypothesis testing to:
- compare 2 or more groups – for example, an intervention group and a control group in a randomised controlled trial, or users before and after they use an app
- assess whether 2 or more variables are correlated – for example, if one increases, does the other decrease?
Hypothesis testing starts with a hypothesis that there’s no difference or relationship between things (the null hypothesis). For example, the use of your app has no relationship, positive or negative, to the number of cigarettes users smoke.
Statisticians calculate the probability of your results having happened by chance. If this probability is low enough, you can reject the null hypothesis and accept the alternative. For example, this app helps users to smoke fewer cigarettes.
The probability of your results having happened by chance is shown as a p-value. For example, a p-value of less than 0.05 would indicate there is less than a 5% chance that the data you collected would have happened by chance under the null hypothesis. 0.05 is the most common threshold for statistical significance.
What your quantitative results show
Your statistical analysis may show that there is a difference between 2 groups or a relationship between variables (correlation). However, it can be difficult to interpret what that means. Why has the difference or the correlation happened? Is it because the digital health product has caused it or for another reason?
For example, you find that increased use of the app correlates with fewer cigarettes smoked. This may not mean that the app caused users to smoke fewer cigarettes, just that the 2 things happened at the same time. It could be coincidental. Maybe users of the app are already trying other things to cut down their smoking.
Causation is the relationship between cause and effect. It indicates that a change is due to a specific cause. For example, you may be able to prove that using the app caused users to smoke fewer cigarettes. The strongest way to prove causation is through the design of your evaluation, for example, by doing a randomised controlled trial.