N-of-1 study: comparative studies
How to use an N-of-1 study to evaluate your digital health product.
This page is part of a collection of guidance on evaluating digital health products.
N-of-1 studies focus on observing changes in individuals (single cases) over time, in comparison to a group-based design in which outcomes are combined for many participants. N-of-1 design (also known as a single-case design study) is used to understand within-person processes, such as changes in an individual’s engagement with your digital product over a period of time.
What to use it for
Similar to group-based designs, an N-of-1 study can be:
- observational (you record what you see)
- experimental (you compare periods when a digital product or features are provided with periods when they are not provided)
This means it can be used during development (formative or iterative evaluation) to find out how to improve your product. It helps you to explore its more nuanced effects and determine, for example, what factors predict higher usage.
An N-of-1 study can also be used to evaluate the effects of your digital product by providing detailed data on how the effect varies for a single person. Because of this, it can be used to inform the design of a comparative study (see Choose evaluation methods: evaluating digital health products).
Pros
Benefits of N-of-1 studies include:
- they can provide more granular data on changes in an outcome than you would get from combining outcome data across participants
- they can identify individuals a product works for and does not work for
- they generally require fewer participants in comparison to more traditional group-based studies
- in an experimental evaluation, each participant acts as their own control, meaning the design is more sensitive to differences in the effects of the intervention
Cons
Drawbacks of N-of-1 studies include:
- they often involve intense data collection that might be a burden for some participants
- you might need expert statistical skills to analyse the data
- there is a risk of missing data because of the repetitive nature of collecting the data – this can make the analysis more complex
How to carry out an N-of-1 study
The most rigorous of this type of study is an N-of-1 trial. In a traditional randomised controlled trial (RCT), groups of participants are randomly allocated to an intervention or a control condition. In an N-of-1 trial, individuals are assigned to different options in a randomly-determined order, so they are exposed to the intervention and control on different days of the trial period (called multiple crossovers).
Because a participant experiences the intervention and the control over different times, some interventions are not appropriate for assessing in an N-of-1 trial. For example, if the:
- intervention takes more time than a typical treatment period to produce an effect (slow onset effect)
- intervention produces a change that remains for some time after the intervention stops (carry-over effect)
- health condition is progressing rapidly
This means N-of-1 trials are good for interventions that are reversible (where the effect wanes over a short time), but not appropriate for interventions which have a long-lasting effect.
Between the periods, you can introduce a washout period, in which, to allow the effects of the previous intervention to diminish, participants receive no intervention. This can help with some slow-onset and carry-over effects. Piloting your digital intervention can also help you to identify any slow-onset and carry-over effects, and how significant they might be.
The statistical power (see ‘Statistical power’ in Design your evaluation: evaluating digital health products) of N-of-1 studies depends on the number of observations rather than the number of participants. N-of-1 designs typically use ecological momentary assessment, which involves frequent data collection at different intervention phases.
An N-of-1 study can include just one person, but typically a series of N-of-1 studies are undertaken. These can either be analysed as separate datasets, or in some cases combined statistically to provide an average effect between participants. If a large number of N-of-1 datasets are combined, then it is possible to identify participant characteristics that are associated with the intervention effect (for example, to identify who the intervention works for and who it does not work for).
The data from an N-of-1 study should be treated as a time series: repeated observations of a particular measurement collected over time. It’s important to note that individual responses tend to be more similar when assessments are carried out close together in time (autocorrelation). For example, when responding to a questionnaire about how you are feeling, your responses given yesterday and tomorrow will usually be more similar to each other than your reponses given a week ago or a week in the future.
You will need to analyse the data using models that account for autocorrelation, either as single cases or combined in a multilevel analysis.
Example: evaluating the effect of goal-setting and self-monitoring on increasing walking
See Sniehotta and others (2012): Testing self-regulation interventions to increase walking using factorial randomized N-of-1 trials.
Researchers wanted to investigate the effectiveness of 2 interventions on walking: goal-setting versus self-monitoring.
They used a factorial N-of-1 trial where they assessed how effective goal-setting and self-monitoring features are in increasing walking. This design has similar principles to group-based factorial RCTs. This was a series of 2 × 2 factorial N-of-1 studies where 10 participants were randomised to either a goal-setting condition or a control and also to either a self-monitoring condition or a control.
In the goal-setting condition, participants were prompted to set themselves a goal to achieve a specific number of steps. The goal-setting control included a goal to consume more fruit and vegetables on that day.
For the self-monitoring condition, participants were given 2 pedometers, one with a visible display and one with a sealed (blinded) display, to monitor their steps.
Participants received a text message prompt each morning, telling them which goal and pedometer they needed to use.
The researchers conducted 10 regression analyses, one per participant, to find out for whom goal-setting and self-monitoring were effective. They controlled for autocorrelation. The study did not have the statistical power to detect interaction effects. Read more about interaction effects in factorial RCTs.
Researchers found that most participants increased their number of steps in both the goal-setting and self-monitoring conditions compared to control days. However, individual analyses showed different effects of the interventions:
- 4 participants significantly increased walking – 2 on self-monitoring days and 2 on goal-setting days
- one participant showed a small decrease in their steps throughout the study
This study showed the variability of the effects of these 2 commonly-used ways to increase activity, suggesting that one size does not fit all.
More information and resources
Dallery and others (2013): Single-case experimental designs to evaluate novel technology-based health interventions. This article discusses how N-of-1 studies can be useful for assessing digital technology.
Agency for Healthcare Research and Quality (2014): Design and implementation of N-of-1 trials: a user’s guide. A comprehensive guide on how to conduct N-of-1 studies.
Kwasnicka and Naughton (2020): N-of-1 methods: A practical guide to exploring trajectories of behaviour change and designing precision behaviour change interventions. This paper provides a guide to analysing N-of-1 data, including how to account for autocorrelations in N-of-1 studies.
Kwasnicka and others (2018): Challenges and solutions for N-of-1 design studies in health psychology. This article outlines the challenges of doing N-of-1 studies and gives solutions for overcoming them.
Examples of N-of-1 studies in digital health
Odineal and others (2020): Effect of mobile device-assisted N-of-1 trial participation on analgesic prescribing for chronic pain: randomized controlled trial. The team conducted an N-of-1 trial and used an app to facilitate the running of the study.
Lee and others (2020): “Asking too much?”: A randomised N-of-1 trial exploring patient preferences and measurement reactivity to frequent use of remote multidimensional pain assessments in children and young people with juvenile idiopathic arthritis. An N-of-1 study to explore the most acceptable way to measure pain levels in children and young people.