Mixed methods study
How to use a combination of quantitative and qualitative data to evaluate your digital health product.
This page is part of a collection of guidance on evaluating digital health products.
A mixed methods study combines quantitative and qualitative data collection and analysis in one study. Individually, these approaches can answer different questions, so combining them can provide you with more in-depth findings. In general, quantitative data is better at answering questions like ‘What is the effect of your digital product?’ and qualitative data can show how and why you got these results.
What to use it for
You can use a mixed methods study at any stage of the development of your digital product or service. It can be used:
- during development (formative or iterative evaluation)
- to describe how well your product works (summative evaluation)
- to find out how to improve or adapt your product to different user groups or environments
Pros
Benefits include:
- combining quantitative and qualitative approaches can balance out the limitations of each method
- it can provide stronger evidence and more confidence in your findings
- it can give you more granular results than each individual method
Cons
Drawbacks include:
- it can be more complex to carry out
- it may require more expertise to collect and analyse data, and to interpret the results, than using one method would
- combining different methods requires extra resources, such as time and money
How to carry out a mixed methods study
Depending on what you want to find out, different quantitative and qualitative components might be most appropriate.
Examples of how a mixed methods study might be used at different stages of product development:
During the development of your digital product
You could use a survey to gather quantitative data combined with open-ended questions to collect qualitative data. This could help you to assess and understand the problems your target population has, to find out how the digital product could be used to help them.
When you have developed your product
A mixed methods study can help you to investigate the effects of your intervention on the outcomes you want to measure. For example, a randomised controlled trial (RCT) can tell you what effect your mindfulness meditation app has on forming a regular meditation routine. Running focus groups with participants in the RCT could then help you to understand why some users stopped using the app after one month.
Adapting your product to different user groups or environments
After your product is launched in one context, you can use a mixed methods study to find out what modifications you need to make to adapt your product for a different context. For example, you could use a before-and-after study to assess if the product can be effective for different populations and follow up with interviews to find out more about the issues experienced by this potential user group. These findings will help you to make relevant cultural adaptations to your product or service.
Example: evaluation of a tailored context-aware smoking cessation app
Naughton and others (2016), A context-sensing mobile phone app (Q sense) for smoking cessation: a mixed-methods study.
The app provides support messages when it identifies that the user is in an environment where they previously reported they have smoked. As the app relies on users’ reporting of their smoking to “learn” about their high-risk places, the researchers wanted to find out if the app was accurate in identifying these places, and whether users were willing to report their immediate smoking behaviour.
They used a mixed methods design where quantitative data collection was followed by qualitative interviews with participants.
The quantitative data collected included geolocation and self-initiated smoking reports. For the qualitative components, after they used the app, participants were interviewed about their perspective of the app’s usefulness to identify any issues and improvements needed.
Fifteen participants who wanted to quit smoking were recruited and interviews were conducted with 13 participants. The quantitative data showed that the app was accurate in collecting geolocation in 97% of smoking reports. However, the researchers found that the participants under-reported their smoking on at least 56% of days.
Five of 9 participants (56%) who were eligible to receive the support (those who were in their high-risk location), received support messages. Engagement data showed that 50% of the messages were tapped within 30 minutes of being sent.
The interview data showed multiple reasons why participants did not report their smoking, such as environmental constraints including driving, not having a phone at hand, and forgetting.
Using a mixed methods study enabled the researchers to assess what happened during the intervention using data from the app (quantitative), and gain an understanding of why this data was observed (qualitative). Exploring the reasons for under-reporting of smoking in this way can help to improve the user-led reporting of smoking so that the app can become more accurate and tailored to the user.
More information and resources
O’Cathain and others (2010), Three techniques for integrating data in mixed methods studies. This resource explains how to integrate data from different components of a mixed methods study.
Shorten and Smith (2017), Mixed methods research: expanding the evidence base. Includes a list of questions to find out what type of mixed methods study design might be most appropriate for your evaluation.
Examples of mixed methods studies in digital health
Rouf and Allman-Farinelli (2018), Messaging for Interventions Aiming to Improve Calcium Intake in Young Adults-A Mixed Methods Study. The team conducted a social media survey with an open-ended component to find the preferred messaging to improve calcium intake in young adults.
Seabrook and others (2020), Understanding How Virtual Reality Can Support Mindfulness Practice: Mixed Methods Study. The team used a survey and interviews to understand the acceptability, effects and user experience of a virtual reality-based mindfulness app.
Greenhalgh and others (2018), Real-World Implementation of Video Outpatient Consultations at Macro, Meso, and Micro Levels: Mixed-Method Study. A mixed-methods study was used to explore how video outpatient consultations can be implemented most efficiently and scaled up across the complex healthcare system.