Think aloud study: qualitative studies
How to use a think aloud study to evaluate your digital health product.
Think aloud methods ask participants to verbalise what they are thinking and doing as they perform a task using your digital health product or service.
What to use it for
It is carried out to understand user experience of your product, for example, to uncover usability problems with navigation, or highlight content that could be improved. It is usually used as a method for carrying out usability testing.
You can use think aloud:
- when you are developing your product, to work out how to make it better (formative or iterative evaluation)
- once you have a product and want to improve it, for example by adding new content or features
Pros
Benefits of think aloud include:
- it provides insight into people’s experiences as they interact with your product
- it can be cheap to carry out
- it can be carried out with low numbers of participants
Cons
Drawbacks of think aloud include:
- it relies on people verbalising thoughts and impressions, rather than objective measures
- participants may say what they believe to be the right answer rather than what they really think (social desirability). This can distort your results and conclusions
- it cannot tell you whether your product has an impact on the outcomes you wanted
How to carry out a think aloud study
Plan the study
Decide what questions you want your study to answer. For example, whether users can find particular content or what their understanding is of the information presented.
Create a written plan, including:
- tasks for the user to complete while using your digital product
- a set of open-ended questions
Decide how many participants you want to recruit and how long you want the sessions to last (45 to 90 minutes works well).
Conduct the study
Typically, you should have a facilitator who runs the study and observers to make notes. Having more than one researcher making notes can increase the reliability of the findings. This is especially important if you’re not planning to record and transcribe the sessions.
Ask the participants to complete the tasks you have planned. This should be uninterrupted as far as possible, although the moderator will probably need to give some prompts. Explain to the participants how a think aloud works: they should tell you their thoughts, reactions and emotions as they occur while they are performing the task (concurrent think-aloud).
Another possible approach is a retrospective think aloud. Here, the participant talks through an interaction they previously performed, often using a video recording of the interaction as a prompt. This avoids the act of thinking aloud from interfering with the performance of the task, but the participant may not remember as much.
Before the think aloud, it may also help to explain to participants that:
- this is not a test and there are no right or wrong answers; it’s about understanding how the product or service works for them
- they can be critical about the product – the researchers won’t be offended
Participants don’t need to provide an explanation of their thoughts. They may feel they need to justify their thoughts, especially in front of a researcher who we may have never met, but there are various ways to overcome this obstacle in a think aloud session. For example:
- you could use a warm-up task to help the participant to focus. For example, you could start the think aloud with another section of your website that isn’t the focus of your research.
- the researcher could sit behind the participant to decrease distraction
If the participant falls silent, prompt them to verbalise their thoughts – for example, ‘what are you thinking?’. Try not to do this too often or to lead the participant towards certain answers.
The qualitative data from a think aloud study is classified into meaningful categories. You can do this using various analysis methods typically used for qualitative studies, such as:
- thematic analysis
- framework analysis
- basic counting of numerical data, such as categories or frequencies
Example: Evaluating digital health products - as we expand, we test the structure and content
While developing this resource, the one you are reading now, we created more content and added new pages. As the resource grew, the structure and navigation of the website became less fit for purpose and we decided to conduct a think aloud study to investigate how it could be improved.
We gave participants a scenario: imagine you are developers of an app designed to help insomnia and you want to evaluate it. We asked them to say everything they thought of out loud as they used the resource to complete the task. We showed participants 3 versions of the website homepage: the existing version, and two alternative prototypes. We alternated the order in which participants saw the different versions.
We conducted 4 think aloud sessions, with a facilitator and 2 observers taking comprehensive notes. We collated the findings using a collaborative online tool, grouping observations into:
- what aspects helped participants
- what didn’t help participants
- any additional features they needed
Each category had a different colour on our board (green – helpful, pink – unhelpful, orange – new feature wanted). We organised them and looked at similarities in two new versions, and differences between versions 1 and 2.
Some findings were the same across all participants:
- adding numbered sections in the guidance helped them to navigate the information
- more bullet points helped them to scan the content
- subheadings helped to break up long lists of content and make them less overwhelming
Based on the feedback, we created a version of the website structure, which can be validated with a second set of user testing.
More information and resources
Participants may find it difficult and unnatural to say everything that comes to their mind out loud. Here is a video by the Nielsen Norman Group explaining what it means to ‘think aloud’. The resource also explains how to create your own demo video for your study.
See also:
-
GOV.UK guidance on using think aloud as part of moderated usability testing
-
GOV.UK user research blog with practical tips for running a think aloud study
Examples of think aloud studies in digital health
McCall and others (2021), Development of a Mobile App to Support Self-Management of Anxiety and Depression in African American Women: A Usability Study. Researchers used think aloud as one of their methods to evaluate the usability of the app prototype tailored to the needs of African American women to help with anxiety and depression.
Szinay and others (2021), Influences on the Uptake of Health and Well-being Apps and Curated App Portals: Think-Aloud and Interview Study. Researchers were interested in how people choose apps for health and wellbeing. They used think aloud to explore what influences the selection process when looking online.