User feedback study: descriptive studies
How to use feedback from marketplace reviews to evaluate your digital health product.
This page is part of a collection of guidance on evaluating digital health products.
Carry out an analysis of user feedback data if you want to find out how users review a digital product that is already publicly available. User feedback can be gathered in different ways. Here we will look at marketplace reviews.
What to use it for
Use a user feedback study when you:
- have developed a product which is already on the market
- want to look at other people’s existing products to inform the development of your own
- want to gain some understanding of what users think of a digital product, for example, their likes, dislikes and comments on functionality
- want to see what problems users experienced so that you can refine your product
Pros
Benefits include:
- data is easy to get: you are working with data that is already available
- if your product is popular, you might have data from a large number of users
Cons
Drawbacks include:
- it might not be possible to do if your product has very few reviews
- you don’t know how much the users engaged with your product before they left a review
- you will not know anything about the type of users and their motivation for leaving a review
How to carry out a user feedback study
You could just focus on your product or analyse the reviews of similar products.
Collate and read publicly-available reviews in an app store or equivalent to get an idea of what users like or dislike about the product. You should do this systematically. Try not to be biased by selecting reviews that tell you what you want or expect to hear.
App store reviews can answer several questions that you might be interested in. For example, negative reviews can help you to improve your digital product. You might also look for comments on functionality or requests for new features. It is important to start by deciding what you would like to find out.
There are various ways to analyse your data. Thematic analysis is common because of its flexibility for capturing and categorising user experience.
Example: user feedback of popular virtual reality exercise games
See Farič and others (2019): What Players of Virtual Reality Exercise Games Want: Thematic Analysis of Web-Based Reviews.
The team wanted to find out what users liked and disliked about virtual reality exercise games (exergames) to inform the design of a new exergame.
They took their sample from the 3 most used virtual reality game marketplaces online. They focused on the most popular exergames, so they filtered their results by popularity or top-selling games.
They took 498 reviews from 29 of the most popular exergames. As they were interested in gaming experience, including likes and dislikes, they extracted the most recent positive and negative reviews. They used thematic analysis to understand and categorise the users’ experience of the exergames.
They found a variety of user experiences, including the preference for highly realistic and intuitive games which gradually increase the difficulty of the exercises.
The results will help the team to inform the design of future exergames to include elements that received positive user feedback and to address gaps in the design shown in the negative user reviews.
More examples of user feedback studies in digital health
Nicholas and others (2017): The Reviews Are in: A Qualitative Content Analysis of Consumer Perspectives on Apps for Bipolar Disorder. This study reports on the analysis of the user feedback of apps addressing bipolar disorder.
Milward and others (2016): User Preferences for Content, Features, and Style for an App to Reduce Harmful Drinking in Young Adults: Analysis of User Feedback in App Stores and Focus Group Interviews. The team wanted to find out user preferences for an app to reduce alcohol consumption in young people. They used user feedback from app stores as background research.
Stawarz and others (2014): Don’t Forget Your Pill! Designing Effective Medication Reminder Apps That Support Users’ Daily Routines. The team carried out analysis of user feedback of medication reminder apps in the app stores.