Guidance

Selecting a health app assessment body

Published 9 October 2017

Introduction

These guidelines should be used in addition to your usual procurement processes and procurement rules.

All app assessment bodies will need to have prior app assessment experience. They will also need to have demonstrated that they understand the apps market - its trends and opportunities - and have strong connections with a range of app developers and the developer community in general.

The supplier you eventually select should:

• have experience of working with high quality health and social care apps, which have been deployed into consumer and clinical settings - ask for references from clients and developers they’ve worked with
• have customers in the health and social care sector
• understand the legal and regulatory landscape around health app development

If you want the assessment body to create a shortlist of apps for review, so that you can see what is already on the market, you should make it clear that all market signalling and shortlisting will be done on an arm’s length basis. The same goes for the final selection process.

When formulating a contract, ask them to outline the principles behind their app assessment process, as these will differ between suppliers. As a minimum they should back up any assessment with:

• case studies of app assessments
• sample results of app assessments

For more information on how to select a supplier, contact Digital@phe.gov.uk.

How they should work with developers

The assessment body must treat app developers in a fair and transparent manner, and identify any potential conflicts of interest. They should explain their process for handling and resolving disputes with developers and other businesses or people they work with.

They should also demonstrate how they will support, inform, guide and educate app developers going through the assessment, to help them improve the quality of their app even if it is not ultimately approved by Public Health England (PHE).

Documenting the assessment process

The assessment process must be fully documented and validated. It must also be standardised and reproducible by any other supplier in principle - though it shouldn’t be shared with any other suppliers and should be protected by a non-disclosure agreement (NDA). The assessor must also demonstrate that:

• how they’ll help protect any intellectual property (IP) created by the developer
• they’ve made reasonable efforts to check that the evidence presented is valid
• they can ensure that scoring will be normalised across different topics and apps to ensure fairness
• they can safely and legally store all the materials and review data they’ll gather (ideally to ISO27001 standards) - this is vital for audit and compliance purposes
• they can provide review data which will help you monitor apps for significant changes that would require revalidation of the app

What you should expect from an assessment body

Your contract with the assessor should clearly state how many apps and topics they are to assess, and the deadline for delivery of their reports. It should set out how both parties will work together to define the app selection criteria. It must also:

• set out how many apps they should review, and the size of the shortlist they should provide you with
• state that you own the results, which you can use to provide advice and guidance to other developers and assessing bodies
• state that you own any IP created during procurement

We expect cost to depend on factors like the complexity of the app, the type of condition it covers and how much specialist reviewer input it will require. You should be able to understand how varying the number of apps reviewed or scale of questioning will vary the cost.

Transparency

The assessment and reporting process should be consistent and transparent. The assessment body should give you access to management reporting dashboards and a live portal containing app assessment progress and results.

In terms of information, the assessment body must confirm that all information collected during an app assessment:

• is accurate, fair and balanced
• will be licensed for publication in the public domain
• will be provided in a format which works with commonly-used software, and can be easily manipulated and translated

Working with reviewers and subject matter experts (SMEs)

The assessment body may employ their reviewers directly, but it’s likely that they will also need expert input to do the detailed review work in a specialist area. They may cover SMEs from within the health system, academia or the tech realm.

These experts will most probably be brought in as contractors. However, if there are specific experts that a commissioner would like to see included then these should be clearly stated in the contract. Support should also be sought from experts already working with PHE, either as employees or through academic partnership.

Contact Digital@phe.gov.uk for information on specific subject areas.

For experts sourced by the assessment agency, they should explain how they have recruited these experts and how they establish their credentials. They should have experts to cover each of the areas of the Digital Assessment Questions. They must also set out how they train these staff, and how they maintain quality and consistency in their work.